feat(vmm): run virtio-net on vhost-net with configurable queue pairs

docs/bridge-networking.md

@@ -4,7 +4,7 @@ By default, dstack-vmm uses **user** networking (QEMU's built-in SLIRP stack, no
 
 ## When to use bridge networking
 
-- High connection concurrency (passt becomes CPU-bound at ~25K+ concurrent connections)
+- High connection concurrency (user-mode networking becomes CPU-bound at ~25K+ concurrent connections)
 - Workloads that need full L2 network access
 - Environments where VMs need to be directly reachable on the LAN
 
@@ -21,11 +21,11 @@ bridge = "virbr0"
 ### Per-VM override
 
 Individual VMs can override the global networking mode via:
-- **CLI**: `vmm-cli.py deploy --net bridge` or `--net passt`
+- **CLI**: `vmm-cli.py deploy --net bridge`, `--net user`, or `--net macvtap`
 - **Web UI**: Networking dropdown in the deploy dialog
 - **API**: `networking: { mode: "bridge" }` in `VmConfiguration`
 
-Only the mode is per-VM; the bridge interface name always comes from the global config.
+The bridge interface name comes from the global config unless the node lists it in `cvm.allowed_bridges`. VMs may also override the vhost and queue settings — see [network-data-plane.md](network-data-plane.md).
 
 ## Host setup
 
@@ -143,28 +143,37 @@ mode = "bridge"
 bridge = "dstack-br0"
 ```
 
-### QEMU bridge helper setup (required for both options)
+### netd is required
 
-The bridge helper allows QEMU to create and attach TAP devices without VMM needing root privileges.
+Bridge networking needs `netd`, the privileged helper that owns every host
+interface a bridge or macvtap NIC uses. It is the same binary:
 
 ```bash
-# Allow QEMU to use the bridge
-sudo mkdir -p /etc/qemu
-echo "allow virbr0" | sudo tee /etc/qemu/bridge.conf
-# Or for manual bridge: echo "allow dstack-br0" | sudo tee /etc/qemu/bridge.conf
-
-# Set setuid on bridge helper
-sudo chmod u+s /usr/lib/qemu/qemu-bridge-helper
+sudo dstack-vmm --config vmm.toml netd
 ```
 
+Nothing else on the node needs `CAP_NET_ADMIN`: the VMM itself still runs
+unprivileged, and `netd` holds the privilege behind a Unix socket whose
+filesystem permissions authorize callers.
+
+This used to be conditional — `netd` built the TAP when libvirt filtering was on
+or when the NIC wanted more than one queue pair, and otherwise QEMU's setuid
+`qemu-bridge-helper` did. Two owners meant two answers to the same questions:
+which netdev QEMU gets, whether vhost is really on, and what a bridge NIC's TAP
+is built with. So a bridge NIC's host interface has one owner now, on every
+node.
+
+`qemu-bridge-helper` is no longer used, and `/etc/qemu/bridge.conf` no longer
+needs an `allow` line for the bridge.
+
 ## How it works
 
-- VMM passes `-netdev bridge,id=net0,br=<bridge>` to QEMU
-- QEMU's bridge helper (setuid) creates a TAP device and attaches it to the bridge
+- `netd` creates a persistent TAP, attaches it to the bridge, binds the nwfilter if the node filters, and the VMM passes `-netdev tap,id=net0,ifname=<tap>,...`
 - Guest MAC address is derived from SHA256 of the VM ID, with an optional configurable prefix (stable across restarts for DHCP IP consistency)
 - The host DHCP server (dnsmasq) assigns an IP to the VM
-- When QEMU exits, the TAP device is automatically destroyed
-- VMM does not need root or `CAP_NET_ADMIN`
+- The TAP outlives QEMU and is deleted when the VMM tears the VM's networking down, so a VM that crashes does not leave its filter rules attached to a name the next VM could take
+- Every interface `netd` creates records which VM of which VMM instance it belongs to, in the kernel's interface alias — see [Who owns an interface](#who-owns-an-interface)
+- The VMM process needs neither root nor `CAP_NET_ADMIN`; `netd` holds that privilege in a separate service
 
 ### MAC address prefix
 
@@ -192,15 +201,124 @@ The remaining bytes are derived from the VM ID hash. The prefix applies to all n
 - Docker's nftables chains (`DOCKER-FORWARD`) run before libvirt's but do not block virbr0 traffic
 - Use `setup-bridge.sh check --bridge <name>` to diagnose missing rules
 
+### Which NIC a port mapping uses
+
+A port mapping says which NIC its traffic enters through:
+
+```bash
+vmm-cli.py deploy ... --port udp:0.0.0.0:7483:51820@0 --port tcp:127.0.0.1:7484:8001@0
+```
+
+Leave `@<nic>` off and the VMM picks the first user-mode NIC — where QEMU's
+`hostfwd=` entries have always gone. If the VM has no user-mode NIC, the mapping
+has no publishing backend and the launch log names it as stranded. A single-NIC
+user-mode VM never needs the suffix.
+
+With several NICs the choice used to be made silently, and not always the way an
+operator would have. A bridge NIC for external traffic beside a user-mode NIC for
+management — the topology multi-NIC was added for — put every published port on
+the *management* NIC: the traffic reached the guest, but over slirp, bypassing
+whatever the bridge NIC's nwfilter was there to enforce and hiding the client's
+address behind the slirp gateway. A second user-mode NIC could never publish
+anything at all, because only the first was ever selected.
+
+A mapping resolves to at most one NIC. The only backend that can carry it is
+QEMU user networking through `hostfwd=`; `netd` builds bridge interfaces but
+does not publish host ports.
+
+### Which ports a bridge NIC can publish
+
+QEMU publishes a port with `hostfwd=` on a user-mode NIC, and that is the only
+mechanism this host has. **The `netd` in this repository builds interfaces; it
+does not forward host ports**, so a bridge NIC cannot carry a port mapping.
+
+`--port …@<nic>` therefore only ever names a user-mode NIC. Pinning to a bridge,
+macvtap or custom NIC is refused at deployment, where the caller is there to be
+told. An unpinned mapping goes to the first user-mode NIC; a VM that has none is
+not refused — it may have been deployed before this — but every mapping it
+strands is named in the launch log.
+
+## Who owns an interface
+
+`netd` names an interface `dt<12 hex>`, a digest of (VMM instance, VM, NIC
+index). That answers "where is this VM's interface" but not "whose is this
+interface" — and the second question is the one a leaked interface poses. So
+`netd` also records the identity on the interface itself:
+
+```console
+$ ip -d link show dtc41d9e0b7a52 | grep alias
+    alias dstack1:0:path-3f9a1c8e7d2b4a60:0a1b2c3d4e5f6071
+```
+
+The kernel holds that for exactly the interface's lifetime, so unlike a file on
+disk it cannot be written late, lost, or left behind. It is a hint, never an
+authority: a record is believed only when re-deriving the interface name from
+it reproduces the name it is written on, so a forged, truncated or ambiguous
+record reads the same as no record at all.
+
+Teardown does not need it — a sweep derives the names it deletes. What needs it
+is an operator, and a host running several VMM instances, where it is the only
+thing that tells one instance's interfaces from another's.
+
+```bash
+# What netd holds on this host
+sudo dstack-vmm netd list
+
+# Everything one VM holds, for a VM whose VMM will never ask again
+sudo dstack-vmm netd remove-vm --instance path-3f9a1c8e7d2b4a60 --vm 0a1b2c3d4e5f6071
+```
+
+### When a release does not land
+
+Every stop and every removal asks `netd` to sweep that VM's interfaces, by
+deriving each of the 256 names its identity could produce. That needs no
+record, and it reaches what a per-NIC teardown cannot: an interface a crash
+left behind before anything on disk pointed at it, or one whose NIC the
+manifest has since dropped.
+
+A removal deletes the VM's directory, and that directory — with its `.removing`
+marker — is the only thing left that says to try again. So it is deleted only
+once the sweep has landed. If `netd` refused, or was not there to ask, the
+directory stays and the next VMM start resumes the removal; `remove_all` is
+idempotent, so the retry costs one round trip. A VM that never asked `netd` for
+an interface is unaffected: there is nothing for `netd` to be holding.
+
+The VMM persists `.netd-pending` before asking netd to prepare an interface and
+clears it only after a successful whole-VM sweep. This cleanup marker survives
+failed launches and network configuration changes, even if the runtime snapshot
+is absent or replaced by a user-mode topology. Older snapshots are promoted to
+the marker before cleanup or replacement. A failed cleanup during an update
+also leaves the old snapshot intact.
+
+On an unfiltered node, an unavailable `libvirtd` does not make an otherwise
+successful TAP sweep fail. Filtered nodes still require confirmation that their
+nwfilter bindings have been released; deleting the TAP alone is not sufficient.
+
+What no VMM will retry is an interface whose VM directory an operator deleted
+by hand, or one recorded under an instance ID no VMM uses any more. `netd list`
+shows both, with the instance and VM they are recorded under:
+
+```bash
+sudo dstack-vmm netd list
+sudo dstack-vmm netd remove-vm --instance <instance> --vm <vm>
+sudo dstack-vmm netd remove-interface dtc41d9e0b7a52
+```
+
+Changing `cvm.instance_id` — or `run_path`, which it is derived from — strands
+interfaces the same way. Running VMs keep working until they stop, and
+`netd list` still shows the old instance ID, which is what `remove-vm` needs.
+
 ### Mixing networking modes
 
-Bridge and passt VMs can coexist. Set the global default in `vmm.toml` and override per-VM as needed:
+Bridge and user-mode VMs can coexist. Set the global default in `vmm.toml` and override per-VM as needed:
 
 ```bash
-# Global default is bridge, but deploy this VM with passt
-vmm-cli.py deploy --name my-vm --image dstack-0.5.6 --compose app.yaml --net passt
+# Global default is bridge, but deploy this VM with user networking
+vmm-cli.py deploy --name my-vm --image dstack-0.5.6 --compose app.yaml --net user
 ```
 
-### vhost-net and TDX
+### vhost-net and multiqueue
+
+Bridge NICs can run on the host kernel's vhost-net data plane and expose several virtio-net queue pairs. Both are off by default and enabled per node or per VM — see [network-data-plane.md](network-data-plane.md) for the knobs, the enablement checklist, the mode support matrix, and how to pick a queue count.
 
-vhost-net (kernel data plane offload for virtio-net) is **not enabled** for bridge mode. TDX encrypts guest memory, which prevents the host kernel from performing DMA-based packet offload. The default QEMU userspace virtio backend is used instead.
+vhost-net works in a TDX guest: the virtio rings and buffers live in shared, unencrypted memory so that a host-side backend can reach them, which is the same mechanism `vhost-vsock-pci` has always relied on.

docs/libvirt-network-filter.md

@@ -14,10 +14,17 @@ host mechanism.
 
 The measurable acceptance criteria are:
 
-- `network_filter = "none"` preserves the existing QEMU `-netdev bridge`
-  behavior and does not require `netd` or libvirt.
+- `network_filter = "none"` installs no nwfilter binding. It does not remove
+  the `netd` dependency: `netd` creates the TAP for every bridge NIC either
+  way, and the VMM uses `-netdev tap` either way. What changes is only whether
+  that TAP carries a binding.
 - `network_filter = "libvirt"` creates the TAP and filter binding before QEMU
   is submitted to Supervisor, and uses QEMU `-netdev tap`.
+- An nwfilter binding outlives the TAP it was bound to, so a teardown clears
+  the binding at every name that VM could have used, whether or not the
+  interface is still there. `dstack-vmm netd list` shows a binding whose
+  interface is already gone as a `binding` row; remove one with
+  `dstack-vmm netd remove-interface <name>`.
 - A failed TAP or filter setup prevents QEMU from starting and rolls back all
   interfaces prepared for that VM.
 - Normal stop and removal delete the filter binding and TAP.
@@ -53,8 +60,11 @@ allowed_macvtap_parents = []
 
 Macvtap is excluded from `allowed_network_modes` by default. Empty bridge and
 macvtap-parent allowlists prevent RPC callers from overriding the respective
-node defaults. If macvtap is explicitly enabled, callers may select only a
-parent in `allowed_macvtap_parents`; the macvtap forwarding mode always comes
+node defaults. If macvtap is explicitly enabled, callers may select a
+parent listed in `allowed_macvtap_parents`, the node's own configured parent, or
+one this VM already holds — restating a value the node would have supplied
+anyway grants nothing new. The same applies to `bridge_name` and
+`allowed_bridges`. The macvtap forwarding mode always comes
 from `[cvm.networking].macvtap_mode` and cannot be selected through deployment
 RPCs. These allowlists authorize attachment targets; an nwfilter is not a
 substitute for that authorization.
@@ -82,7 +92,8 @@ For libvirt mode, startup is:
 2. Create the TAP for the configured QEMU UID and attach it to the bridge.
 3. Create a libvirt nwfilter binding for the TAP.
 4. Bring the TAP up and return success.
-5. Start QEMU directly with `-netdev tap,script=no,downscript=no`.
+5. Start QEMU directly with `-netdev tap,script=no,downscript=no`, carrying
+   `vhost=on|off` and, above one queue pair, `queues=N`.
 
 Teardown stops QEMU first, removes the binding, and deletes the TAP. Operations
 are serialized by `netd`. The design intentionally does not add ownership
@@ -98,20 +109,65 @@ arguments. It never accepts a command, executable path, TAP name, or raw XML
 from a client. Filter XML is generated internally with XML escaping and is
 validated by libvirt.
 
+Teardown by identity only reaches the NIC indices its caller still has a record
+of, and that record is written *after* the interface exists — a VMM killed in
+between leaves a TAP nothing on disk points at, and a manifest that lost a NIC
+leaves the same thing behind. `remove_all` names a VM instead of an interface
+and derives every name that VM could occupy, so neither has to be recorded for
+teardown to work. The VMM sweeps before preparing a launch as well as on stop,
+which makes a launch self-healing regardless of what the record says.
+
+A bridge prepare also carries two things `netd` does not need to build the TAP.
+`workdir` names the VM's directory on the host: untrusted, never read for a
+decision, and present only so an operator reading `netd`'s log can get from an
+opaque TAP name back to the VM. `ingress` states the host ports that NIC should make
+reachable at its guest, which the VMM cannot arrange itself — it runs without
+`CAP_NET_ADMIN` by design, and QEMU's `hostfwd=` entries need a user-mode netdev
+that a bridge NIC does not have. The `netd` in this repository builds interfaces
+and does not forward ports; it says so by leaving `ingress` out of its response,
+the same reading `queues` gets, so a caller can tell "this netd does not do that"
+from "nothing was asked for" instead of assuming ports were forwarded because a
+TAP came back.
+
 ## Deployment modes
 
-Production should run one shared service. `netd` reads only the `[netd]`
-section, so its root-owned configuration can be small and independent of every
-VMM instance:
+Production should run one shared service. `netd` reads the `[netd]` section,
+plus `cvm.network_filter.mode` if the file has one, so its root-owned
+configuration can be small and independent of every VMM instance:
 
 ```toml
 # /etc/dstack/netd.toml
 [netd]
 socket = "/run/dstack/netd.sock"
 socket_mode = 0o660
 libvirt_uri = "qemu:///system"
+
+# Required here because this file has no [cvm] section for netd to read the
+# node's policy from.
+[netd.network_filter]
+mode = "libvirt"
+filter = "clean-traffic"
+parameters = {}
 ```
 
+`[netd.network_filter]` is netd's own copy of the invariant, not a convenience.
+netd is the privileged side of the socket, and anything that can reach the
+socket can ask for an unfiltered TAP on a host bridge — a request a filtering
+node has to refuse in the daemon rather than in its caller. When netd and the
+VMM share one `vmm.toml`, leaving it unset derives it from
+`[cvm.network_filter]` so the two cannot drift apart; a malformed section is a
+startup error rather than a silent fallback to "filter nothing".
+
+The request says only *whether* to bind a filter, never which one. A caller that
+named the filter could name `allow-arp`, which contains no drop rule at all, or
+pin `clean-traffic` to the gateway's MAC and IP through its parameters, and
+still satisfy a policy that asked for "some filter".
+
+A macvtap parent is refused when filtering is required and the parent is a host
+bridge or is enslaved to one: nwfilter does not apply to macvtap, so that
+request is the same unfiltered access to the same segment, spelled with a
+different operation.
+
 Production deployments can use systemd socket activation. The socket unit
 owns the filesystem mode and ownership; `netd.socket_mode` applies only to the
 standalone bind path.
@@ -162,10 +218,38 @@ sudo dstack-vmm --config ./vmm.toml \
   --netd-socket /run/dstack-dev/netd.sock
 ```
 
-User networking and bridge networking with `mode = "none"` never connect to
-`netd`. Libvirt mode fails closed if `netd` is unavailable.
-
-Filtered TAP netdevs currently set `vhost=off`. This keeps the initial backend
-on the directly bound TAP path and avoids adding `/dev/vhost-net` permissions
-to the QEMU user. It is a deliberate security-first throughput tradeoff; a
-future configurable vhost mode requires equivalent filter integration tests.
+User networking and a caller-supplied netdev never ask `netd` to build an
+interface. The VMM still contacts the socket for such a VM -- every launch and
+every stop releases whatever the VM held, before it decides whether it needs
+anything built -- but nothing about the VM depends on the answer. Bridge and
+macvtap do ask, and fail closed if `netd` is unavailable.
+
+Filtered TAP netdevs follow the node's `vhost` and `queues` settings like any
+other TAP-backed NIC (see [network-data-plane.md](network-data-plane.md)). The
+nwfilter binding is installed on the host TAP interface, so packets traverse it
+whether they were written by QEMU or by a vhost worker; filtering is unaffected
+by the data plane choice. Enabling vhost does require the QEMU user to be able
+to open `/dev/vhost-net`.
+
+`netd` creates the TAP for unfiltered bridge NICs too. Those TAPs carry no
+nwfilter binding, so a bridge node needs `netd` even when
+`network_filter.mode = "none"` — see
+[bridge-networking.md](bridge-networking.md) for why the host interface has a
+single owner.
+
+An empty filter name is what selects that unfiltered TAP, so `mode = "libvirt"`
+with an empty `filter` is rejected at config load rather than quietly producing
+an unbound TAP.
+
+Removal carries the same distinction: the VMM tells `netd` whether the interface
+it is asking about was created with a binding, from a record made when it was
+built rather than from configuration that may have changed since. A binding it
+was told about must be gone before `netd` returns; otherwise `netd` still asks
+libvirt to clear one — an interface name is reused by the same VM, and a
+leftover binding's rules would be inherited — but a `libvirtd` it cannot reach
+is a warning rather than a failure. So a node with `virsh` installed and no
+running `libvirtd` can create and destroy multiqueue TAPs. The flag defaults to
+true on the wire, so an older VMM's removals still drop their bindings.
+
+`netd` requires the `virsh` binary to be present whatever the filter mode; it is
+`libvirtd` that unfiltered work does not need.