feat(vmm): give port mappings a NIC, and say which NIC publishes them

docs/bridge-networking.md

@@ -143,35 +143,37 @@ mode = "bridge"
 bridge = "dstack-br0"
 ```
 
-### QEMU bridge helper setup (needed unless every bridge NIC goes through netd)
+### netd is required
 
-The bridge helper allows QEMU to create and attach TAP devices without VMM needing root privileges.
-It is used only on the single-queue bridge paths; a NIC that `netd` builds never touches it, so a
-node that runs `netd` for all of its bridge VMs does not need it at all.
-
-The VMM probes `/usr/lib/qemu/qemu-bridge-helper`, `/usr/libexec/qemu-bridge-helper` and
-`/usr/local/libexec/qemu-bridge-helper`. Set `cvm.qemu_bridge_helper` in `vmm.toml` for a path
-outside that list.
+Bridge networking needs `netd`, the privileged helper that owns every host
+interface a bridge or macvtap NIC uses. It is the same binary:
 
 ```bash
-# Allow QEMU to use the bridge
-sudo mkdir -p /etc/qemu
-echo "allow virbr0" | sudo tee /etc/qemu/bridge.conf
-# Or for manual bridge: echo "allow dstack-br0" | sudo tee /etc/qemu/bridge.conf
-
-# Set setuid on bridge helper
-sudo chmod u+s /usr/lib/qemu/qemu-bridge-helper
+sudo dstack-vmm --config vmm.toml netd
 ```
 
+Nothing else on the node needs `CAP_NET_ADMIN`: the VMM itself still runs
+unprivileged, and `netd` holds the privilege behind a Unix socket whose
+filesystem permissions authorize callers.
+
+This used to be conditional — `netd` built the TAP when libvirt filtering was on
+or when the NIC wanted more than one queue pair, and otherwise QEMU's setuid
+`qemu-bridge-helper` did. Two owners meant two answers to the same questions:
+which netdev QEMU gets, whether vhost is really on, and what a bridge NIC's TAP
+is built with. So a bridge NIC's host interface has one owner now, on every
+node.
+
+`qemu-bridge-helper` is no longer used, and `/etc/qemu/bridge.conf` no longer
+needs an `allow` line for the bridge.
+
 ## How it works
 
-- With more than one queue pair, or with libvirt filtering on, `netd` creates the TAP and the VMM passes `-netdev tap,id=net0,ifname=<tap>,...` — this is the usual case on a node running `netd` with multi-vCPU VMs, since queue pairs default to the VM's vCPU count. Without `netd`, a bridge NIC that took that default drops back to one queue pair and takes a helper path below
-- Otherwise the VMM passes `-netdev tap,id=net0,br=<bridge>,helper=<qemu-bridge-helper>,vhost=on`, or `-netdev bridge,id=net0,br=<bridge>` when vhost is off or no helper is found
-- QEMU's bridge helper (setuid) creates a TAP device and attaches it to the bridge on the two helper paths
+- `netd` creates a persistent TAP, attaches it to the bridge, binds the nwfilter if the node filters, and the VMM passes `-netdev tap,id=net0,ifname=<tap>,...`
 - Guest MAC address is derived from SHA256 of the VM ID, with an optional configurable prefix (stable across restarts for DHCP IP consistency)
 - The host DHCP server (dnsmasq) assigns an IP to the VM
-- On the two bridge-helper paths the TAP disappears when QEMU exits; a `netd`-created TAP is persistent and is deleted when the VMM tears the VM's networking down
-- The VMM process itself needs neither root nor `CAP_NET_ADMIN` on any path; the `netd` path moves that privilege into a separate root service instead
+- The TAP outlives QEMU and is deleted when the VMM tears the VM's networking down, so a VM that crashes does not leave its filter rules attached to a name the next VM could take
+- Every interface `netd` creates records which VM of which VMM instance it belongs to, in the kernel's interface alias — see [Who owns an interface](#who-owns-an-interface)
+- The VMM process needs neither root nor `CAP_NET_ADMIN`; `netd` holds that privilege in a separate service
 
 ### MAC address prefix
 
@@ -199,6 +201,101 @@ The remaining bytes are derived from the VM ID hash. The prefix applies to all n
 - Docker's nftables chains (`DOCKER-FORWARD`) run before libvirt's but do not block virbr0 traffic
 - Use `setup-bridge.sh check --bridge <name>` to diagnose missing rules
 
+### Which NIC a port mapping uses
+
+A port mapping says which NIC its traffic enters through:
+
+```bash
+vmm-cli.py deploy ... --port udp:0.0.0.0:7483:51820@0 --port tcp:127.0.0.1:7484:8001@0
+```
+
+Leave `@<nic>` off and the VMM picks the first user-mode NIC — where QEMU's
+`hostfwd=` entries have always gone — and failing that the first bridge NIC. A
+single-NIC VM never needs it.
+
+With several NICs the choice used to be made silently, and not always the way an
+operator would have. A bridge NIC for external traffic beside a user-mode NIC for
+management — the topology multi-NIC was added for — put every published port on
+the *management* NIC: the traffic reached the guest, but over slirp, bypassing
+whatever the bridge NIC's nwfilter was there to enforce and hiding the client's
+address behind the slirp gateway. A second user-mode NIC could never publish
+anything at all, because only the first was ever selected.
+
+A mapping resolves to exactly one NIC, and that NIC's backend decides the
+mechanism: `hostfwd=` for user mode, `netd` for a bridge. Nothing can be claimed
+by both.
+
+### Which ports a bridge NIC can publish
+
+QEMU publishes a port with `hostfwd=` on a user-mode NIC, and that is the only
+mechanism this host has. **The `netd` in this repository builds interfaces; it
+does not forward host ports**, so a bridge NIC cannot carry a port mapping.
+
+`--port …@<nic>` therefore only ever names a user-mode NIC. Pinning to a bridge,
+macvtap or custom NIC is refused at deployment, where the caller is there to be
+told. An unpinned mapping goes to the first user-mode NIC; a VM that has none is
+not refused — it may have been deployed before this — but every mapping it
+strands is named in the launch log.
+
+## Who owns an interface
+
+`netd` names an interface `dt<12 hex>`, a digest of (VMM instance, VM, NIC
+index). That answers "where is this VM's interface" but not "whose is this
+interface" — and the second question is the one a leaked interface poses. So
+`netd` also records the identity on the interface itself:
+
+```console
+$ ip -d link show dtc41d9e0b7a52 | grep alias
+    alias dstack1:0:path-3f9a1c8e7d2b4a60:0a1b2c3d4e5f6071
+```
+
+The kernel holds that for exactly the interface's lifetime, so unlike a file on
+disk it cannot be written late, lost, or left behind. It is a hint, never an
+authority: a record is believed only when re-deriving the interface name from
+it reproduces the name it is written on, so a forged, truncated or ambiguous
+record reads the same as no record at all.
+
+Teardown does not need it — a sweep derives the names it deletes. What needs it
+is an operator, and a host running several VMM instances, where it is the only
+thing that tells one instance's interfaces from another's.
+
+```bash
+# What netd holds on this host
+sudo dstack-vmm netd list
+
+# Everything one VM holds, for a VM whose VMM will never ask again
+sudo dstack-vmm netd remove-vm --instance path-3f9a1c8e7d2b4a60 --vm 0a1b2c3d4e5f6071
+```
+
+### When a release does not land
+
+Every stop and every removal asks `netd` to sweep that VM's interfaces, by
+deriving each of the 256 names its identity could produce. That needs no
+record, and it reaches what a per-NIC teardown cannot: an interface a crash
+left behind before anything on disk pointed at it, or one whose NIC the
+manifest has since dropped.
+
+A removal deletes the VM's directory, and that directory — with its `.removing`
+marker — is the only thing left that says to try again. So it is deleted only
+once the sweep has landed. If `netd` refused, or was not there to ask, the
+directory stays and the next VMM start resumes the removal; `remove_all` is
+idempotent, so the retry costs one round trip. A VM that never asked `netd` for
+an interface is unaffected: there is nothing for `netd` to be holding.
+
+What no VMM will retry is an interface whose VM directory an operator deleted
+by hand, or one recorded under an instance ID no VMM uses any more. `netd list`
+shows both, with the instance and VM they are recorded under:
+
+```bash
+sudo dstack-vmm netd list
+sudo dstack-vmm netd remove-vm --instance <instance> --vm <vm>
+sudo dstack-vmm netd remove-interface dtc41d9e0b7a52
+```
+
+Changing `cvm.instance_id` — or `run_path`, which it is derived from — strands
+interfaces the same way. Running VMs keep working until they stop, and
+`netd list` still shows the old instance ID, which is what `remove-vm` needs.
+
 ### Mixing networking modes
 
 Bridge and user-mode VMs can coexist. Set the global default in `vmm.toml` and override per-VM as needed:

docs/libvirt-network-filter.md

@@ -14,13 +14,17 @@ host mechanism.
 
 The measurable acceptance criteria are:
 
-- `network_filter = "none"` installs no nwfilter binding. It still uses `netd`
-  for any NIC with more than one queue pair, and a `tap` netdev behind
-  `qemu-bridge-helper` whenever vhost is on; only a single-queue, non-vhost
-  bridge NIC keeps the historical `-netdev bridge` path with no `netd` or
-  libvirt dependency.
+- `network_filter = "none"` installs no nwfilter binding. It does not remove
+  the `netd` dependency: `netd` creates the TAP for every bridge NIC either
+  way, and the VMM uses `-netdev tap` either way. What changes is only whether
+  that TAP carries a binding.
 - `network_filter = "libvirt"` creates the TAP and filter binding before QEMU
   is submitted to Supervisor, and uses QEMU `-netdev tap`.
+- An nwfilter binding outlives the TAP it was bound to, so a teardown clears
+  the binding at every name that VM could have used, whether or not the
+  interface is still there. `dstack-vmm netd list` shows a binding whose
+  interface is already gone as a `binding` row; remove one with
+  `dstack-vmm netd remove-interface <name>`.
 - A failed TAP or filter setup prevents QEMU from starting and rolls back all
   interfaces prepared for that VM.
 - Normal stop and removal delete the filter binding and TAP.
@@ -105,6 +109,26 @@ arguments. It never accepts a command, executable path, TAP name, or raw XML
 from a client. Filter XML is generated internally with XML escaping and is
 validated by libvirt.
 
+Teardown by identity only reaches the NIC indices its caller still has a record
+of, and that record is written *after* the interface exists — a VMM killed in
+between leaves a TAP nothing on disk points at, and a manifest that lost a NIC
+leaves the same thing behind. `remove_all` names a VM instead of an interface
+and derives every name that VM could occupy, so neither has to be recorded for
+teardown to work. The VMM sweeps before preparing a launch as well as on stop,
+which makes a launch self-healing regardless of what the record says.
+
+A bridge prepare also carries two things `netd` does not need to build the TAP.
+`workdir` names the VM's directory on the host: untrusted, never read for a
+decision, and present only so an operator reading `netd`'s log can get from an
+opaque TAP name back to the VM. `ingress` states the host ports that NIC should make
+reachable at its guest, which the VMM cannot arrange itself — it runs without
+`CAP_NET_ADMIN` by design, and QEMU's `hostfwd=` entries need a user-mode netdev
+that a bridge NIC does not have. The `netd` in this repository builds interfaces
+and does not forward ports; it says so by leaving `ingress` out of its response,
+the same reading `queues` gets, so a caller can tell "this netd does not do that"
+from "nothing was asked for" instead of assuming ports were forwarded because a
+TAP came back.
+
 ## Deployment modes
 
 Production should run one shared service. `netd` reads the `[netd]` section,
@@ -194,11 +218,11 @@ sudo dstack-vmm --config ./vmm.toml \
   --netd-socket /run/dstack-dev/netd.sock
 ```
 
-User networking never asks `netd` to build an interface; the VMM still opens a
-short liveness-probe connection to the netd socket on every launch and when
-describing a stopped VM. Libvirt mode fails closed if `netd`
-is unavailable. Bridge networking with `mode = "none"` connects only when it
-needs more than one queue pair, as described below.
+User networking and a caller-supplied netdev never ask `netd` to build an
+interface. The VMM still contacts the socket for such a VM -- every launch and
+every stop releases whatever the VM held, before it decides whether it needs
+anything built -- but nothing about the VM depends on the answer. Bridge and
+macvtap do ask, and fail closed if `netd` is unavailable.
 
 Filtered TAP netdevs follow the node's `vhost` and `queues` settings like any
 other TAP-backed NIC (see [network-data-plane.md](network-data-plane.md)). The
@@ -207,10 +231,11 @@ whether they were written by QEMU or by a vhost worker; filtering is unaffected
 by the data plane choice. Enabling vhost does require the QEMU user to be able
 to open `/dev/vhost-net`.
 
-`netd` also creates the TAP for unfiltered bridge NICs that ask for more than
-one queue pair, because `qemu-bridge-helper` returns a single descriptor and
-cannot create a `multi_queue` device. Those TAPs carry no nwfilter binding, so
-a multiqueue bridge node needs `netd` even when `network_filter.mode = "none"`.
+`netd` creates the TAP for unfiltered bridge NICs too. Those TAPs carry no
+nwfilter binding, so a bridge node needs `netd` even when
+`network_filter.mode = "none"` — see
+[bridge-networking.md](bridge-networking.md) for why the host interface has a
+single owner.
 
 An empty filter name is what selects that unfiltered TAP, so `mode = "libvirt"`
 with an empty `filter` is rejected at config load rather than quietly producing

docs/network-data-plane.md

@@ -64,8 +64,8 @@ refuses a request for four queue pairs should not hand out sixteen by itself.
 The hard ceiling from any source is 64.
 
 Without vhost the default is a single queue pair. The QEMU main loop drains
-every queue on one thread, so extra queues buy little while still costing a
-netd interface, more MSI-X vectors, and a changed guest device. An explicit
+every queue on one thread, so extra queues buy little while still costing more
+MSI-X vectors and a changed guest device. An explicit
 queue count is still honoured without vhost, since that combination is a
 deliberate request rather than a default. The two defaults travelling together
 also means a node that never sets `vhost` keeps building the device its VMs
@@ -150,9 +150,9 @@ measured, so attestation and app identity are unaffected. Before flipping it:
    restarting. The VMM warns at startup when its own access fails, but it
    cannot refuse on that basis — QEMU need not share its credentials.
 
-2. **Restart `netd` before or together with the VMM.** Multiqueue bridge NICs
-   are prepared by `netd`, and the VMM checks that `netd` echoes the queue
-   count it built. An older `netd` fails that check; the launch is rolled back
+2. **Restart `netd` before or together with the VMM.** Every bridge and
+   macvtap NIC is prepared by `netd`, and the VMM checks that `netd` echoes the
+   queue count it built. An older `netd` fails that check; the launch is rolled back
    and fails with the reason in the VMM log, but the VM does not start until
    `netd` is upgraded.
 
@@ -165,35 +165,22 @@ measured, so attestation and app identity are unaffected. Before flipping it:
 | Mode | netdev | vhost | queues > 1 |
 |---|---|---|---|
 | `user` | `user,...` | no backend | not supported |
-| `bridge` | `tap,ifname=` via netd, else `tap,br=,helper=`, else `bridge,br=` | yes | yes, through netd |
-| `bridge` with libvirt filtering | `tap,ifname=` | yes | yes, through netd |
+| `bridge` | `tap,ifname=` via netd | yes | yes |
 | `macvtap` | `tap,fd=` / `tap,fds=` | yes | yes |
 | `custom` | operator's own string | operator's own string | no, not settable |
 
-QEMU's `bridge` netdev accepts neither `vhost=` nor `queues=`, so enabling
-vhost switches bridge mode to a `tap` netdev driven by the same setuid
-`qemu-bridge-helper`. The VMM still needs no network privileges. The helper has
-no compiled-in default path for the `tap` netdev, so the VMM probes the known
-distribution locations; set `cvm.qemu_bridge_helper` if yours is elsewhere. If
-no helper is found the NIC falls back to the non-vhost `bridge` netdev with a
-warning, because a node-wide setting must not stop a node from booting VMs
-that never asked for it.
-
-The helper returns exactly one descriptor, which is why more than one queue
-pair in bridge mode is created by `netd` instead: it adds a persistent
-`multi_queue` TAP that QEMU then opens once per queue. `netd` requires the
-`virsh` binary to be installed even when nothing is filtered, though it does
-not require a reachable `libvirtd`. That applies whether or
-not libvirt filtering is on, so a bridge node needs `netd` to get the default
-queue count (see [libvirt-network-filter.md](libvirt-network-filter.md)).
-Without it, bridge NICs fall back to a single queue pair with a warning rather
-than failing to launch; a VM that asked for a queue count explicitly still
-fails, so the caller learns their request was not met. `netd` is probed by
-connecting, not by looking for its socket file, because a `netd` that died
-leaves the socket behind. One-shot `dstack-vmm run` has no netd lifecycle at
-all and behaves like a node without it. `netd` reports back the
-queue count it created, and the VMM refuses to launch on a mismatch — a `netd`
-deployed separately as a root service can be older than the VMM asking it for
+QEMU's `bridge` netdev accepts neither `vhost=` nor `queues=`, and the setuid
+`qemu-bridge-helper` behind its `tap` netdev returns exactly one descriptor. So
+bridge mode runs on a TAP that `netd` creates: persistent, `multi_queue` when
+asked for, and opened once per queue by QEMU. That is true of every bridge NIC,
+filtered or not, single-queue or not — see
+[bridge-networking.md](bridge-networking.md) for why the host interface has one
+owner. `netd` requires the `virsh` binary to be installed even when nothing is
+filtered, though it does not require a reachable `libvirtd`. One-shot
+`dstack-vmm run` does not manage netd interface lifecycle, so it refuses bridge
+and macvtap NICs outside `--dry-run`. `netd` reports back the queue count it
+created, and the VMM refuses to launch on a mismatch — a `netd` deployed
+separately as a root service can be older than the VMM asking it for
 multiqueue, and QEMU would otherwise reject the interface from inside the
 per-VM launcher.
 
@@ -253,7 +240,7 @@ multiqueue line rate with zero `swiotlb buffer is full` events.
 `vectors` is derived, never configured: `2N + 2`, one vector per queue
 direction plus config and control. One queue pair emits no `mq=on` or
 `vectors=` at all, leaving the guest device line byte for byte identical to the
-one before this feature. The `-netdev` half does change wherever vhost is on,
+one before this feature. The `-netdev` half carries `vhost=on|off` either way,
 since that is what selects the backend.
 
 ## Requirements

docs/onboarding.md

@@ -149,7 +149,7 @@ The deploy command:
 
 Pass `--vcpu`, `--memory`, or `--disk` to change the app resources before you deploy.
 
-The `--port 8080:80` mapping means `host_port:vm_port` and uses TCP on `127.0.0.1`. The full accepted forms are `vm`, `host:vm`, `proto:host:vm`, and `proto:addr:host:vm`. Use `tcp` or `udp` for `proto`. Fixed host and VM ports must be between 1 and 65535. If you omit the host port, or use `auto` or `0`, `dstack` picks a free localhost port and prints the selected mapping after deploy.
+The `--port 8080:80` mapping means `host_port:vm_port` and uses TCP on `127.0.0.1`. The full accepted forms are `vm`, `host:vm`, `proto:host:vm`, and `proto:addr:host:vm`, each optionally suffixed with `@<nic>` to name which NIC the traffic enters through (a single-NIC VM never needs it). Use `tcp` or `udp` for `proto`. Fixed host and VM ports must be between 1 and 65535. If you omit the host port, or use `auto` or `0`, `dstack` picks a free localhost port and prints the selected mapping after deploy.
 
 Open the app from the host:
 

docs/vmm-cli-user-guide.md

@@ -290,6 +290,11 @@ Expose services running in your VM:
 
 # Multiple ports
 --port tcp:8080:80 --port tcp:8443:443
+
+# Pin a mapping to one NIC: protocol[:host_address]:host_port:vm_port@<nic>
+# Without @<nic> the mapping enters through the first user-mode NIC, or the
+# first bridge NIC when the VM has no user-mode one.
+--port tcp:0.0.0.0:8443:443@0
 ```
 
 #### GPU Assignment