vmm: sanitize GPUs through vfio-ioctls hot reset

[kvinwang]

Summary

• switch GPU sanitize-on-attach from root-only sysfs writes to VFIO_DEVICE_PCI_HOT_RESET
• use rust-vmm/vfio abstractions instead of maintaining handwritten VFIO ioctl numbers, ABI structs, and variable-length buffer parsing in dstack
• validate the kernel-reported reset scope and reject affected devices outside the GPU's IOMMU group
• keep devices bound to vfio-pci across the reset
• add dstack-vmm sanitize-gpu <slot>... for standalone validation and operations

Dependency strategy

rust-vmm/vfio already provides the VFIO bindings and container/group/device lifecycle abstractions, but its public API does not yet expose PCI hot reset operations. This PR temporarily pins the following fork commit:

• kvinwang/vfio@1b560c2

The fork adds pci_hot_reset_info and pci_hot_reset APIs. After validating the complete flow on GPU hardware, we plan to submit those APIs upstream and replace the Git dependency with a released crates.io version.

Motivation

The existing sanitize path writes Bridge Control through sysfs and re-probes devices through /sys/bus/pci/drivers_probe. Both operations require root privileges. The VMM normally runs as a dedicated unprivileged user that already has access to the VFIO group nodes required for GPU passthrough.

The VFIO hot-reset ioctl asks the kernel to perform the bus reset and authorizes it through ownership of every affected VFIO group. This is the same mechanism used by VMM implementations such as QEMU.

Safety

• retain the dedicated-bridge topology check
• query the kernel-reported affected-device set before resetting
• reject any affected device outside the target GPU's IOMMU group
• pass the owned VFIO group to the reset ioctl as proof of ownership
• drop all VFIO objects before QEMU starts

Tests

• cargo check -p dstack-vmm
• cargo test -p dstack-vmm — 119 passed
• cargo clippy -p dstack-vmm --all-targets -- -D warnings

The new reset path has not yet been validated on target GPU hardware. The standalone sanitize-gpu command is included to exercise the exact launch-time path during that validation.

Relationship to #1058

This is an alternative implementation of #1058 that addresses the review feedback to use vfio-ioctls. PR #1058 has not been modified.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Leechael]

Validated this on real Blackwell hardware — an 8× B200 host with Intel TDX, all GPUs bound to vfio-pci, one GPU per IOMMU group, CC mode on. Built from this branch and exercised through the new sanitize-gpu subcommand.

The hot-reset path works. Single GPU and all eight, with the GPUs idle:

sanitize-gpu <one slot>    -> all sanitized GPUs are VFIO-ready count=1 elapsed_ms=2024
sanitize-gpu <eight slots> -> all sanitized GPUs are VFIO-ready count=8 elapsed_ms=2099

Reset scope per GPU is reported as the GPU alone, consistent with the one-GPU-per-group topology. CC mode survives the reset, and the devices come back bound to vfio-pci with their cdev nodes intact.

Two observations from the run:

1. sanitize-gpu opens the legacy group node, while the VMM hands QEMU the iommufd cdev. The reset path goes through /dev/vfio/<group>, but VMs are started with -object iommufd plus /dev/vfio/devices/vfioN. Those are mutually exclusive in the kernel, so while a device is still attached to an iommufd context the group open returns EBUSY. The gpu_reset.rs module doc says the group nodes "are the same ones QEMU opens", which stopped being true once configure_gpus moved to -object iommufd.

This matters in practice on a confidential-VM host, where the device stays attached for a while after the guest has powered down and QEMU has exited — lsof shows no holder, yet both the group open and VFIO_DEVICE_BIND_IOMMUFD fail until the VM's memory teardown completes. So sanitize-gpu is usable only once the VMM reports the VM fully stopped, not merely when the guest is down. Worth a line in the subcommand's help or docs.

2. Non-fatal error during the 8-GPU run. One GPU logged, from vfio-ioctls cleanup:

ERROR vfio_ioctls::vfio_ioctls::vfio_syscall: VFIO_GROUP_UNSET_CONTAINER ioctl failed: Invalid argument (os error 22)
ERROR vfio_ioctls::vfio_device: Could not unbind VFIO group: <n>

The reset itself succeeded and the remaining seven proceeded normally, so it looks like a teardown-path issue rather than a functional one, but it was reproducible.

Separately: switching away from the root-only sysfs bridge-control write is a real operational improvement. On the previous path the VMM, running as an unprivileged user, needed group-writable config on both each GPU and its upstream bridge, which does not survive a reboot without extra machinery. Removing that requirement is worth the change on its own.

(Testing note, not a defect of this PR: we ran this branch against 0.6.0-rc0 images, which it is not targeted at — Image::version_tuple parses each dot-separated component as u16, so a semver pre-release suffix fails to parse and make_sys_config then reports Unsupported image version: (0, 0, 0). We patched that locally to get the branch onto our images. Only relevant to whoever backports this.)