[yummybomb]

Summary

Deleting 16 or more vGPU guests on one host at the same moment failed most of the deletes with a 500, about a second before their VMMs had exited. The grace a VMM with VFIO devices gets after SIGTERM now grows with the number of such teardowns in flight.

Why it happened

A delete asks the guest to shut down and waits up to 5 s, then sends the VMM SIGTERM and waits 5 s, then SIGKILL and two waits of 2 s. A VMM with VFIO devices releases them on its way out, and the host driver is slower at that when many go at once. On a host with two L40S, with 6 GB guests on a 3 GB vGPU profile, on a build of #552, which does not change how a delete stops the VMM:

Deleted together	Guests	Deletes that failed	VMMs gone, from the delete request
1	forked from a snapshot	0	5.4 s
8	forked	0	5.9 to 6.1 s
16	forked	15 of 16	14.0 to 15.3 s
26	forked	15 of 26	10.4 to 15.3 s
16	cold-booted	15 of 16	14.0 to 15.4 s

SIGTERM goes out 5 s in, so the VMMs needed up to 10 s after it. SIGKILL followed at 10 s and the request gave up at 14 s. Kernel stacks sampled while they waited were in the vGPU driver, under nv_vgpu_vfio_migration_set_state and nv_vgpu_vfio_close; the kernel log showed no errors.

Every failed delete succeeded on retry and no device was left behind. But 42 VMMs across the two forked batches were hard-killed while they were releasing their device, which is what the SIGTERM grace is there to prevent.

What changes

terminateThenKill counts the VFIO teardowns in flight. The wait after SIGTERM is the configured grace plus one second for every other teardown in flight, capped at one minute, and it is re-evaluated while it waits, so a VMM that started exiting alone still gets the time when others join it. A guest terminated alone waits exactly as before. The SIGKILL waits are unchanged.

Testing

• TestKillHypervisorWaitsLongerWhileOtherVFIOTeardownsAreInFlight: a process that takes 0.6 s to exit after SIGTERM is hard-killed when it is alone with a 100 ms grace and exits cleanly when two other teardowns are in flight. TestKillHypervisorCountsVFIOTeardownsInFlight: two real teardowns, where the second one only gets its time because the first is counted. Four mutants (no allowance, an allowance when alone, a teardown that never joins the count, one that never leaves it) each fail one of them.
• The same deletes on the same host, on a build of Add opt-in vGPU snapshots and forks on a managed QEMU runtime #552 plus this change:

Deleted together	Guests	Deletes that failed	VMMs gone, from the delete request
12	forked	0	11.6 to 12.8 s
16	forked	0	13.9 to 15.3 s
26	forked	0	10.4 to 15.4 s
16	cold-booted	0	14.1 to 15.6 s

No VMM was hard-killed and no device was left behind.

Not covered

• The deletes still take as long as the driver needs: about 15 s per request for 16 or more together, against 5.5 s for one alone. Of that, 5 s is the graceful shutdown wait, which these guests never finished within.
• More than 26 guests, or other GPU models and vGPU profiles. The allowance is a second per teardown because 16 to 26 needed about 10 s in total; I have no measurement that says how that grows beyond 26.