[yummybomb]

Summary

Deleting a VM leaves two pooled connections and a cached balloon target behind in hypeman. None of them is dropped until hypeman restarts.

QMP client. Hypeman keeps one QMP client per QEMU socket in a process-wide pool (lib/hypervisor/qemu/pool.go). Pool entries are only dropped by hv.Shutdown(), a QMP error, or a later start that reuses the same socket path. Delete never calls any of those. It shuts the guest down over vsock or kills QEMU directly, and the socket path is unique per instance. So every deleted QEMU VM leaves its QMP client in the pool, which keeps the unix socket fd open. Stop has the same gap when the guest shuts down by itself.

Guest-agent connection. Delete closes the pooled guest-agent gRPC connection, then sends the graceful shutdown RPC, which opens and pools it again. Nothing closes that second connection. A gRPC client whose peer is gone keeps three goroutines. Stop and standby already close it after the VM exits. This one is not specific to QEMU.

Balloon target. QEMU and Cloud Hypervisor cache the last balloon target per socket path and only clear it in Shutdown() and DeleteVM(). A VM that is deleted, or whose guest shuts down or crashes by itself, keeps the entry. When that VM is started again on the same socket path, hypeman reports the previous boot's target for a process whose balloon was never set, and the memory controller only writes a target when it differs from the cached one.

On a production GPU host, the count of dead unix connections (ESTAB, no local path, peer inode 0) tracks the count of deletes since the last hypeman restart: 880 connections against 882 deletes over about 29 hours. Hypeman goroutines on QEMU hosts climb by about 2.5 per deleted VM, and they only reset when hypeman restarts.

The goroutines come from the guest-agent connection, not from QMP. QEMU closes its QMP socket cleanly both when it exits and when it is killed, so go-qemu's reader goroutines exit by themselves and only the fd stays open.

Changes

• hypervisor.RegisterVMMStateReleaser / hypervisor.ReleaseVMMState: a per-type hook that drops what a hypervisor type caches for a socket whose VMM has exited. Hypervisor types that cache nothing ignore it.
• QEMU (standard and microvm) registers a releaser that closes the pooled client and its fd, and forgets the cached balloon target. Cloud Hypervisor registers one that forgets its balloon target.
• Starting a QEMU or Cloud Hypervisor process drops the same cached state first. This covers a VMM that exited by itself, which is restarted without going through stop or delete. resetClient, which only dropped the QEMU client at start, is replaced by this.
• BalloonTargetCache.Load no longer hands a recreated socket the previous target. The cache is keyed by socket path, device and inode, but on a miss Load fell back to the key last indexed for the path, which is how the old process's target came back. The fallback now only applies when the socket file is missing, which can happen while its VMM is still running. On a filesystem that reuses the freed inode the key itself still matches, so the clear at start is needed as well.
• deleteInstanceWithOptions releases the client once the VMM is confirmed gone, whether through graceful guest shutdown or kill, and closes the guest-agent connection again at the same point. A failed kill still returns before this.
• stopInstance releases the client next to the existing stale-socket cleanup. Standby and the stop fallback already evict through hv.Shutdown(), so they don't change.
• A QEMU start that fails after connecting to QMP now drops its pooled client in the start cleanup. Before, a failed create left it pooled.

Testing

Leaked per deleted VM, measured with real QEMU 7.2 under KVM (4 VMs per mode, goroutines classified from a goroutine profile):

	delete with graceful shutdown	delete with SkipGracefulShutdown
main	1 socket fd, 3 goroutines	1 socket fd, 0 goroutines
this PR	0	0

All leaked goroutines were grpcsync.(*CallbackSerializer).run, and none were in go-qemu. With only the QMP release and not the guest-agent close, the fds were gone and the 3 goroutines per graceful delete remained. The scratch test that took these numbers is not part of this PR, and the numbers predate the balloon-target changes.

Balloon target after the guest exits by itself and the instance is started again, measured with real QEMU 7.2 and Cloud Hypervisor v51.1 (2048 MiB assigned, target set to 1536 MiB on the first boot):

	QEMU	Cloud Hypervisor
before the start-path clear	1536 MiB (stale)	1536 MiB (stale)
with the start-path clear	2048 MiB	2048 MiB

These were taken before the Load fallback change, on a filesystem that does not reuse inodes.

• TestReleaseVMMStateClosesPooledConnection (qemu): a pooled client connected to a fake QMP server is removed from the pool, the server sees the connection close, and the cached balloon target is gone. Covers both QEMU types. It fails if the releaser isn't registered or doesn't clear the target.
• TestDeleteAndStopReleaseVMMState (instances): delete and stop of a running instance each call the registered releaser with the instance socket. Each subtest fails if its call site is removed.
• TestDeleteClosesGuestConn (instances, Linux): deleting a running instance closes the guest-agent connection that the shutdown RPC pooled. It fails without the new close.
• TestCleanupFailedStartDropsPooledClient (qemu): the failed-start cleanup removes the pooled client. It fails without the release. The call from startQEMUProcess into that cleanup is not covered by a test.
• TestStartQEMUProcessDropsStaleBalloonTarget (qemu): starting a real QEMU process on a socket path with a cached target leaves no target. It is skipped when QEMU is not installed, and fails without the release at start.
• TestBalloonTargetCacheMissesForRecreatedSocket and TestBalloonTargetCacheSurvivesMissingSocket (hypervisor): a recreated socket misses, and a missing socket still finds its target. The first fails with the old unconditional fallback, the second with no fallback at all.
• TestReleaseVMMStateForgetsBalloonTarget (cloudhypervisor): the releaser clears the cached target. It fails if the releaser isn't registered. The clear on Cloud Hypervisor's start path has no unit test; the table above is its only check.
• go vet ./lib/hypervisor/... ./lib/instances/ is clean, and go test -race passes for the new tests and the existing lifecycle, stop and delete unit tests.
• I haven't validated this on a GPU host. After deploy, the check is that dead unix connections on QEMU hosts stay near zero and that the goroutine count stops climbing with deletes.

Not covered

A guest RPC that races a delete can still create and pool a guest-agent connection after delete's second close. Exec closes it again when the dial to the dead guest fails. The stat and cp handlers do not. That race predates this PR.