[yummybomb]

Summary

A fork that was still being brought up when Hypeman was killed stayed behind as an ordinary instance, holding its VMM, its memory and its vGPU. A fork is now marked pending until its request succeeds, and at startup Hypeman deletes the marked forks whose API process is gone.

Why it happened

A fork is made in two steps: its metadata is saved, then it is brought to the state the caller asked for. If the process dies in between, the fork stays wherever it got to. Nothing on disk says it is unfinished, so after the restart it looks like any other standby or paused instance. The caller was disconnected without a response and has no ID to delete.

For a QEMU guest with a vGPU this is expensive: a restore that was cut short keeps its VMM, the guest memory it had pinned and its virtual function until someone deletes the instance by hand.

What changes

• StoredMetadata.ForkPending is set when a fork's metadata is first saved, for instance forks and snapshot forks, and cleared once the fork has reached its target state. It names the API process that is making the fork, by PID, start time and boot ID. If clearing it fails, the fork request fails and the fork is cleaned up like any other failed fork, so a fork whose caller got a success is never still marked.
• RecoverInterruptedForks runs once at startup, before the API listens and before the vGPU reconciler starts. It deletes a marked fork only when the process named in the mark no longer exists: its PID is gone, belongs to a process started at another time, or was recorded before a reboot. When it cannot tell, it leaves the fork alone and logs a warning.
• Snapshot metadata never carries the mark, so nothing restored or forked from a snapshot of a pending fork inherits it.

Forks that completed are not touched, and a graceful restart still lets in-flight forks finish.

The owner check is there because nothing stops a second API process from starting against the same data directory, and the binary has no flags: running it to look for a version starts it. Without the check, that process would delete the forks the running one was still bringing up.

Testing

• TestForkIsPendingOnlyUntilItsRequestSucceeds covers the mark through an instance fork, a snapshot fork, a fork left half-made and a snapshot of that fork. TestRecoverInterruptedForksDeletesOnlyForksWhoseProcessIsGone runs the startup pass over a fork whose process exited (deleted), one whose process is still running (kept) and a finished one (kept). Two more tests cover the owner check: an exited process, a reused PID, an earlier boot, and a live PID with nothing else recorded.
• Nine mutants each fail at least one of them: the mark never cleared for either kind of fork, the mark kept in snapshots, the mark not naming the process, recovery ignoring the owner, recovery deleting nothing, and the owner check ignoring the PID, the start time or the boot ID.
• On a host with two GPUs I killed Hypeman with SIGKILL while four vGPU snapshot forks were restoring, on a build of Add opt-in vGPU snapshots and forks on a managed QEMU runtime #552 plus this change:

Killed after	Fork requests answered	Deleted at startup	Left after restart
0.3 s	0 of 4	4	nothing
0.7 s	0 of 4	4	nothing
1.0 s	0 of 4	4	nothing
1.5 s	0 of 4	4	nothing
2.2 s	1 of 4	3	the answered fork, running and usable

No VMM, virtual function, TAP or guest directory was left in any case, and the next fork passed. Without the change the same kill at 1.0 s left two forks Paused and two in Standby, with two VMMs still holding two virtual functions (two runs).

• A graceful restart 1.0 s into four forks: all four requests succeeded, nothing was deleted at startup, and all four guests were running and usable.

Not covered

• Two API processes on one data directory were not run on a host. The test above stands in for it.
• The mark is only cleared on success, so a fork that a failed request leaves behind without cleaning it up is deleted at the next start too. A fork from a running instance whose source cannot be restored afterwards looks like such a case from the code; I did not exercise it.
• A VMM started in the instant before the kill, before its control socket exists and before its PID is saved, cannot be found by the delete. That window is the same as for any start and is not changed here.
• A fork from a running instance puts its source in standby first. If Hypeman dies during that, the source stays in standby. This change only deals with the fork.
• On macOS there is no start time or boot ID to compare, so the check has only the PID. A fork whose PID has been reused by another process is kept, as it is today.