Detect wedged vGPU VFs from the guest and report to the health store

[yummybomb]

Summary

Top half of the wedged-VF work, stacked on #462 (the VF health store, placement exclusion, admission, and /resources fields). This PR adds the detection path that feeds that store.

• The guest agent watches /dev/kmsg for kernel-facility NVRM: ... RmInitAdapter failed! records and exposes the current GPU initialization state over GetGPUInitStatus via vsock.
• A host sentinel controller polls assigned vendor-VFIO instances with up to 64 concurrent checks and reports failures and successes into the VF health store from Quarantine unhealthy vGPU VFs via a persisted health store #462.
• Only instances with a live VMM (control socket present) are polled; a stopped or standby instance whose failed release retained its assignment is skipped, since QEMU vsock dials by guest CID alone and a stale CID could be reused by an unrelated instance.

Safety and failure handling

• GPU initialization success is terminal in the guest agent, so a delayed failure watcher cannot replace a confirmed success.
• The host trusts only the guest-agent RPC; workload output on the shared serial console cannot influence the tally.
• A slow nvidia-smi attempt is killed after 30 seconds. The guest agent waits for that process to be reaped before retrying, so an attempt stuck in uninterruptible I/O cannot accumulate concurrent probes; the independent kmsg watcher still reports the underlying init failure.
• Unreachable guests do not serialize a host-wide scan; checks run with a fixed concurrency limit.
• Repeated polls and controller restarts do not double-count an assignment. The sentinel revalidates the VF assignment before reporting, and the health store deduplicates reports by assignment.
• Create allocates the vGPU immediately before metadata persistence to minimize the unpersisted window.
• Reconciliation preserves assignments when hypervisor liveness is uncertain and records that condition in logs and metrics.

Observability

• hypeman_instances_vgpu_sentinel_init_failures_total
• hypeman_instances_vgpu_sentinel_quarantines_total
• hypeman_instances_vgpu_sentinel_checks_total by result (ok, failed, unknown, rpc_error, unsupported_agent, or list_error)
• hypeman_instances_vgpu_quarantined_vfs
• hypeman_instances_vgpu_vf_health_store_unavailable
• hypeman_instances_vgpu_reconcile_liveness_uncertain_total

lib/devices/GPU.md gains the detection and sentinel documentation.

Out of scope

An operator force-cycle endpoint. Recovery remains the documented manual DCGM quiesce, SR-IOV cycle, state edit, restart, and verification flow.

Testing

Passed:

go test -race ./lib/system/guest_agent ./lib/guest ./lib/instances -run 'Test(VGPUSentinel|ListVGPUSentinel|ProbeGPUInit|RunGPUProbeAttempt|GPUInitReporter)' -count=1
go vet ./lib/system/guest_agent ./lib/guest ./lib/instances ./lib/devices ./lib/providers ./lib/resources
go test ./lib/devices ./lib/providers ./lib/resources

The full lib/instances race suite did not complete cleanly locally because its integration tests require additional host networking and image tooling and surfaced unrelated dependency races.

The underlying wedge signal and manual recovery sequence were previously validated on L40S hardware. A live end-to-end run of the guest watcher and host controller is still required before merge, including two failed assignments and the success path.

---

Note

Medium Risk
Changes vGPU placement safety, automatic VF quarantine, and create/reconcile timing on GPU hosts; incorrect guest signals or assignment confirmation could remove capacity, though running instances are not detached.

Overview
Adds an automatic detection path for wedged vendor-VFIO virtual functions: the guest agent watches /dev/kmsg for kernel NVRM RmInitAdapter failed! lines (and optionally probes with nvidia-smi -L), exposes GetGPUInitStatus over vsock, and the new VGPUSentinelController polls running vendor-VFIO instances every 5s (up to 64 concurrent) to record failures and successes into the VF health store from #462.

Host wiring: Wire injects VGPUSentinelController and main starts it alongside other background controllers. Create path now allocates the vGPU immediately before metadata persistence so assignments are less likely to exist only in memory.

VF health store: Adds RepairVFHealthStore (used once per sentinel tick); failure/success reports no longer auto-retry a failed persist on every call—they fail closed until repair succeeds. vGPU reconcile treats ambiguous hypervisor liveness as “preserve assignment” with a warning and hypeman_instances_vgpu_reconcile_liveness_uncertain_total.

Docs/metrics: GPU.md documents sentinel behavior, metrics (sentinel_checks_total, quarantine gauges, store-unavailable), and tightened error text for failed-create retention stubs.

^{Reviewed by Cursor Bugbot for commit b23a654. Bugbot is set up for automated code reviews on this repo. Configure here.}

[yummybomb]

one blocking test-quality issue. the sentinel/store boundaries, fail-closed placement behavior, assignment revalidation, and persistence rollback paths otherwise look coherent in context. targeted race tests and go vet pass locally; the current linux CI also exposes the changed-test failure below.

[yummybomb]

blocker: this 10ms deadline is shorter than normal shell startup under race/loaded CI, so the nominally successful second attempt can time out too. The current Linux job failed here with actual: 3, and go test -race ... -count=100 reproduces it locally. Please make the attempt runner/clock injectable and use a deterministic fake, or otherwise give the success attempt independent scheduling margin; asserting exactly two real process launches with this deadline is flaky.

[yummybomb]

fixed in 88f3c3d by injecting the probe attempt into the retry loop and testing the timeout/success sequence directly. go test -race ./lib/system/guest_agent -run TestProbeGPUInitRetriesAfterAttemptTimeout -count=100, the package race suite, and go vet pass.

[yummybomb]

two code-quality findings. the store/placement boundaries, persistence rollback, assignment revalidation, and tests otherwise look coherent in repo context. the new functions peak at cyclomatic complexity 16 (loadLocked / selectLeastLoadedVF); those paths remain cohesive and well-covered, so I don't see mechanical decomposition as a merge requirement.

[yummybomb]

The tail offset is process-local, but a successful report removes the only persistent dedup record. After FAILED -> OK, restarting hypeman starts at offset 0, records the historical failure again, increments init_failures_total, then clears it again on the historical OK. With vf_quarantine_threshold: 1, replay also emits a fresh quarantine/rescind and increments both counters on every restart; if that replayed clear fails, a healthy VF stays quarantined. Please retain a bounded last-resolved assignment/watermark per VF (or coalesce a replay before mutating state) and add a restart regression test.

[yummybomb]

non-blocking: please thread ctx into scanSentinelLog and check cancellation between reads (and target scans). On the first pass after controller/process start every assignment begins at byte 0, so a host with many logs near the 50 MB rotation limit can keep Run inside this loop and hold the process-level errgroup.Wait well after shutdown was requested.

[yummybomb]

reviewed the change against hypeship/vendor-vfio-vgpu and traced the sentinel through guest output, host log scanning, durable health state, placement, and admission. the persistence rollback and assignment-identity handling are coherent, but the serial marker is not actually source-authenticated, and timed-out probes can overlap in the exact uninterruptible-ioctl case this code targets.

leaving the first finding as a blocker; github does not allow this account to request changes on its own PR.

[yummybomb]

blocker: a standalone line does not establish that the guest agent wrote it. In exec mode the customer entrypoint's stdout/stderr is wired directly to the same serial console (lib/system/init/mode_exec.go:132-133), so an ordinary workload can print a complete line matching this regex without root or /dev/kmsg access. Repeating that across assignments can quarantine healthy VFs, and the runbook's statement that only a root guest can forge this is therefore incorrect. Please carry the report over a channel whose source the host can distinguish (for example a dedicated guest-agent RPC/transport), or otherwise authenticate the source; matching the log shape is not sufficient here.

[yummybomb]

this bounds how long the caller waits, but not the attempt's lifetime. If nvidia-smi is stuck in the uninterruptible ioctl described above, Kill does not make cmd.Wait return; the process and waiter goroutine remain live while probeGPUInitUntil launches another attempt after 15s. The 10-minute loop can therefore accumulate many concurrent stuck probes on the VF. Can we retain the in-flight attempt and avoid launching another until it is reaped (or stop retrying once an attempt cannot be killed), so there is at most one outstanding nvidia-smi?

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit b23a654. Configure here.}