fix(guest-agent): make GPU telemetry cost nothing without a GPU and survive a stale sample - #1184
Merged
kvinwang merged 6 commits intoSep 7, 2026
Conversation
dstack-util's boot attestation gate already counted display-class PCI devices through sysfs. The guest agent needs the same answer to decide whether GPU telemetry is worth collecting at all, and a second copy of the vendor/class matching is how the two drift apart. The two callers want opposite failure policies -- the boot gate must fail closed when the inventory cannot be read, a telemetry gate wants to report no GPUs -- so the shared function returns the counts and each caller keeps its own policy on top.
nvml-wrapper's Device::is_cc_enabled calls nvmlSystemGetConfComputeSettings and never touches the device handle, so cc_enabled was a system-wide setting copied onto every row and described as per-GPU. It moves next to cc_ready, which is system-wide for the same reason. sample_age_ms is added because the agent now serves the last known snapshot rather than nothing when a sample is overdue. A consumer that cannot see the age cannot decide whether the numbers still mean anything. GpuDevice.error becomes repeated errors. It was a "; "-joined string that the metrics template split back apart to count failures, which any NVML message containing that separator would inflate.
NVML calls cannot be cancelled, so the sample belongs in a process the agent can kill. dstack-util is where it goes: it already links nvml-wrapper and already calls Nvml::init during boot setup, so this adds a subcommand rather than a dependency, and an operator can run it by hand inside the CVM. One process per sample instead of a resident helper. Nvml::init runs fresh every time, so a driver that loads after the agent started is picked up on the next sample rather than being cached as "no GPU" for the agent's lifetime. stdout is the JSON document and stderr carries the log, so the subscriber is pointed at stderr for this subcommand only. The previous in-agent helper installed no subscriber at all to protect its stdout protocol, which made every NVML warning it logged a no-op.
Three problems with sampling from inside the agent: A guest with no NVIDIA card paid for the feature. Any GpuInfo call, any /metrics scrape, and any dashboard load -- including on apps with public_sysinfo off, where the result was rendered nowhere -- forked a helper that then stayed resident answering "no driver" forever. The PCI scan now runs once per process and every later request on such a guest is an atomic load returning a constant. A card present without the kernel module is reported as that, rather than as no GPU. /metrics never returned GPU data. The non-blocking path served a placeholder whenever the snapshot was older than the 5s TTL, which is every scrape at any realistic Prometheus interval: dstack_gpu_nvml_up 0 and no series at all on a healthy GPU. Callers now get the last snapshot whatever its age, with dstack_gpu_sample_age_seconds alongside, and the refresh happens behind them. The persistent line protocol had no cancellation safety. The RPC timeout sat 1s above the sampler's, and a future dropped between writing the request and reading the reply left the pipe desynchronised for good. One process per sample removes the state that could desynchronise. Sampling itself moves to `dstack-util gpu-info`; TtlCell replaces the hand-rolled snapshot cache, which is where get_allow_stale comes from.
A collector that hangs burns the sample timeout and is then killed. On the success cadence the next scrape respawned it immediately, so a guest whose driver has wedged sat in a near-continuous spawn-and-kill loop -- useless work, and the state most likely to strand a process in an uninterruptible driver call where SIGKILL only queues. Serving is unchanged: the last outcome is always returned whatever its age. Only the decision to resample is delayed. Also records what public_sysinfo now exposes. GPU UUID and PCI bus address are new identifiers on that surface and belong in the table operators read before turning the switch on.
The module justified its cache by implying sampling is expensive. It is not: one `dstack-util gpu-info` run, fork to exit, measures 94-99 ms on a single H200 in CC mode. Leaving the wrong reason in place invites the next reader to delete the cache once they measure it themselves. The reasons that survive the measurement are that `/metrics` carries CPU, memory, disk and container state that must not sit behind a wedged driver call, that three surfaces can scrape at once and would each fork their own collector, and that enumeration cost grows with card count where only one card has been measured. Recorded alongside the number. Also states what lazy refreshing means for freshness. A served snapshot is about one scrape interval old, not one TTL old, which is what the two live scrapes showed at 14.41 s and 19.96 s against a 5 s TTL. That is the design working, but it is not what the constant's name suggests, and a consumer aligning these series with a host-side exporter needs to know the offset is there.
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
dstack-util’s main matches on cli.command twice by value, causing a use-after-move compile error in the new logging setup.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR is a follow-up to the GPU telemetry work in #1178, focusing on making telemetry effectively free on GPU-less guests, resilient to stale samples, and safer under NVML/driver failure modes by moving sampling out-of-process into dstack-util.
Changes:
- Add a shared sysfs-based PCI GPU inventory (
lspci::sysfs) and reuse it for both boot gating and guest-agent telemetry gating. - Rework guest-agent GPU telemetry to serve the last known snapshot regardless of age (with
sample_age_ms/dstack_gpu_sample_age_seconds) and refresh lazily in the background, with failure backoff. - Move NVML sampling into a new
dstack-util gpu-infosubcommand and update proto/templates/tests to reflect API changes (errors[], system-widecc_enabled,sample_age_ms).
File summaries
| File | Description |
|---|---|
| dstack/lspci/src/sysfs.rs | New sysfs PCI inventory helper + unit tests for GPU counting |
| dstack/lspci/src/lib.rs | Exposes the new sysfs module |
| dstack/lspci/Cargo.toml | Adds tempfile for sysfs inventory tests |
| dstack/guest-api/src/lib.rs | Extends proto roundtrip tests for new GPU fields (errors, sample_age_ms, cc_enabled) |
| dstack/guest-api/proto/guest_api.proto | Updates GPU telemetry schema (system-wide CC fields, errors[], sample_age_ms) |
| dstack/guest-agent/templates/metrics.tpl | Always exports last snapshot + adds sample-age and errors.len() counting |
| dstack/guest-agent/templates/dashboard.html | Updates GPU dashboard rendering for CC fields, sample age, and error list |
| dstack/guest-agent/src/models.rs | Adds a shared gpu_labels filter and template-level regression tests |
| dstack/guest-agent/src/main.rs | Removes the old hidden helper mode flag/logic |
| dstack/guest-agent/src/lib.rs | Drops re-export of the removed helper entrypoint |
| dstack/guest-agent/src/http_routes.rs | Avoids sampling for private sysinfo dashboard loads; uses cached gpu_info() for /metrics |
| dstack/guest-agent/src/guest_api_service.rs | Increases GPU RPC timeout and routes RPC to awaited-first-sample path |
| dstack/guest-agent/src/gpu_info.rs | New cached + out-of-process sampling implementation with lazy refresh + backoff |
| dstack/guest-agent/Cargo.toml | Swaps deps to support new approach (cached-cell, lspci; drops tokio io-util feature) |
| dstack/dstack-util/src/system_setup.rs | Reuses lspci::sysfs inventory for attestation gating |
| dstack/dstack-util/src/main.rs | Adds GpuInfo subcommand + special log routing for machine-readable stdout |
| dstack/dstack-util/src/gpu_info.rs | Implements one-shot NVML sampler producing JSON on stdout |
| dstack/dstack-util/Cargo.toml | Adds guest-api + lspci dependencies needed for GPU sampling/inventory |
| dstack/Cargo.lock | Locks new dependencies |
| docs/security/cvm-boundaries.md | Documents that public_sysinfo covers GPU metrics and exposed identifiers |
Review details
- Files reviewed: 19/20 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+1581
to
+1584
| match cli.command { | ||
| Commands::GpuInfo => builder.with_writer(std::io::stderr).init(), | ||
| _ => builder.init(), | ||
| } |
Comment on lines
58
to
+60
| let handler = GuestApiHandler::construct(context) | ||
| .map_err(|e| format!("Failed to construct RPC handler: {}", e))?; | ||
| let gpu_info = handler.gpu_info().await.unwrap_or_default(); | ||
| let system_info = handler.sys_info().await.unwrap_or_default(); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up review fixes on top of #1178, based on a stack of that branch, so this PR shows only the six commits added to it. Merging it updates #1178 in place.
The feature is right and the design write-up on #1178 is right. What follows are things the code does not yet do, found by reading it against a live GPU host and then confirmed on one.
What is fixed
A guest with no NVIDIA card paid for the feature. Any
GpuInfocall, any/metricsscrape and any dashboard load forked a helper that then stayed resident answering "no driver" forever — including on apps withpublic_sysinfooff, where the result was rendered nowhere. The PCI scan now runs once per process and every later request on such a guest is an atomic load returning a constant. A card present without the kernel module loaded is reported as that, rather than as no GPU./metricsnever returned GPU data. The non-blocking path served a placeholder whenever the snapshot was older than the 5 s TTL — which is every scrape at any realistic Prometheus interval. A healthy H200 reporteddstack_gpu_nvml_up 0and no device series at all. Callers now get the last snapshot whatever its age, withdstack_gpu_sample_age_secondsalongside, and the refresh runs behind them. Reproduced live before and after: see Validation item 8.The persistent line protocol had no cancellation safety. The RPC timeout sat 1 s above the sampler's, and a future dropped between writing the request and reading the reply left the pipe desynchronised for good. Sampling moves to
dstack-util gpu-info, one process per sample, which removes the state that could desynchronise.dstack-utilalready linksnvml-wrapperand already callsNvml::initduring boot setup, so this is a subcommand rather than a dependency, and an operator can run it by hand inside the CVM. A freshNvml::initper sample also means a driver that loads after the agent started is picked up on the next sample instead of being cached as "no GPU" for the agent's lifetime.A wedged driver was resampled on the success cadence. A collector that hangs burns the sample timeout and is then killed; the next scrape respawned it immediately, leaving such a guest in a near-continuous spawn-and-kill loop — useless work, and the state most likely to strand a process in an uninterruptible driver call where
SIGKILLonly queues. Failures now back off for 30 s. Serving is unchanged: the last outcome is always returned whatever its age.cc_enabledwas described as per-GPU but is not.nvml-wrapper'sDevice::is_cc_enabledcallsnvmlSystemGetConfComputeSettingsand never touches the device handle, so it was a system-wide setting copied onto every row. It moves next tocc_ready, which is system-wide for the same reason.GpuDevice.errorcould not be counted. It was a"; "-joined string that the metrics template split back apart to count failures, so any NVML message containing that separator inflateddstack_gpu_query_errors. It becomesrepeated errors.The sysfs GPU scan was about to exist twice.
dstack-util's boot attestation gate already counted display-class PCI devices; the agent needs the same answer. Shared throughlspci::sysfs, with the failure policy left to each caller: the boot gate must fail closed when the inventory cannot be read, a telemetry gate wants to report no GPUs.API changes on top of #1178
GpuInfoResponse.cc_enabledmoves fromGpuDeviceto the response, next tocc_ready.GpuInfoResponse.sample_age_msis added. The agent now serves the last known snapshot rather than nothing when a sample is overdue; a consumer that cannot see the age cannot decide whether the numbers still mean anything.GpuDevice.error(string) becomesGpuDevice.errors(repeated string)./metricsgainsdstack_gpu_sample_age_seconds.Both proto sides ship together, as #1178 already notes:
ProxiedGuestApidecodes and re-encodes, so a field the VMM does not know is dropped silently.Validation
Live on an H200 SXM 141 GB (
19:00.0, CC mode) in a real TDX CVM, with a VMM built from this branch merged with #1065 (needed for GPU assignment),sanitize_on_attach = false. Nine checks, all passing:dstack-util gpu-infoagainst the real cardProxiedGuestApi.GpuInfothrough the VMMcc_enabled/sample_age_ms/errorsall survive the re-encode/metricson the GPU CVM/metricson a GPU-less CVMnvml_up 1, zero device series79.7 WItem 8 is the one that matters most here. Both ages are far past the 5 s TTL, and the old code returned
dstack_gpu_nvml_up 0and zero series in both — that is, a healthy H200 reporting "no GPU" at every realistic scrape interval.Three unknowns from #1178's Known limits are now measured:
NotSupportedunder CC mode is empty, at least on a single H200.query_errors = 0, stderr silent, SM utilization, all three memory fields, temperature and power all available,cc_enabled = 1andcc_ready = 1.dlopen,nvmlInit_v2, enumeration, every field query and JSON serialization. The 10 sSAMPLE_TIMEOUThas two orders of magnitude of headroom. Independently corroborated by item 8: the age of 19.96 s at a 20 s interval puts the refresh triggered by the first scrape at ~40 ms.gpu-infoprocesses, 0dstack-utilprocesses, 0 NVIDIA devices on PCI.memory_total_bytes = 150754820096(140.40 GB) matches an H200 SXM 141 GB. UUID and PCI BDF formats align with the DCGM exporter label set, as intended.Also on a TDX lab host without GPUs:
cargo fmt --check,cargo clippy -p dstack-guest-agent -- -D warnings, and 133 + 5 + 4 tests green acrossdstack-guest-agent,guest-apiandlspci.Not addressed
sample_age_msanddstack_gpu_sample_age_secondsexist so a consumer can correct for the offset when aligning against a host-side exporter. A background ticker would cap the age at the TTL for a permanent ~2% of a core on every GPU guest, to publish a number the consumer can already derive. The last commit records this so the next reader does not have to rediscover it.SAMPLE_TIMEOUTstays at 10 s despite the 94-99 ms measurement.nvmlInit_v2enumerates every card and only one card has been measured; trading an untested case for faster recovery in the tested one is the wrong direction until an eight-card CVM has been through this.