diff --git a/HACKING.md b/HACKING.md index 21b384487..c34292da4 100644 --- a/HACKING.md +++ b/HACKING.md @@ -985,8 +985,45 @@ resident. The idle processor polls the device error sink, and the first fault is sent once through the machine's fatal channel. Fatal GPU errors reject reset and restart before touching CPU planes; a successful quiescent reset after a recoverable rejection rearms notification. -Newport still rejects `[graphics] raster = "wgpu"` until M3; full opcode and -lifecycle coverage are future work. The legacy prototype remains +Stopped resident snapshots take one lifecycle write lock against PIO, DMA and +monitor producers. They require an empty published GFIFO, no worker or active +display cut and no incomplete HostRW transfer, then capture register/context +state and both complete GPU planes at one submitted stream position. The +GFIFO itself is ephemeral and is not serialized. Restore stages validated +register and full-plane halves, uploads both planes into a new epoch, and only +then publishes the registers and permits processor/display restart. Old +scanout bindings and composed captures are invalidated. Missing, duplicate or +failed halves leave the board stopped rather than exposing partial state. +Disk snapshots, CI rollback, live debugger checkpoints and a second Newport +head share this transaction; reset clears authoritative GPU planes without +refreshing stale CPU shadows. + +Resident telemetry is aggregate-only and checked. It classifies primitive and +address kinds, mode features, conservative clipped coverage, HostRW transfer +sizes, nonempty batch sizes and flush reasons; board counters cover GFIFO +occupancy, reservation failures, `BUS_BUSY`, frame cuts and render completion. +Snapshots reject epoch changes, decreasing counters and overflow rather than +wrapping. The ordinary resident path performs no telemetry map or readback. +An explicit profiling option waits for each nonempty batch and maps one +64-byte metadata record with timestamps, busy-tile totals and the generated +indirect dispatch. Profiling metadata is separate from raw-plane and composed +capture traffic. + +In addition to capacity and architectural boundaries, a continuously fed +resident batch is submitted at the next GFIFO command boundary after 4 ms. +This cannot split a command, HostRW transfer or bounded-set wait. A replay of +66,741 embedded-PROM command-monitor records on Vulkan and D3D12 accepted +20,987 operations with no unsupported operation or read mismatch. It formed +43-51 real batches; 25-31 were latency flushes. Eight direct presentations of +the replayed state had conservative p95 bounds of 2.10 ms for render +completion and 0.26 ms for the frame cut with no capture readback. HostR was +four operations and reused two selected-plane maps, so the measured design +keeps full-plane HostR snapshots. The same evidence retains 8 x 8 tiles, +GPU-generated active-tile dispatch, 1,024 state slots and three reusable sets. +These are gated replay measurements, not production IRIX application timing. + +Newport still rejects `[graphics] raster = "wgpu"` until production activation; +live frontend integration remains future work. The legacy prototype remains CPU-canonical. Residency tests can opt into replacing each CPU shadow with its own diff --git a/TESTING.md b/TESTING.md index 56aedb53d..c62b45daf 100644 --- a/TESTING.md +++ b/TESTING.md @@ -435,17 +435,17 @@ Remove-Item Env:\IRIS_NO_JIT -ErrorAction SilentlyContinue .\scripts\test-resident-gate.ps1 ``` -The exact 103-case script requires 28 `resident_` CPU tests, four flat-fill -tests, two line tests, two HostW walker tests, one HostW interpreter/JIT source -test, one shade-DDA test, one pattern-state test, two resident HostR +The exact 130-case script requires 43 `resident_` CPU tests, four flat-fill +tests, two line tests, two HostW walker tests, one HostW interpreter/JIT +source test, one shade-DDA test, one pattern-state test, two resident HostR port/retirement tests, three command-ABI tests and three direct-state tests, -then **19 resident GPU tests and nine direct scanout tests on each backend**: -103 tests in total. The window test requires an available native desktop and a -GPU that can present to it. The script checks each libtest pass count, rejects -hardware skips and prints the selected adapter; both Vulkan and D3D12 must -succeed for the full gate. `.\scripts\test-resident-gate.ps1 -Backend vulkan` -or `-Backend dx12` runs just one hardware backend and is not the complete -two-backend gate. For manual PowerShell runs set +then **24 resident GPU tests and ten direct scanout tests on each backend**. +That is 130 tests in total. The window test requires an available native +desktop and a GPU that can present to it. The script checks each libtest pass +count, rejects hardware skips and prints the selected adapter; both Vulkan and +D3D12 must succeed for the full gate. `.\scripts\test-resident-gate.ps1 +-Backend vulkan` or `-Backend dx12` runs just one hardware backend and is not +the complete two-backend gate. For manual PowerShell runs set `$env:REX3_RESIDENT_BACKEND = 'vulkan'` or `'dx12'` before each command instead of the shell prefix. A host that lacks either required hardware backend cannot claim the complete gate. Metal is not covered by this script. @@ -460,6 +460,49 @@ selected display mode unchanged. Assertions distinguish per-mode composed capture attempts, successes and bytes from attributed raw-plane `Compatibility` observations. +The lifecycle cases stop and serialize producers at an empty GFIFO boundary, +capture registers and both complete authoritative planes once, and restore +them into a new epoch before restart. They cover disk snapshots, CI rollback, +live debugger checkpoints, a second Newport head, reset, monitor exclusion, +HostW continuation and sticky failure. Hardware cases round-trip complete +GPU-written planes and both resident display modes while the CPU framebuffer +mappings remain protected. Missing, duplicate, incomplete or failed snapshot +halves must not publish registers, pixels, captures or a restartable board. +Metal remains outside this gate. + +The M8 cases add checked histogram/delta accounting, exact GFIFO +full-versus-contention telemetry, bounded-set wait accounting, a 4 ms +continuous-stream batch-age boundary and profiling-only validation of the +GPU-generated active-tile dispatch. The independent mixed-workload hardware +gate compares complete planes and context against the CPU engine after fills, +copies, integer/fractional/antialiased lines, HostW, HostR, direct +presentations and explicit observations. It also requires zero ordinary +raw-plane/composed readback and reports typed batch, GFIFO and latency +aggregates. + +An optional local recorder file can be replayed through the resident backend +without printing records or pixels. The file is not a test artifact and must +not be committed. `REX3_RESIDENT_PROFILE` enables a 64-byte metadata map and +wait per nonempty batch; omit it to verify the default map-free path. +`REX3_RESIDENT_REPLAY_FRAMES` optionally presents the final replayed state +through direct wgpu scanout that many times without requesting capture. The +recording must contain valid final display state when that option is used. +For example, on PowerShell: + +```powershell +$env:REX3_RESIDENT_REPLAY = 'C:\local\newport.rec' +$env:REX3_RESIDENT_BACKEND = 'vulkan' # repeat with dx12 +$env:REX3_RESIDENT_PROFILE = '1' +$env:REX3_RESIDENT_REPLAY_FRAMES = '8' +cargo test --locked -p iris --lib --features wgpu,rex-jit,rexdiag,gputrace resident_replay_aggregate_file -- --ignored --nocapture --test-threads=1 +``` + +The aggregate reports records and checked reads, accepted/no-op/unsupported +operations, typed primitive/mode/coverage/HostRW distributions, nonempty +batches and flush reasons, GFIFO/backpressure, attributed transfer bytes and +conservative latency buckets. Replay throughput and optional post-replay +presentations are test-path measurements, not live guest wall-clock timing. + To deliberately regenerate, set `REX3_CORPUS_OUT` to an existing relative directory with a pre-existing regular `.rex3-corpus-root` file and run `rex3_corpus_generate -- --ignored --nocapture` with the strict CPU feature diff --git a/clanker/plans/GPUTodo.md b/clanker/plans/GPUTodo.md index 26c86d352..9d2150c3e 100644 --- a/clanker/plans/GPUTodo.md +++ b/clanker/plans/GPUTodo.md @@ -1,6 +1,6 @@ # REX3 proper GPU mode: implementation plan -Status: upstream integration in progress 2026-11-13. Upstream +Status: upstream integration in progress 2026-11-15. Upstream [#569](https://github.com/techomancer/iris/pull/569) merged M0, the gated test-only M1 resident slice and a minimum M2 scanout slice. [#731](https://github.com/techomancer/iris/pull/731) merged the test-only @@ -10,9 +10,11 @@ common wgpu presenter on a native window, test-only) is on `main` too (checked 2026-11-15 at `dcd54c20`). #748 reduced DID dispatch latency and #749 completed the gated M2 readback compatibility slice. #750 is the open upstream M3 branch and is being expanded to carry paired M3+M4 coverage. The -remaining M3-M10 work exists in an unmerged candidate stack, not in upstream -`main`. -No production GPU-authoritative Newport mode has landed. The tile-binned +local cumulative paired branches now carry M5+M6 and M7+M8. M0-M8 are +complete for the gated, test-only candidate path, but those paired branches +are not yet in upstream `main`. Production activation, application-level M8 +acceptance and M9/M10 remain open. No production GPU-authoritative Newport +mode has landed, and live `raster = "wgpu"` remains rejected. The tile-binned `IRIS_REX3_GPU` prototype is separate and CPU-canonical, not the target architecture. @@ -747,9 +749,10 @@ Partial, test-only: a read-only same-device binding and GFIFO frame marker order resident fills, direct scanout and a GPU frame consumer. Screenshot, frame-request, headless and VNC tests capture composed output only. Ordinary frames after seeding have zero raw-plane upload/readback, CPU-plane access -and CPU fallback. In the presenter follow-up (now on `main`), a test-only `WgpuRenderer` -handoff presents two ordered draws and a resize on a real native window using -the same GPU. An explicit diagnostic reads the presenter's unpacked frame +and CPU fallback. In the presenter follow-up (now on `main`), a test-only +`WgpuRenderer` handoff makes three presentations, including a resize, on a +real native window using the same GPU. An explicit diagnostic reads the +presenter's unpacked frame texture for pixel parity with the software compositor; it does not inspect final swapchain or on-screen pixels. This threaded test is available on Windows and Linux; macOS needs a main-thread window harness. Resident refresh @@ -772,14 +775,14 @@ It is deliberately slow and never an automatic fallback. M2 is complete within this gated test-only scope. Production live integration and final on-screen validation remain open; `iris-gui` cannot select this mode. -The Windows gate `.\scripts\test-resident-gate.ps1` checks 103 cases: 28 +The Windows gate `.\scripts\test-resident-gate.ps1` checks 130 cases: 43 resident CPU, 4 flat-fill, 2 line, 1 shade-DDA, 1 pattern-state, 3 command-ABI and 3 direct-state tests, 2 HostW walker tests, 1 HostW interpreter/JIT source -oracle, 2 resident HostR port/retirement tests, plus 19 resident GPU and 9 +oracle, 2 resident HostR port/retirement tests, plus 24 resident GPU and 10 direct-scanout tests on each of Vulkan and D3D12. It requires the named backend on an integrated/discrete adapter and a native desktop, and fails rather than skipping when unavailable; a single-backend run is not the -complete 103-test gate. See +complete 130-test gate. See `TESTING.md` for commands and prerequisites. ### M3 - Command ABI and complete uber-shader @@ -1056,34 +1059,109 @@ resident mode yet. ### M7 - Snapshot, reset and failure lifecycle -- [ ] Define a quiescent GPU snapshot boundary. -- [ ] Serialize full planes and required command/context state. -- [ ] Restore by uploading before command processing resumes. +- [x] Define a quiescent GPU snapshot boundary. +- [x] Serialize full planes and required command/context state. +- [x] Restore by uploading before command processing resumes. - [x] Define device-loss behavior without CPU-current checkpoints. -- [ ] Re-run Stop/reset/restore/debugger/frontend lifecycle tests. -- [ ] Validate Vulkan and D3D12; unblock Metal only after safe completion and +- [x] Re-run Stop/reset/restore/debugger/frontend lifecycle tests. +- [x] Validate Vulkan and D3D12; unblock Metal only after safe completion and destruction are demonstrated. Complete when snapshot round trips are exact and GPU failure cannot return success with stale or partially recovered framebuffer state. -Device loss is sticky and fatal in the gated resident path, with one machine -notification and no restart from stale CPU planes. Production lifecycle and -snapshot support remain open. +Complete within the gated resident backend. A lifecycle write lock excludes +PIO, DMA and monitor producers while Stop joins the display and GFIFO workers. +The snapshot boundary requires a stopped worker, a fully published empty +FIFO, no active display frame and no incomplete HostRW transfer. It then +captures the complete register/context/display state and one attributed +two-plane `Snapshot` observation under the same lock. GFIFO contents are +deliberately not serialized. + +Restore validates and stages the register and full RGB/auxiliary halves +without publishing either one. Only after both planes have been uploaded into +a new resident epoch are the registers committed, old scanout bindings and +captures invalidated, and restart allowed. Invalid, duplicate, missing or +partially written halves leave the board stopped and require a complete retry. +Machine disk snapshots, CI rollback, live debugger checkpoints and a second +Newport head use the same atomic checkpoint. Reset zeros the GPU planes and +registers without reading or writing stale CPU shadows. + +CPU/test-double cases cover producer exclusion, Stop/restart, reset, debugger +commands, invalid input and sticky failure. Vulkan and D3D12 cases round-trip +complete GPU-written planes, HostW continuation state and direct/compatibility +display output while protected CPU planes remain untouched. Device loss or a +failed upload remains sticky and cannot return success or restart with +partial state. Metal remains blocked until its completion and destruction +behavior is exercised by an equivalent gate. ### M8 - Optimize only after complete live execution -- [ ] Capture representative live Newport workloads. -- [ ] Record primitive, mode, area and HostRW transfer distributions. -- [ ] Add GFIFO occupancy/backpressure telemetry. -- [ ] Tune tile size, active-tile dispatch and batch limits. -- [ ] Bound batching by both capacity and presentation latency. -- [ ] Compare full-plane and region HOSTR readback only if HOSTR is material. -- [ ] Re-run correctness after every scheduling/layout change. +- [x] Capture a representative, redistributable live Newport workload. +- [x] Record primitive, mode, area and HostRW transfer distributions. +- [x] Add GFIFO occupancy/backpressure telemetry. +- [x] Tune tile size, active-tile dispatch and batch limits. +- [x] Bound batching by both capacity and presentation latency. +- [x] Compare full-plane and region HOSTR readback only if HOSTR is material. +- [x] Re-run correctness after every scheduling/layout change. Complete when the selected design passes correctness and end-to-end gates on representative applications, not only synthetic fills. +The gated implementation now records checked, typed aggregate telemetry for +primitive and address kinds, mode features, clipped area and tiles, HostW and +HostR transfer sizes, nonempty batch sizes, flush reasons, command-generation +and submission time, bounded-set waits, GFIFO occupancy/backpressure and +frame-cut/presentation latency. Counter overflow, reset/epoch mismatch and +decreasing snapshots fail closed. Default operation maps no profiling data. +Explicit profiling waits for each nonempty batch and maps one 64-byte metadata +record containing timestamps, busy-tile totals and the GPU-generated indirect +dispatch; it never maps framebuffer data. + +Batching now has a 4 ms maximum age in addition to existing state, payload and +tile-list capacity boundaries. The processor checks that age after each GFIFO +command, so it cannot split one architectural operation or preempt a bounded +GPU-set wait. Idle, display, HostR, copy-hazard, observation, restore and exit +boundaries retain their existing ordering. Empty flush checks are not counted +as batches. + +A local recording of the embedded SGI PROM command monitor exercised real +guest `hinv`, `printenv`, `help` and `version` redraws without proprietary +installation media. Aggregate-only replay on both Vulkan and D3D12 executed +66,741 records, checked 398 reads with zero differences, accepted 20,987 +operations with 206 no-ops and zero unsupported operations, and issued 20,777 +GPU commands. It contained 20,781 draws, two HostW and four HostR operations; +12,443 draws used Z pattern, 150 used scissor and 109 used CID testing. It did +not contain copy or line primitives, which remain covered by the independent +mixed-workload hardware gate. + +After excluding empty submit checks, the replay formed 43-51 nonempty batches. +The latency guard caused 25-31 flushes under the continuously fed replay, +alongside 13-15 capacity flushes. Eight post-replay direct presentations on +each backend used no raw or composed readback and reported conservative p95 +bounds of 2.10 ms for render completion and 0.26 ms for the frame cut. These +are replay/test-path measurements, not guest wall-clock application timing. +The fastest producer drove GFIFO occupancy to 29,329-29,713 entries, but no +reservation was refused as full or contended. + +The captured HostR traffic was only four requests among 20,987 accepted +operations. Two generation-keyed selected-plane maps transferred 16 MiB +total, so region readback is not justified by this evidence. Profiling mapped +only 43-51 metadata records (2,752-3,264 bytes total), measured +165,234-167,299 busy tiles and 1,067,889 tile/primitive pairs, and confirmed +that GPU-generated dispatch matched the busy list. Measurements therefore +retain 8 x 8 tiles, GPU-generated active-tile dispatch, 1,024 state slots, +three reusable sets and full selected-plane HostR snapshots. + +The exact 130-case Vulkan+D3D12 gate covers the scheduling change and includes +an independent mixed workload with fills, copies, all line families, HostW, +HostR, direct presentation and explicit observation. The M8 implementation +checklist is complete within the gated backend. Its milestone acceptance for +normal use remains open until the production path runs representative IRIX +desktop applications and satisfies the end-to-end throughput, CPU-use and +frame-rate criteria below; the embedded PROM capture does not claim that +coverage. + ### M9/M10 - Production activation and frontends - [ ] Compile the resident path in normal wgpu builds after its prerequisites diff --git a/scripts/test-resident-gate.ps1 b/scripts/test-resident-gate.ps1 index 8367c85ac..3dbabf57d 100644 --- a/scripts/test-resident-gate.ps1 +++ b/scripts/test-resident-gate.ps1 @@ -66,7 +66,7 @@ function Invoke-ResidentTests { } try { - Invoke-ResidentTests 'resident_' 28 $false + Invoke-ResidentTests 'resident_' 43 $false Invoke-ResidentTests 'dev::ng1::rex3_flat_fill::tests' 4 $false Invoke-ResidentTests 'dev::ng1::wgpu::line::tests' 2 $false Invoke-ResidentTests 'dev::ng1::wgpu::host::tests' 2 $false @@ -78,8 +78,8 @@ try { Invoke-ResidentTests 'direct_ng1_scanout_' 3 $false foreach ($name in $requestedBackends) { $env:REX3_RESIDENT_BACKEND = $name - Invoke-ResidentTests 'resident_gpu_' 19 $true - Invoke-ResidentTests 'dev::ng1::wgpu::direct_tests' 9 $true + Invoke-ResidentTests 'resident_gpu_' 24 $true + Invoke-ResidentTests 'dev::ng1::wgpu::direct_tests' 10 $true } } finally { [Environment]::SetEnvironmentVariable('REX3_RESIDENT_BACKEND', $previousBackend, 'Process') diff --git a/src/dev/ng1/rex3.rs b/src/dev/ng1/rex3.rs index ea2c10497..3f964cb8f 100644 --- a/src/dev/ng1/rex3.rs +++ b/src/dev/ng1/rex3.rs @@ -28,6 +28,12 @@ mod plane; #[path = "rex3_frame.rs"] pub(super) mod frame; #[cfg(all(test, feature = "wgpu"))] +#[path = "rex3_lifecycle.rs"] +mod lifecycle; +#[cfg(all(test, feature = "wgpu"))] +#[path = "rex3_lifecycle_tests.rs"] +mod lifecycle_tests; +#[cfg(all(test, feature = "wgpu"))] pub use plane::CpuPlane; #[cfg(not(all(test, feature = "wgpu")))] pub type CpuPlane = Box<[u32]>; @@ -1403,6 +1409,10 @@ pub struct Rex3 { resident_fatal_notified: AtomicBool, #[cfg(all(test, feature = "wgpu"))] resident_frame: frame::ResidentFrameControl, + #[cfg(all(test, feature = "wgpu"))] + resident_lifecycle: RwLock, + #[cfg(all(test, feature = "wgpu"))] + resident_telemetry: Mutex, pub gfifo: GFifo, pub vc2: Mutex, @@ -1426,6 +1436,8 @@ pub struct Rex3 { /// Handle to the REX3-Processor thread, set once when it starts, used by /// `gfifo_push` to unpark it. OnceLock gives lock-free reads on the hot path. processor_unparker: std::sync::OnceLock, + #[cfg(all(test, feature = "wgpu"))] + resident_unparker: Mutex>, /// Original coarse dirty flag, retained verbatim when partial uploads are /// not compiled in. #[cfg(not(feature = "rex-dirty-upload"))] @@ -1614,6 +1626,10 @@ impl Rex3 { resident_fatal_notified: AtomicBool::new(false), #[cfg(all(test, feature = "wgpu"))] resident_frame: frame::ResidentFrameControl::default(), + #[cfg(all(test, feature = "wgpu"))] + resident_lifecycle: RwLock::new(lifecycle::ResidentLifecycle::default()), + #[cfg(all(test, feature = "wgpu"))] + resident_telemetry: Mutex::new(crate::dev::ng1::wgpu::telemetry::Board::default()), gfifo: GFifo::new(), vc2: Mutex::new(Vc2::new()), xmap0: Mutex::new(Xmap9::new()), @@ -1629,6 +1645,8 @@ impl Rex3 { refresh_thread: Mutex::new(None), processor_parked: AtomicBool::new(false), processor_unparker: std::sync::OnceLock::new(), + #[cfg(all(test, feature = "wgpu"))] + resident_unparker: Mutex::new(None), #[cfg(not(feature = "rex-dirty-upload"))] fb_dirty: AtomicBool::new(true), #[cfg(feature = "rex-dirty-upload")] @@ -1755,9 +1773,9 @@ impl Rex3 { .set_telemetry_enabled(enabled) } - /// The caller must own the stopped board exclusively. The resident board - /// has no refresh thread or CPU-plane observers; explicit backend - /// observation is a separate test-only operation. + /// The caller serializes producers and lifecycle transitions. A stopped + /// worker and an empty, fully published FIFO are the only boundary at + /// which the context and both authoritative planes can be snapshotted. #[cfg(all(test, feature = "wgpu"))] fn check_resident_quiescent( &self, @@ -1772,6 +1790,13 @@ impl Rex3 { { return Err(ResidentError::InvalidState("REX3 requires a stopped, idle board")); } + if !self.resident_frame.quiescent() { + return Err(ResidentError::InvalidState("REX3 has an active display frame")); + } + let context = unsafe { &*self.context.get() }; + if context.host_len != 1 || context.host_cursor != 0 { + return Err(ResidentError::InvalidState("REX3 has an unfinished HOSTRW transfer")); + } #[cfg(feature = "gputrace")] if self.tap.load(Ordering::Acquire) != 0 || self.rec.lock().is_some() @@ -1779,12 +1804,14 @@ impl Rex3 { { return Err(ResidentError::InvalidState("REX3 cannot attach while recording or tracing")); } - if self.head.renderer_slot().lock().is_some() { - return Err(ResidentError::InvalidState("REX3 live resident scanout is not implemented")); - } Ok(()) } + #[cfg(all(test, feature = "wgpu"))] + pub(crate) fn resident_active(&self) -> bool { + self.resident_frame.is_active() + } + /// Attach a test double through the same gate as the GPU implementation. #[cfg(all(test, feature = "wgpu"))] pub(crate) fn attach_resident_raster( @@ -1793,6 +1820,9 @@ impl Rex3 { ) -> Result<(), String> { self.raster.get_mut().kind().ensure_available(RasterModeKind::Resident)?; self.check_resident_quiescent().map_err(|error| error.to_string())?; + if self.head.renderer_slot().lock().is_some() { + return Err("REX3 live resident scanout is not implemented".into()); + } self.processor_unparker.take(); *self.raster.get_mut() = RasterMode::Resident(crate::dev::ng1::wgpu::resident::ResidentRaster::new(backend)); @@ -1806,48 +1836,40 @@ impl Rex3 { pub(crate) fn activate_resident_gpu( &mut self, gpu: Arc, + ) -> Result<(), crate::dev::ng1::wgpu::resident::ResidentError> { + self.activate_resident_gpu_with_profiling(gpu, false) + } + + #[cfg(all(test, feature = "wgpu"))] + pub(crate) fn activate_resident_gpu_with_profiling( + &mut self, + gpu: Arc, + profiling: bool, ) -> Result<(), crate::dev::ng1::wgpu::resident::ResidentError> { use crate::dev::ng1::wgpu::resident::{ResidentError, ResidentGpu, ResidentRaster}; self.raster.get_mut().kind().ensure_available(RasterModeKind::Resident) .map_err(ResidentError::Failed)?; self.check_resident_quiescent()?; - let backend = ResidentGpu::activate(gpu, self.fb_rgb.get_mut(), self.fb_aux.get_mut())?; + if self.head.renderer_slot().lock().is_some() { + return Err(ResidentError::InvalidState("REX3 live resident scanout is not implemented")); + } + let mut backend = ResidentGpu::activate(gpu, self.fb_rgb.get_mut(), self.fb_aux.get_mut())?; + if profiling { + backend.enable_profiling()?; + } self.processor_unparker.take(); *self.raster.get_mut() = RasterMode::Resident(ResidentRaster::new(backend)); self.resident_frame.activate(); Ok(()) } - /// Reset only while exclusively owned and quiescent. Power-on supplies - /// fresh CPU seed data; the old GPU planes are never read back. + /// Test seam retained for existing callers; power-on uses the same + /// stopped resident lifecycle. #[cfg(all(test, feature = "wgpu"))] pub(crate) fn reset_resident_gpu( &mut self, ) -> Result<(), crate::dev::ng1::wgpu::resident::ResidentError> { - use crate::dev::ng1::wgpu::resident::ResidentError; - self.check_resident_quiescent()?; - match self.raster.get_mut() { - RasterMode::Resident(raster) => { - if let Some(error @ (ResidentError::DeviceLost(_) | ResidentError::Failed(_))) = raster.fault() { - return Err(error.clone()); - } - } - _ => return Err(ResidentError::InvalidState("resident raster is not active")), - } - // No other owner or worker can see this temporary mode. A reset - // discards old GPU contents rather than treating stale CPU data as a - // fallback; a failed reseed restores a sticky resident fault. - let mut mode = std::mem::replace(self.raster.get_mut(), RasterMode::Cpu); - self.power_on(); - let RasterMode::Resident(raster) = &mut mode else { unreachable!() }; - let result = raster.reset(self.fb_rgb.get_mut(), self.fb_aux.get_mut()); - *self.raster.get_mut() = mode; - if result.is_ok() { - self.resident_fatal_notified.store(false, Ordering::Release); - self.processor_unparker.take(); - self.resident_frame.activate(); - } - result + self.reset_resident_board() } #[cfg(all(test, feature = "wgpu"))] @@ -1897,9 +1919,10 @@ impl Rex3 { #[cfg(all(test, feature = "wgpu"))] fn submit_resident( &self, + reason: crate::dev::ng1::wgpu::telemetry::FlushReason, ) -> Option> { let result = match &mut *self.raster.lock() { - RasterMode::Resident(raster) => Some(raster.submit()), + RasterMode::Resident(raster) => Some(raster.submit_for(reason)), _ => None, }; if let Some(Err(error)) = &result { @@ -1923,6 +1946,70 @@ impl Rex3 { result.unwrap_or(Err(ResidentError::InvalidState("resident raster is not active"))) } + #[cfg(all(test, feature = "wgpu"))] + pub(crate) fn resident_telemetry_snapshot( + &self, + ) -> Result { + self.resident_snapshot_from_status(self.resident_status()?) + } + + #[cfg(all(test, feature = "wgpu"))] + pub(crate) fn resident_telemetry_diagnostics( + &self, + ) -> Result< + (crate::dev::ng1::wgpu::telemetry::Snapshot, Option), + crate::dev::ng1::wgpu::resident::ResidentError, + > { + use crate::dev::ng1::wgpu::resident::ResidentError; + let diagnostics = match &*self.raster.lock() { + RasterMode::Resident(raster) => raster.diagnostics(), + _ => Err(ResidentError::InvalidState("resident raster is not active")), + }?; + Ok((self.resident_snapshot_from_status(diagnostics.status)?, diagnostics.fault)) + } + + #[cfg(all(test, feature = "wgpu"))] + fn resident_snapshot_from_status( + &self, + status: crate::dev::ng1::wgpu::resident::ResidentStatus, + ) -> Result { + use crate::dev::ng1::wgpu::resident::{ObservationReason, ResidentError}; + use crate::dev::ng1::wgpu::telemetry::{Snapshot, Traffic}; + let board = self.resident_telemetry.lock().clone(); + if board.errors { + return Err(ResidentError::InvalidState("board telemetry overflow")); + } + let mut traffic = Traffic { raw_maps: status.raw_plane_map_attempts, ..Traffic::default() }; + for (index, reason) in [ + ObservationReason::HostRead, ObservationReason::Snapshot, ObservationReason::Monitor, + ObservationReason::Recording, ObservationReason::Validation, ObservationReason::Compatibility, + ].into_iter().enumerate() { + let counts = status.observations.for_reason(reason); + traffic.raw_attempts[index] = counts.attempts; + traffic.raw_bytes[index] = counts.successful_bytes; + } + for (index, mode) in [ + frame::ResidentDisplayMode::DirectWgpu, frame::ResidentDisplayMode::ReadbackCompatibility, + ].into_iter().enumerate() { + let counts = self.resident_composed_captures(mode); + traffic.composed_attempts[index] = counts.attempts; + traffic.composed_bytes[index] = counts.successful_bytes; + } + Ok(Snapshot { epoch: status.accepted.epoch.value(), workload: status.workload, board, traffic }) + } + + #[cfg(all(test, feature = "wgpu"))] + fn flush_resident_if_due(&self) { + if !self.resident_frame.is_active() { return; } + let result = match &mut *self.raster.lock() { + RasterMode::Resident(raster) => Some(raster.flush_if_due(std::time::Instant::now())), + _ => None, + }; + if let Some(Err(error)) = result { + self.fail_resident(error); + } + } + #[cfg(all(test, feature = "wgpu"))] fn poll_resident_device_health(&self) { let result = match &mut *self.raster.lock() { @@ -1966,18 +2053,20 @@ impl Rex3 { &self, ) -> Result<(), crate::dev::ng1::wgpu::resident::ResidentError> { use crate::dev::ng1::wgpu::resident::ResidentError; + let lifecycle = self.resident_lifecycle.write(); if self.raster.lock().kind() != RasterModeKind::Resident { return Err(ResidentError::InvalidState("resident raster is not active")); } - // The processor does not select a display; the explicitly chosen - // display mode validates its renderer when it starts and renders. - let mut processor = self.processor_thread.lock(); - if let Some(error) = self.resident_fault() { - return Err(error); - } - if self.running.swap(true, Ordering::SeqCst) { + self.check_resident_health(false)?; + if self.running.load(Ordering::Acquire) { return Ok(()); } + self.check_resident_quiescent()?; + if !lifecycle.ready() { + return Err(ResidentError::InvalidState("resident restore is incomplete")); + } + let mut processor = self.processor_thread.lock(); + self.running.store(true, Ordering::SeqCst); let rex3 = unsafe { std::mem::transmute::<&Rex3, &'static Rex3>(self) }; unsafe { (*self.context.get()).host = rex3 as *const Rex3; } match thread::Builder::new().name("REX3-Resident-Processor-Test".into()) @@ -1990,7 +2079,9 @@ impl Rex3 { Ok(handle) => *processor = Some(handle), Err(error) => { self.running.store(false, Ordering::Release); - return Err(ResidentError::Failed(format!("start resident processor: {error}"))); + let error = ResidentError::Failed(format!("start resident processor: {error}")); + self.fail_resident(error.clone()); + return Err(error); } } Ok(()) @@ -2199,7 +2290,11 @@ impl Rex3 { /// Program VC2 with a host-side Newport timing preset (see `[graphics] resolution`). pub fn apply_display_resolution(&self, mode: crate::dev::ng1::vc2_timings::NewportResolution) { #[cfg(all(test, feature = "wgpu"))] - self.reject_resident_operation("display resolution changes").unwrap(); + if self.resident_active() { + self.apply_resident_display_resolution(mode, false) + .unwrap_or_else(|error| panic!("REX3 {error}")); + return; + } if mode.is_guest() { return; } @@ -3064,7 +3159,30 @@ impl Rex3 { // blend has spec-cited factor semantics — all of which a second copy would // get subtly wrong. + fn wake_gfifo_processor(&self) { + if !self.processor_parked.load(Ordering::Acquire) { + return; + } + #[cfg(all(test, feature = "wgpu"))] + if self.resident_frame.is_active() { + if let Some(t) = self.resident_unparker.lock().as_ref() { + t.unpark(); + } + return; + } + if let Some(t) = self.processor_unparker.get() { + t.unpark(); + } + } + fn gfifo_push(&self, addr: u32, val: u64) { + #[cfg(all(test, feature = "wgpu"))] + if self.resident_frame.is_active() { + while !self.gfifo_try_push(addr, val) { + std::hint::spin_loop(); + } + return; + } #[cfg(feature = "developer")] { let len = self.gfifo.len() + 1; @@ -3077,11 +3195,7 @@ impl Rex3 { // Wake the consumer if it parked on an empty fifo (idle desktop). Cheap // on the hot path: a relaxed-ish load that is false whenever the // processor is actively draining. - if self.processor_parked.load(Ordering::Acquire) { - if let Some(t) = self.processor_unparker.get() { - t.unpark(); - } - } + self.wake_gfifo_processor(); } /// Non-blocking `gfifo_push`: returns `false` when the queue is full or a @@ -3100,14 +3214,21 @@ impl Rex3 { if len > hwm { Some(len) } else { None } }); } - if !self.gfifo.try_push(addr, val) { + #[cfg(all(test, feature = "wgpu"))] + let accepted = if self.resident_frame.is_active() { + let before = self.gfifo.len(); + let result = self.gfifo.try_push_traced(addr, val); + self.note_resident_reservation(before, result); + result.is_ok() + } else { + self.gfifo.try_push(addr, val) + }; + #[cfg(not(all(test, feature = "wgpu")))] + let accepted = self.gfifo.try_push(addr, val); + if !accepted { return false; } - if self.processor_parked.load(Ordering::Acquire) { - if let Some(t) = self.processor_unparker.get() { - t.unpark(); - } - } + self.wake_gfifo_processor(); true } @@ -3131,14 +3252,21 @@ impl Rex3 { if len > hwm { Some(len) } else { None } }); } - if !self.gfifo.try_push2(addr0, val0, addr1, val1) { + #[cfg(all(test, feature = "wgpu"))] + let accepted = if self.resident_frame.is_active() { + let before = self.gfifo.len(); + let result = self.gfifo.try_push2_traced(addr0, val0, addr1, val1); + self.note_resident_reservation(before, result); + result.is_ok() + } else { + self.gfifo.try_push2(addr0, val0, addr1, val1) + }; + #[cfg(not(all(test, feature = "wgpu")))] + let accepted = self.gfifo.try_push2(addr0, val0, addr1, val1); + if !accepted { return false; } - if self.processor_parked.load(Ordering::Acquire) { - if let Some(t) = self.processor_unparker.get() { - t.unpark(); - } - } + self.wake_gfifo_processor(); true } @@ -3155,17 +3283,37 @@ impl Rex3 { if len > hwm { Some(len) } else { None } }); } - if !self.gfifo.try_push_batch(token, vals.len() as u64, vals) { + #[cfg(all(test, feature = "wgpu"))] + let accepted = if self.resident_frame.is_active() { + let before = self.gfifo.len(); + let result = self.gfifo.try_push_batch_traced(token, vals.len() as u64, vals); + self.note_resident_reservation(before, result); + result.is_ok() + } else { + self.gfifo.try_push_batch(token, vals.len() as u64, vals) + }; + #[cfg(not(all(test, feature = "wgpu")))] + let accepted = self.gfifo.try_push_batch(token, vals.len() as u64, vals); + if !accepted { return false; } - if self.processor_parked.load(Ordering::Acquire) { - if let Some(t) = self.processor_unparker.get() { - t.unpark(); - } - } + self.wake_gfifo_processor(); true } + #[cfg(all(test, feature = "wgpu"))] + fn note_resident_reservation(&self, before: usize, result: Result<(), crate::gfifo::PushFailure>) { + let occupancy = before.max(self.gfifo.len()); + self.resident_telemetry.lock().reservation(occupancy, result); + } + + #[cfg(all(test, feature = "wgpu"))] + fn note_resident_bus_busy(&self) { + if self.resident_frame.is_active() { + self.resident_telemetry.lock().bus_busy(); + } + } + fn wait_gfifo_idle(&self) { loop { // Publish before testing: unlike the bus read paths this one has no @@ -3203,7 +3351,8 @@ impl Rex3 { } thread::yield_now(); } - self.submit_resident().ok_or(ResidentError::InvalidState("resident raster is not active"))? + self.submit_resident(crate::dev::ng1::wgpu::telemetry::FlushReason::WaitIdle) + .ok_or(ResidentError::InvalidState("resident raster is not active"))? } pub(crate) fn wait_idle(&self) { @@ -3227,7 +3376,12 @@ impl Rex3 { return BUS_ERR; } drop(mode); - return if self.gfifo_try_push(token, count) { BUS_OK } else { BUS_BUSY }; + return if self.gfifo_try_push(token, count) { + BUS_OK + } else { + self.note_resident_bus_busy(); + BUS_BUSY + }; } } self.gfifo_push(token, count); @@ -3246,6 +3400,13 @@ impl Rex3 { fn register_processor(&self) { // Publish our thread handle so gfifo_push can unpark us when we park. + #[cfg(all(test, feature = "wgpu"))] + if self.resident_frame.is_active() { + *self.resident_unparker.lock() = Some(thread::current()); + } else { + let _ = self.processor_unparker.set(thread::current()); + } + #[cfg(not(all(test, feature = "wgpu")))] let _ = self.processor_unparker.set(thread::current()); let backoff = crossbeam_utils::Backoff::new(); let mut is_busy = false; @@ -3257,6 +3418,8 @@ impl Rex3 { if addr == GFIFO_RESIDENT_FRAME { self.process_resident_frame_marker(val); self.gfifo.consume(); + #[cfg(all(test, feature = "wgpu"))] + self.flush_resident_if_due(); continue; } @@ -3297,17 +3460,19 @@ impl Rex3 { if exit { #[cfg(all(test, feature = "wgpu"))] - let _ = self.submit_resident(); + let _ = self.submit_resident(crate::dev::ng1::wgpu::telemetry::FlushReason::Exit); #[cfg(feature = "rex-gpu")] self.flush_gpu_replay(FlushReason::Exit); self.gfxbusy.store(false, Ordering::Release); self.gfifo.flush_head(); break; } // GFIFO_EXIT + #[cfg(all(test, feature = "wgpu"))] + self.flush_resident_if_due(); } else { if is_busy { #[cfg(all(test, feature = "wgpu"))] - let _ = self.submit_resident(); + let _ = self.submit_resident(crate::dev::ng1::wgpu::telemetry::FlushReason::Idle); #[cfg(feature = "rex-gpu")] self.flush_gpu_replay(FlushReason::Idle); self.gfxbusy.store(false, Ordering::Release); @@ -3834,7 +3999,7 @@ impl Rex3 { // is tracked separately from this ordered submission fence. GFIFO_DISP_SYNC => { #[cfg(all(test, feature = "wgpu"))] - if matches!(self.submit_resident(), Some(Err(_))) { + if matches!(self.submit_resident(crate::dev::ng1::wgpu::telemetry::FlushReason::DisplayFence), Some(Err(_))) { // A failed prefix is not a completed display fence. return false; } @@ -4062,16 +4227,27 @@ impl Rex3 { /// Save RGB/aux planes with a custom filename prefix (e.g. `rex3_head1`). pub fn save_framebuffers_named(&self, dir: &std::path::Path, prefix: &str) -> std::io::Result<()> { #[cfg(all(test, feature = "wgpu"))] - self.reject_resident_operation("framebuffer snapshots").map_err(std::io::Error::other)?; + if self.resident_frame.is_active() { + return self.save_resident_framebuffers(dir, prefix); + } self.wait_idle(); let rgb = unsafe { &*self.fb_rgb.get() }; + let aux = unsafe { &*self.fb_aux.get() }; + Self::write_framebuffer_files(dir, prefix, rgb, aux) + } + + fn write_framebuffer_files( + dir: &std::path::Path, + prefix: &str, + rgb: &[u32], + aux: &[u32], + ) -> std::io::Result<()> { let mut bytes = Vec::with_capacity(rgb.len() * 4); for &word in rgb.iter() { bytes.extend_from_slice(&word.to_be_bytes()); } std::fs::write(dir.join(format!("{prefix}_rgb.bin")), &bytes)?; - let aux = unsafe { &*self.fb_aux.get() }; bytes.clear(); for &word in aux.iter() { bytes.extend_from_slice(&word.to_be_bytes()); @@ -4085,7 +4261,9 @@ impl Rex3 { /// rollback checkpoint; bypasses the byte-shuffle the disk path needs. pub fn snapshot_framebuffers_inmem(&self) -> (Vec, Vec) { #[cfg(all(test, feature = "wgpu"))] - self.reject_resident_operation("framebuffer snapshots").unwrap(); + if self.resident_frame.is_active() { + return self.snapshot_resident_planes().unwrap_or_else(|error| panic!("REX3 {error}")); + } self.wait_idle(); let rgb = unsafe { &*self.fb_rgb.get() }; let aux = unsafe { &*self.fb_aux.get() }; @@ -4097,7 +4275,10 @@ impl Rex3 { /// framebuffer size. pub fn restore_framebuffers_inmem(&self, rgb: &[u32], aux: &[u32]) { #[cfg(all(test, feature = "wgpu"))] - self.reject_resident_operation("framebuffer restores").unwrap(); + if self.resident_frame.is_active() { + self.stage_resident_planes(rgb, aux).unwrap_or_else(|error| panic!("REX3 {error}")); + return; + } self.wait_idle(); let dst_rgb = unsafe { &mut *self.fb_rgb.get() }; let n = rgb.len().min(dst_rgb.len()); @@ -4167,7 +4348,9 @@ impl Rex3 { pub fn load_framebuffers_named(&self, dir: &std::path::Path, prefix: &str) -> std::io::Result<()> { #[cfg(all(test, feature = "wgpu"))] - self.reject_resident_operation("framebuffer restores").map_err(std::io::Error::other)?; + if self.resident_frame.is_active() { + return self.load_resident_framebuffers(dir, prefix); + } self.wait_idle(); let mut rgb_changed = false; let path_rgb = dir.join(format!("{prefix}_rgb.bin")); @@ -4250,6 +4433,7 @@ impl Device for Rex3 { fn stop(&self) { #[cfg(all(test, feature = "wgpu"))] if self.raster.lock().kind() == RasterModeKind::Resident { + let _lifecycle = self.resident_lifecycle.write(); self.head.display_stop().request(); self.running.store(false, Ordering::SeqCst); self.resident_frame.wake(); @@ -4262,11 +4446,13 @@ impl Device for Rex3 { } let mut processor = self.processor_thread.lock(); if let Some(handle) = processor.take() { + let mut sent_exit = false; // Unlike a blocking FIFO push, this also terminates if a // test worker has already failed with the queue still full. while !handle.is_finished() { if self.gfifo_try_push(GFIFO_EXIT, 0) { handle.thread().unpark(); + sent_exit = true; break; } thread::yield_now(); @@ -4275,11 +4461,20 @@ impl Device for Rex3 { self.fail_resident(crate::dev::ng1::wgpu::resident::ResidentError::Failed( "resident GFIFO worker panicked".into(), )); + } else if !sent_exit && self.resident_fault().is_none() { + self.fail_resident(crate::dev::ng1::wgpu::resident::ResidentError::Failed( + "resident GFIFO worker exited unexpectedly".into(), + )); } } - // After joining, normal FIFO consumption can safely discard - // leftovers even if a producer races teardown. Bound the drain - // so a continuing producer cannot make stop block indefinitely. + *self.resident_unparker.lock() = None; + // Leftovers cannot be presented as a successfully stopped + // prefix. Drain only after making that condition sticky. + if !self.gfifo.is_empty() && self.resident_fault().is_none() { + self.fail_resident(crate::dev::ng1::wgpu::resident::ResidentError::Failed( + "resident GFIFO was not drained at stop".into(), + )); + } for _ in 0..GFIFO_DEPTH { if self.gfifo.peek().is_none() { break; } self.gfifo.consume(); @@ -4287,7 +4482,7 @@ impl Device for Rex3 { self.gfifo.flush_head(); drop(processor); self.gfxbusy.store(false, Ordering::Release); - let _ = self.submit_resident(); + let _ = self.submit_resident(crate::dev::ng1::wgpu::telemetry::FlushReason::Exit); return; } // Close a recording and trace while the processor can still drain, @@ -4438,6 +4633,9 @@ impl Device for Rex3 { return Err(format!("Usage: {} debug | {} status", cmd, cmd)); } + #[cfg(all(test, feature = "wgpu"))] + let _resident_monitor = self.resident_monitor_guard()?; + if cmd == "rex" && args[0] == "status" { let ctx = unsafe { &*self.context.get() }; @@ -4884,9 +5082,13 @@ impl Rex3 { // Contention here means a producer is mid-push, which is itself // "not empty" — report busy and let the CPU retry. if !self.gfifo.publish_tail() { + #[cfg(all(test, feature = "wgpu"))] + self.note_resident_bus_busy(); return BusRead32::busy(); } if self.gfxbusy.load(Ordering::Acquire) || !self.gfifo.is_empty() { + #[cfg(all(test, feature = "wgpu"))] + self.note_resident_bus_busy(); return BusRead32::busy(); } BusRead32::ok($val) @@ -5190,6 +5392,8 @@ impl Rex3 { // as the queue stays full. if !self.gfifo_try_push(offset, val as u64) { crate::pf_level!(self.pf_fifo_full, crate::perfetto::events::REX3_FIFO_FULL, true); + #[cfg(all(test, feature = "wgpu"))] + self.note_resident_bus_busy(); return BUS_BUSY; } crate::pf_level!(self.pf_fifo_full, crate::perfetto::events::REX3_FIFO_FULL, false); @@ -5216,9 +5420,13 @@ impl Rex3 { if reg_offset64r == REX3_HOSTRW0 { // Same rule as busy_or_val! on the 32-bit path: publish, then test. if !self.gfifo.publish_tail() { + #[cfg(all(test, feature = "wgpu"))] + self.note_resident_bus_busy(); return BusRead64::busy(); } if self.gfxbusy.load(Ordering::Acquire) || !self.gfifo.is_empty() { + #[cfg(all(test, feature = "wgpu"))] + self.note_resident_bus_busy(); return BusRead64::busy(); } #[cfg(all(test, feature = "wgpu"))] @@ -5317,6 +5525,8 @@ impl Rex3 { // the transfer ordered correctly against concurrent CPU register writes // — the words arrive in the stream at the point the producer put them. if !self.gfifo_try_push_batch(REX3_DMA_BATCH_W, vals) { + #[cfg(all(test, feature = "wgpu"))] + self.note_resident_bus_busy(); return BUS_BUSY; } BUS_OK @@ -5384,6 +5594,8 @@ impl Rex3 { // Sole effect of this write, so a full queue reports BUS_BUSY and // the CPU retries the store (see write32's default arm). if !self.gfifo_try_push(REX3_HOSTRW64 | (offset & 0x0800), val) { + #[cfg(all(test, feature = "wgpu"))] + self.note_resident_bus_busy(); return BUS_BUSY; } return BUS_OK; @@ -5414,6 +5626,8 @@ impl Rex3 { // `is_go` puts the GO bit on the low word, matching the two-write order. let go_lo = if is_go { off_lo | 0x0800 } else { off_lo }; if !self.gfifo_try_push2(off_hi, high as u64, go_lo, low as u64) { + #[cfg(all(test, feature = "wgpu"))] + self.note_resident_bus_busy(); return BUS_BUSY; } BUS_OK @@ -5536,8 +5750,27 @@ impl Rex3 { } } +#[cfg(all(test, feature = "wgpu"))] +macro_rules! resident_bus_gate { + ($board:expr, $on_error:expr) => { + let _resident_bus = match $board.resident_bus_guard() { + Ok(guard) => guard, + Err(status) => { + if status == BUS_BUSY { $board.note_resident_bus_busy(); } + return ($on_error)(status); + } + }; + }; +} + +#[cfg(not(all(test, feature = "wgpu")))] +macro_rules! resident_bus_gate { + ($board:expr, $on_error:expr) => {}; +} + impl BusDevice for Rex3 { fn read8(&self, addr: u64) -> BusRead8 { + resident_bus_gate!(self, |status| BusRead8 { status, data: 0 }); let addr = addr as u32; // REX3 decodes only its 8 KB window below 4 GB #[cfg(feature = "gputrace")] { @@ -5550,6 +5783,7 @@ impl BusDevice for Rex3 { } fn read16(&self, addr: u64) -> BusRead16 { + resident_bus_gate!(self, |status| BusRead16 { status, data: 0 }); let addr = addr as u32; // REX3 decodes only its 8 KB window below 4 GB #[cfg(feature = "gputrace")] { @@ -5562,6 +5796,7 @@ impl BusDevice for Rex3 { } fn read32(&self, addr: u64) -> BusRead32 { + resident_bus_gate!(self, |status| BusRead32 { status, data: 0 }); let addr = addr as u32; // REX3 decodes only its 8 KB window below 4 GB #[cfg(feature = "gputrace")] { @@ -5574,6 +5809,7 @@ impl BusDevice for Rex3 { } fn read64(&self, addr: u64) -> BusRead64 { + resident_bus_gate!(self, |status| BusRead64 { status, data: 0 }); let addr = addr as u32; // REX3 decodes only its 8 KB window below 4 GB #[cfg(feature = "gputrace")] { @@ -5586,6 +5822,7 @@ impl BusDevice for Rex3 { } fn write8(&self, addr: u64, val: u8) -> u32 { + resident_bus_gate!(self, |status| status); let addr = addr as u32; // REX3 decodes only its 8 KB window below 4 GB #[cfg(feature = "gputrace")] { @@ -5598,6 +5835,7 @@ impl BusDevice for Rex3 { } fn write16(&self, addr: u64, val: u16) -> u32 { + resident_bus_gate!(self, |status| status); let addr = addr as u32; // REX3 decodes only its 8 KB window below 4 GB #[cfg(feature = "gputrace")] { @@ -5610,6 +5848,7 @@ impl BusDevice for Rex3 { } fn write32(&self, addr: u64, val: u32) -> u32 { + resident_bus_gate!(self, |status| status); let addr = addr as u32; // REX3 decodes only its 8 KB window below 4 GB #[cfg(feature = "gputrace")] { @@ -5622,6 +5861,7 @@ impl BusDevice for Rex3 { } fn write64(&self, addr: u64, val: u64) -> u32 { + resident_bus_gate!(self, |status| status); let addr = addr as u32; // REX3 decodes only its 8 KB window below 4 GB #[cfg(feature = "gputrace")] { @@ -5634,6 +5874,7 @@ impl BusDevice for Rex3 { } fn dma_read64(&self, addr: u64) -> BusRead64 { + resident_bus_gate!(self, |status| BusRead64 { status, data: 0 }); let addr = addr as u32; // REX3 decodes only its 8 KB window below 4 GB #[cfg(feature = "gputrace")] { @@ -5646,6 +5887,7 @@ impl BusDevice for Rex3 { } fn dma_write64_bulk(&self, addr: u64, vals: &[u64]) -> u32 { + resident_bus_gate!(self, |status| status); let addr = addr as u32; // REX3 decodes only its 8 KB window below 4 GB #[cfg(feature = "gputrace")] { @@ -5658,6 +5900,7 @@ impl BusDevice for Rex3 { } fn dma_read64_bulk(&self, addr: u64, out: &mut [u64]) -> u32 { + resident_bus_gate!(self, |status| status); let addr = addr as u32; // REX3 decodes only its 8 KB window below 4 GB #[cfg(feature = "gputrace")] { @@ -5759,7 +6002,8 @@ impl crate::gfx_display::GfxDisplay for Rex3 { fn frame_available(&self) -> bool { #[cfg(all(test, feature = "wgpu"))] if self.resident_frame.is_active() { - return self.resident_frame.frame_available(); + return !self.resident_fatal_notified.load(Ordering::Acquire) + && self.resident_frame.frame_available(); } true } @@ -5789,7 +6033,18 @@ impl crate::gfx_display::GfxDisplay for Rex3 { fn hw_cursor(&self) -> Option { use crate::dev::ng1::vc2::{VC2_REG_CURRENT_CURSOR_X, VC2_REG_CURSOR_Y_LOC, VC2_REG_CURSOR_ENTRY_PTR, VC2_REG_DISPLAY_CONTROL, VC2_CTRL_CURSOR_EN, VC2_CTRL_CURSOR_SIZE}; - let x_adjust = self.head.screen().lock().cursor_x_adjust; + let screen = self.head.screen().lock(); + #[cfg(all(test, feature = "wgpu"))] + if self.resident_frame.is_active() + && (self.resident_fatal_notified.load(Ordering::Acquire) || screen.width == 0 || screen.height == 0) + { + return None; + } + let x_adjust = screen.cursor_x_adjust; + #[cfg(all(test, feature = "wgpu"))] + let _resident_screen = self.resident_frame.is_active().then_some(screen); + #[cfg(not(all(test, feature = "wgpu")))] + drop(screen); let vc2 = self.vc2.lock(); let ctrl = vc2.regs[VC2_REG_DISPLAY_CONTROL as usize]; if ctrl & VC2_CTRL_CURSOR_EN == 0 { @@ -5820,22 +6075,12 @@ impl crate::gfx_display::GfxDisplay for Rex3 { } } -impl Resettable for Rex3 { - fn power_on(&self) { - #[cfg(all(test, feature = "wgpu"))] - self.reject_resident_operation("reset").unwrap(); - // Reset drawing context +impl Rex3 { + fn reset_registers(&self) { unsafe { *self.context.get() = Rex3Context::power_on_default(); } - // Reset config registers self.config.config.store(0, Ordering::Relaxed); self.config.status.store(0, Ordering::Relaxed); *self.dcb.lock() = Rex3DcbState::default(); - // Clear framebuffers - self.with_cpu_rgb_write(|fb| fb.fill(0)); - unsafe { (*self.fb_aux.get()).fill(0); } - self.mark_framebuffer_full(DamagePlane::Rgb); - self.mark_framebuffer_full(DamagePlane::Aux); - // Reset Vc2, Xmap, Cmap *self.vc2.lock() = crate::dev::ng1::vc2::Vc2::new(); *self.xmap0.lock() = crate::dev::ng1::xmap9::Xmap9::new(); *self.xmap1.lock() = crate::dev::ng1::xmap9::Xmap9::new(); @@ -5845,6 +6090,21 @@ impl Resettable for Rex3 { } } +impl Resettable for Rex3 { + fn power_on(&self) { + #[cfg(all(test, feature = "wgpu"))] + if self.resident_frame.is_active() { + self.reset_resident_board().unwrap_or_else(|error| panic!("REX3 {error}")); + return; + } + self.reset_registers(); + self.with_cpu_rgb_write(|fb| fb.fill(0)); + unsafe { (*self.fb_aux.get()).fill(0); } + self.mark_framebuffer_full(DamagePlane::Rgb); + self.mark_framebuffer_full(DamagePlane::Aux); + } +} + /// Serialize Rex3Context to a flat TOML table. All fixed-point fields are stored as raw u32 bits. fn save_rex3_context(ctx: &Rex3Context) -> toml::Value { let mut tbl = toml::map::Map::new(); @@ -5873,6 +6133,10 @@ fn save_rex3_context(ctx: &Rex3Context) -> toml::Value { tbl.insert("slopegrn".into(), hex_u32(ctx.slopegrn as u32)); tbl.insert("slopeblue".into(), hex_u32(ctx.slopeblue as u32)); tbl.insert("host_shifter".into(), hex_u64(ctx.host_shifter)); + tbl.insert("hostrw0".into(), hex_u64(ctx.hostrw[0])); + tbl.insert("pat_bit".into(), hex_u8(ctx.pat_bit)); + tbl.insert("zpat_bit".into(), hex_u8(ctx.zpat_bit)); + tbl.insert("mid_primitive".into(), hex_u8(u8::from(ctx.mid_primitive))); tbl.insert("drawmode0".into(), hex_u32(ctx.drawmode0.0)); tbl.insert("drawmode1".into(), hex_u32(ctx.drawmode1.0)); tbl.insert("lsmode".into(), hex_u32(ctx.lsmode.0)); @@ -5908,6 +6172,13 @@ fn load_rex3_context(ctx: &mut Rex3Context, v: &toml::Value) { if let Some(x) = get_field(v, "slopegrn") { ctx.slopegrn = toml_u32(x).unwrap_or(0) as i32; } if let Some(x) = get_field(v, "slopeblue") { ctx.slopeblue = toml_u32(x).unwrap_or(0) as i32; } if let Some(x) = get_field(v, "host_shifter") { ctx.host_shifter = toml_u64(x).unwrap_or(0); } + if let Some(x) = get_field(v, "hostrw0") { + ctx.hostrw[0] = toml_u64(x).unwrap_or(0); + ctx.hostrw_arm_single(); + } + if let Some(x) = get_field(v, "pat_bit") { ctx.pat_bit = toml_u8(x).unwrap_or(0); } + if let Some(x) = get_field(v, "zpat_bit") { ctx.zpat_bit = toml_u8(x).unwrap_or(0); } + if let Some(x) = get_field(v, "mid_primitive") { ctx.mid_primitive = toml_u8(x).unwrap_or(0) != 0; } if let Some(x) = get_field(v, "drawmode0") { ctx.drawmode0.0 = toml_u32(x).unwrap_or(0); } if let Some(x) = get_field(v, "drawmode1") { ctx.drawmode1.0 = toml_u32(x).unwrap_or(0); } if let Some(x) = get_field(v, "lsmode") { ctx.lsmode.0 = toml_u32(x).unwrap_or(0); } @@ -5915,10 +6186,8 @@ fn load_rex3_context(ctx: &mut Rex3Context, v: &toml::Value) { if let Some(x) = get_field(v, "bresrndinc2") { ctx.bresrndinc2.0 = toml_u32(x).unwrap_or(0); } } -impl Saveable for Rex3 { - fn save_state(&self) -> toml::Value { - #[cfg(all(test, feature = "wgpu"))] - self.reject_resident_operation("state snapshots").unwrap(); +impl Rex3 { + fn save_state_fields(&self) -> toml::Value { let mut tbl = toml::map::Map::new(); // Drawing context @@ -5982,9 +6251,7 @@ impl Saveable for Rex3 { toml::Value::Table(tbl) } - fn load_state(&self, v: &toml::Value) -> Result<(), String> { - #[cfg(all(test, feature = "wgpu"))] - self.reject_resident_operation("state restores")?; + fn load_state_fields(&self, v: &toml::Value) { if let Some(ctx_v) = get_field(v, "context") { let ctx = unsafe { &mut *self.context.get() }; load_rex3_context(ctx, ctx_v); @@ -5997,6 +6264,7 @@ impl Saveable for Rex3 { if let Some(x) = get_field(cfg_v, "dcbmode") { dcb.dcbmode = toml_u32(x).unwrap_or(0); } if let Some(x) = get_field(cfg_v, "dcbdata0") { dcb.dcbdata0 = toml_u32(x).unwrap_or(0); } if let Some(x) = get_field(cfg_v, "dcbdata1") { dcb.dcbdata1 = toml_u32(x).unwrap_or(0); } + dcb.backbusy_until = None; } if let Some(vv) = get_field(v, "vc2") { @@ -6031,6 +6299,24 @@ impl Saveable for Rex3 { self.gfifo.reset(); self.gfxbusy.store(false, Ordering::Relaxed); + } +} + +impl Saveable for Rex3 { + fn save_state(&self) -> toml::Value { + #[cfg(all(test, feature = "wgpu"))] + if self.resident_frame.is_active() { + return self.snapshot_resident_registers().unwrap_or_else(|error| panic!("REX3 {error}")); + } + self.save_state_fields() + } + + fn load_state(&self, v: &toml::Value) -> Result<(), String> { + #[cfg(all(test, feature = "wgpu"))] + if self.resident_frame.is_active() { + return self.stage_resident_registers(v); + } + self.load_state_fields(v); Ok(()) } } @@ -6041,6 +6327,7 @@ fn save_xmap9(xmap: &crate::dev::ng1::xmap9::Xmap9) -> toml::Value { tbl.insert("cursor_cmap_msb".into(), hex_u32(xmap.cursor_cmap_msb as u32)); tbl.insert("popup_cmap_msb".into(), hex_u32(xmap.popup_cmap_msb as u32)); tbl.insert("mode_addr".into(), hex_u32(xmap.mode_addr as u32)); + tbl.insert("fifo_avail".into(), hex_u32(xmap.fifo_avail as u32)); tbl.insert("mode_table".into(), u32_slice_to_toml(&xmap.mode_table)); toml::Value::Table(tbl) } @@ -6050,6 +6337,7 @@ fn load_xmap9(xmap: &mut crate::dev::ng1::xmap9::Xmap9, v: &toml::Value) { if let Some(x) = get_field(v, "cursor_cmap_msb") { xmap.cursor_cmap_msb = toml_u32(x).unwrap_or(0) as u8; } if let Some(x) = get_field(v, "popup_cmap_msb") { xmap.popup_cmap_msb = toml_u32(x).unwrap_or(0) as u8; } if let Some(x) = get_field(v, "mode_addr") { xmap.mode_addr = toml_u32(x).unwrap_or(0) as u8; } + if let Some(x) = get_field(v, "fifo_avail") { xmap.fifo_avail = toml_u32(x).unwrap_or(0) as u8; } if let Some(r) = get_field(v, "mode_table") { load_u32_slice(r, &mut xmap.mode_table); } xmap.dirty = true; } @@ -6058,7 +6346,11 @@ fn save_cmap(cmap: &crate::dev::ng1::cmap::Cmap) -> toml::Value { let mut tbl = toml::map::Map::new(); tbl.insert("addr_lo".into(), hex_u32(cmap.addr_lo as u32)); tbl.insert("addr_hi".into(), hex_u32(cmap.addr_hi as u32)); + tbl.insert("rgb_counter".into(), hex_u32(cmap.rgb_counter as u32)); + tbl.insert("red_temp".into(), hex_u32(cmap.red_temp as u32)); + tbl.insert("green_temp".into(), hex_u32(cmap.green_temp as u32)); tbl.insert("command".into(), hex_u32(cmap.command as u32)); + tbl.insert("revision".into(), hex_u32(cmap.revision as u32)); tbl.insert("palette".into(), u32_slice_to_toml(&cmap.palette)); toml::Value::Table(tbl) } @@ -6066,7 +6358,11 @@ fn save_cmap(cmap: &crate::dev::ng1::cmap::Cmap) -> toml::Value { fn load_cmap(cmap: &mut crate::dev::ng1::cmap::Cmap, v: &toml::Value) { if let Some(x) = get_field(v, "addr_lo") { cmap.addr_lo = toml_u32(x).unwrap_or(0) as u8; } if let Some(x) = get_field(v, "addr_hi") { cmap.addr_hi = toml_u32(x).unwrap_or(0) as u8; } + if let Some(x) = get_field(v, "rgb_counter") { cmap.rgb_counter = toml_u32(x).unwrap_or(0) as u8; } + if let Some(x) = get_field(v, "red_temp") { cmap.red_temp = toml_u32(x).unwrap_or(0) as u8; } + if let Some(x) = get_field(v, "green_temp") { cmap.green_temp = toml_u32(x).unwrap_or(0) as u8; } if let Some(x) = get_field(v, "command") { cmap.command = toml_u32(x).unwrap_or(0) as u8; } + if let Some(x) = get_field(v, "revision") { cmap.revision = toml_u32(x).unwrap_or(0) as u8; } if let Some(r) = get_field(v, "palette") { load_u32_slice(r, &mut cmap.palette); } cmap.dirty = true; } diff --git a/src/dev/ng1/rex3_frame.rs b/src/dev/ng1/rex3_frame.rs index e6bf97e36..393b870c4 100644 --- a/src/dev/ng1/rex3_frame.rs +++ b/src/dev/ng1/rex3_frame.rs @@ -14,6 +14,7 @@ use crate::dev::ng1::wgpu::direct::{ use crate::dev::ng1::wgpu::resident::{ Observation, ObservationReason, PlaneMask, Position, ResidentError, }; +use crate::dev::ng1::wgpu::telemetry::{duration_ns, FlushReason}; use crate::dev::ng1::wgpu::Ng1Scanout; use crate::disp::{BarStats, Rex3Screen, StatusBar, StatusBarTexture, FB_STRIDE}; use crate::debug_overlay::DebugOverlay; @@ -138,29 +139,41 @@ impl ResidentFrameControl { self.captured.store(true, Ordering::Release); } - fn invalidate(&self) { + pub(super) fn invalidate(&self) { self.captured.store(false, Ordering::Release); } + pub(super) fn quiescent(&self) -> bool { + let Some(_serial) = self.serial.try_lock() else { return false; }; + self.marker.lock().active.is_none() + } + fn capture_count(&self, mode: ResidentDisplayMode) -> CaptureCount { *self.captures.lock().for_mode(mode) } - fn capture_attempt(&self, mode: ResidentDisplayMode, bytes: u64) { - self.captures.lock().for_mode(mode).attempt(bytes); + fn capture_attempt(&self, mode: ResidentDisplayMode, bytes: u64) -> Result<(), ResidentError> { + self.captures.lock().for_mode(mode).attempt(bytes) } - fn capture_success(&self, mode: ResidentDisplayMode, bytes: u64) { - self.captures.lock().for_mode(mode).success(bytes); + fn capture_success(&self, mode: ResidentDisplayMode, bytes: u64) -> Result<(), ResidentError> { + self.captures.lock().for_mode(mode).success(bytes) } - fn capture_delta(&self, mode: ResidentDisplayMode, before: CaptureCount, after: CaptureCount) { + fn capture_delta(&self, mode: ResidentDisplayMode, before: CaptureCount, after: CaptureCount) -> Result<(), ResidentError> { let mut totals = self.captures.lock(); let total = totals.for_mode(mode); - total.attempts += after.attempts - before.attempts; - total.attempted_bytes += after.attempted_bytes - before.attempted_bytes; - total.successes += after.successes - before.successes; - total.successful_bytes += after.successful_bytes - before.successful_bytes; + let add_delta = |current: u64, old: u64, new: u64| { + new.checked_sub(old).and_then(|delta| current.checked_add(delta)) + .ok_or(ResidentError::InvalidState("composed capture counter decreased or overflowed")) + }; + *total = CaptureCount { + attempts: add_delta(total.attempts, before.attempts, after.attempts)?, + attempted_bytes: add_delta(total.attempted_bytes, before.attempted_bytes, after.attempted_bytes)?, + successes: add_delta(total.successes, before.successes, after.successes)?, + successful_bytes: add_delta(total.successful_bytes, before.successful_bytes, after.successful_bytes)?, + }; + Ok(()) } pub(super) fn wake(&self) { @@ -222,6 +235,7 @@ impl Rex3 { } fn resident_frame_cut(&self, timeout: Duration) -> Result, ResidentError> { + let started = Instant::now(); let control = &self.resident_frame; let deadline = Instant::now() + timeout; let serial = control.serial.try_lock_for(timeout) @@ -262,7 +276,10 @@ impl Rex3 { match result { Ok((position, topscan)) => { drop(state); - return Ok(FrameCut { control, id, position, topscan, requests, _serial: serial }); + let cut = FrameCut { control, id, position, topscan, requests, _serial: serial }; + let elapsed = duration_ns(started.elapsed()).map_err(ResidentError::InvalidState)?; + self.resident_telemetry.lock().cut(elapsed); + return Ok(cut); } Err(error) => { state.active = None; @@ -318,14 +335,24 @@ impl Rex3 { gpu: Arc, mode: ResidentDisplayMode, ) -> Result<(), ResidentError> { + let _lifecycle = self.resident_lifecycle.write(); if !self.running.load(Ordering::Acquire) { return Err(ResidentError::InvalidState("resident GFIFO is stopped")); } + self.check_resident_health(false)?; let mut thread_slot = self.refresh_thread.lock(); if thread_slot.is_some() { return Err(ResidentError::InvalidState("resident display already started")); } - let mut display = ResidentDisplay::new(self, &gpu, mode)?; + let mut display = match ResidentDisplay::new(self, &gpu, mode) { + Ok(display) => display, + Err(error) => { + if matches!(error, ResidentError::DeviceLost(_) | ResidentError::Failed(_)) { + self.fail_resident(error.clone()); + } + return Err(error); + } + }; if !self.running.load(Ordering::Acquire) { return Err(ResidentError::InvalidState("resident GFIFO is stopped")); } @@ -335,8 +362,15 @@ impl Rex3 { if std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| display.run(rex))).is_err() { rex.fail_resident(ResidentError::Failed("resident display worker panicked".into())); } - }) - .map_err(|error| ResidentError::Failed(format!("start resident display: {error}")))?; + }); + let handle = match handle { + Ok(handle) => handle, + Err(error) => { + let error = ResidentError::Failed(format!("start resident display: {error}")); + self.fail_resident(error.clone()); + return Err(error); + } + }; *thread_slot = Some(handle); Ok(()) } @@ -492,6 +526,7 @@ impl ResidentCompatibilityDisplay { if raster.status()?.accepted != cut.position { return Err(ResidentError::InvalidState("resident frame marker stream cut changed")); } + raster.submit_for(FlushReason::FrameCut)?; let observation = raster.observe(ObservationReason::Compatibility, PlaneMask::Both)?; if observation.position() != cut.position { return Err(ResidentError::InvalidState("compatibility observation used the wrong stream cut")); @@ -549,8 +584,8 @@ impl ResidentCompatibilityDisplay { let (width, height) = (screen.width, screen.height); if capture { let bytes = (width * height * size_of::()) as u64; - self.capture.attempt(bytes); - rex.resident_frame.capture_attempt(ResidentDisplayMode::ReadbackCompatibility, bytes); + self.capture.attempt(bytes)?; + rex.resident_frame.capture_attempt(ResidentDisplayMode::ReadbackCompatibility, bytes)?; let fault = self.capture_fault.take(); if matches!(fault, Some(CaptureFault::BeforeCapture)) { return Err(ResidentError::Failed("injected composed capture failure before capture".into())); @@ -565,8 +600,8 @@ impl ResidentCompatibilityDisplay { screen.rgba[row..row + width] .copy_from_slice(&compositor.pixels()[row..row + width]); } - self.capture.success(bytes); - rex.resident_frame.capture_success(ResidentDisplayMode::ReadbackCompatibility, bytes); + self.capture.success(bytes)?; + rex.resident_frame.capture_success(ResidentDisplayMode::ReadbackCompatibility, bytes)?; publish_capture(rex, &mut screen, requests, self.save_screenshots); } Ok(ResidentDisplayFrame { position: cut.position, width: width as u32, height: height as u32 }) @@ -669,7 +704,7 @@ impl ResidentDirectDisplay { return Err(ResidentError::InvalidState("resident frame marker stream cut changed")); } - raster.submit()?; + raster.submit_for(FlushReason::FrameCut)?; let frame = self.scanout.compose(&screen.display_state(), raster)?; if frame.position != cut.position { return Err(ResidentError::InvalidState("resident scanout used the wrong stream cut")); @@ -690,7 +725,7 @@ impl ResidentDirectDisplay { let before = self.scanout.traffic().composed_readbacks; let result = self.scanout.capture(stamp, raster, &mut self.capture); let after = self.scanout.traffic().composed_readbacks; - rex.resident_frame.capture_delta(ResidentDisplayMode::DirectWgpu, before, after); + rex.resident_frame.capture_delta(ResidentDisplayMode::DirectWgpu, before, after)?; result?; for y in 0..height { let row = y * FB_STRIDE; @@ -780,19 +815,23 @@ impl ResidentDisplay { timeout: Duration, at_marker: impl FnOnce(Position), ) -> Result { + let started = Instant::now(); let requests = rex.resident_frame.requests(); // Retain a completed capture across uncaptured presentations; otherwise // a CI consumer can miss it before it acquires the screen lock. if requests.needs_frame || requests.needs_screenshot || rex.head.capturing() { rex.resident_frame.invalidate(); } - match &mut self.driver { + let frame = match &mut self.driver { ResidentDisplayDriver::Direct(display) => { let stamp = display.render(rex, timeout, at_marker)?; Ok(ResidentDisplayFrame { position: stamp.position, width: stamp.width, height: stamp.height }) } ResidentDisplayDriver::Compatibility(display) => display.render(rex, timeout, at_marker), - } + }?; + let elapsed = duration_ns(started.elapsed()).map_err(ResidentError::InvalidState)?; + rex.resident_telemetry.lock().presentation(elapsed); + Ok(frame) } fn run(&mut self, rex: &Rex3) { @@ -869,9 +908,9 @@ mod tests { control.invalidate(); assert!(!control.frame_available(), "an unpublished display state is not a captured frame"); - control.capture_attempt(ResidentDisplayMode::DirectWgpu, 64); - control.capture_success(ResidentDisplayMode::DirectWgpu, 64); - control.capture_attempt(ResidentDisplayMode::ReadbackCompatibility, 32); + control.capture_attempt(ResidentDisplayMode::DirectWgpu, 64).unwrap(); + control.capture_success(ResidentDisplayMode::DirectWgpu, 64).unwrap(); + control.capture_attempt(ResidentDisplayMode::ReadbackCompatibility, 32).unwrap(); control.activate(); assert_eq!(control.capture_count(ResidentDisplayMode::DirectWgpu), CaptureCount { attempts: 1, successes: 1, attempted_bytes: 64, successful_bytes: 64, @@ -879,6 +918,15 @@ mod tests { assert_eq!(control.capture_count(ResidentDisplayMode::ReadbackCompatibility), CaptureCount { attempts: 1, successes: 0, attempted_bytes: 32, successful_bytes: 0, }); + let exhausted = CaptureCount { attempts: u64::MAX, attempted_bytes: 64, successes: 1, successful_bytes: 64 }; + control.captures.lock().direct = exhausted; + assert!(control.capture_attempt(ResidentDisplayMode::DirectWgpu, 1).is_err()); + assert_eq!(control.capture_count(ResidentDisplayMode::DirectWgpu), exhausted); + assert!(control.capture_delta( + ResidentDisplayMode::ReadbackCompatibility, + CaptureCount { attempts: 1, ..CaptureCount::default() }, + CaptureCount::default(), + ).is_err()); } #[test] @@ -939,6 +987,14 @@ mod tests { }); rex.running.store(false, Ordering::Release); rex.gfifo.reset(); + assert!(rex.snapshot_resident_registers().is_ok()); + let serial = rex.resident_frame.serial.lock(); + assert!(rex.snapshot_resident_registers().is_err()); + drop(serial); + rex.resident_frame.marker.lock().active = Some(Marker { id: 1, reached: None }); + assert!(rex.snapshot_resident_registers().is_err()); + rex.resident_frame.marker.lock().active = None; + assert!(rex.snapshot_resident_registers().is_ok()); }).unwrap().join().unwrap(); } diff --git a/src/dev/ng1/rex3_lifecycle.rs b/src/dev/ng1/rex3_lifecycle.rs new file mode 100644 index 000000000..a86ae3050 --- /dev/null +++ b/src/dev/ng1/rex3_lifecycle.rs @@ -0,0 +1,430 @@ +//! Stopped, test-only resident snapshot, restore and reset boundaries. + +use std::io; +use std::path::Path; +use std::sync::atomic::Ordering; + +use parking_lot::{RwLockReadGuard, RwLockWriteGuard}; + +use super::{RasterMode, Rex3}; +use crate::dev::ng1::wgpu::resident::{ + ObservationReason, PlaneMask, ResidentError, PLANE_WORDS, +}; +use crate::traits::{BUS_BUSY, BUS_ERR}; + +pub(super) struct ResidentLifecycle { + registers: Option, + planes: Option<(Vec, Vec)>, + rejected: bool, + timing_preset_pending: bool, +} + +impl Default for ResidentLifecycle { + fn default() -> Self { + Self { registers: None, planes: None, rejected: false, timing_preset_pending: true } + } +} + +impl ResidentLifecycle { + pub(super) fn ready(&self) -> bool { + !self.rejected && self.registers.is_none() && self.planes.is_none() + } + + fn reject_pending(&mut self) { + if !self.ready() { + self.registers = None; + self.planes = None; + self.rejected = true; + } + } + + fn resume_after_rejection(&mut self) { + if self.rejected { + let timing_preset_pending = self.timing_preset_pending; + *self = Self::default(); + self.timing_preset_pending = timing_preset_pending; + } + } +} + +impl Rex3 { + /// Bus operations retain this guard through all producer side effects, + /// including the return of a bulk HOSTR transfer. + pub(super) fn resident_bus_guard( + &self, + ) -> Result>, u32> { + if !self.resident_frame.is_active() { + return Ok(None); + } + let guard = self.resident_lifecycle.try_read().ok_or(BUS_BUSY)?; + if !self.running.load(Ordering::Acquire) + || self.resident_fatal_notified.load(Ordering::Acquire) + { + return Err(BUS_ERR); + } + Ok(Some(guard)) + } + + pub(super) fn resident_monitor_guard( + &self, + ) -> Result>, String> { + if !self.resident_frame.is_active() { + return Ok(None); + } + let guard = self.resident_lifecycle.try_write() + .ok_or_else(|| "REX3 resident lifecycle is busy".to_owned())?; + if !guard.ready() { + return Err("resident restore is incomplete".into()); + } + self.check_resident_health(false).map_err(|error| error.to_string())?; + if self.running.load(Ordering::Acquire) { + self.wait_resident_idle().map_err(|error| error.to_string())?; + } else { + self.check_resident_quiescent().map_err(|error| error.to_string())?; + } + Ok(Some(guard)) + } + + pub(super) fn check_resident_health(&self, reset: bool) -> Result<(), ResidentError> { + let result = { + let mut mode = self.raster.lock(); + match &mut *mode { + RasterMode::Resident(raster) => match raster.fault() { + Some(ResidentError::DeviceLost(_) | ResidentError::Failed(_)) => { + Err(raster.fault().unwrap().clone()) + } + Some(_) if reset => Ok(()), + Some(error) => Err(error.clone()), + None => raster.poll_device_health(), + }, + _ => Err(ResidentError::InvalidState("resident raster is not active")), + } + }; + if let Err(error) = &result { + if !matches!(error, ResidentError::InvalidState("resident raster is not active")) { + self.fail_resident(error.clone()); + } + } + result + } + + fn resident_boundary(&self, reset: bool) -> Result<(), ResidentError> { + self.check_resident_health(reset)?; + self.check_resident_quiescent() + } + + fn invalidate_resident_capture(&self) { + // A consumer checks availability again while holding this same lock. + let mut screen = self.head.screen().lock(); + self.resident_frame.invalidate(); + screen.width = 0; + screen.height = 0; + } + + pub(crate) fn apply_resident_display_resolution( + &self, + mode: crate::dev::ng1::vc2_timings::NewportResolution, + initial_only: bool, + ) -> Result<(), ResidentError> { + let mut lifecycle = self.resident_lifecycle.write(); + self.check_resident_health(false)?; + if !lifecycle.ready() { + return Err(ResidentError::InvalidState("resident restore is incomplete")); + } + if (initial_only && !lifecycle.timing_preset_pending) || mode.is_guest() { + return Ok(()); + } + self.check_resident_quiescent()?; + crate::dev::ng1::vc2_timings::apply_newport_resolution(&mut self.vc2.lock(), mode); + lifecycle.timing_preset_pending = false; + // The usual host background is a CPU-shadow fill; resident planes + // remain authoritative and are deliberately left untouched. + self.invalidate_resident_capture(); + Ok(()) + } + + pub(super) fn snapshot_resident_registers(&self) -> Result { + let lifecycle = self.resident_lifecycle.write(); + self.resident_boundary(false)?; + if !lifecycle.ready() { + return Err(ResidentError::InvalidState("resident restore is incomplete")); + } + Ok(self.save_state_fields()) + } + + pub(super) fn snapshot_resident_planes(&self) -> Result<(Vec, Vec), ResidentError> { + let lifecycle = self.resident_lifecycle.write(); + self.observe_resident_planes(&lifecycle) + } + + pub(crate) fn snapshot_resident_checkpoint( + &self, + ) -> Result<(toml::Value, (Vec, Vec)), ResidentError> { + let lifecycle = self.resident_lifecycle.write(); + let planes = self.observe_resident_planes(&lifecycle)?; + Ok((self.save_state_fields(), planes)) + } + + fn observe_resident_planes(&self, lifecycle: &ResidentLifecycle) -> Result<(Vec, Vec), ResidentError> { + self.resident_boundary(false)?; + if !lifecycle.ready() { + return Err(ResidentError::InvalidState("resident restore is incomplete")); + } + let result = { + let mut mode = self.raster.lock(); + match &mut *mode { + RasterMode::Resident(raster) => raster.observe(ObservationReason::Snapshot, PlaneMask::Both), + _ => Err(ResidentError::InvalidState("resident raster is not active")), + } + }; + let observation = match result { + Ok(observation) => observation, + Err(error) => { + self.fail_resident(error.clone()); + return Err(error); + } + }; + let decode = |bytes: Option<&[u8]>| -> Option> { + let bytes = bytes.filter(|bytes| bytes.len() == PLANE_WORDS * size_of::())?; + Some(bytes.chunks_exact(4).map(|word| { + u32::from_ne_bytes(word.try_into().expect("four-byte framebuffer word")) + }).collect()) + }; + let planes = if observation.reason() == ObservationReason::Snapshot + && observation.planes() == PlaneMask::Both + { + decode(observation.rgb()).zip(decode(observation.aux())) + } else { + None + }; + match planes { + Some(planes) => Ok(planes), + None => { + let error = ResidentError::Failed("incomplete resident snapshot observation".into()); + self.fail_resident(error.clone()); + Err(error) + } + } + } + + pub(super) fn save_resident_framebuffers(&self, dir: &Path, prefix: &str) -> io::Result<()> { + let lifecycle = self.resident_lifecycle.write(); + let (rgb, aux) = self.observe_resident_planes(&lifecycle).map_err(io::Error::other)?; + Self::write_captured_resident_framebuffers(dir, prefix, &rgb, &aux) + } + + pub(crate) fn write_captured_resident_framebuffers( + dir: &Path, + prefix: &str, + rgb: &[u32], + aux: &[u32], + ) -> io::Result<()> { + validate_planes(rgb.len(), aux.len()).map_err(io::Error::other)?; + let pending = incomplete_framebuffer_save(dir, prefix); + std::fs::write(&pending, b"REX3 resident framebuffer save is incomplete\n")?; + Self::write_framebuffer_files(dir, prefix, rgb, aux)?; + std::fs::remove_file(pending) + } + + pub(super) fn stage_resident_registers(&self, state: &toml::Value) -> Result<(), String> { + let mut lifecycle = self.resident_lifecycle.write(); + self.resident_boundary(false).map_err(|error| error.to_string())?; + if let Err(error) = validate_state(&self.save_state_fields(), state, "REX3") + .and_then(|()| validate_continuation(state)) + { + lifecycle.reject_pending(); + return Err(error); + } + if lifecycle.registers.is_some() { + lifecycle.reject_pending(); + return Err("resident register restore is already staged".into()); + } + lifecycle.resume_after_rejection(); + lifecycle.registers = Some(state.clone()); + self.invalidate_resident_capture(); + self.commit_resident_restore(&mut lifecycle).map_err(|error| error.to_string()) + } + + pub(super) fn stage_resident_planes(&self, rgb: &[u32], aux: &[u32]) -> Result<(), ResidentError> { + let mut lifecycle = self.resident_lifecycle.write(); + self.resident_boundary(false)?; + if let Err(error) = validate_planes(rgb.len(), aux.len()) { + lifecycle.reject_pending(); + return Err(error); + } + if lifecycle.planes.is_some() { + lifecycle.reject_pending(); + return Err(ResidentError::InvalidState("resident plane restore is already staged")); + } + lifecycle.resume_after_rejection(); + lifecycle.planes = Some((rgb.to_vec(), aux.to_vec())); + self.invalidate_resident_capture(); + self.commit_resident_restore(&mut lifecycle) + } + + pub(super) fn load_resident_framebuffers(&self, dir: &Path, prefix: &str) -> io::Result<()> { + let mut lifecycle = self.resident_lifecycle.write(); + self.resident_boundary(false).map_err(io::Error::other)?; + let read = |plane: &str| -> io::Result> { + let bytes = std::fs::read(dir.join(format!("{prefix}_{plane}.bin")))?; + if bytes.len() != PLANE_WORDS * size_of::() { + return Err(io::Error::new(io::ErrorKind::InvalidData, format!("incomplete resident {plane} plane"))); + } + Ok(bytes.chunks_exact(4).map(|word| { + u32::from_be_bytes(word.try_into().expect("four-byte framebuffer word")) + }).collect()) + }; + let planes = (|| { + if incomplete_framebuffer_save(dir, prefix).try_exists()? { + return Err(io::Error::new(io::ErrorKind::InvalidData, "resident framebuffer save is incomplete")); + } + Ok((read("rgb")?, read("aux")?)) + })(); + let (rgb, aux) = match planes { + Ok(planes) => planes, + Err(error) => { + lifecycle.reject_pending(); + return Err(error); + } + }; + if lifecycle.planes.is_some() { + lifecycle.reject_pending(); + return Err(io::Error::other("resident plane restore is already staged")); + } + lifecycle.resume_after_rejection(); + lifecycle.planes = Some((rgb, aux)); + self.invalidate_resident_capture(); + self.commit_resident_restore(&mut lifecycle).map_err(io::Error::other) + } + + fn commit_resident_restore(&self, lifecycle: &mut ResidentLifecycle) -> Result<(), ResidentError> { + let (Some(state), Some((rgb, aux))) = (&lifecycle.registers, &lifecycle.planes) else { + return Ok(()); + }; + let result = { + let mut mode = self.raster.lock(); + match &mut *mode { + RasterMode::Resident(raster) => raster.restore(rgb, aux), + _ => Err(ResidentError::InvalidState("resident raster is not active")), + } + }; + if let Err(error) = result { + lifecycle.reject_pending(); + self.fail_resident(error.clone()); + return Err(error); + } + self.load_state_fields(state); + self.xmap_fence.store(0, Ordering::Release); + self.gfifo_fence.store(0, Ordering::Release); + *lifecycle = ResidentLifecycle::default(); + lifecycle.timing_preset_pending = false; + self.invalidate_resident_capture(); + Ok(()) + } + + pub(super) fn reset_resident_board(&self) -> Result<(), ResidentError> { + let mut lifecycle = self.resident_lifecycle.write(); + self.resident_boundary(true)?; + self.invalidate_resident_capture(); + let zeros = vec![0u32; PLANE_WORDS]; + let result = { + let mut mode = self.raster.lock(); + match &mut *mode { + RasterMode::Resident(raster) => raster.reset(&zeros, &zeros), + _ => Err(ResidentError::InvalidState("resident raster is not active")), + } + }; + if let Err(error) = result { + self.fail_resident(error.clone()); + return Err(error); + } + self.reset_registers(); + self.gfifo.reset(); + self.gfxbusy.store(false, Ordering::Release); + self.xmap_fence.store(0, Ordering::Release); + self.gfifo_fence.store(0, Ordering::Release); + *lifecycle = ResidentLifecycle::default(); + self.resident_fatal_notified.store(false, Ordering::Release); + Ok(()) + } +} + +fn incomplete_framebuffer_save(dir: &Path, prefix: &str) -> std::path::PathBuf { + dir.join(format!("{prefix}_resident_incomplete")) +} + +fn validate_planes(rgb: usize, aux: usize) -> Result<(), ResidentError> { + for (plane, actual) in [("RGB", rgb), ("auxiliary", aux)] { + if actual != PLANE_WORDS { + return Err(ResidentError::InvalidSeed { plane, expected: PLANE_WORDS, actual }); + } + } + Ok(()) +} + +fn validate_state(template: &toml::Value, state: &toml::Value, path: &str) -> Result<(), String> { + match (template, state) { + (toml::Value::Table(expected), toml::Value::Table(actual)) => { + for (key, value) in expected { + let name = format!("{path}.{key}"); + let entry = actual.get(key).ok_or_else(|| format!("missing {name}"))?; + validate_state(value, entry, &name)?; + } + Ok(()) + } + (toml::Value::Array(expected), toml::Value::Array(actual)) if expected.len() == actual.len() => { + for (value, entry) in expected.iter().zip(actual) { + validate_state(value, entry, path)?; + } + Ok(()) + } + (toml::Value::String(example), value) => { + let bits = example.len().saturating_sub(2) * 4; + let max = if bits >= 64 { u64::MAX } else { (1u64 << bits) - 1 }; + if crate::snapshot::toml_u64(value).is_some_and(|number| number <= max) { + Ok(()) + } else { + Err(format!("invalid {path}")) + } + } + _ => Err(format!("incomplete or invalid {path}")), + } +} + +fn validate_continuation(state: &toml::Value) -> Result<(), String> { + for (section, key, max) in [ + ("context", "hostcnt", 8), + ("context", "pat_bit", 31), + ("context", "zpat_bit", 31), + ("context", "mid_primitive", 1), + ("cmap0", "rgb_counter", 2), + ("cmap1", "rgb_counter", 2), + ("bt445", "rgb_counter", 2), + ] { + let number = state.get(section).and_then(|v| v.get(key)).and_then(crate::snapshot::toml_u64); + if !number.is_some_and(|number| number <= max) { + return Err(format!("invalid REX3.{section}.{key}")); + } + } + for (section, keys) in [ + ("vc2", &["index"][..]), + ("xmap0", &["config", "cursor_cmap_msb", "popup_cmap_msb", "mode_addr", "fifo_avail"][..]), + ("xmap1", &["config", "cursor_cmap_msb", "popup_cmap_msb", "mode_addr", "fifo_avail"][..]), + ("cmap0", &["addr_lo", "addr_hi", "rgb_counter", "red_temp", "green_temp", "command", "revision"][..]), + ("cmap1", &["addr_lo", "addr_hi", "rgb_counter", "red_temp", "green_temp", "command", "revision"][..]), + ] { + for key in keys { + let value = state.get(section).and_then(|v| v.get(*key)).and_then(crate::snapshot::toml_u64); + if !value.is_some_and(|number| number <= u8::MAX as u64) { + return Err(format!("invalid REX3.{section}.{key}")); + } + } + } + let regs = state.get("vc2").and_then(|v| v.get("regs")).and_then(toml::Value::as_array); + if !regs.is_some_and(|regs| regs.iter().all(|value| { + crate::snapshot::toml_u64(value).is_some_and(|number| number <= u16::MAX as u64) + })) { + return Err("invalid REX3.vc2.regs".into()); + } + Ok(()) +} diff --git a/src/dev/ng1/rex3_lifecycle_tests.rs b/src/dev/ng1/rex3_lifecycle_tests.rs new file mode 100644 index 000000000..8e2a705ee --- /dev/null +++ b/src/dev/ng1/rex3_lifecycle_tests.rs @@ -0,0 +1,758 @@ +use super::*; +use crate::dev::ng1::wgpu::resident::{ + Observation, ObservationReason, PlaneMask, ResidentBackend, ResidentError, ResidentStatus, + PLANE_WORDS, +}; +use crate::gfx_display::{copy_frame, GfxDisplay}; +use crate::test_board::TestBoard; +use std::io; +use std::path::Path; +use std::sync::mpsc; +use std::time::Duration; + +struct ProbeState { + rgb: Vec, + aux: Vec, + observations: usize, + resets: usize, + restores: usize, + health_error: Option, + status_error: Option, + observation_error: Option, + reset_error: Option, + restore_error: Option, + observation_barrier: Option<(mpsc::Sender<()>, mpsc::Receiver<()>)>, + reset_barrier: Option<(mpsc::Sender<()>, mpsc::Receiver<()>)>, +} + +struct Probe(Arc>); + +impl ResidentBackend for Probe { + fn enqueue(&mut self, _: &mut Rex3Context) -> Result<(), ResidentError> { Ok(()) } + fn submit(&mut self) -> Result<(), ResidentError> { Ok(()) } + fn poll_device_health(&mut self) -> Result<(), ResidentError> { + self.0.lock().health_error.clone().map_or(Ok(()), Err) + } + fn status(&mut self) -> Result { + self.0.lock().status_error.clone().map_or_else(|| Ok(ResidentStatus::for_test()), Err) + } + fn observe(&mut self, reason: ObservationReason, planes: PlaneMask) -> Result { + assert_eq!((reason, planes), (ObservationReason::Snapshot, PlaneMask::Both)); + let mut state = self.0.lock(); + state.observations += 1; + if let Some(error) = &state.observation_error { + return Err(error.clone()); + } + if let Some((entered, release)) = state.observation_barrier.take() { + entered.send(()).unwrap(); + drop(state); + release.recv_timeout(Duration::from_secs(5)).unwrap(); + state = self.0.lock(); + } + Ok(Observation::for_test( + reason, planes, + Some(bytemuck::cast_slice(&state.rgb).to_vec()), + Some(bytemuck::cast_slice(&state.aux).to_vec()), + )) + } + fn reset(&mut self, rgb: &[u32], aux: &[u32]) -> Result<(), ResidentError> { + let mut state = self.0.lock(); + state.resets += 1; + if let Some(error) = &state.reset_error { return Err(error.clone()); } + if let Some((entered, release)) = state.reset_barrier.take() { + entered.send(()).unwrap(); + drop(state); + release.recv_timeout(Duration::from_secs(5)).unwrap(); + state = self.0.lock(); + } + assert_eq!((rgb.len(), aux.len()), (PLANE_WORDS, PLANE_WORDS)); + assert!(rgb.iter().all(|&word| word == 0) && aux.iter().all(|&word| word == 0)); + state.rgb.copy_from_slice(rgb); + state.aux.copy_from_slice(aux); + Ok(()) + } + fn restore(&mut self, rgb: &[u32], aux: &[u32]) -> Result<(), ResidentError> { + let mut state = self.0.lock(); + state.restores += 1; + if let Some(error) = state.restore_error.clone() { + state.rgb[0] ^= 0xffff_ffff; + return Err(error); + } + assert_eq!((rgb.len(), aux.len()), (PLANE_WORDS, PLANE_WORDS)); + state.rgb.copy_from_slice(rgb); + state.aux.copy_from_slice(aux); + Ok(()) + } +} + +fn on_large_stack(f: impl FnOnce() + Send + 'static) { + thread::Builder::new().name("resident-lifecycle-test".into()).stack_size(64 * 1024 * 1024) + .spawn(f).unwrap().join().unwrap_or_else(|panic| std::panic::resume_unwind(panic)); +} + +fn probe_state() -> Arc> { + let state = Arc::new(Mutex::new(ProbeState { + rgb: vec![0x1234_5678; PLANE_WORDS], + aux: vec![0x89ab_cdef; PLANE_WORDS], + observations: 0, resets: 0, restores: 0, + health_error: None, status_error: None, observation_error: None, reset_error: None, restore_error: None, + observation_barrier: None, reset_barrier: None, + })); + { + let mut data = state.lock(); + data.rgb[PLANE_WORDS - 1] = 0xfedc_ba98; + data.aux[PLANE_WORDS - 1] = 0x7654_3210; + } + state +} + +fn fixture() -> ( + TestBoard, + Arc, + Arc>, + plane::PlaneProtection, + plane::PlaneProtection, +) { + let mut rex = Box::new(Rex3::new(crate::disp::StatusSource::detached())); + rex.fb_rgb.get_mut().isolate().unwrap(); + rex.fb_aux.get_mut().isolate().unwrap(); + let state = probe_state(); + rex.attach_resident_raster(Probe(Arc::clone(&state))).unwrap(); + rex.fb_rgb.get_mut().reset_access_counts(); + rex.fb_aux.get_mut().reset_access_counts(); + let rgb_guard = unsafe { (&*rex.fb_rgb.get()).protect() }.unwrap(); + let aux_guard = unsafe { (&*rex.fb_aux.get()).protect() }.unwrap(); + let rex = Arc::::from(rex); + (TestBoard::new(Arc::clone(&rex)), rex, state, rgb_guard, aux_guard) +} + +fn assert_no_cpu_plane_access(rex: &Rex3) { + assert_eq!(unsafe { (&*rex.fb_rgb.get()).access_counts() }, Default::default()); + assert_eq!(unsafe { (&*rex.fb_aux.get()).access_counts() }, Default::default()); +} + +fn panics(f: impl FnOnce()) { + assert!(std::panic::catch_unwind(std::panic::AssertUnwindSafe(f)).is_err()); +} + +#[test] +fn resident_lifecycle_restores_registers_planes_and_continuations_atomically() { + on_large_stack(|| { + let (_owner, rex, state, _rgb_guard, _aux_guard) = fixture(); + let ctx = unsafe { &mut *rex.context.get() }; + ctx.colorred = 0x123456; + ctx.host_shifter = 0x1234_5678_9abc_def0; + ctx.hostcnt = 5; + ctx.hostrw[0] = 0xfedc_ba98_7654_3210; + ctx.pat_bit = 7; + ctx.zpat_bit = 11; + ctx.mid_primitive = true; + rex.config.config.store(0x45678, Ordering::Relaxed); + rex.dcb.lock().dcbdata1 = 0xabc_def0; + rex.vc2.lock().ram[123] = 0x5a5a; + rex.xmap1.lock().mode_table[9] = 0x123456; + { + let mut cmap = rex.cmap0.lock(); + cmap.addr_lo = 3; + cmap.write_crs(crate::dev::ng1::cmap::CMAP_REG_PALETTE, 0x12); + cmap.write_crs(crate::dev::ng1::cmap::CMAP_REG_PALETTE, 0x34); + } + rex.bt445.lock().palette[17] = [3, 5, 7]; + + let registers = rex.save_state(); + let (rgb, aux) = rex.snapshot_framebuffers_inmem(); + assert_eq!(state.lock().observations, 1); + assert_eq!((rgb[0], rgb[PLANE_WORDS - 1]), (0x1234_5678, 0xfedc_ba98)); + assert_eq!((aux[0], aux[PLANE_WORDS - 1]), (0x89ab_cdef, 0x7654_3210)); + + rex.power_on(); + assert_eq!(state.lock().resets, 1); + assert!(state.lock().rgb.iter().all(|&word| word == 0)); + assert_eq!(unsafe { (*rex.context.get()).colorred }, 0); + rex.load_state(®isters).unwrap(); + assert_eq!(unsafe { (*rex.context.get()).colorred }, 0, "staged registers are not published"); + assert_eq!(rex.start_resident_processor(), Err(ResidentError::InvalidState("resident restore is incomplete"))); + assert!(rex.apply_resident_display_resolution( + crate::dev::ng1::vc2_timings::NewportResolution::Res1024x768, true, + ).is_err()); + assert!(rex.execute_command("bt445", &["identity"], Box::new(io::sink())).is_err()); + panics(|| { let _ = rex.save_state(); }); + assert!(rex.snapshot_resident_planes().is_err()); + + rex.restore_framebuffers_inmem(&rgb, &aux); + assert_eq!(state.lock().restores, 1); + assert_eq!(rex.save_state(), registers); + let ctx = unsafe { &*rex.context.get() }; + assert_eq!((ctx.host_shifter, ctx.hostcnt, ctx.hostrw[0]), (0x1234_5678_9abc_def0, 5, 0xfedc_ba98_7654_3210)); + assert_eq!((ctx.pat_bit, ctx.zpat_bit, ctx.mid_primitive), (7, 11, true)); + { + let mut cmap = rex.cmap0.lock(); + cmap.write_crs(crate::dev::ng1::cmap::CMAP_REG_PALETTE, 0x56); + assert_eq!(cmap.palette[3], 0x563412); + } + assert_eq!(&state.lock().rgb, &rgb); + assert_eq!(&state.lock().aux, &aux); + + let expected = rex.save_state(); + let (entered_tx, entered_rx) = mpsc::channel(); + let (release_tx, release_rx) = mpsc::channel(); + state.lock().observation_barrier = Some((entered_tx, release_rx)); + let snapshot_board = Arc::clone(&rex); + let snapshot = thread::Builder::new().stack_size(64 * 1024 * 1024) + .spawn(move || snapshot_board.snapshot_resident_checkpoint()).unwrap(); + entered_rx.recv_timeout(Duration::from_secs(5)).unwrap(); + assert!(rex.resident_lifecycle.try_write().is_none()); + let (reset_tx, reset_rx) = mpsc::channel(); + let reset_board = Arc::clone(&rex); + let reset = thread::Builder::new().stack_size(64 * 1024 * 1024) + .spawn(move || reset_tx.send(reset_board.reset_resident_board()).unwrap()).unwrap(); + assert!(matches!(reset_rx.recv_timeout(Duration::from_millis(100)), Err(mpsc::RecvTimeoutError::Timeout))); + release_tx.send(()).unwrap(); + let (saved_state, (saved_rgb, saved_aux)) = snapshot.join().unwrap().unwrap(); + assert_eq!(saved_state, expected); + assert_eq!(saved_rgb, rgb); + assert_eq!(saved_aux, aux); + reset_rx.recv_timeout(Duration::from_secs(5)).unwrap().unwrap(); + reset.join().unwrap(); + assert_eq!(unsafe { (*rex.context.get()).colorred }, 0); + assert_no_cpu_plane_access(&rex); + }); +} + +#[test] +fn resident_lifecycle_rejects_nonquiescent_and_incomplete_transfers() { + on_large_stack(|| { + let (_owner, rex, state, _rgb_guard, _aux_guard) = fixture(); + let registers = rex.save_state(); + rex.start(); + assert!(rex.save_framebuffers_named(Path::new("target"), "resident-m7-must-not-exist").is_err()); + panics(|| { let _ = rex.snapshot_framebuffers_inmem(); }); + panics(|| { let _ = rex.save_state(); }); + assert!(rex.load_state(®isters).is_err()); + panics(|| rex.power_on()); + assert_eq!(state.lock().observations, 0); + rex.stop(); + + assert_eq!(rex.write32((REX3_BASE | REX3_WRMASK) as u64, 1), BUS_ERR); + assert!(rex.gfifo.try_push(REX3_WRMASK, 5)); + assert!(rex.snapshot_resident_planes().is_err()); + assert!(rex.start_resident_processor().is_err()); + panics(|| rex.power_on()); + rex.gfifo.reset(); + unsafe { (*rex.context.get()).host_len = 2; } + assert!(rex.snapshot_resident_planes().is_err()); + assert!(rex.load_state(®isters).is_err()); + assert!(rex.execute_command("rex", &["status"], Box::new(io::sink())).is_err()); + unsafe { (*rex.context.get()).host_len = 1; (*rex.context.get()).host_cursor = 1; } + assert!(rex.snapshot_resident_planes().is_err()); + unsafe { (*rex.context.get()).host_cursor = 0; } + assert!(rex.snapshot_resident_planes().is_ok()); + assert_no_cpu_plane_access(&rex); + }); +} + +#[test] +fn resident_lifecycle_reset_hides_frontend_before_upload() { + on_large_stack(|| { + use crate::dev::ng1::vc2::{VC2_CTRL_CURSOR_EN, VC2_REG_DISPLAY_CONTROL}; + let (_owner, rex, state, _rgb_guard, _aux_guard) = fixture(); + { + let mut screen = rex.head.screen().lock(); + screen.width = 32; + screen.height = 32; + } + rex.vc2.lock().regs[VC2_REG_DISPLAY_CONTROL as usize] = VC2_CTRL_CURSOR_EN; + assert!(rex.hw_cursor().is_some()); + + let (entered_tx, entered_rx) = mpsc::channel(); + let (release_tx, release_rx) = mpsc::channel(); + state.lock().reset_barrier = Some((entered_tx, release_rx)); + let resetting = Arc::clone(&rex); + let resetting = thread::Builder::new().stack_size(64 * 1024 * 1024) + .spawn(move || resetting.power_on()).unwrap(); + entered_rx.recv_timeout(Duration::from_secs(2)).unwrap(); + let hidden_during_upload = rex.hw_cursor().is_none(); + let debugger_busy = rex.execute_command("bt445", &["identity"], Box::new(io::sink())).is_err(); + release_tx.send(()).unwrap(); + resetting.join().unwrap(); + assert!(hidden_during_upload); + assert!(debugger_busy); + assert!(rex.hw_cursor().is_none()); + rex.execute_command("bt445", &["identity"], Box::new(io::sink())).unwrap(); + assert_eq!(rex.bt445.lock().palette[1], [1, 1, 1]); + assert_no_cpu_plane_access(&rex); + }); +} + +#[test] +fn resident_lifecycle_stop_waits_for_producers_and_restart_rebinds_wakeup() { + on_large_stack(|| { + let (_owner, rex, _state, _rgb_guard, _aux_guard) = fixture(); + rex.start(); + rex.execute_command("rex", &["status"], Box::new(io::sink())).unwrap(); + while rex.resident_unparker.lock().is_none() { thread::yield_now(); } + let first = rex.resident_unparker.lock().as_ref().unwrap().id(); + let (held_tx, held_rx) = mpsc::channel(); + let (release_tx, release_rx) = mpsc::channel(); + let producer = Arc::clone(&rex); + let producer = thread::spawn(move || { + let _guard = producer.resident_bus_guard().unwrap().unwrap(); + held_tx.send(()).unwrap(); + release_rx.recv().unwrap(); + }); + held_rx.recv_timeout(Duration::from_secs(2)).unwrap(); + let (done_tx, done_rx) = mpsc::channel(); + let stopping = Arc::clone(&rex); + let stopping = thread::spawn(move || { + stopping.stop(); + done_tx.send(()).unwrap(); + }); + assert!(done_rx.recv_timeout(Duration::from_millis(100)).is_err()); + release_tx.send(()).unwrap(); + producer.join().unwrap(); + done_rx.recv_timeout(Duration::from_secs(2)).unwrap(); + stopping.join().unwrap(); + assert_eq!(rex.read32((REX3_BASE | REX3_WRMASK) as u64).status, BUS_ERR); + + rex.start(); + while rex.resident_unparker.lock().is_none() { thread::yield_now(); } + assert_ne!(rex.resident_unparker.lock().as_ref().unwrap().id(), first); + let deadline = std::time::Instant::now() + Duration::from_secs(2); + while !rex.processor_parked.load(Ordering::Acquire) { + assert!(std::time::Instant::now() < deadline); + thread::yield_now(); + } + assert_eq!(rex.write32((REX3_BASE | REX3_WRMASK) as u64, 0x12345678), BUS_OK); + rex.wait_resident_idle().unwrap(); + assert_eq!(unsafe { (*rex.context.get()).wrmask }, 0x12345678); + rex.stop(); + assert_no_cpu_plane_access(&rex); + }); +} + +#[test] +fn resident_lifecycle_machine_head1_checkpoint_round_trip() { + const CHILD: &str = "IRIS_RESIDENT_M7_HEAD1_CHILD"; + if std::env::var_os(CHILD).is_none() { + struct Remove(std::path::PathBuf); + impl Drop for Remove { + fn drop(&mut self) { let _ = std::fs::remove_dir_all(&self.0); } + } + let path = std::env::current_dir().unwrap().join("target") + .join(format!("resident-m7-head1-{}", std::process::id())); + std::fs::create_dir(&path).unwrap(); + let _cleanup = Remove(path.clone()); + let output = std::process::Command::new(std::env::current_exe().unwrap()) + .current_dir(&path) + .env(CHILD, &path) + .env("IRIS_REX_JIT_PROFILE", "profile.bin") + .args([ + "--exact", "dev::ng1::rex3::lifecycle_tests::resident_lifecycle_machine_head1_checkpoint_round_trip", + "--nocapture", "--test-threads=1", + ]) + .output().unwrap(); + assert!(output.status.success(), "head 1 child failed:\n{}\n{}", + String::from_utf8_lossy(&output.stdout), String::from_utf8_lossy(&output.stderr)); + return; + } + + let path = std::path::PathBuf::from(std::env::var_os(CHILD).unwrap()); + assert_eq!(std::env::current_dir().unwrap(), path); + assert!(path.file_name().and_then(|name| name.to_str()) + .is_some_and(|name| name.starts_with("resident-m7-head1-"))); + assert_eq!(path.parent().and_then(|parent| parent.file_name()), Some(std::ffi::OsStr::new("target"))); + on_large_stack(|| { + let mut cfg = crate::bench_runner::bench_config([16, 0, 0, 0], &crate::cpu::parts::R4400_150); + cfg.headless = false; + cfg.graphics.heads = 2; + cfg.nvram.clear(); + cfg.nveeprom.clear(); + cfg.monitor_port = Some(0); + cfg.graphics.resolution = crate::dev::ng1::vc2_timings::NewportResolution::Res1024x768; + let mut machine = Box::new(crate::machine::Machine::new(cfg)); + let primary = machine.get_rex3().expect("primary Newport head"); + let rex = machine.get_rex3_head1().expect("second Newport head"); + primary.start(); + rex.start(); + let incomplete_checkpoint = machine.capture_live_checkpoint(); + assert!(primary.check_resident_quiescent().is_ok()); + assert!(rex.check_resident_quiescent().is_ok()); + let primary_state = probe_state(); + { + let mut data = primary_state.lock(); + data.rgb[0] = 0xaabb_ccdd; + data.aux[0] = 0x1122_3344; + } + let state = probe_state(); + let attach = |board: &Rex3, data: &Arc>| { + unsafe { + (&mut *board.fb_rgb.get()).isolate().unwrap(); + (&mut *board.fb_aux.get()).isolate().unwrap(); + (&mut *board.fb_rgb.get()).reset_access_counts(); + (&mut *board.fb_aux.get()).reset_access_counts(); + } + { + let mut raster = board.raster.lock(); + assert_eq!(raster.kind(), RasterModeKind::Cpu); + *raster = RasterMode::Resident(crate::dev::ng1::wgpu::resident::ResidentRaster::new( + Probe(Arc::clone(data)), + )); + } + board.resident_frame.activate(); + ( + unsafe { (&*board.fb_rgb.get()).protect() }.unwrap(), + unsafe { (&*board.fb_aux.get()).protect() }.unwrap(), + ) + }; + let primary_guards = attach(&primary, &primary_state); + let head1_guards = attach(&rex, &state); + assert!(machine.restore_live_checkpoint(&incomplete_checkpoint).unwrap_err().contains("head 1")); + assert_eq!((primary_state.lock().restores, state.lock().restores), (0, 0)); + drop(incomplete_checkpoint); + let visible = |board: &Rex3| { + let vc2 = board.vc2.lock(); + crate::dev::ng1::vc2_timings::decode_vc2_visible_size(&vc2.regs, &vc2.ram) + }; + machine.start(); + machine.stop(); + assert_eq!(visible(&primary), (1024, 768)); + assert_eq!(visible(&rex), (1024, 768)); + assert_eq!(primary_state.lock().rgb[0], 0xaabb_ccdd); + assert_eq!(state.lock().rgb[0], 0x1234_5678); + unsafe { + (*primary.context.get()).colorred = 0x456789; + (*rex.context.get()).colorred = 0x123456; + } + primary.vc2.lock().ram[123] = 0x4567; + rex.vc2.lock().ram[123] = 0x1234; + primary.start(); + rex.start(); + + let checkpoint = machine.capture_live_checkpoint(); + unsafe { + (*primary.context.get()).colorred = 0x112233; + (*rex.context.get()).colorred = 0xabcdef; + } + { + let mut data = state.lock(); + data.rgb[0] = 0; + data.aux[0] = 0; + } + { + let mut data = primary_state.lock(); + data.rgb[0] = 0; + data.aux[0] = 0; + } + machine.restore_live_checkpoint(&checkpoint).unwrap(); + assert!(primary.is_running()); + assert!(rex.is_running()); + assert_eq!(unsafe { (*primary.context.get()).colorred }, 0x456789); + assert_eq!(unsafe { (*rex.context.get()).colorred }, 0x123456); + assert_eq!((primary.vc2.lock().ram[123], rex.vc2.lock().ram[123]), (0x4567, 0x1234)); + { + let data = primary_state.lock(); + assert_eq!((data.rgb[0], data.aux[0]), (0xaabb_ccdd, 0x1122_3344)); + assert_eq!((data.observations, data.restores), (1, 1)); + } + { + let data = state.lock(); + assert_eq!((data.rgb[0], data.aux[0]), (0x1234_5678, 0x89ab_cdef)); + assert_eq!((data.observations, data.restores), (1, 1)); + } + + machine.save_snapshot("m7-head1").unwrap(); + let snapshot = Path::new("saves").join("m7-head1"); + assert!(snapshot.join("rex3_head1_rgb.bin").is_file()); + assert!(snapshot.join("rex3_head1_aux.bin").is_file()); + machine.stop(); + unsafe { + (*primary.context.get()).colorred = 0x998877; + (*rex.context.get()).colorred = 0xfedcba; + } + { + let mut data = state.lock(); + data.rgb[0] = 0; + data.aux[0] = 0; + } + { + let mut data = primary_state.lock(); + data.rgb[0] = 0; + data.aux[0] = 0; + } + machine.load_snapshot_paused("m7-head1").unwrap(); + assert_eq!(unsafe { (*primary.context.get()).colorred }, 0x456789); + assert_eq!(unsafe { (*rex.context.get()).colorred }, 0x123456); + assert_eq!((primary.vc2.lock().ram[123], rex.vc2.lock().ram[123]), (0x4567, 0x1234)); + { + let data = primary_state.lock(); + assert_eq!((data.rgb[0], data.aux[0]), (0xaabb_ccdd, 0x1122_3344)); + assert_eq!((data.observations, data.restores), (2, 2)); + } + machine.start(); + assert_eq!((primary.vc2.lock().ram[123], rex.vc2.lock().ram[123]), (0x4567, 0x1234)); + machine.stop(); + primary.apply_display_resolution(crate::dev::ng1::vc2_timings::NewportResolution::Res1280x960); + assert_eq!(visible(&primary).1, 960); + assert_eq!(primary_state.lock().rgb[0], 0xaabb_ccdd); + { + let data = state.lock(); + assert_eq!((data.rgb[0], data.aux[0]), (0x1234_5678, 0x89ab_cdef)); + assert_eq!((data.observations, data.restores), (2, 2)); + } + let rgb_path = snapshot.join("rex3_head1_rgb.bin"); + let rgb_file = std::fs::read(&rgb_path).unwrap(); + machine.ci_restore("m7-head1").unwrap(); + std::fs::remove_file(&rgb_path).unwrap(); + machine.stop(); + unsafe { (*rex.context.get()).colorred = 0x111111; } + { + let mut data = state.lock(); + data.rgb[0] = 0; + data.aux[0] = 0; + } + machine.ci_rollback().unwrap(); + machine.stop(); + assert_eq!(unsafe { (*rex.context.get()).colorred }, 0x123456); + { + let data = state.lock(); + assert_eq!((data.rgb[0], data.aux[0]), (0x1234_5678, 0x89ab_cdef)); + assert_eq!((data.observations, data.restores), (3, 4)); + } + assert!(machine.load_snapshot_paused("m7-head1").is_err()); + assert!(rex.start_resident_processor().is_err()); + std::fs::write(&rgb_path, rgb_file).unwrap(); + let state_path = snapshot.join("rex3_head1.bin"); + let state_file = std::fs::read(&state_path).unwrap(); + std::fs::remove_file(&state_path).unwrap(); + assert!(machine.load_snapshot_paused("m7-head1").is_err()); + assert_eq!(state.lock().restores, 4); + std::fs::write(&state_path, state_file).unwrap(); + let chunks_path = snapshot.join("chunks.bin"); + let chunks_file = std::fs::read(&chunks_path).unwrap(); + let mut chunks: crate::snapshot::ChunksManifest = postcard::from_bytes(&chunks_file).unwrap(); + chunks.framebuffer_chunks = None; + std::fs::write(&chunks_path, postcard::to_allocvec(&chunks).unwrap()).unwrap(); + assert!(machine.load_snapshot_paused("m7-head1").unwrap_err().contains("both planes")); + assert!(primary.start_resident_processor().is_err()); + assert_eq!(primary_state.lock().restores, 4); + machine.reset(); + assert_eq!(visible(&primary), (1024, 768)); + assert_eq!(visible(&rex), (1024, 768)); + assert!(primary_state.lock().rgb.iter().all(|&word| word == 0)); + assert!(state.lock().rgb.iter().all(|&word| word == 0)); + std::fs::write(&chunks_path, chunks_file).unwrap(); + std::fs::remove_file(&rgb_path).unwrap(); + std::fs::create_dir(&rgb_path).unwrap(); + assert!(machine.save_snapshot("m7-head1").is_err()); + assert!(snapshot.join("resident_snapshot_incomplete").exists()); + assert!(snapshot.join("rex3_head1_resident_incomplete").exists()); + let reset_counts = (primary_state.lock().resets, state.lock().resets); + assert!(machine.load_snapshot_paused("m7-head1").unwrap_err().contains("incomplete")); + assert_eq!((primary_state.lock().resets, state.lock().resets), reset_counts); + std::fs::remove_dir(&rgb_path).unwrap(); + machine.save_snapshot("m7-head1").unwrap(); + assert!(!snapshot.join("resident_snapshot_incomplete").exists()); + assert!(!snapshot.join("rex3_head1_resident_incomplete").exists()); + machine.load_snapshot_paused("m7-head1").unwrap(); + + let lost = ResidentError::DeviceLost("injected snapshot completion failure".into()); + primary_state.lock().status_error = Some(lost.clone()); + assert!(machine.save_snapshot("m7-head1").is_err()); + assert!(snapshot.join("resident_snapshot_incomplete").exists()); + assert_eq!(primary.resident_fault(), Some(lost.clone())); + assert_eq!(primary.start_resident_processor(), Err(lost)); + assert!(machine.load_snapshot_paused("m7-head1").unwrap_err().contains("incomplete")); + assert_no_cpu_plane_access(&primary); + assert_no_cpu_plane_access(&rex); + machine.stop(); + drop(machine); + drop(head1_guards); + drop(primary_guards); + }); +} + +#[test] +fn resident_lifecycle_invalid_inputs_cannot_mix_staged_snapshots() { + on_large_stack(|| { + let (_owner, rex, data, _rgb_guard, _aux_guard) = fixture(); + let registers = rex.save_state(); + let rgb = data.lock().rgb.clone(); + let aux = data.lock().aux.clone(); + unsafe { (*rex.context.get()).colorred = 0x7777; } + + let mut bad = registers.clone(); + bad["vc2"]["regs"].as_array_mut().unwrap()[0] = toml::Value::String("0x00010000".into()); + rex.restore_framebuffers_inmem(&rgb, &aux); + assert!(rex.load_state(&bad).is_err()); + assert!(rex.start_resident_processor().is_err()); + assert_eq!(unsafe { (*rex.context.get()).colorred }, 0x7777); + assert_eq!(data.lock().restores, 0); + + rex.load_state(®isters).unwrap(); + assert!(rex.stage_resident_planes(&rgb[..rgb.len() - 1], &aux).is_err()); + assert!(rex.start_resident_processor().is_err()); + rex.restore_framebuffers_inmem(&rgb, &aux); + assert!(rex.start_resident_processor().is_err(), "rejected state cannot be paired with later planes"); + rex.load_state(®isters).unwrap(); + assert_eq!(data.lock().restores, 1); + assert_eq!(rex.save_state(), registers); + + rex.load_state(®isters).unwrap(); + assert!(rex.load_state(®isters).is_err(), "a duplicate register half is ambiguous"); + rex.restore_framebuffers_inmem(&rgb, &aux); + assert!(rex.start_resident_processor().is_err()); + rex.load_state(®isters).unwrap(); + assert_eq!(data.lock().restores, 2); + + rex.restore_framebuffers_inmem(&rgb, &aux); + assert!(rex.stage_resident_planes(&rgb, &aux).is_err(), "a duplicate plane half is ambiguous"); + rex.load_state(®isters).unwrap(); + assert!(rex.start_resident_processor().is_err()); + rex.restore_framebuffers_inmem(&rgb, &aux); + assert_eq!(data.lock().restores, 3); + + let mut bad = registers.clone(); + bad["xmap0"]["config"] = toml::Value::String("0x00000100".into()); + assert!(rex.load_state(&bad).is_err()); + bad = registers.clone(); + bad["context"]["hostcnt"] = toml::Value::String("0x00000009".into()); + assert!(rex.load_state(&bad).is_err()); + assert_eq!(rex.save_state(), registers); + assert_no_cpu_plane_access(&rex); + }); +} + +#[test] +fn resident_lifecycle_device_failures_are_sticky_and_never_publish_partial_state() { + on_large_stack(|| { + for stage in ["health", "observation", "restore", "reset"] { + let (_owner, rex, data, _rgb_guard, _aux_guard) = fixture(); + let (sender, receiver) = mpsc::channel(); + rex.set_fatal_sender(sender); + unsafe { (*rex.context.get()).colorred = 0x1111; } + let registers = rex.save_state(); + unsafe { (*rex.context.get()).colorred = 0x2222; } + let error = if stage == "health" || stage == "observation" { + ResidentError::DeviceLost(format!("injected {stage}")) + } else { + ResidentError::Failed(format!("injected {stage}")) + }; + match stage { + "health" => { + data.lock().health_error = Some(error.clone()); + assert_eq!(rex.snapshot_resident_planes(), Err(error.clone())); + } + "observation" => { + data.lock().observation_error = Some(error.clone()); + assert_eq!(rex.snapshot_resident_planes(), Err(error.clone())); + } + "restore" => { + let rgb = data.lock().rgb.clone(); + let aux = data.lock().aux.clone(); + rex.load_state(®isters).unwrap(); + data.lock().restore_error = Some(error.clone()); + assert_eq!(rex.stage_resident_planes(&rgb, &aux), Err(error.clone())); + } + "reset" => { + data.lock().reset_error = Some(error.clone()); + assert_eq!(rex.reset_resident_board(), Err(error.clone())); + } + _ => unreachable!(), + } + assert_eq!(rex.resident_fault(), Some(error.clone())); + assert_eq!(unsafe { (*rex.context.get()).colorred }, 0x2222); + assert!(rex.load_state(®isters).is_err()); + assert_eq!(rex.snapshot_resident_planes(), Err(error.clone())); + assert_eq!(rex.reset_resident_board(), Err(error.clone())); + assert_eq!(rex.start_resident_processor(), Err(error.clone())); + let vc2_regs = rex.vc2.lock().regs; + assert_eq!(rex.apply_resident_display_resolution( + crate::dev::ng1::vc2_timings::NewportResolution::Res1024x768, true, + ), Err(error.clone())); + assert_eq!(rex.vc2.lock().regs, vc2_regs); + assert!(rex.execute_command("rex", &["status"], Box::new(io::sink())).is_err()); + panics(|| { let _ = rex.save_state(); }); + panics(|| rex.restore_framebuffers_inmem(&[], &[])); + assert!(rex.save_framebuffers_named(Path::new("target"), "resident-m7-failed").is_err()); + assert!(rex.load_framebuffers_named(Path::new("target"), "resident-m7-failed").is_err()); + assert!(copy_frame(&*rex, &mut Vec::new()).is_none()); + let notification = receiver.recv_timeout(Duration::from_secs(2)).unwrap(); + assert_eq!(notification.message, error.to_string()); + assert!(receiver.try_recv().is_err(), "only the first failure is reported"); + assert_no_cpu_plane_access(&rex); + } + }); +} + +#[test] +fn resident_lifecycle_disk_restore_requires_two_complete_planes() { + on_large_stack(|| { + struct Remove(std::path::PathBuf); + impl Drop for Remove { + fn drop(&mut self) { let _ = std::fs::remove_dir_all(&self.0); } + } + let path = Path::new("target").join(format!( + "resident-m7-files-{}-{:?}", std::process::id(), thread::current().id(), + ).replace(['(', ')'], "")); + std::fs::create_dir_all(&path).unwrap(); + let _cleanup = Remove(path.clone()); + let (_owner, rex, state, _rgb_guard, _aux_guard) = fixture(); + let registers = rex.save_state(); + rex.save_framebuffers_named(&path, "board").unwrap(); + assert_eq!(state.lock().observations, 1); + let rgb = std::fs::read(path.join("board_rgb.bin")).unwrap(); + let aux = std::fs::read(path.join("board_aux.bin")).unwrap(); + assert_eq!(&rgb[..4], &0x1234_5678u32.to_be_bytes()); + assert_eq!(&aux[..4], &0x89ab_cdefu32.to_be_bytes()); + unsafe { (*rex.context.get()).colorred = 0x7777; } + rex.load_state(®isters).unwrap(); + std::fs::write(path.join("board_aux.bin"), &aux[..7]).unwrap(); + assert_eq!(rex.load_framebuffers_named(&path, "board").unwrap_err().kind(), io::ErrorKind::InvalidData); + assert_eq!(unsafe { (*rex.context.get()).colorred }, 0x7777); + assert!(rex.start_resident_processor().is_err()); + assert_eq!(state.lock().restores, 0); + std::fs::remove_file(path.join("board_aux.bin")).unwrap(); + assert_eq!(rex.load_framebuffers_named(&path, "board").unwrap_err().kind(), io::ErrorKind::NotFound); + std::fs::write(path.join("board_aux.bin"), &aux).unwrap(); + rex.load_framebuffers_named(&path, "board").unwrap(); + assert!(rex.start_resident_processor().is_err()); + rex.load_state(®isters).unwrap(); + assert_eq!(rex.save_state(), registers); + assert_eq!(state.lock().restores, 1); + + { + let mut data = state.lock(); + data.rgb[0] = 0x0102_0304; + data.aux[0] = 0x0506_0708; + } + let aux_path = path.join("board_aux.bin"); + std::fs::remove_file(&aux_path).unwrap(); + std::fs::create_dir(&aux_path).unwrap(); + assert!(rex.save_framebuffers_named(&path, "board").is_err()); + assert!(path.join("board_resident_incomplete").exists()); + std::fs::remove_dir(&aux_path).unwrap(); + std::fs::write(&aux_path, &aux).unwrap(); + assert_eq!(&std::fs::read(path.join("board_rgb.bin")).unwrap()[..4], &0x0102_0304u32.to_be_bytes()); + rex.load_state(®isters).unwrap(); + assert_eq!(rex.load_framebuffers_named(&path, "board").unwrap_err().kind(), io::ErrorKind::InvalidData); + assert!(rex.start_resident_processor().is_err()); + let (rgb, aux) = { + let data = state.lock(); + (data.rgb.clone(), data.aux.clone()) + }; + rex.restore_framebuffers_inmem(&rgb, &aux); + assert!(rex.start_resident_processor().is_err()); + rex.load_state(®isters).unwrap(); + rex.save_framebuffers_named(&path, "board").unwrap(); + assert!(!path.join("board_resident_incomplete").exists()); + rex.load_state(®isters).unwrap(); + rex.load_framebuffers_named(&path, "board").unwrap(); + assert_eq!(state.lock().restores, 3); + assert_eq!(state.lock().observations, 3); + { + let data = state.lock(); + assert_eq!((data.rgb[0], data.aux[0]), (0x0102_0304, 0x0506_0708)); + } + assert_no_cpu_plane_access(&rex); + }); +} diff --git a/src/dev/ng1/rex3_rec.rs b/src/dev/ng1/rex3_rec.rs index cc768f1c4..a7f6873dd 100644 --- a/src/dev/ng1/rex3_rec.rs +++ b/src/dev/ng1/rex3_rec.rs @@ -802,7 +802,7 @@ pub(crate) mod tests { replay_recs_with_hook(rex, bus, recs, check_hashes, |_, _| {}) } - pub(super) fn replay_recs_with_hook( + pub(crate) fn replay_recs_with_hook( rex: &Rex3, bus: &dyn BusDevice, recs: &[Rec], check_hashes: bool, mut before_record: impl FnMut(usize, &Rec), ) -> Report { @@ -908,6 +908,15 @@ pub(crate) mod tests { // A failed access can leave the processor blocked or already dead. // Only a completed replay may synchronously drain it. if rep.access_error.is_none() { + #[cfg(feature = "wgpu")] + if rex.resident_active() { + if rex.wait_resident_idle().is_err() { + rep.access_error = Some("resident replay failed while draining".into()); + } + } else { + rex.wait_idle(); + } + #[cfg(not(feature = "wgpu"))] rex.wait_idle(); } rep diff --git a/src/dev/ng1/rex3_tests.rs b/src/dev/ng1/rex3_tests.rs index 3e9f74609..386eba300 100644 --- a/src/dev/ng1/rex3_tests.rs +++ b/src/dev/ng1/rex3_tests.rs @@ -364,19 +364,20 @@ mod resident_gate_tests { } #[test] - fn resident_gate_rejects_unimplemented_cpu_plane_operations() { + fn resident_gate_rejects_unsupported_observers_and_invalid_restores() { with_big_stack(|| { let mut rex = Box::new(Rex3::new(crate::disp::StatusSource::detached())); let before = unsafe { (*rex.fb_rgb.get())[0] }; unsafe { (*rex.context.get()).colorred = 0x1234; } rex.attach_resident_raster(probe(Ok(())).0).unwrap(); let path = std::path::Path::new("."); - assert!(rex.save_framebuffers_named(path, "unused").is_err()); - assert!(rex.load_framebuffers_named(path, "unused").is_err()); assert!(rex.dump_framebuffer_debug(path).is_err()); assert!(rex.load_state(&toml::Value::Table(Default::default())).is_err()); assert!(rex.execute_command("rex", &["fbdump"], Box::new(std::io::sink())).is_err()); assert!(rex.gfifo.is_empty()); + assert!(rex.save_state().get("context").is_some()); + assert!(rex.save_framebuffers_named(path, "unused").is_err()); + assert!(rex.load_framebuffers_named(path, "unused").is_err()); let rejected = |operation: &dyn Fn()| { assert!(std::panic::catch_unwind(std::panic::AssertUnwindSafe(operation)).is_err()); @@ -385,7 +386,7 @@ mod resident_gate_tests { rejected(&|| rex.restore_framebuffers_inmem(&[0], &[0])); rejected(&|| rex.power_on()); rejected(&|| { let _ = rex.save_state(); }); - rex.wait_resident_idle().unwrap(); + assert!(rex.wait_resident_idle().is_err()); #[cfg(feature = "gputrace")] { assert!(rex.set_recording(None, false).is_err()); @@ -745,18 +746,18 @@ mod resident_gate_tests { let mut rex = Box::new(Rex3::new(crate::disp::StatusSource::detached())); rex.attach_resident_raster(ResetBackend).unwrap(); rex.start(); - while rex.processor_unparker.get().is_none() { + while rex.resident_unparker.lock().is_none() { std::thread::yield_now(); } - let first = rex.processor_unparker.get().unwrap().id(); + let first = rex.resident_unparker.lock().as_ref().unwrap().id(); rex.stop(); rex.reset_resident_gpu().unwrap(); - assert!(rex.processor_unparker.get().is_none()); + assert!(rex.resident_unparker.lock().is_none()); rex.start(); - while rex.processor_unparker.get().is_none() { + while rex.resident_unparker.lock().is_none() { std::thread::yield_now(); } - assert_ne!(rex.processor_unparker.get().unwrap().id(), first); + assert_ne!(rex.resident_unparker.lock().as_ref().unwrap().id(), first); rex.stop(); }); } @@ -804,9 +805,9 @@ fn test_hostr_board_ports_and_batch_retirement() { let rex = TestBoard::new(Arc::::from(rex)); rex.gfifo_push(REX3_WRMASK, 0); - assert_eq!(rex.read32(set_addr(REX3_HOSTRW0)).status, BUS_BUSY); - assert_eq!(rex.read32(go_addr(REX3_HOSTRW1)).status, BUS_BUSY); - assert_eq!(rex.read64(go_addr(REX3_HOSTRW0)).status, BUS_BUSY); + assert_eq!(rex.read32(set_addr(REX3_HOSTRW0)).status, BUS_ERR); + assert_eq!(rex.read32(go_addr(REX3_HOSTRW1)).status, BUS_ERR); + assert_eq!(rex.read64(go_addr(REX3_HOSTRW0)).status, BUS_ERR); assert_eq!(rex.gfifo.len(), 1, "busy reads must not enqueue GO"); let mut not_started = [LATCH; 2]; assert_eq!(rex.dma_read64(go_addr(REX3_HOSTRW0)).status, BUS_ERR); @@ -9662,6 +9663,46 @@ fn fill_24bpp_and_sample(rex: &Rex3, dm1: u32) -> u32 { read_pixel(rex, 0, 0) } +#[test] +fn cpu_saveable_restores_partial_command_and_palette_state() { + with_big_stack(|| { + let rex = make_rex3_unstarted(); + let ctx = unsafe { &mut *rex.context.get() }; + ctx.hostrw[0] = 0x1234_5678_9abc_def0; + ctx.host_shifter = 0xfedc_ba98_7654_3210; + ctx.hostcnt = 5; + ctx.pat_bit = 7; + ctx.zpat_bit = 11; + ctx.mid_primitive = true; + { + let mut cmap = rex.cmap0.lock(); + cmap.addr_lo = 3; + cmap.write_crs(crate::dev::ng1::cmap::CMAP_REG_PALETTE, 0x12); + cmap.write_crs(crate::dev::ng1::cmap::CMAP_REG_PALETTE, 0x34); + } + let saved = rex.save_state(); + rex.power_on(); + rex.load_state(&saved).unwrap(); + assert_eq!(rex.save_state(), saved); + { + let mut cmap = rex.cmap0.lock(); + cmap.write_crs(crate::dev::ng1::cmap::CMAP_REG_PALETTE, 0x56); + assert_eq!(cmap.palette[3], 0x563412); + } + + let mut old = saved; + for field in ["hostrw0", "host_shifter", "pat_bit", "zpat_bit", "mid_primitive"] { + old["context"].as_table_mut().unwrap().remove(field); + } + for section in ["cmap0", "cmap1"] { + for field in ["rgb_counter", "red_temp", "green_temp"] { + old[section].as_table_mut().unwrap().remove(field); + } + } + rex.load_state(&old).unwrap(); + }); +} + /// After reset, DRAWMODE1 is at IRIS's reset value, not the hardware's. /// /// The spec's Table 11 init column has PLANES=001, RGBMODE=1, FASTCLEAR=1 and diff --git a/src/dev/ng1/testdata/batch_1025_fence.txt b/src/dev/ng1/testdata/batch_1025_fence.txt index 0e5b675cb..20adb7788 100644 --- a/src/dev/ng1/testdata/batch_1025_fence.txt +++ b/src/dev/ng1/testdata/batch_1025_fence.txt @@ -1,5 +1,5 @@ # Generated synthetic REX3 recording; see README.md. -H 317fd7b43803fe6e1da1a7696a469a960a90b77eb712f7050d72d8231c4f816f +H 43645fbfb97832efd648c4905324f8496afea26ddaf5d9a5bbae54665c12ea19 W 32 0 3000f019 W 32 220 ffffff W 32 1328 1e00 @@ -1042,6 +1042,6 @@ W 32 154 10051005 W 32 950 10051005 W 32 238 250 W 32 240 400 -H e774f5523c92a435e16df9c4781555a07af44365b4589168918b451a7b064735 +H 812a21c5e3000e466b9dcaa57a0bad471c2843cf91566ec463ba449f4e976135 R 32 4 306 -H e774f5523c92a435e16df9c4781555a07af44365b4589168918b451a7b064735 +H 812a21c5e3000e466b9dcaa57a0bad471c2843cf91566ec463ba449f4e976135 diff --git a/src/dev/ng1/testdata/clip_ci8_mixed.txt b/src/dev/ng1/testdata/clip_ci8_mixed.txt index 139dea0e0..b808b8111 100644 --- a/src/dev/ng1/testdata/clip_ci8_mixed.txt +++ b/src/dev/ng1/testdata/clip_ci8_mixed.txt @@ -1,5 +1,5 @@ # Generated synthetic REX3 recording; see README.md. -H 317fd7b43803fe6e1da1a7696a469a960a90b77eb712f7050d72d8231c4f816f +H 43645fbfb97832efd648c4905324f8496afea26ddaf5d9a5bbae54665c12ea19 W 32 0 3000f019 W 32 220 ffffff W 32 1328 1e00 @@ -21,7 +21,7 @@ W 32 20c 30000 W 32 4 306 W 32 154 10281027 W 32 950 101e101e -H bad0b5555794051123f9a4266033ca3abbcb2b2c34680e2c6477511fe7a89040 +H 32d5a3963ce47d82ced9508da68f543aaa80c7abbab306a7b6288604a77713be W 32 1328 1e00 W 32 0 30007009 W 32 220 ff @@ -38,4 +38,4 @@ W 32 4 306 W 32 154 104b1023 W 32 950 1046101e R 32 4 306 -H ac8e484d20d88cebe5f118795b074084c8410fe48468ce1a95814f807e8ddf3d +H 631e37d740b33caf4a3fa611ddc4adb87526debebe4c301641de80d29149dc10 diff --git a/src/dev/ng1/testdata/dma_hostrw.txt b/src/dev/ng1/testdata/dma_hostrw.txt index 7e9783240..3bebfa3f4 100644 --- a/src/dev/ng1/testdata/dma_hostrw.txt +++ b/src/dev/ng1/testdata/dma_hostrw.txt @@ -1,5 +1,5 @@ # Generated synthetic REX3 recording; see README.md. -H 317fd7b43803fe6e1da1a7696a469a960a90b77eb712f7050d72d8231c4f816f +H 43645fbfb97832efd648c4905324f8496afea26ddaf5d9a5bbae54665c12ea19 W 32 0 3000f019 W 32 220 ffffff W 32 1328 1e00 @@ -18,14 +18,14 @@ W 32 154 101b1014 W 32 150 10141014 W 32 4 146 B a30 1 1122334455667788 -H a08365462272adb0347ab475136b2e859f5a32af4ef56ec23d40a2b8bd9d37c1 +H f95c8dbd5f64b8293acdf42547d24a773ca11167e4a410e60eb5b3c313df2e59 W 32 154 101b1014 W 32 150 10141014 W 32 4 165 b a30 1 1122334455667788 -H 6196e2e9ad86517091ab20b2f5906cc694a4e296a2e64ddd2cdb50ab35d79e4a +H ed92b8dc932aab4dce369deef4364e155072d64d32f4b6fdad45daf5729d381f W 32 154 101b1014 W 32 150 10141014 W 32 4 165 D a30 1122334455667788 -H 6196e2e9ad86517091ab20b2f5906cc694a4e296a2e64ddd2cdb50ab35d79e4a +H ed92b8dc932aab4dce369deef4364e155072d64d32f4b6fdad45daf5729d381f diff --git a/src/dev/ng1/testdata/hostrw_residual.txt b/src/dev/ng1/testdata/hostrw_residual.txt index f13173dda..ed31d4a82 100644 --- a/src/dev/ng1/testdata/hostrw_residual.txt +++ b/src/dev/ng1/testdata/hostrw_residual.txt @@ -1,5 +1,5 @@ # Generated synthetic REX3 recording; see README.md. -H 317fd7b43803fe6e1da1a7696a469a960a90b77eb712f7050d72d8231c4f816f +H 43645fbfb97832efd648c4905324f8496afea26ddaf5d9a5bbae54665c12ea19 W 32 0 3000f019 W 32 220 ffffff W 32 1328 1e00 @@ -18,12 +18,12 @@ W 32 154 10081008 W 32 150 10081008 W 32 4 46 W 32 a30 ff123456 -H cefcd2c533990b2f39b4e28c59240055a8288ba1777bed0c67fb9f109b100fef +H eab5a77cade82deac673fdab966d224db7358d75c2314caf43cc1e480822b4f5 W 32 154 10081008 W 32 150 10081008 W 32 804 365 R 32 230 ff123456 -H 956a19106bca359203773efcc394b975f88edb9c8a533bd8ff19d2878290a83c +H ad521053338cab0caa72ec4583769ed68bee97f721c3a16f52fda8276d1f978b W 32 0 30007189 W 32 220 ff W 32 154 10161014 @@ -31,4 +31,4 @@ W 32 150 10141014 W 32 4 46 W 32 a30 abcdef91 R 32 4 46 -H 33077e74e5278c889540d63a4165d083d070ca065f2832ae1041c3f1c6444e0b +H 5c4c76496e1f21a9bf6789337217ed1a7419adc3e9cfe6c93685ab05f933d608 diff --git a/src/dev/ng1/testdata/long_spans.txt b/src/dev/ng1/testdata/long_spans.txt index 91ab26bf0..656f41729 100644 --- a/src/dev/ng1/testdata/long_spans.txt +++ b/src/dev/ng1/testdata/long_spans.txt @@ -1,5 +1,5 @@ # Generated synthetic REX3 recording; see README.md. -H 317fd7b43803fe6e1da1a7696a469a960a90b77eb712f7050d72d8231c4f816f +H 43645fbfb97832efd648c4905324f8496afea26ddaf5d9a5bbae54665c12ea19 W 32 0 3000f019 W 32 220 ffffff W 32 1328 1e00 @@ -74,6 +74,6 @@ W 32 154 14b91046 W 32 950 100a1046 W 32 154 14b91047 W 32 950 100a1047 -H 977b70b60ae13a8c42c12e479cf5451c429437aee98a1d99fa50a6e68164f489 +H a05c36d8c7b74d91d1899bae1de1794cc93d582cc12adf566388fe81d35c1457 R 32 4 102 -H 977b70b60ae13a8c42c12e479cf5451c429437aee98a1d99fa50a6e68164f489 +H a05c36d8c7b74d91d1899bae1de1794cc93d582cc12adf566388fe81d35c1457 diff --git a/src/dev/ng1/testdata/manifest.toml b/src/dev/ng1/testdata/manifest.toml index 104d9496e..af226b3dd 100644 --- a/src/dev/ng1/testdata/manifest.toml +++ b/src/dev/ng1/testdata/manifest.toml @@ -40,7 +40,7 @@ compiled_shapes = 1 gpu_handled = 4 gpu_interpreter_fallback = 0 display_fences = 0 -blake3 = "e067045becc111d19257f26640073e640a4743915d4184f88be11e0831f6df74" +blake3 = "ac33152e2796df78595fa526ef7dbfdfa527301e1d64d403a079fb14b50364d4" [case.gpu_fallback_reasons] @@ -94,7 +94,7 @@ compiled_shapes = 1 gpu_handled = 32 gpu_interpreter_fallback = 0 display_fences = 0 -blake3 = "d777e684bee8d0a0008c8de606ea8b8af3ba3ec1f887ca044523a3f98475b967" +blake3 = "e004c7ae1dae152ecaa65c9755f6372c0aa6f85240cbe90182b28947e6d11e15" [case.gpu_fallback_reasons] @@ -140,7 +140,7 @@ compiled_shapes = 2 gpu_handled = 88 gpu_interpreter_fallback = 0 display_fences = 0 -blake3 = "54270e6b74b904213f095f0b2ad159f9eedbca18d64655abf1faccd59efca0c2" +blake3 = "6132da489ef494aa5460c37ded69913a0824fd6a8471e216c23d2218dd5b87e9" [case.gpu_fallback_reasons] @@ -200,7 +200,7 @@ gpu_interpreter_fallback = 3 display_fences = 0 residual_shifter = "9100000000000000" residual_count = 1 -blake3 = "1ccbc1c226fbceb26712563c7f49077d59bc47688a482afd5334d02f6313d871" +blake3 = "9b3e0cd162c37fce1163736b95ae11696d3d77fccc82a13a6036fa6dc42b69d5" [case.gpu_fallback_reasons] opcode = 1 @@ -255,7 +255,7 @@ compiled_shapes = 3 gpu_handled = 1 gpu_interpreter_fallback = 3 display_fences = 0 -blake3 = "c6f05d44d88aa4c5e7eaf0f23b44209e36c4a2977d6b7a2fa8a567ffa8415896" +blake3 = "a17d938558b8607a45a944de04795f3533a6bf45880e2d716982b4f713cde3e8" [case.gpu_fallback_reasons] opcode = 2 @@ -308,7 +308,7 @@ compiled_shapes = 3 gpu_handled = 2 gpu_interpreter_fallback = 2 display_fences = 0 -blake3 = "0758416f6b7135af66c2a601f1ffd565e583450be1b3cbec9eb4de4c8b1aaa1f" +blake3 = "7532d5f32592aca180eaefd7ba0605eab894ff9e6cece0b52d353a26b666468d" [case.gpu_fallback_reasons] "clip mode" = 1 @@ -366,7 +366,7 @@ compiled_shapes = 1 gpu_handled = 1025 gpu_interpreter_fallback = 0 display_fences = 1 -blake3 = "270091d0548fcbc7fec3aae4862cc150e40cf2e8992885bf0044295d38a62de1" +blake3 = "b18345343677e0b8693462b2320dd2f8998624f53ae374389272553e599c790a" [case.gpu_fallback_reasons] diff --git a/src/dev/ng1/testdata/overlap_windows.txt b/src/dev/ng1/testdata/overlap_windows.txt index 78cc14c23..7116a7deb 100644 --- a/src/dev/ng1/testdata/overlap_windows.txt +++ b/src/dev/ng1/testdata/overlap_windows.txt @@ -1,5 +1,5 @@ # Generated synthetic REX3 recording; see README.md. -H 317fd7b43803fe6e1da1a7696a469a960a90b77eb712f7050d72d8231c4f816f +H 43645fbfb97832efd648c4905324f8496afea26ddaf5d9a5bbae54665c12ea19 W 32 0 3000f019 W 32 220 ffffff W 32 1328 1e00 @@ -30,7 +30,7 @@ W 32 20c 1e000 W 32 4 306 W 32 154 103c1064 W 32 950 10141055 -H d48160f0f6dcbcfe58917e1076d9dd3377e293d6791318e71daa9d3ec013acc4 +H af00823c8b7e6c1a036ea9f437a379a8ad304d5ed5ea5e2e816c48567e3f0e86 R 32 1338 3 R 32 4 306 -H d48160f0f6dcbcfe58917e1076d9dd3377e293d6791318e71daa9d3ec013acc4 +H af00823c8b7e6c1a036ea9f437a379a8ad304d5ed5ea5e2e816c48567e3f0e86 diff --git a/src/dev/ng1/testdata/tiny_tiles.txt b/src/dev/ng1/testdata/tiny_tiles.txt index 2b64b5764..868443bbf 100644 --- a/src/dev/ng1/testdata/tiny_tiles.txt +++ b/src/dev/ng1/testdata/tiny_tiles.txt @@ -1,5 +1,5 @@ # Generated synthetic REX3 recording; see README.md. -H 317fd7b43803fe6e1da1a7696a469a960a90b77eb712f7050d72d8231c4f816f +H 43645fbfb97832efd648c4905324f8496afea26ddaf5d9a5bbae54665c12ea19 W 32 0 3000f019 W 32 220 ffffff W 32 1328 1e00 @@ -194,6 +194,6 @@ W 32 154 107210fa W 32 950 107210fa W 32 154 107310fa W 32 950 107310fa -H bb516dd21429874c8ee9b580891f84f6af1d903deb0d8595184e62730d462277 +H 7d8dfae080b22dc9f8e271fb95f09fc7ae94b75ad4bdd3e8e3c093cf418fe715 R 32 4 306 -H bb516dd21429874c8ee9b580891f84f6af1d903deb0d8595184e62730d462277 +H 7d8dfae080b22dc9f8e271fb95f09fc7ae94b75ad4bdd3e8e3c093cf418fe715 diff --git a/src/dev/ng1/wgpu/direct.rs b/src/dev/ng1/wgpu/direct.rs index 0e5f5384a..90fd3ec0e 100644 --- a/src/dev/ng1/wgpu/direct.rs +++ b/src/dev/ng1/wgpu/direct.rs @@ -98,14 +98,20 @@ pub(crate) struct CaptureCount { } impl CaptureCount { - pub(crate) fn attempt(&mut self, bytes: u64) { - self.attempts += 1; - self.attempted_bytes += bytes; + pub(crate) fn attempt(&mut self, bytes: u64) -> Result<(), ResidentError> { + let attempts = self.attempts.checked_add(1).ok_or(ResidentError::InvalidState("composed capture counter exhausted"))?; + let attempted_bytes = self.attempted_bytes.checked_add(bytes).ok_or(ResidentError::InvalidState("composed capture bytes exhausted"))?; + self.attempts = attempts; + self.attempted_bytes = attempted_bytes; + Ok(()) } - pub(crate) fn success(&mut self, bytes: u64) { - self.successes += 1; - self.successful_bytes += bytes; + pub(crate) fn success(&mut self, bytes: u64) -> Result<(), ResidentError> { + let successes = self.successes.checked_add(1).ok_or(ResidentError::InvalidState("composed capture counter exhausted"))?; + let successful_bytes = self.successful_bytes.checked_add(bytes).ok_or(ResidentError::InvalidState("composed capture bytes exhausted"))?; + self.successes = successes; + self.successful_bytes = successful_bytes; + Ok(()) } } @@ -346,7 +352,7 @@ impl Ng1ResidentScanout { return Err(ResidentError::InvalidState("resident capture output is too small")); } let bytes = pixels as u64 * 4; - self.traffic.composed_readbacks.attempt(bytes); + self.traffic.composed_readbacks.attempt(bytes)?; let fault = self.capture_fault.take(); if matches!(fault, Some(CaptureFault::BeforeCapture)) { return Err(ResidentError::Failed("injected composed capture failure before capture".into())); @@ -359,7 +365,7 @@ impl Ng1ResidentScanout { } self.scanout.gpu().check().map_err(scanout_error)?; out[..pixels].copy_from_slice(&self.capture_scratch); - self.traffic.composed_readbacks.success(bytes); + self.traffic.composed_readbacks.success(bytes)?; Ok(()) } diff --git a/src/dev/ng1/wgpu/direct_tests.rs b/src/dev/ng1/wgpu/direct_tests.rs index cc74f333c..21ce36e25 100644 --- a/src/dev/ng1/wgpu/direct_tests.rs +++ b/src/dev/ng1/wgpu/direct_tests.rs @@ -37,7 +37,7 @@ use crate::dev::ng1::vc2::{ use crate::dev::ng1::xmap9::Xmap9; use crate::disp::{Ng1DisplaySource, Rex3Screen, FB_STRIDE}; use crate::gfx_display::GfxDisplay; -use crate::traits::{BusDevice, Device, BUS_BUSY, BUS_OK}; +use crate::traits::{BusDevice, Device, Saveable, BUS_BUSY, BUS_OK}; use crate::wgpu::{Gpu, GpuOptions, Staging}; const WIDTH: usize = 64; @@ -1076,7 +1076,7 @@ fn direct_ng1_scanout_gpu_uses_the_board_binding_seam() { .unwrap_or_else(|panic| std::panic::resume_unwind(panic)); } -fn configure_board_display(rex: &Rex3) { +pub(super) fn configure_board_display(rex: &Rex3) { let display = Display::new(); *rex.vc2.lock() = display.vc2.into_inner(); *rex.xmap0.lock() = display.xmap.into_inner(); @@ -1140,11 +1140,11 @@ fn software_display_frame(display: &mut Display, rgb: &[u32], aux: &[u32]) -> Ve pixels } -struct QueuePresenter { - gpu: Option>, - accepts: bool, - outputs: Arc>>, - resizes: Arc>>, +pub(super) struct QueuePresenter { + pub(super) gpu: Option>, + pub(super) accepts: bool, + pub(super) outputs: Arc>>, + pub(super) resizes: Arc>>, } struct CompatibilityPresenter { @@ -2387,3 +2387,118 @@ fn resident_frame_gpu_headless_requests_and_vnc_capture() { .join() .unwrap_or_else(|panic| std::panic::resume_unwind(panic)); } + +#[test] +#[ignore = "requires the requested conformant hardware GPU backend"] +fn resident_frame_gpu_restore_invalidates_capture_and_rebinds_display() { + std::thread::Builder::new().name("resident-frame-restore".into()).stack_size(64 * 1024 * 1024) + .spawn(|| { + use std::time::Duration; + + let gpu = required_gpu(); + let (rgb, aux) = seed_planes(); + for mode in [ResidentDisplayMode::DirectWgpu, ResidentDisplayMode::ReadbackCompatibility] { + let mut expected_rgb = rgb.clone(); + let mut expected_aux = aux.clone(); + let mut rex = Box::new(Rex3::new(crate::disp::StatusSource::detached())); + rex.fb_rgb.get_mut().copy_from_slice(&rgb); + rex.fb_aux.get_mut().copy_from_slice(&aux); + rex.fb_rgb.get_mut().isolate().unwrap(); + rex.fb_aux.get_mut().isolate().unwrap(); + configure_board_display(&rex); + rex.activate_resident_gpu(Arc::clone(&gpu)).unwrap(); + rex.fb_rgb.get_mut().reset_access_counts(); + rex.fb_aux.get_mut().reset_access_counts(); + let rgb_guard = unsafe { (&*rex.fb_rgb.get()).protect() }.unwrap(); + let aux_guard = unsafe { (&*rex.fb_aux.get()).protect() }.unwrap(); + let board = SharedTestBoard(Arc::from(rex)); + let display: Arc = board.0.clone(); + let ps2 = Arc::new(crate::dev::ps2::Ps2Controller::new(None)); + let frontend = crate::hostinput::HostInput::new(Some(display), ps2); + board.0.start(); + board.0.start_resident_display(Arc::clone(&gpu), mode).unwrap(); + let mut first = fill_context(); + first.drawmode0.0 |= 1 << 5; + enqueue_fill(&board.0, &first); + draw_reference(first, &mut expected_rgb, &mut expected_aux); + let mut captured = Vec::new(); + crate::gfx_display::fresh_frame(&*board.0, &mut captured, Duration::from_secs(10)) + .expect("capture initial drawn frame"); + assert_eq!(frontend.screen_size(), (WIDTH, HEIGHT)); + assert!(board.0.hw_cursor().is_some()); + assert_pixels_eq("snapshot display", &captured, &software_frame(&board.0, &expected_rgb, &expected_aux)); + let original_capture = captured.clone(); + board.0.stop(); + let registers = board.0.save_state(); + let (saved_rgb, saved_aux) = board.0.snapshot_framebuffers_inmem(); + assert_eq!((&saved_rgb, &saved_aux), (&expected_rgb, &expected_aux)); + let saved_epoch = board.0.resident_status().unwrap().accepted.epoch; + + board.0.start(); + board.0.start_resident_display(Arc::clone(&gpu), mode).unwrap(); + let mut replacement = fill_context(); + replacement.drawmode0.0 |= 1 << 5; + replacement.colorred = 0xee << 11; + replacement.colorgrn = 0x11 << 11; + replacement.colorblue = 0x33 << 11; + enqueue_fill(&board.0, &replacement); + draw_reference(replacement, &mut expected_rgb, &mut expected_aux); + let (_, _, changed_seq) = + crate::gfx_display::fresh_frame(&*board.0, &mut captured, Duration::from_secs(10)) + .expect("capture replacement frame"); + assert_pixels_eq("replacement display", &captured, &software_frame(&board.0, &expected_rgb, &expected_aux)); + assert_ne!(captured, original_capture); + board.0.stop(); + + let mut source = &*board.0; + let mut stale = Ng1Scanout::new_resident(&gpu, &source).unwrap(); + let old_stamp = { + let mut screen = board.0.head.screen().lock(); + stale.compose(&screen.display_state(), &mut source).unwrap().stamp() + }; + assert!(board.0.frame_available()); + board.0.load_state(®isters).unwrap(); + assert!(!board.0.frame_available()); + assert_eq!(frontend.screen_size(), (0, 0)); + assert!(board.0.hw_cursor().is_none()); + assert!(crate::gfx_display::copy_frame(&*board.0, &mut captured).is_none()); + board.0.request_frame(); + assert!(board.0.frame_pending()); + board.0.restore_framebuffers_inmem(&saved_rgb, &saved_aux); + assert!(board.0.frame_pending(), "restore must retain pending requests"); + assert!(crate::gfx_display::copy_frame(&*board.0, &mut captured).is_none()); + let new_epoch = board.0.resident_status().unwrap().accepted.epoch; + assert_ne!(new_epoch, saved_epoch); + assert!(matches!( + stale.capture(old_stamp, &mut source, &mut captured), + Err(ResidentError::InvalidState("resident composed frame belongs to an old epoch")) + )); + { + let mut screen = board.0.head.screen().lock(); + assert!(matches!( + stale.compose(&screen.display_state(), &mut source), + Err(ResidentError::InvalidState("resident scanout bindings belong to an old or different epoch")) + )); + } + + board.0.start(); + board.0.start_resident_display(Arc::clone(&gpu), mode).unwrap(); + let deadline = Instant::now() + Duration::from_secs(10); + while board.0.frame_pending() && Instant::now() < deadline { + std::thread::sleep(Duration::from_millis(2)); + } + assert!(!board.0.frame_pending(), "new display must complete the pending capture"); + let (_, _, restored_seq) = crate::gfx_display::copy_frame(&*board.0, &mut captured) + .expect("capture restored frame"); + assert!(restored_seq > changed_seq); + assert_eq!(frontend.screen_size(), (WIDTH, HEIGHT)); + assert!(board.0.hw_cursor().is_some()); + assert_pixels_eq("restored display", &captured, &original_capture); + assert_pixels_eq("restored software oracle", &captured, &software_frame(&board.0, &saved_rgb, &saved_aux)); + board.0.stop(); + assert_no_cpu_plane_access(&board.0); + drop(aux_guard); + drop(rgb_guard); + } + }).unwrap().join().unwrap_or_else(|panic| std::panic::resume_unwind(panic)); +} diff --git a/src/dev/ng1/wgpu/host.rs b/src/dev/ng1/wgpu/host.rs index 84bea28b0..465dd949c 100644 --- a/src/dev/ng1/wgpu/host.rs +++ b/src/dev/ng1/wgpu/host.rs @@ -8,6 +8,7 @@ use crate::dev::ng1::rex3_generic::{calculate_fb_address, DynMode, Mode}; use crate::dev::ng1::rex3_shape; use super::shade::ShadeDda; +use super::telemetry::Coverage; #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub(super) enum HostKind { @@ -41,6 +42,7 @@ pub(super) struct PreparedHostFill { completion: HostCompletion, payload_qwords: usize, fast_clear: bool, + coverage: Option, } impl PreparedHostFill { @@ -56,6 +58,10 @@ impl PreparedHostFill { self.fast_clear } + pub(super) fn coverage(&self) -> Option { + self.coverage + } + pub(super) fn finish(self, context: &mut Rex3Context) { context.xstart = self.completion.xstart; context.ystart = self.completion.ystart; @@ -180,14 +186,14 @@ pub(super) fn prepare_host_fill(context: &Rex3Context) -> Result 1 && stream.cursor < stream.length { let before = stream.cursor; @@ -198,14 +204,14 @@ pub(super) fn prepare_host_fill(context: &Rex3Context) -> Result Result, ) { match kind { - HostKind::Block => run_block(context, mode, walk, stream, write_address_seen), - HostKind::Span => run_span(context, mode, walk, stream, write_address_seen), + HostKind::Block => run_block(context, mode, walk, stream, coverage), + HostKind::Span => run_span(context, mode, walk, stream, coverage), } } @@ -283,7 +290,7 @@ fn visit_pixel( y: i32, walk: &WalkState, stream: &mut HostStream, - write_address_seen: &mut bool, + coverage: &mut Option, ) { let pattern = pattern_result(context, mode, walk); if !pattern.draws { @@ -292,7 +299,12 @@ fn visit_pixel( if !pattern.background { stream.consume(context); } - *write_address_seen |= calculate_fb_address::(mode, x, y, context).is_some(); + if let Some(address) = calculate_fb_address::(mode, x, y, context) { + match coverage { + Some(bounds) => bounds.include_address(address), + None => *coverage = Some(Coverage::from_address(address)), + } + } } fn advance_pixel(context: &Rex3Context, mode: &DynMode, walk: &mut WalkState) { @@ -323,7 +335,7 @@ fn run_block( mode: &DynMode, walk: &mut WalkState, stream: &mut HostStream, - write_address_seen: &mut bool, + coverage: &mut Option, ) { let stop_on_y = mode.stopony() != 0; let color_host = mode.colorhost() != 0; @@ -360,7 +372,7 @@ fn run_block( walk.xstart > context.xend }; if !(skip_first || x_end_reached && skip_last || lr_skip) { - visit_pixel(context, mode, x, y, walk, stream, write_address_seen); + visit_pixel(context, mode, x, y, walk, stream, coverage); } advance_pixel(context, mode, walk); @@ -414,7 +426,7 @@ fn run_span( mode: &DynMode, walk: &mut WalkState, stream: &mut HostStream, - write_address_seen: &mut bool, + coverage: &mut Option, ) { if mode.lronly() != 0 && context.bresoctinc1.octant() & OCTANT_XDEC != 0 { return; @@ -438,7 +450,7 @@ fn run_span( walk.xstart += 1 << 11; let x_end_reached = walk.xstart > context.xend; if !(first && skip_first || x_end_reached && skip_last) { - visit_pixel(context, mode, x, y, walk, stream, write_address_seen); + visit_pixel(context, mode, x, y, walk, stream, coverage); } advance_pixel(context, mode, walk); if x_end_reached { diff --git a/src/dev/ng1/wgpu/line.rs b/src/dev/ng1/wgpu/line.rs index fa32c90de..7d9e6f530 100644 --- a/src/dev/ng1/wgpu/line.rs +++ b/src/dev/ng1/wgpu/line.rs @@ -14,6 +14,7 @@ use crate::dev::ng1::rex3_shape; use super::host::HostStream; use super::shade::ShadeDda; +use super::telemetry::Coverage; const SKIPPED_PIXEL: u32 = u32::MAX; const BACKGROUND_PIXEL: u32 = 1 << 31; @@ -346,6 +347,7 @@ pub(super) struct PreparedHostLine { completion: HostLineCompletion, payload_qwords: usize, fast_clear: bool, + coverage: Option, } impl PreparedHostLine { @@ -361,6 +363,10 @@ impl PreparedHostLine { self.fast_clear } + pub(super) fn coverage(&self) -> Option { + self.coverage + } + pub(super) fn finish(self, context: &mut Rex3Context) { context.xstart = self.completion.xstart; context.ystart = self.completion.ystart; @@ -393,6 +399,7 @@ pub(super) fn prepare_host_line( let mut simulated = *context; let mut stream = HostStream::from_context(context); let mut line = None; + let mut coverage: Option = None; let mut previous_cursor = None; loop { let prepared = prepare_line_for(&simulated, LineOperation::Host)?; @@ -410,13 +417,18 @@ pub(super) fn prepare_host_line( if !background { stream.consume(context); } - invocation_writes |= calculate_fb_address::( + if let Some(address) = calculate_fb_address::( &mode, pixel.x, pixel.y, &simulated, - ) - .is_some(); + ) { + invocation_writes = true; + match &mut coverage { + Some(bounds) => bounds.include_address(address), + None => coverage = Some(Coverage::from_address(address)), + } + } } } let last = pixel.index + 1 == geometry.pixel_count; @@ -468,6 +480,7 @@ pub(super) fn prepare_host_line( }, payload_qwords: context.host_len as usize, fast_clear, + coverage, }) } diff --git a/src/dev/ng1/wgpu/mod.rs b/src/dev/ng1/wgpu/mod.rs index 173ef916a..ef38b95d6 100644 --- a/src/dev/ng1/wgpu/mod.rs +++ b/src/dev/ng1/wgpu/mod.rs @@ -43,6 +43,9 @@ pub(crate) mod shade; #[cfg(test)] pub(crate) mod resident; +#[cfg(test)] +pub(crate) mod telemetry; + #[cfg(test)] pub(crate) mod direct; diff --git a/src/dev/ng1/wgpu/resident.rs b/src/dev/ng1/wgpu/resident.rs index aa2b558db..2b3485480 100644 --- a/src/dev/ng1/wgpu/resident.rs +++ b/src/dev/ng1/wgpu/resident.rs @@ -7,7 +7,7 @@ use std::num::NonZeroUsize; use std::sync::atomic::{AtomicU64, Ordering}; use std::sync::{Arc, Weak}; -use std::time::Duration; +use std::time::{Duration, Instant}; use parking_lot::Mutex; @@ -48,14 +48,18 @@ use super::command::{ use super::copy::prepare_copy_fill; use super::host::prepare_host_fill; use super::line::{prepare_copy_line, prepare_host_line, prepare_line}; +use super::telemetry::{duration_ns, Coverage, DrawSample, FlushReason, Workload}; const STRIDE: u32 = 2048; const HEIGHT: u32 = 1024; -const PLANE_WORDS: usize = (STRIDE * HEIGHT) as usize; +pub(crate) const PLANE_WORDS: usize = (STRIDE * HEIGHT) as usize; const PLANE_BYTES: u64 = PLANE_WORDS as u64 * size_of::() as u64; const READBACK_BYTES: u64 = 2 * PLANE_BYTES; const MAX_SLOTS: u32 = 1024; const MAX_SETS: usize = 3; +/// Flush at the next GFIFO command boundary after four milliseconds. A single +/// long command or a bounded GPU-set wait cannot be preempted by this policy. +pub(crate) const MAX_BATCH_AGE: Duration = Duration::from_millis(4); const READBACK_CALLBACK_TIMEOUT: Duration = Duration::from_secs(30); const FLAT: Fragment = Fragment::new("resident_flat.wgsl", include_str!("shaders/resident_flat.wgsl")); @@ -117,6 +121,10 @@ fn gpu_error(error: String) -> ResidentError { #[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)] pub(crate) struct Epoch(u64); +impl Epoch { + pub(crate) fn value(self) -> u64 { self.0 } +} + #[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)] pub(crate) struct Sequence(u64); @@ -200,6 +208,22 @@ impl Observation { pub(crate) fn aux(&self) -> Option<&[u8]> { self.aux.as_deref() } pub(crate) fn rgb_generation(&self) -> Option { self.rgb_generation } pub(crate) fn aux_generation(&self) -> Option { self.aux_generation } + + #[cfg(test)] + pub(crate) fn for_test( + reason: ObservationReason, + planes: PlaneMask, + rgb: Option>, + aux: Option>, + ) -> Self { + let position = ResidentStatus::for_test().accepted; + Self { + position, reason, planes, + rgb_generation: rgb.as_ref().map(|_| position), + aux_generation: aux.as_ref().map(|_| position), + rgb, aux, + } + } } /// Counts observations and their selected full-plane transfers separately. @@ -289,7 +313,7 @@ pub(crate) struct ResidentStatus { pub(crate) aux: PlaneStatus, pub(crate) initialization_uploads: u32, pub(crate) initialization_bytes: u64, - /// Successfully queued restore-prefix uploads, not GPU completions. + /// Successfully completed restore-prefix uploads. pub(crate) restore_uploads: u64, pub(crate) restore_bytes: u64, /// Cumulative across resets and restores for this backend instance. @@ -299,6 +323,7 @@ pub(crate) struct ResidentStatus { /// Accepted HostW GOs and immutable qwords staged for them. pub(crate) host_write_transfers: u64, pub(crate) host_write_qwords: u64, + pub(crate) workload: Workload, } impl ResidentStatus { @@ -324,6 +349,7 @@ impl ResidentStatus { raw_plane_map_attempts: 0, host_write_transfers: 0, host_write_qwords: 0, + workload: Workload::default(), } } @@ -339,6 +365,7 @@ impl ResidentStatus { next.raw_plane_map_attempts = previous.raw_plane_map_attempts; next.host_write_transfers = previous.host_write_transfers; next.host_write_qwords = previous.host_write_qwords; + next.workload = previous.workload.clone(); Ok(next) } @@ -367,6 +394,7 @@ impl ResidentStatus { raw_plane_map_attempts: self.raw_plane_map_attempts, host_write_transfers: self.host_write_transfers, host_write_qwords: self.host_write_qwords, + workload: self.workload.clone(), }) } @@ -541,6 +569,14 @@ pub(crate) trait ResidentBackend: Send { Err(ResidentError::Unsupported("submit")) } + fn submit_for(&mut self, _reason: FlushReason) -> Result<(), ResidentError> { + self.submit() + } + + fn flush_if_due(&mut self, _now: Instant) -> Result<(), ResidentError> { Ok(()) } + + fn host_read_result(&mut self, _qwords: u64, _generation_ns: u64) -> Result<(), ResidentError> { Ok(()) } + /// Submit the accepted prefix and return the selected full planes at its /// stream cut. This does not install or acknowledge a CPU shadow. fn observe( @@ -573,7 +609,7 @@ pub(crate) trait ResidentBackend: Send { } /// With GFIFO and scanout quiescent, discard the old stream and seed new - /// planes from the board's complete CPU-shadow allocations. + /// planes from the supplied complete data, without acknowledging shadows. fn reset(&mut self, _rgb: &[u32], _aux: &[u32]) -> Result<(), ResidentError> { Err(ResidentError::Unsupported("reset")) } @@ -651,7 +687,7 @@ impl ResidentRaster { // A rejected GO must not strand an already accepted prefix. // A device failure while submitting that prefix takes priority. let error = if matches!(&error, ResidentError::Unsupported(_)) { - self.backend.submit().err().unwrap_or(error) + self.backend.submit_for(FlushReason::Unsupported).err().unwrap_or(error) } else { error }; @@ -720,7 +756,9 @@ impl ResidentRaster { } else { self.host_read.words.resize(PLANE_WORDS, 0); } + let initial_cursor = context.host_cursor; self.enqueue_backend(context)?; + let generation_start = Instant::now(); let mode = rex3_shape::unpack( context.drawmode0.0, context.drawmode1.0, @@ -742,6 +780,10 @@ impl ResidentRaster { } } } + let produced = context.host_cursor.checked_sub(initial_cursor) + .ok_or(ResidentError::InvalidState("HostR cursor decreased"))?; + let generation_ns = duration_ns(generation_start.elapsed()).map_err(ResidentError::InvalidState)?; + self.run(|backend| backend.host_read_result(u64::from(produced), generation_ns))?; context.host_len = 1; context.host_cursor = 0; Ok(()) @@ -756,7 +798,15 @@ impl ResidentRaster { } pub(crate) fn submit(&mut self) -> Result<(), ResidentError> { - self.run(|backend| backend.submit()) + self.submit_for(FlushReason::Explicit) + } + + pub(crate) fn submit_for(&mut self, reason: FlushReason) -> Result<(), ResidentError> { + self.run(|backend| backend.submit_for(reason)) + } + + pub(crate) fn flush_if_due(&mut self, now: Instant) -> Result<(), ResidentError> { + self.run(|backend| backend.flush_if_due(now)) } pub(crate) fn poll_device_health(&mut self) -> Result<(), ResidentError> { @@ -808,6 +858,7 @@ impl ResidentRaster { match self.backend.reset(rgb, aux) { Ok(()) => { self.fault = None; + self.host_read = HostReadSnapshot::default(); Ok(()) } Err(error) => { @@ -818,7 +869,9 @@ impl ResidentRaster { } pub(crate) fn restore(&mut self, rgb: &[u32], aux: &[u32]) -> Result<(), ResidentError> { - self.run(|backend| backend.restore(rgb, aux)) + self.run(|backend| backend.restore(rgb, aux))?; + self.host_read = HostReadSnapshot::default(); + Ok(()) } pub(in crate::dev::ng1) fn scanout_bindings( @@ -866,6 +919,8 @@ pub(crate) struct ResidentGpu { aux_shadow: usize, max_slots: u32, max_sets: NonZeroUsize, + batch_started: Option, + current_coverage: Option, } struct RasterSet { @@ -880,6 +935,19 @@ impl ResidentGpu { Self::activate_with_limits(gpu, rgb, aux, MAX_SLOTS, NonZeroUsize::new(MAX_SETS).unwrap(), None) } + /// Profiling deliberately maps 64 bytes and waits for every nonempty + /// batch. It is never inherited from the ordinary batch environment. + pub(crate) fn enable_profiling(&mut self) -> Result<(), ResidentError> { + let status = self.progress.lock(); + if status.accepted != status.submitted || status.workload.profiling { + return Err(ResidentError::InvalidState("profiling requires an idle unprofiled raster")); + } + drop(status); + self.batch.enable_profiling().map_err(ResidentError::Failed)?; + self.progress.lock().workload.profiling = true; + Ok(()) + } + fn activate_with_limits( gpu: Arc, rgb: &[u32], @@ -1034,16 +1102,13 @@ impl ResidentGpu { RasterSet { payload, group } }) .collect(); - let mut batch = Batch::new_bounded( + let batch = Batch::new_bounded_unprofiled( &gpu, &pipelines, TileGrid::new(REX3_SCREEN_WIDTH as u32, REX3_SCREEN_HEIGHT as u32), max_slots, max_sets, ); - // The shared optional batch timer maps and waits at every flush. - // A resident submission must remain a submission-only boundary. - batch.timing = None; gpu.instance.poll_all(false); gpu.check().map_err(gpu_error)?; @@ -1069,6 +1134,8 @@ impl ResidentGpu { rgb_shadow: rgb.as_ptr() as usize, aux_shadow: aux.as_ptr() as usize, max_slots, max_sets, + batch_started: None, + current_coverage: None, }) } @@ -1133,7 +1200,11 @@ impl ResidentGpu { }; // This is the complete accepted prefix at entry. Later drawing cannot // be enqueued through this exclusively borrowed backend until it ends. - self.submit()?; + self.submit_for(if reason == ObservationReason::HostRead { + FlushReason::HostRead + } else { + FlushReason::Observation + })?; let (rgb, aux) = readback(self, planes)?; let valid = |bytes: &Option>, selected: bool| match bytes { Some(bytes) => selected && bytes.len() == PLANE_BYTES as usize, @@ -1188,10 +1259,12 @@ impl ResidentGpu { if line_shape { let prepared = prepare_copy_line(context).map_err(ResidentError::Unsupported)?; let line = prepared.line(); + self.current_coverage = line.and_then(|line| + Coverage::new(line.min_x, line.min_y, line.max_x, line.max_y)); if line.is_some() && prepared.payload().len() > MAX_PAYLOAD_WORDS { return Err(ResidentError::Unsupported("copy line payload length")); } - self.submit()?; + self.submit_for(FlushReason::CopyHazard)?; let position = self.progress.lock().next_position()?; if let Some(line) = line { let slot = RexStateSlot::for_copy(context); @@ -1222,12 +1295,14 @@ impl ResidentGpu { } self.progress.lock().accept(position, line.map(|line| line.plane)); prepared.finish(context); - return self.submit(); + return self.submit_for(FlushReason::CopyHazard); } let prepared = prepare_copy_fill(context).map_err(ResidentError::Unsupported)?; let fill = prepared.fill(); - self.submit()?; + self.current_coverage = fill.and_then(|fill| + Coverage::new(fill.min_x, fill.min_y, fill.max_x, fill.max_y)); + self.submit_for(FlushReason::CopyHazard)?; let position = self.progress.lock().next_position()?; if let Some(fill) = fill { let slot = RexStateSlot::for_copy(context); @@ -1251,13 +1326,15 @@ impl ResidentGpu { } self.progress.lock().accept(position, fill.map(|fill| fill.plane)); prepared.finish(context); - self.submit() + self.submit_for(FlushReason::CopyHazard) } fn enqueue_line(&mut self, context: &mut Rex3Context) -> Result<(), ResidentError> { let command = prepare_line(context).map_err(ResidentError::Unsupported)?; let position = self.progress.lock().next_position()?; let line = command.line(); + self.current_coverage = line.and_then(|line| + Coverage::new(line.min_x, line.min_y, line.max_x, line.max_y)); if let Some(line) = line { if command.payload().len() > MAX_PAYLOAD_WORDS { return Err(ResidentError::Unsupported("line payload length")); @@ -1268,7 +1345,7 @@ impl ResidentGpu { let payload = match self.payload.append_iter(command.payload()) { Ok(payload) => payload, Err(Full) if !self.batch.is_empty() => { - self.submit()?; + self.submit_for(FlushReason::Capacity)?; continue; } Err(Full) => { @@ -1299,7 +1376,7 @@ impl ResidentGpu { } Err(Full) if !self.batch.is_empty() => { self.payload.rollback(checkpoint); - self.submit()?; + self.submit_for(FlushReason::Capacity)?; } Err(Full) => { self.payload.rollback(checkpoint); @@ -1322,7 +1399,7 @@ impl ResidentGpu { make_command: impl Fn(u32, PayloadRef) -> RexCommand, ) -> Result<(), ResidentError> { if !self.batch.is_empty() && !self.serial_host_batch { - self.submit()?; + self.submit_for(FlushReason::HostTransition)?; } loop { let checkpoint = self.payload.checkpoint(); @@ -1336,7 +1413,7 @@ impl ResidentGpu { })) { Ok(payload) => payload, Err(Full) if !self.batch.is_empty() => { - self.submit()?; + self.submit_for(FlushReason::Capacity)?; continue; } Err(Full) => { @@ -1359,7 +1436,7 @@ impl ResidentGpu { } Err(Full) if !self.batch.is_empty() => { self.payload.rollback(checkpoint); - self.submit()?; + self.submit_for(FlushReason::Capacity)?; } Err(Full) => { self.payload.rollback(checkpoint); @@ -1388,6 +1465,7 @@ impl ResidentGpu { status.next_position()? }; let line = prepared.line(); + self.current_coverage = line.and_then(|_| prepared.coverage()); if let Some(line) = line { let fast_clear = prepared.fast_clear(); let payload_words = prepared @@ -1452,6 +1530,7 @@ impl ResidentGpu { status.next_position()? }; let fill = prepared.fill(); + self.current_coverage = fill.and_then(|_| prepared.coverage()); if let Some(fill) = fill { let fast_clear = prepared.fast_clear(); let payload_words = prepared @@ -1504,7 +1583,7 @@ impl ResidentGpu { let payload = match self.payload.append(words) { Ok(payload) => payload, Err(Full) if !self.batch.is_empty() => { - self.submit()?; + self.submit_for(FlushReason::Capacity)?; continue; } Err(Full) => { @@ -1536,7 +1615,7 @@ impl ResidentGpu { } Err(Full) if !self.batch.is_empty() => { self.payload.rollback(checkpoint); - self.submit()?; + self.submit_for(FlushReason::Capacity)?; } Err(Full) => { self.payload.rollback(checkpoint); @@ -1545,17 +1624,7 @@ impl ResidentGpu { } } } -} - -impl ResidentBackend for ResidentGpu { - fn gpu(&self) -> Option<&Arc> { Some(&self.gpu) } - - fn poll_device_health(&mut self) -> Result<(), ResidentError> { - self.gpu.instance.poll_all(false); - self.gpu.check().map_err(gpu_error) - } - - fn enqueue(&mut self, context: &mut Rex3Context) -> Result<(), ResidentError> { + fn enqueue_inner(&mut self, context: &mut Rex3Context) -> Result<(), ResidentError> { self.gpu.check().map_err(gpu_error)?; let opcode = context.drawmode0.opcode(); if opcode == DRAWMODE0_OPCODE_NOOP { @@ -1580,7 +1649,7 @@ impl ResidentBackend for ResidentGpu { return self.enqueue_host(context); } if self.serial_host_batch { - self.submit()?; + self.submit_for(FlushReason::HostTransition)?; } if matches!( context.drawmode0.adrmode(), @@ -1593,6 +1662,11 @@ impl ResidentBackend for ResidentGpu { let command = prepare_resident_flat_fill(context).map_err(ResidentError::Unsupported)?; let position = self.progress.lock().next_position()?; let fill = command.fill(); + self.current_coverage = fill.and_then(|fill| Coverage::new( + fill.min_x as i32, fill.min_y as i32, + (fill.min_x + fill.width - 1) as i32, + (fill.min_y + fill.height - 1) as i32, + )); if let Some(fill) = fill { let slot = RexStateSlot::for_flat_fill(context, fill); let shade_payload = command.shade_payload(); @@ -1607,18 +1681,100 @@ impl ResidentBackend for ResidentGpu { Ok(()) } + fn enqueue_at(&mut self, context: &mut Rex3Context, now: Instant) -> Result<(), ResidentError> { + self.flush_if_due(now)?; + let start = Instant::now(); + let sample = DrawSample::from_context(context); + let requested_qwords = u64::from(context.host_len.max(1)); + let (rgb, aux, host_qwords, submit_ns) = { + let status = self.progress.lock(); + (status.rgb.last_write, status.aux.last_write, status.host_write_qwords, + status.workload.submission_ns) + }; + self.current_coverage = None; + let result = self.enqueue_inner(context); + let elapsed = duration_ns(start.elapsed()).map_err(ResidentError::InvalidState)?; + let pending = { + let mut status = self.progress.lock(); + let nested_submit = status.workload.submission_ns.checked_sub(submit_ns) + .ok_or(ResidentError::InvalidState("submission time decreased"))?; + let generation = elapsed.checked_sub(nested_submit) + .ok_or(ResidentError::InvalidState("nested submission exceeds command time"))?; + if matches!(result, Err(ResidentError::Unsupported(_))) { + status.workload.unsupported = status.workload.unsupported.checked_add(1) + .ok_or(ResidentError::InvalidState("unsupported counter exhausted"))?; + } + if result.is_ok() { + let noop = !sample.is_host_read() + && status.rgb.last_write == rgb && status.aux.last_write == aux; + let transfer = if sample.is_host_write() { + status.host_write_qwords.checked_sub(host_qwords) + .ok_or(ResidentError::InvalidState("HostW qword counter decreased"))? + } else { + requested_qwords + }; + status.workload.record_draw(sample, self.current_coverage, noop, transfer, generation) + .map_err(ResidentError::InvalidState)?; + } + status.accepted != status.submitted + }; + if pending && self.batch_started.is_none() { + self.batch_started = Some(now); + } + result + } +} + +impl ResidentBackend for ResidentGpu { + fn gpu(&self) -> Option<&Arc> { Some(&self.gpu) } + + fn poll_device_health(&mut self) -> Result<(), ResidentError> { + self.gpu.instance.poll_all(false); + self.gpu.check().map_err(gpu_error) + } + + fn enqueue(&mut self, context: &mut Rex3Context) -> Result<(), ResidentError> { + self.enqueue_at(context, Instant::now()) + } + fn submit(&mut self) -> Result<(), ResidentError> { + self.submit_for(FlushReason::Explicit) + } + + fn flush_if_due(&mut self, now: Instant) -> Result<(), ResidentError> { + if self.batch_started.is_some_and(|start| + now.checked_duration_since(start).is_some_and(|age| age >= MAX_BATCH_AGE)) + { + self.submit_for(FlushReason::Latency)?; + } + Ok(()) + } + + fn host_read_result(&mut self, qwords: u64, generation_ns: u64) -> Result<(), ResidentError> { + self.progress.lock().workload.record_host_read(qwords, generation_ns) + .map_err(ResidentError::InvalidState) + } + + fn submit_for(&mut self, reason: FlushReason) -> Result<(), ResidentError> { self.gpu.check().map_err(gpu_error)?; let (accepted, submitted) = { let status = self.progress.lock(); (status.accepted, status.submitted) }; if accepted == submitted { + self.batch_started = None; return Ok(()); } if self.batch.is_empty() && !self.payload.is_empty() { return Err(ResidentError::InvalidState("payload exists without a command")); } + let start = Instant::now(); + let commands = self.batch.len(); + let tile_pairs = self.batch.tile_pair_bound(); + let waits_before = self.batch.wait_stats(); + let profile_before = self.batch.timing.as_ref().map(|timing| ( + timing.batches, timing.tiles, timing.pairs, timing.bin_ns, timing.raster_ns, + )); let result = if self.batch.is_empty() { self.batch.try_flush(&self.pipelines, &[]) } else { @@ -1645,7 +1801,58 @@ impl ResidentBackend for ResidentGpu { } self.progress.lock().submit(accepted); record_completion(&self.gpu, &self.progress, accepted); - self.gpu.check().map_err(gpu_error) + self.gpu.check().map_err(gpu_error)?; + let elapsed = duration_ns(start.elapsed()).map_err(ResidentError::InvalidState)?; + let waits_after = self.batch.wait_stats(); + let waits = waits_after.waits.checked_sub(waits_before.waits) + .ok_or(ResidentError::InvalidState("bounded-set wait count decreased"))?; + let wait_ns = waits_after.nanoseconds.checked_sub(waits_before.nanoseconds) + .ok_or(ResidentError::InvalidState("bounded-set wait time decreased"))?; + let profile_error = self.batch.timing.as_ref().and_then(|timing| timing.failure); + let mut status = self.progress.lock(); + status.workload.record_flush(reason, commands, tile_pairs, elapsed, waits, wait_ns) + .map_err(ResidentError::InvalidState)?; + if commands != 0 { + if let (Some(before), Some(after)) = (profile_before, self.batch.timing.as_ref()) { + let workload = &mut status.workload; + workload.profile_maps = workload.profile_maps.checked_add(1) + .ok_or(ResidentError::InvalidState("profiling map counter exhausted"))?; + workload.profile_bytes = workload.profile_bytes.checked_add(64) + .ok_or(ResidentError::InvalidState("profiling byte counter exhausted"))?; + let samples = after.batches.checked_sub(before.0) + .ok_or(ResidentError::InvalidState("profiling sample count decreased"))?; + let tiles = after.tiles.checked_sub(before.1) + .ok_or(ResidentError::InvalidState("profiling busy tiles decreased"))?; + let pairs = after.pairs.checked_sub(before.2) + .ok_or(ResidentError::InvalidState("profiling pairs decreased"))?; + if samples > 1 || tiles > u64::from(self.batch.grid().tiles()) || pairs > tile_pairs || tiles > pairs { + return Err(ResidentError::InvalidState("profiling busy-tile bounds are invalid")); + } + let elapsed = |after: f64, before: f64| -> Result { + let delta = after - before; + if !delta.is_finite() || delta < 0.0 || delta >= u64::MAX as f64 { + return Err(ResidentError::InvalidState("profiling duration invalid")); + } + Ok(delta as u64) + }; + workload.profile_samples = workload.profile_samples.checked_add(samples) + .ok_or(ResidentError::InvalidState("profiling sample counter exhausted"))?; + workload.profile_busy_tiles = workload.profile_busy_tiles.checked_add(tiles) + .ok_or(ResidentError::InvalidState("profiling tile counter exhausted"))?; + workload.profile_tile_pairs = workload.profile_tile_pairs.checked_add(pairs) + .ok_or(ResidentError::InvalidState("profiling pair counter exhausted"))?; + workload.profile_bin_ns = workload.profile_bin_ns.checked_add(elapsed(after.bin_ns, before.3)?) + .ok_or(ResidentError::InvalidState("profiling bin time exhausted"))?; + workload.profile_raster_ns = workload.profile_raster_ns.checked_add(elapsed(after.raster_ns, before.4)?) + .ok_or(ResidentError::InvalidState("profiling raster time exhausted"))?; + } + } + drop(status); + self.batch_started = None; + if let Some(error) = profile_error { + return Err(ResidentError::Failed(format!("resident profiling: {error}"))); + } + Ok(()) } fn observe( @@ -1695,36 +1902,54 @@ impl ResidentBackend for ResidentGpu { } fn reset(&mut self, rgb: &[u32], aux: &[u32]) -> Result<(), ResidentError> { - let replacement = Self::activate_with_limits( + let rgb_is_shadow = rgb.as_ptr() as usize == self.rgb_shadow; + let aux_is_shadow = aux.as_ptr() as usize == self.aux_shadow; + let profiling = self.progress.lock().workload.profiling; + let progress = Arc::new(Mutex::new(self.progress.lock().clone())); + let mut replacement = Self::activate_with_limits( Arc::clone(&self.gpu), rgb, aux, self.max_slots, self.max_sets, - Some((Arc::clone(&self.progress), self.readback.clone())), + Some((progress, self.readback.clone())), )?; + if profiling { + replacement.batch.enable_profiling().map_err(ResidentError::Failed)?; + } + replacement.gpu.wait(None).map_err(gpu_error)?; + replacement.rgb_shadow = self.rgb_shadow; + replacement.aux_shadow = self.aux_shadow; + { + let mut status = replacement.progress.lock(); + if !rgb_is_shadow { status.rgb.cpu_shadow = None; } + if !aux_is_shadow { status.aux.cpu_shadow = None; } + let seed = status.submitted; + status.confirm(seed); + } *self = replacement; Ok(()) } fn restore(&mut self, rgb: &[u32], aux: &[u32]) -> Result<(), ResidentError> { // Submit the old prefix first so untouched suffixes include its writes. - self.submit()?; + self.submit_for(FlushReason::Restore)?; let epoch = next_epoch()?; let rgb_words = rgb.len().min(PLANE_WORDS); let aux_words = aux.len().min(PLANE_WORDS); - let next = self.progress.lock().restored(epoch, rgb_words, aux_words)?; + let mut next = self.progress.lock().restored(epoch, rgb_words, aux_words)?; if rgb_words != 0 { self.gpu.write(&self.planes.rgb, 0, bytemuck::cast_slice(&rgb[..rgb_words])); } if aux_words != 0 { self.gpu.write(&self.planes.aux, 0, bytemuck::cast_slice(&aux[..aux_words])); } - self.gpu.submit(self.gpu.encoder("resident rex3 restore")); + let submission = self.gpu.submit(self.gpu.encoder("resident rex3 restore")); + self.gpu.wait(Some(submission)).map_err(gpu_error)?; let seed = next.submitted; + next.confirm(seed); *self.progress.lock() = next; - record_completion(&self.gpu, &self.progress, seed); - self.gpu.check().map_err(gpu_error) + Ok(()) } fn scanout_bindings( diff --git a/src/dev/ng1/wgpu/resident_tests.rs b/src/dev/ng1/wgpu/resident_tests.rs index 66f2c9364..75eab27fa 100644 --- a/src/dev/ng1/wgpu/resident_tests.rs +++ b/src/dev/ng1/wgpu/resident_tests.rs @@ -18,7 +18,7 @@ use crate::dev::ng1::rex3::{ }; use crate::dev::ng1::rex3_generic::{draw_with_fb, fline_setup, Framebuffers}; use crate::dev::ng1::rex3_shape; -use crate::traits::{BusDevice, Device, Resettable, BUS_BUSY, BUS_OK}; +use crate::traits::{BusDevice, Device, Resettable, Saveable, BUS_BUSY, BUS_OK}; use crate::wgpu::GpuOptions; fn on_large_stack(f: impl FnOnce() + Send + 'static) { @@ -234,6 +234,80 @@ fn cpu_host_draw(ctx: &mut Rex3Context, rgb: &mut [u32], aux: &mut [u32]) { ctx.host_cursor = 0; } +#[test] +#[ignore = "requires the requested conformant hardware GPU backend"] +fn resident_gpu_age_threshold_and_explicit_busy_tile_profile() { + on_large_stack(|| { + let gpu = required_gpu(); + let rgb = seeded_plane(0x3926_18af); + let aux = seeded_plane(0x4e03_d2b7); + let mut expected_rgb = rgb.clone(); + let mut expected_aux = aux.clone(); + let mut resident = ResidentGpu::activate(Arc::clone(&gpu), &rgb, &aux).unwrap(); + assert!(resident.batch.timing.is_none(), "ordinary resident work must not inherit profiling"); + + let mut first = context(DRAWMODE1_PLANES_RGB, true, false, (96, 0), (103, 7), u32::MAX, 0x2233_4455); + let mut expected = first; + resident.enqueue(&mut first).unwrap(); + cpu_draw(&mut expected, &mut expected_rgb, &mut expected_aux); + let started = resident.batch_started.expect("first accepted GO starts the batch clock"); + resident.flush_if_due(started + MAX_BATCH_AGE - std::time::Duration::from_nanos(1)).unwrap(); + assert_eq!(resident.status().unwrap().submitted.sequence, Sequence(0)); + resident.flush_if_due(started + MAX_BATCH_AGE).unwrap(); + let unprofiled = resident.status().unwrap(); + assert_eq!(unprofiled.workload.flush[FlushReason::Latency as usize], 1); + assert_eq!((unprofiled.raw_plane_map_attempts, unprofiled.workload.profile_maps), (0, 0)); + assert!(!unprofiled.workload.profiling); + + resident.enable_profiling().expect("explicit timestamp and busy-list profiling"); + assert!(resident.enable_profiling().is_err(), "profiling cannot be toggled twice"); + let profiled_start = resident.status().unwrap().workload; + assert!(profiled_start.delta_since(&unprofiled.workload).is_err()); + for (start, end, color) in [ + ((0, 0), (7, 7), 0x0011_2233), + ((0, 0), (7, 7), 0x0044_5566), + ((32, 0), (39, 7), 0x0077_8899), + ] { + let mut actual = context(DRAWMODE1_PLANES_RGB, true, false, start, end, u32::MAX, color); + let mut expected = actual; + resident.enqueue(&mut actual).unwrap(); + cpu_draw(&mut expected, &mut expected_rgb, &mut expected_aux); + } + resident.submit_for(FlushReason::Explicit).unwrap(); + let status = resident.status().unwrap(); + let measured = status.workload.delta_since(&profiled_start).unwrap(); + assert_eq!((measured.profile_maps, measured.profile_bytes, measured.profile_samples), (1, 64, 1)); + assert_eq!((measured.profile_busy_tiles, measured.profile_tile_pairs), (2, 3)); + assert_eq!(status.raw_plane_map_attempts, 0, "profiling maps only timestamps and indirect metadata"); + let observed = resident.observe(ObservationReason::Validation, PlaneMask::Both).unwrap(); + compare_observed(observed.rgb().unwrap(), &expected_rgb, "profiled RGB"); + compare_observed(observed.aux().unwrap(), &expected_aux, "profiled aux"); + + let mut bounded = ResidentGpu::activate_with_limits( + Arc::clone(&gpu), &rgb, &aux, 8, NonZeroUsize::new(1).unwrap(), None, + ).unwrap(); + let mut draw = context(DRAWMODE1_PLANES_RGB, true, false, (12, 24), (19, 31), u32::MAX, 0x0011_3355); + bounded.enqueue(&mut draw).unwrap(); + bounded.submit().unwrap(); + gpu.wait(None).unwrap(); + let (entered_tx, entered_rx) = std::sync::mpsc::channel(); + let done = bounded.batch.force_wait_for_test(entered_tx).unwrap(); + let release = std::thread::spawn(move || { + entered_rx.recv_timeout(std::time::Duration::from_secs(10)).unwrap(); + done.store(true, Ordering::Release); + }); + let mut next = context(DRAWMODE1_PLANES_RGB, true, false, (20, 24), (27, 31), u32::MAX, 0x0055_3311); + bounded.enqueue(&mut next).unwrap(); + bounded.submit().unwrap(); + release.join().unwrap(); + let wait = bounded.status().unwrap().workload; + assert_eq!(wait.set_waits, 1, "bounded-set reuse waited exactly once"); + assert!(wait.set_wait_ns > 0); + assert_eq!((wait.profile_maps, bounded.status().unwrap().raw_plane_map_attempts), (0, 0)); + gpu.check().unwrap(); + }); +} + #[test] fn resident_limits_cover_plane_bindings_and_batch_dispatch() { let good = wgpu::Limits::default(); @@ -426,7 +500,7 @@ fn resident_gpu_observation_selection_positions_and_shadow_acknowledgement() { } let pending = raster.status().unwrap(); assert_eq!(pending.accepted.sequence, Sequence(3)); - assert_eq!(pending.submitted.sequence, Sequence(0)); + assert!(pending.submitted.sequence < pending.accepted.sequence); assert_eq!((pending.rgb.last_write.sequence, pending.aux.last_write.sequence), (Sequence(1), Sequence(2))); assert_eq!(pending.observations, ObservationCounters::default()); assert_eq!(pending.raw_plane_map_attempts, 0); @@ -972,7 +1046,8 @@ fn resident_gpu_linear_fills_match_complete_cpu_planes() { assert!(after_reset.rgb.cpu_shadow_current() && after_reset.aux.cpu_shadow_current()); let completed = after_reset.completed; progress.lock().confirm(old_position); - assert_eq!(progress.lock().completed, completed, "stale completion changed reset status"); + resident.progress.lock().confirm(old_position); + assert_eq!(resident.status().unwrap().completed, completed, "stale completion changed reset status"); let reset_seed = resident.observe(ObservationReason::Validation, PlaneMask::Both).unwrap(); compare_observed(reset_seed.rgb().unwrap(), &seed_rgb, "reset RGB seed"); compare_observed(reset_seed.aux().unwrap(), &seed_aux, "reset aux seed"); @@ -2356,6 +2431,7 @@ fn resident_gpu_host_fills_match_complete_cpu_planes() { None, ) .expect("activate HostW resident planes"); + let batch_clock = Instant::now(); // Isolate capacity and host-transition boundaries from age flushes. struct Case { label: &'static str, @@ -2739,7 +2815,7 @@ fn resident_gpu_host_fills_match_complete_cpu_planes() { expected_prims += 1; } resident - .enqueue(&mut actual) + .enqueue_at(&mut actual, batch_clock) .unwrap_or_else(|error| panic!("enqueue {}: {error}", case.label)); cpu_host_draw(&mut expected, &mut expected_rgb, &mut expected_aux); assert_context_eq(&actual, &expected, case.label); @@ -3063,7 +3139,7 @@ fn resident_gpu_host_fills_match_complete_cpu_planes() { expected_prims += 1; } resident - .enqueue(&mut actual) + .enqueue_at(&mut actual, batch_clock) .unwrap_or_else(|error| panic!("enqueue {label}: {error}")); cpu_host_draw(&mut expected, &mut expected_rgb, &mut expected_aux); assert_context_eq(&actual, &expected, &label); @@ -3107,7 +3183,7 @@ fn resident_gpu_host_fills_match_complete_cpu_planes() { "{label}" ); resident - .enqueue(&mut actual) + .enqueue_at(&mut actual, batch_clock) .unwrap_or_else(|error| panic!("enqueue {label}: {error}")); cpu_host_draw(&mut expected, &mut expected_rgb, &mut expected_aux); assert_context_eq(&actual, &expected, &label); @@ -3143,7 +3219,7 @@ fn resident_gpu_host_fills_match_complete_cpu_planes() { 0x00ff_33aa, ); let mut expected = ordinary; - resident.enqueue(&mut ordinary).unwrap(); + resident.enqueue_at(&mut ordinary, batch_clock).unwrap(); cpu_draw(&mut expected, &mut expected_rgb, &mut expected_aux); assert_context_eq(&ordinary, &expected, "ordinary draw after zoom batch"); expected_prims += 1; @@ -3168,7 +3244,7 @@ fn resident_gpu_host_fills_match_complete_cpu_planes() { boundary_host.drawmode1.0 |= (1 << 7) | (3 << 8); set_payload(&mut boundary_host, &[0xff33_6699_0000_0000]); let mut expected = boundary_host; - resident.enqueue(&mut boundary_host).unwrap(); + resident.enqueue_at(&mut boundary_host, batch_clock).unwrap(); cpu_host_draw(&mut expected, &mut expected_rgb, &mut expected_aux); assert_context_eq( &boundary_host, @@ -3260,7 +3336,7 @@ fn resident_gpu_host_fills_match_complete_cpu_planes() { 1 ); let mut expected = actual; - resident.enqueue(&mut actual).expect("enqueue wrapping HostW"); + resident.enqueue_at(&mut actual, batch_clock).expect("enqueue wrapping HostW"); cpu_host_draw( &mut expected, &mut wrap_expected_rgb, @@ -3733,6 +3809,52 @@ impl Drop for TestRex { } } +struct ProducerTelemetryBackend { + entered: std::sync::mpsc::Sender<()>, + resume: std::sync::mpsc::Receiver<()>, +} + +impl ResidentBackend for ProducerTelemetryBackend { + fn enqueue(&mut self, _context: &mut Rex3Context) -> Result<(), ResidentError> { + self.entered.send(()).map_err(|_| ResidentError::InvalidState("producer test receiver closed"))?; + self.resume.recv_timeout(std::time::Duration::from_secs(10)) + .map_err(|_| ResidentError::Timeout("producer test"))?; + Ok(()) + } + fn submit(&mut self) -> Result<(), ResidentError> { Ok(()) } + fn status(&mut self) -> Result { Ok(ResidentStatus::seeded(Epoch(1))) } +} + +#[test] +fn resident_board_gfifo_full_and_bus_busy_are_counted_at_producer() { + on_large_stack(|| { + let mut board = TestRex(Box::new(Rex3::new(crate::disp::StatusSource::detached()))); + let (entered_tx, entered_rx) = std::sync::mpsc::channel(); + let (resume_tx, resume_rx) = std::sync::mpsc::channel(); + board.0.attach_resident_raster(ProducerTelemetryBackend { entered: entered_tx, resume: resume_rx }).unwrap(); + board.0.start(); + write_reg(&board.0, REX3_DRAWMODE0, DRAWMODE0_OPCODE_NOOP, true); + entered_rx.recv_timeout(std::time::Duration::from_secs(10)).unwrap(); + let address = (REX3_BASE | REX3_COLORRED) as u64; + let mut pushed = 0u64; + loop { + match board.0.write32(address, pushed as u32) { + BUS_OK => pushed += 1, + BUS_BUSY => break, + status => panic!("producer test write failed: {status:#x}"), + } + assert!(pushed < crate::dev::ng1::rex3::GFIFO_DEPTH as u64); + } + assert_eq!(board.0.read32(address).status, BUS_BUSY); + resume_tx.send(()).unwrap(); + board.0.wait_resident_idle().unwrap(); + let snapshot = board.0.resident_telemetry_snapshot().unwrap(); + assert_eq!(snapshot.board.occupancy.high_water, (crate::dev::ng1::rex3::GFIFO_DEPTH - 1) as u64); + assert_eq!(snapshot.board.reservations, pushed + 2); + assert_eq!((snapshot.board.backpressure, snapshot.board.full, snapshot.board.bus_busy), (1, 1, 2)); + }); +} + fn write_reg(rex: &Rex3, offset: u32, value: u32, go: bool) { let addr = (REX3_BASE | offset | if go { 0x800 } else { 0 }) as u64; loop { @@ -3775,6 +3897,18 @@ impl ResidentBackend for SharedResident { self.0.lock().submit() } + fn submit_for(&mut self, reason: FlushReason) -> Result<(), ResidentError> { + self.0.lock().submit_for(reason) + } + + fn flush_if_due(&mut self, now: Instant) -> Result<(), ResidentError> { + self.0.lock().flush_if_due(now) + } + + fn host_read_result(&mut self, qwords: u64, generation_ns: u64) -> Result<(), ResidentError> { + self.0.lock().host_read_result(qwords, generation_ns) + } + fn observe(&mut self, reason: ObservationReason, planes: PlaneMask) -> Result { self.0.lock().observe(reason, planes) } @@ -3805,6 +3939,567 @@ impl ResidentBackend for SharedResident { } } +struct AgeGateResident { + resident: SharedResident, + first: bool, + accepted: std::sync::mpsc::Sender<()>, + resume: std::sync::mpsc::Receiver<()>, + flushed: std::sync::mpsc::Sender<()>, + continue_after_flush: std::sync::mpsc::Receiver<()>, +} + +impl ResidentBackend for AgeGateResident { + fn enqueue(&mut self, context: &mut Rex3Context) -> Result<(), ResidentError> { + let mut resident = self.resident.0.lock(); + resident.enqueue(context)?; + if self.first { + self.first = false; + resident.batch_started = Some(Instant::now() - MAX_BATCH_AGE); + drop(resident); + self.accepted.send(()).map_err(|_| ResidentError::InvalidState("age test receiver closed"))?; + self.resume.recv_timeout(std::time::Duration::from_secs(10)) + .map_err(|_| ResidentError::Timeout("age test producer"))?; + } + Ok(()) + } + + fn flush_if_due(&mut self, now: Instant) -> Result<(), ResidentError> { + let mut resident = self.resident.0.lock(); + let before = resident.progress.lock().workload.flush[FlushReason::Latency as usize]; + resident.flush_if_due(now)?; + let after = resident.progress.lock().workload.flush[FlushReason::Latency as usize]; + drop(resident); + if after > before { + self.flushed.send(()).map_err(|_| ResidentError::InvalidState("age test receiver closed"))?; + self.continue_after_flush.recv_timeout(std::time::Duration::from_secs(10)) + .map_err(|_| ResidentError::Timeout("age test observer"))?; + } + Ok(()) + } + + fn submit_for(&mut self, reason: FlushReason) -> Result<(), ResidentError> { + self.resident.submit_for(reason) + } + fn submit(&mut self) -> Result<(), ResidentError> { self.resident.submit() } + fn poll_device_health(&mut self) -> Result<(), ResidentError> { self.resident.poll_device_health() } + fn status(&mut self) -> Result { self.resident.status() } + fn recorded_status(&self) -> Result { self.resident.recorded_status() } +} + +#[test] +#[ignore = "requires the requested conformant hardware GPU backend"] +fn resident_gpu_continuous_gfifo_flushes_on_age_without_idle_or_fence() { + on_large_stack(|| { + let gpu = required_gpu(); + let rgb = seeded_plane(0x3718_529a); + let aux = seeded_plane(0x6482_7c10); + let mut board = TestRex(Box::new(Rex3::new(crate::disp::StatusSource::detached()))); + board.0.fb_rgb.get_mut().copy_from_slice(&rgb); + board.0.fb_aux.get_mut().copy_from_slice(&aux); + let backend = Arc::new(Mutex::new(ResidentGpu::activate(Arc::clone(&gpu), &rgb, &aux).unwrap())); + let (accepted_tx, accepted_rx) = std::sync::mpsc::channel(); + let (resume_tx, resume_rx) = std::sync::mpsc::channel(); + let (flushed_tx, flushed_rx) = std::sync::mpsc::channel(); + let (continue_tx, continue_rx) = std::sync::mpsc::channel(); + board.0.attach_resident_raster(AgeGateResident { + resident: SharedResident(Arc::clone(&backend)), + first: true, + accepted: accepted_tx, + resume: resume_rx, + flushed: flushed_tx, + continue_after_flush: continue_rx, + }).unwrap(); + let mut draw = context(DRAWMODE1_PLANES_RGB, true, false, (12, 12), (19, 19), u32::MAX, 0x4411_3322); + draw.drawmode0.0 |= 1 << 5; + board.0.start(); + enqueue_context(&board.0, &draw, (12, 12), (19, 19)); + accepted_rx.recv_timeout(std::time::Duration::from_secs(10)).unwrap(); + for value in 0..2048 { + write_reg(&board.0, REX3_COLORRED, value, false); + } + resume_tx.send(()).unwrap(); + flushed_rx.recv_timeout(std::time::Duration::from_secs(10)).unwrap(); + assert!(board.0.gfifo.len() > 0, "the age flush happened while the FIFO remained fed"); + continue_tx.send(()).unwrap(); + board.0.wait_resident_idle().unwrap(); + let status = board.0.resident_telemetry_snapshot().unwrap(); + assert_eq!(status.workload.flush[FlushReason::Latency as usize], 1); + assert_eq!(status.workload.flush[FlushReason::Capacity as usize], 0); + assert_eq!(status.workload.flush[FlushReason::DisplayFence as usize], 0); + assert_eq!(status.workload.batches.sum, 1); + assert_eq!((status.traffic.raw_maps, status.workload.profile_maps), (0, 0)); + board.0.stop(); + }); +} + +#[test] +#[ignore = "requires the requested conformant hardware GPU backend"] +fn resident_gpu_independent_mixed_workload_gate() { + on_large_stack(|| { + use crate::dev::ng1::rex3::frame::{ResidentDisplay, ResidentDisplayMode}; + use crate::dev::ng1::rex3::REX3_XYMOVE; + use crate::dev::ng1::wgpu::direct_tests::{configure_board_display, QueuePresenter}; + use crate::dev::ng1::wgpu::telemetry::{AddressMode, ModeFeature, PrimitiveKind}; + + let gpu = required_gpu(); + let rgb = seeded_plane(0x5638_4129); + let aux = seeded_plane(0x8921_76bc); + let mut resident = TestRex(Box::new(Rex3::new(crate::disp::StatusSource::detached()))); + let mut cpu = TestRex(Box::new(Rex3::new(crate::disp::StatusSource::detached()))); + for board in [&mut resident, &mut cpu] { + board.0.fb_rgb.get_mut().copy_from_slice(&rgb); + board.0.fb_aux.get_mut().copy_from_slice(&aux); + } + #[cfg(feature = "rex-jit")] + cpu.0.jit_enabled.store(false, Ordering::Relaxed); + configure_board_display(&resident.0); + resident.0.activate_resident_gpu(Arc::clone(&gpu)).unwrap(); + let outputs = Arc::new(Mutex::new(Vec::new())); + *resident.0.head.renderer_slot().lock() = Some(Box::new(QueuePresenter { + gpu: Some(Arc::clone(&gpu)), + accepts: true, + outputs: Arc::clone(&outputs), + resizes: Arc::new(Mutex::new(Vec::new())), + })); + let baseline = resident.0.resident_telemetry_snapshot().unwrap(); + resident.0.start(); + cpu.0.start(); + + for board in [&resident.0, &cpu.0] { + write_reg(board, REX3_CLIPMODE, 0xf << CLIPMODE_CIDMATCH_SHIFT, false); + } + let draw = |start, end, plane, color, span| { + let mut context = context(plane, plane == DRAWMODE1_PLANES_RGB, span, start, end, u32::MAX, color); + context.drawmode0.0 |= 1 << 5; + for board in [&resident.0, &cpu.0] { + enqueue_context(board, &context, start, end); + } + }; + for (start, end, plane, color) in [ + ((0, 0), (63, 31), DRAWMODE1_PLANES_RGB, 0x0010_2020), + ((5, 3), (44, 24), DRAWMODE1_PLANES_RGB, 0x0040_3020), + ((18, 8), (59, 27), DRAWMODE1_PLANES_RGB, 0x0020_3050), + ((23, 10), (32, 19), DRAWMODE1_PLANES_CID, 0x0000_0003), + ] { + draw(start, end, plane, color, false); + } + draw((1, 2), (60, 2), DRAWMODE1_PLANES_RGB, 0x00ee_cc88, true); + draw((3, 29), (62, 29), DRAWMODE1_PLANES_RGB, 0x0044_66bb, true); + for glyph in 0..16 { + let x = 3 + glyph * 3; + draw((x, 13), (x + 1, 15), DRAWMODE1_PLANES_RGB, 0x00cc_cccc, false); + } + let mut display = ResidentDisplay::new(&resident.0, &gpu, ResidentDisplayMode::DirectWgpu).unwrap(); + display.render(&resident.0, std::time::Duration::from_secs(10), |_| {}).unwrap(); + let first = resident.0.resident_telemetry_snapshot().unwrap(); + assert_eq!((first.traffic.raw_maps, first.traffic.composed_attempts), (0, [0, 0])); + assert_eq!(display.capture_count().attempts, 0); + + for (start, end) in [((0, 30), (63, 1)), ((0, 6), (63, 6))] { + let mut line = context(DRAWMODE1_PLANES_RGB, true, true, start, end, u32::MAX, 0x0080_a0e0); + line.drawmode0.0 = DRAWMODE0_OPCODE_DRAW | DRAWMODE0_ADRMODE_I_LINE_SH + | (1 << 5) | (1 << 8) | (1 << 9); + for board in [&resident.0, &cpu.0] { + enqueue_context(board, &line, start, end); + } + } + for (start, end, movement) in [ + ((5, 3), (28, 12), (4i16, 3i16)), + ((36, 5), (48, 16), (-4i16, 1i16)), + ] { + let mut copy = context(DRAWMODE1_PLANES_RGB, true, false, start, end, u32::MAX, 0); + copy.drawmode0.0 = (copy.drawmode0.0 & !3) | DRAWMODE0_OPCODE_SCR2SCR | (1 << 5); + let xy_move = ((movement.0 as u16 as u32) << 16) | movement.1 as u16 as u32; + for board in [&resident.0, &cpu.0] { + write_reg(board, REX3_XYMOVE, xy_move, false); + enqueue_context(board, ©, start, end); + } + } + let start = (8, 21); + let end = (15, 22); + let mut host = context(DRAWMODE1_PLANES_RGB, true, false, start, end, 0x00ff_ffff, 0); + host.drawmode0.0 |= (1 << 5) | (1 << 6); + host.drawmode1.0 |= (1 << 7) | (1 << 8) | (1 << 10); + let host_go = (REX3_BASE | 0x0800 | REX3_HOSTRW0) as u64; + for board in [&resident.0, &cpu.0] { + write_reg(board, REX3_DRAWMODE1, host.drawmode1.0, false); + write_reg(board, REX3_WRMASK, host.wrmask, false); + write_reg(board, REX3_XYENDI, xy(end.0, end.1), false); + write_reg(board, REX3_XYSTARTI, xy(start.0, start.1), false); + write_reg(board, REX3_DRAWMODE0, host.drawmode0.0, false); + loop { + match board.dma_write64_bulk(host_go, &[0x1020_3040_5060_7080, 0x90a0_b0c0_d0e0_f001]) { + BUS_OK => break, + BUS_BUSY => std::thread::yield_now(), + status => panic!("independent HostW failed: {status:#x}"), + } + } + } + let mut noop = context(DRAWMODE1_PLANES_RGB, true, false, (0, 0), (1, 1), 0, 0); + noop.drawmode0.0 |= 1 << 5; + for board in [&resident.0, &cpu.0] { + enqueue_context(board, &noop, (0, 0), (1, 1)); + } + display.render(&resident.0, std::time::Duration::from_secs(10), |_| {}).unwrap(); + let before_hostr = resident.0.resident_telemetry_snapshot().unwrap(); + assert_eq!((before_hostr.traffic.raw_maps, before_hostr.traffic.composed_attempts), (0, [0, 0])); + assert_eq!(display.direct_traffic().unwrap().composed_readbacks.attempts, 0); + assert_eq!(outputs.lock().len(), 2, "both frames reached the same-device presenter"); + + let read_mode = (host.drawmode0.0 & !(3 | (1 << 6))) | DRAWMODE0_OPCODE_READ; + let read_drawmode1 = (host.drawmode1.0 & !(3 << 8)) | (3 << 8); + for _ in 0..2 { + let mut results = [[0u64; 2]; 2]; + for (index, board) in [&resident.0, &cpu.0].into_iter().enumerate() { + write_reg(board, REX3_DRAWMODE1, read_drawmode1, false); + write_reg(board, REX3_XYENDI, xy(11, 21), false); + write_reg(board, REX3_XYSTARTI, xy(8, 21), false); + write_reg(board, REX3_DRAWMODE0, read_mode, false); + loop { + match board.dma_read64_bulk(host_go, &mut results[index]) { + BUS_OK => break, + BUS_BUSY => std::thread::yield_now(), + status => panic!("independent HostR failed: {status:#x}"), + } + } + } + assert_eq!(results[0], results[1], "HostR qwords match the interpreter"); + } + resident.0.wait_resident_idle().unwrap(); + cpu.0.wait_idle(); + let snapshot = resident.0.resident_telemetry_snapshot().unwrap(); + let delta = snapshot.delta_since(&baseline).unwrap(); + let workload = &delta.workload; + assert_eq!(workload.accepted, 30); + assert_eq!((workload.noops, workload.unsupported), (1, 0)); + assert_eq!(workload.primitive[PrimitiveKind::Draw as usize], 25); + assert_eq!(workload.primitive[PrimitiveKind::Copy as usize], 2); + assert_eq!(workload.primitive[PrimitiveKind::HostWrite as usize], 1); + assert_eq!(workload.primitive[PrimitiveKind::HostRead as usize], 2); + assert_eq!(workload.address[AddressMode::Block as usize], 26); + assert_eq!(workload.address[AddressMode::Span as usize], 2); + assert_eq!(workload.address[AddressMode::IntegerLine as usize], 2); + assert_eq!(workload.features[ModeFeature::Auxiliary as usize], 1); + assert_eq!((workload.host_write_qwords.sum, workload.host_write_words), (2, 4)); + assert_eq!((workload.host_read_produced_qwords.sum, workload.host_read_words), (4, 8)); + assert!(workload.batches.samples > 0 && workload.batches.sum > 0); + assert!(workload.clipped_area.samples >= 25 && workload.clipped_tiles.samples >= 25); + assert_eq!((delta.traffic.raw_maps, delta.traffic.raw_bytes[0]), (1, PLANE_BYTES)); + assert_eq!(delta.traffic.composed_attempts, [0, 0]); + assert_eq!((delta.board.cut_latency_ns.samples, delta.board.presentation_latency_ns.samples), (2, 2)); + assert!(delta.board.occupancy.samples > 0); + assert_eq!((workload.profile_maps, workload.profile_bytes), (0, 0)); + #[cfg(feature = "rexdiag")] + assert_eq!((resident.0.interp_go_count.load(Ordering::Relaxed), resident.0.jit_go_count.load(Ordering::Relaxed)), (0, 0)); + eprintln!("independent mixed workload aggregate (not a live capture): {delta}"); + + resident.0.stop(); + cpu.0.stop(); + let (actual_rgb, actual_aux) = resident.0.snapshot_framebuffers_inmem(); + compare_plane(&actual_rgb, unsafe { &*cpu.0.fb_rgb.get() }, "mixed workload RGB"); + compare_plane(&actual_aux, unsafe { &*cpu.0.fb_aux.get() }, "mixed workload aux"); + gpu.check().unwrap(); + }); +} + +#[cfg(feature = "gputrace")] +struct ResidentReplayBus<'a> { + board: &'a Rex3, + deadline: Instant, +} + +#[cfg(feature = "gputrace")] +impl ResidentReplayBus<'_> { + fn status(&self, status: u32) -> u32 { + if status == BUS_BUSY + && (Instant::now() >= self.deadline || self.board.resident_status().is_err()) + { + crate::traits::BUS_ERR + } else { + status + } + } +} + +#[cfg(feature = "gputrace")] +macro_rules! resident_replay_read { + ($name:ident, $result:ty) => { + fn $name(&self, address: u64) -> $result { + let mut result = self.board.$name(address); + result.status = self.status(result.status); + result + } + }; +} + +#[cfg(feature = "gputrace")] +macro_rules! resident_replay_write { + ($name:ident, $value:ty) => { + fn $name(&self, address: u64, value: $value) -> u32 { + self.status(self.board.$name(address, value)) + } + }; +} + +#[cfg(feature = "gputrace")] +impl BusDevice for ResidentReplayBus<'_> { + resident_replay_read!(read8, crate::traits::BusRead8); + resident_replay_read!(read16, crate::traits::BusRead16); + resident_replay_read!(read32, crate::traits::BusRead32); + resident_replay_read!(read64, crate::traits::BusRead64); + resident_replay_read!(dma_read64, crate::traits::BusRead64); + resident_replay_write!(write8, u8); + resident_replay_write!(write16, u16); + resident_replay_write!(write32, u32); + resident_replay_write!(write64, u64); + + fn dma_write64_bulk(&self, address: u64, values: &[u64]) -> u32 { + self.status(self.board.dma_write64_bulk(address, values)) + } + + fn dma_read64_bulk(&self, address: u64, values: &mut [u64]) -> u32 { + self.status(self.board.dma_read64_bulk(address, values)) + } +} + +/// Replays a local, potentially sensitive recorder input but emits only aggregate +/// counters. Replay throughput and optional post-replay presentations are not +/// live workload timing data. +#[cfg(feature = "gputrace")] +#[test] +#[ignore = "optional local recording; set REX3_RESIDENT_REPLAY"] +fn resident_replay_aggregate_file() { + on_large_stack(|| { + use crate::dev::ng1::rex3::rec; + let path = std::env::var("REX3_RESIDENT_REPLAY").expect("set REX3_RESIDENT_REPLAY"); + let mut records = rec::load(&path).unwrap_or_else(|_| panic!("cannot load local REX3 recording")); + if let Some(limit) = std::env::var("REX3_RESIDENT_REPLAY_LIMIT").ok().and_then(|v| v.parse::().ok()) { + records.truncate(limit); + } + let gpu = required_gpu(); + let mut board = TestRex(Box::new(Rex3::new(crate::disp::StatusSource::detached()))); + if let Some(state) = rec::tests::start_state(&records) { + board.0.rec_state_decode(state).unwrap_or_else(|_| panic!("cannot decode local recording state")); + } + let profile = std::env::var_os("REX3_RESIDENT_PROFILE").is_some(); + board.0.activate_resident_gpu_with_profiling(Arc::clone(&gpu), profile).unwrap(); + board.0.start(); + let before = board.0.resident_telemetry_snapshot().unwrap(); + let bus = ResidentReplayBus { + board: &board.0, + deadline: Instant::now() + std::time::Duration::from_secs(120), + }; + let report = rec::tests::replay_recs_with_hook(&board.0, &bus, &records, false, |_, _| {}); + if let Some(frames) = std::env::var("REX3_RESIDENT_REPLAY_FRAMES").ok() + .and_then(|value| value.parse::().ok()) + { + use crate::dev::ng1::rex3::frame::{ResidentDisplay, ResidentDisplayMode}; + let mut display = ResidentDisplay::new(&board.0, &gpu, ResidentDisplayMode::DirectWgpu) + .unwrap_or_else(|_| panic!("cannot create the replay presentation path")); + for _ in 0..frames { + display.render(&board.0, std::time::Duration::from_secs(10), |_| {}) + .unwrap_or_else(|_| panic!("cannot present the replayed resident state")); + } + } + let drained = report.access_error.is_none(); + let (after, fault) = board.0.resident_telemetry_diagnostics() + .unwrap_or_else(|_| panic!("resident replay telemetry is unavailable")); + let delta = after.delta_since(&before).unwrap(); + let complete = drained && fault.is_none() && report.reads_differ == 0; + eprintln!( + "local recording replay aggregate: records={} reads_checked={} reads_differ={} partial={} {delta}", + report.executed, report.reads_checked, report.reads_differ, !complete, + ); + assert!(complete, "resident replay diverged or stopped after {} records", report.executed); + board.0.stop(); + }); +} + +#[test] +#[ignore = "requires the requested conformant hardware GPU backend"] +fn resident_gpu_snapshot_restore_reset_round_trip_without_cpu_shadows() { + on_large_stack(|| { + let gpu = required_gpu(); + let seed_rgb = seeded_plane(0x3487_5612); + let seed_aux = seeded_plane(0xabcd_9876); + let mut expected_rgb = seed_rgb.clone(); + let mut expected_aux = seed_aux.clone(); + let mut rex = TestRex(Box::new(Rex3::new(crate::disp::StatusSource::detached()))); + rex.0.fb_rgb.get_mut().copy_from_slice(&seed_rgb); + rex.0.fb_aux.get_mut().copy_from_slice(&seed_aux); + rex.0.fb_rgb.get_mut().isolate().unwrap(); + rex.0.fb_aux.get_mut().isolate().unwrap(); + rex.0.activate_resident_gpu(Arc::clone(&gpu)).unwrap(); + let initial_telemetry = rex.0.resident_telemetry_snapshot().unwrap(); + rex.0.fb_rgb.get_mut().reset_access_counts(); + rex.0.fb_aux.get_mut().reset_access_counts(); + let rgb_guard = unsafe { (&*rex.0.fb_rgb.get()).protect() }.unwrap(); + let aux_guard = unsafe { (&*rex.0.fb_aux.get()).protect() }.unwrap(); + let original_binding = rex.0.resident_scanout_bindings(&gpu).unwrap(); + rex.0.start(); + let cases = [ + (DRAWMODE1_PLANES_RGB, true, (10, 20), (22, 23), 0x7f33_2211), + (DRAWMODE1_PLANES_CID, false, (12, 21), (18, 24), 0x0000_0003), + ]; + write_reg(&rex.0, REX3_CLIPMODE, 0xf << CLIPMODE_CIDMATCH_SHIFT, false); + for (plane, rgbmode, start, end, color) in cases { + let mut draw = context(plane, rgbmode, false, start, end, u32::MAX, color); + draw.drawmode0.0 |= 1 << 5; + enqueue_context(&rex.0, &draw, start, end); + cpu_draw(&mut draw, &mut expected_rgb, &mut expected_aux); + } + assert!(std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { + rex.0.snapshot_framebuffers_inmem() + })).is_err()); + rex.0.stop(); + let before = rex.0.resident_status().unwrap(); + assert_eq!(before.accepted.sequence, Sequence(2)); + assert!(!before.rgb.cpu_shadow_current() && !before.aux.cpu_shadow_current()); + let ctx = unsafe { &mut *rex.0.context.get() }; + ctx.hostrw[0] = 0x1234_5678_9abc_def0; + ctx.host_shifter = 0xfedc_ba98_7654_3210; + ctx.hostcnt = 5; + ctx.pat_bit = 9; + ctx.zpat_bit = 3; + ctx.mid_primitive = true; + rex.0.cmap0.lock().rgb_counter = 2; + let registers = rex.0.save_state(); + let (rgb, aux) = rex.0.snapshot_framebuffers_inmem(); + compare_plane(&rgb, &expected_rgb, "snapshot RGB"); + compare_plane(&aux, &expected_aux, "snapshot aux"); + let observed = rex.0.resident_status().unwrap(); + let counts = observed.observations.for_reason(ObservationReason::Snapshot); + assert_eq!((counts.attempts, counts.successes), (1, 1)); + assert_eq!((counts.successful_transfers, counts.successful_bytes), (2, 2 * PLANE_BYTES)); + assert_eq!(observed.completed, Some(observed.submitted)); + assert_eq!((observed.rgb.cpu_shadow, observed.aux.cpu_shadow), (before.rgb.cpu_shadow, before.aux.cpu_shadow)); + + rex.0.power_on(); + let reset = rex.0.resident_status().unwrap(); + assert!(rex.0.resident_telemetry_snapshot().unwrap().delta_since(&initial_telemetry).is_err()); + assert_ne!(reset.accepted.epoch, original_binding.epoch); + assert_eq!((reset.rgb.cpu_shadow, reset.aux.cpu_shadow), (None, None)); + let (reset_rgb, reset_aux) = rex.0.snapshot_framebuffers_inmem(); + assert!(reset_rgb.iter().all(|&word| word == 0)); + assert!(reset_aux.iter().all(|&word| word == 0)); + + rex.0.restore_framebuffers_inmem(&rgb, &aux); + assert!(std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| rex.0.start())).is_err()); + rex.0.load_state(®isters).unwrap(); + let restored = rex.0.resident_status().unwrap(); + assert_ne!(restored.accepted.epoch, reset.accepted.epoch); + assert_eq!(restored.completed, Some(restored.submitted)); + assert_eq!((restored.rgb.cpu_shadow, restored.aux.cpu_shadow), (None, None)); + assert_eq!(restored.restore_uploads, 2); + assert_eq!(rex.0.resident_scanout_bindings(&gpu).unwrap().epoch, restored.accepted.epoch); + assert_eq!(rex.0.save_state(), registers); + let (restored_rgb, restored_aux) = rex.0.snapshot_framebuffers_inmem(); + compare_plane(&restored_rgb, &expected_rgb, "restored RGB"); + compare_plane(&restored_aux, &expected_aux, "restored aux"); + + rex.0.start(); + let start = (30, 40); + let end = (35, 44); + let mut draw = context(DRAWMODE1_PLANES_RGB, true, false, start, end, u32::MAX, 0x4499_8877); + draw.drawmode0.0 |= 1 << 5; + enqueue_context(&rex.0, &draw, start, end); + cpu_draw(&mut draw, &mut expected_rgb, &mut expected_aux); + rex.0.stop(); + let (resumed_rgb, resumed_aux) = rex.0.snapshot_framebuffers_inmem(); + compare_plane(&resumed_rgb, &expected_rgb, "resumed RGB"); + compare_plane(&resumed_aux, &expected_aux, "resumed aux"); + assert_eq!(unsafe { (&*rex.0.fb_rgb.get()).access_counts() }, Default::default()); + assert_eq!(unsafe { (&*rex.0.fb_aux.get()).access_counts() }, Default::default()); + drop(aux_guard); + drop(rgb_guard); + }); +} + +#[test] +#[ignore = "requires the requested conformant hardware GPU backend"] +fn resident_gpu_hostw_continues_across_snapshot_restore() { + on_large_stack(|| { + let gpu = required_gpu(); + let seed_rgb = seeded_plane(0x1357_2468); + let seed_aux = seeded_plane(0x8642_7531); + let mut resident = TestRex(Box::new(Rex3::new(crate::disp::StatusSource::detached()))); + let mut cpu = TestRex(Box::new(Rex3::new(crate::disp::StatusSource::detached()))); + for board in [&mut resident, &mut cpu] { + board.0.fb_rgb.get_mut().copy_from_slice(&seed_rgb); + board.0.fb_aux.get_mut().copy_from_slice(&seed_aux); + } + #[cfg(feature = "rex-jit")] + cpu.0.jit_enabled.store(false, Ordering::Relaxed); + resident.0.fb_rgb.get_mut().isolate().unwrap(); + resident.0.fb_aux.get_mut().isolate().unwrap(); + resident.0.activate_resident_gpu(Arc::clone(&gpu)).unwrap(); + resident.0.fb_rgb.get_mut().reset_access_counts(); + resident.0.fb_aux.get_mut().reset_access_counts(); + let rgb_guard = unsafe { (&*resident.0.fb_rgb.get()).protect() }.unwrap(); + let aux_guard = unsafe { (&*resident.0.fb_aux.get()).protect() }.unwrap(); + resident.0.start(); + cpu.0.start(); + + let start = (100, 100); + let end = (115, 100); + let mut transfer = context(DRAWMODE1_PLANES_RGB, true, false, start, end, u32::MAX, 0); + transfer.drawmode0.0 |= 1 << 6; + transfer.drawmode0.0 &= !(1 << 9); + transfer.drawmode1.0 |= (1 << 7) | (1 << 8) | (1 << 10); + let host_go = (REX3_BASE | 0x0800 | REX3_HOSTRW0) as u64; + let send = |board: &Rex3, word: u64| loop { + match board.write64(host_go, word) { + BUS_OK => break, + BUS_BUSY => std::thread::yield_now(), + status => panic!("HostW continuation producer failed: {status:#x}"), + } + }; + for board in [&resident.0, &cpu.0] { + write_reg(board, REX3_DRAWMODE1, transfer.drawmode1.0, false); + write_reg(board, REX3_WRMASK, transfer.wrmask, false); + write_reg(board, REX3_CLIPMODE, transfer.clipmode, false); + write_reg(board, REX3_XYSTARTI, xy(start.0, start.1), false); + write_reg(board, REX3_XYENDI, xy(end.0, end.1), false); + write_reg(board, REX3_DRAWMODE0, transfer.drawmode0.0, false); + send(board, 0x1020_3040_5060_7080); + } + resident.0.wait_resident_idle().unwrap(); + cpu.0.wait_idle(); + assert_context_eq( + unsafe { &*resident.0.context.get() }, unsafe { &*cpu.0.context.get() }, "before HostW checkpoint", + ); + assert!(unsafe { (*resident.0.context.get()).mid_primitive }); + resident.0.stop(); + let registers = resident.0.save_state(); + let (rgb, aux) = resident.0.snapshot_framebuffers_inmem(); + compare_plane(&rgb, unsafe { &*cpu.0.fb_rgb.get() }, "HostW checkpoint RGB"); + compare_plane(&aux, unsafe { &*cpu.0.fb_aux.get() }, "HostW checkpoint aux"); + resident.0.power_on(); + resident.0.load_state(®isters).unwrap(); + resident.0.restore_framebuffers_inmem(&rgb, &aux); + assert_context_eq( + unsafe { &*resident.0.context.get() }, unsafe { &*cpu.0.context.get() }, "restored HostW checkpoint", + ); + + resident.0.start(); + send(&resident.0, 0x90a0_b0c0_d0e0_f001); + send(&cpu.0, 0x90a0_b0c0_d0e0_f001); + resident.0.wait_resident_idle().unwrap(); + cpu.0.wait_idle(); + assert_context_eq( + unsafe { &*resident.0.context.get() }, unsafe { &*cpu.0.context.get() }, "resumed HostW", + ); + resident.0.stop(); + let (rgb, aux) = resident.0.snapshot_framebuffers_inmem(); + compare_plane(&rgb, unsafe { &*cpu.0.fb_rgb.get() }, "resumed HostW RGB"); + compare_plane(&aux, unsafe { &*cpu.0.fb_aux.get() }, "resumed HostW aux"); + assert_eq!(unsafe { (&*resident.0.fb_rgb.get()).access_counts() }, Default::default()); + assert_eq!(unsafe { (&*resident.0.fb_aux.get()).access_counts() }, Default::default()); + drop(aux_guard); + drop(rgb_guard); + }); +} + #[test] #[ignore = "requires the requested conformant host GPU"] fn resident_gpu_gfifo_draws_match_complete_cpu_planes() { @@ -3876,6 +4571,9 @@ fn resident_gpu_gfifo_draws_match_complete_cpu_planes() { rex.0.wait_resident_idle(), Err(ResidentError::Unsupported(_)) )); + let (rejected, fault) = rex.0.resident_telemetry_diagnostics().unwrap(); + assert!(matches!(fault, Some(ResidentError::Unsupported(_)))); + assert_eq!(rejected.workload.unsupported, 1); gpu.wait(None).unwrap(); let mut guard = resident.lock(); let prefix_status = guard.status().unwrap(); @@ -4047,7 +4745,7 @@ fn resident_gpu_gfifo_activation_reset_and_fault() { assert_ne!(reset.accepted.epoch, initial.accepted.epoch); assert_ne!(rex.0.resident_scanout_bindings(&gpu).unwrap().epoch, initial_bindings.epoch); assert_eq!(reset.accepted.sequence, Sequence(0)); - assert!(reset.rgb.cpu_shadow_current() && reset.aux.cpu_shadow_current()); + assert!(!reset.rgb.cpu_shadow_current() && !reset.aux.cpu_shadow_current()); let reset_cpu_rgb = unsafe { (*rex.0.fb_rgb.get())[20 * STRIDE as usize + 10] }; rex.0.start(); write_reg( @@ -4152,6 +4850,6 @@ fn resident_gpu_gfifo_activation_reset_and_fault() { let recovered = rex.0.resident_status().unwrap(); assert_ne!(recovered.accepted.epoch, reset.accepted.epoch); assert_eq!(recovered.accepted.sequence, Sequence(0)); - assert!(recovered.rgb.cpu_shadow_current() && recovered.aux.cpu_shadow_current()); + assert!(!recovered.rgb.cpu_shadow_current() && !recovered.aux.cpu_shadow_current()); }); } diff --git a/src/dev/ng1/wgpu/telemetry.rs b/src/dev/ng1/wgpu/telemetry.rs new file mode 100644 index 000000000..1da94d32e --- /dev/null +++ b/src/dev/ng1/wgpu/telemetry.rs @@ -0,0 +1,649 @@ +//! Bounded, aggregate-only accounting for the test-gated Newport raster. + +use std::fmt; + +use crate::dev::ng1::rex3::{ + Rex3Context, DRAWMODE0_ADRMODE_A_LINE, DRAWMODE0_ADRMODE_BLOCK, + DRAWMODE0_ADRMODE_F_LINE, DRAWMODE0_ADRMODE_I_LINE, DRAWMODE0_ADRMODE_SPAN, + DRAWMODE0_OPCODE_DRAW, DRAWMODE0_OPCODE_NOOP, DRAWMODE0_OPCODE_READ, + DRAWMODE0_OPCODE_SCR2SCR, +}; +use crate::wgpu::tiles::{TILE_H, TILE_W}; + +const BUCKETS: usize = 32; + +macro_rules! add { + ($field:expr, $value:expr) => { + $field = $field.checked_add($value).ok_or("resident telemetry counter overflow")? + }; +} + +macro_rules! subtract { + ($after:ident, $before:ident, $($field:ident),+ $(,)?) => { + $( + $after.$field = $after.$field.checked_sub($before.$field) + .ok_or(concat!(stringify!($field), " counter decreased"))?; + )+ + }; +} + +/// Bucket 0 is zero; bucket i is 2^(i-1)..2^i-1, with the final bucket +/// unbounded. Quantiles are conservative upper bounds, not raw samples. +#[derive(Clone, Debug, Default, PartialEq, Eq)] +pub(crate) struct Histogram { + pub(crate) buckets: [u64; BUCKETS], + pub(crate) samples: u64, + pub(crate) sum: u64, + /// Lifetime high-water mark, including in a delta report and after reset. + pub(crate) high_water: u64, +} + +impl Histogram { + pub(crate) fn record(&mut self, value: u64) -> Result<(), &'static str> { + let bucket = if value == 0 { 0 } else { + (u64::BITS - value.leading_zeros()) as usize + }.min(BUCKETS - 1); + let count = self.buckets[bucket].checked_add(1).ok_or("histogram bucket overflow")?; + let samples = self.samples.checked_add(1).ok_or("histogram sample overflow")?; + let sum = self.sum.checked_add(value).ok_or("histogram sum overflow")?; + self.buckets[bucket] = count; + self.samples = samples; + self.sum = sum; + self.high_water = self.high_water.max(value); + Ok(()) + } + + pub(crate) fn delta_since(&self, before: &Self) -> Result { + if self.high_water < before.high_water { + return Err("histogram maximum decreased"); + } + let mut delta = self.clone(); + subtract!(delta, before, samples, sum); + for (bucket, previous) in delta.buckets.iter_mut().zip(before.buckets) { + *bucket = bucket.checked_sub(previous).ok_or("histogram bucket decreased")?; + } + Ok(delta) + } + + pub(crate) fn percentile_upper(&self, percent: u32) -> Option { + if self.samples == 0 || !(1..=100).contains(&percent) { + return None; + } + let rank = ((self.samples as u128 * percent as u128 + 99) / 100) as u64; + let mut seen = 0u64; + for (index, &count) in self.buckets.iter().enumerate() { + seen = seen.checked_add(count)?; + if seen >= rank { + return match index { + 0 => Some(0), + _ if index == BUCKETS - 1 => Some(self.high_water), + _ => Some((1u64 << index) - 1), + }; + } + } + None + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +#[repr(usize)] +pub(crate) enum PrimitiveKind { + Draw, + Copy, + HostWrite, + HostRead, + Noop, +} + +const PRIMITIVE_KINDS: usize = 5; + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +#[repr(usize)] +pub(crate) enum AddressMode { + Span, + Block, + IntegerLine, + FractionalLine, + AntialiasedLine, + Other, +} + +const ADDRESS_MODES: usize = 6; + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +#[repr(usize)] +pub(crate) enum ModeFeature { + Blend, + Shade, + ZPattern, + LinePattern, + Dither, + FastClear, + Scissor, + CidTest, + Auxiliary, + ColorHost, + AlphaHost, + PackedHost, + DoubleHost, + XyOffset, + YFlip, + AlphaCompare, +} + +const MODE_FEATURES: usize = 16; + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +#[repr(usize)] +pub(crate) enum FlushReason { + Capacity, + HostTransition, + CopyHazard, + Idle, + WaitIdle, + DisplayFence, + FrameCut, + Observation, + HostRead, + Explicit, + Exit, + Latency, + Restore, + Unsupported, +} + +const FLUSH_REASONS: usize = 14; + +/// Screen-clipped conservative destination bounds. This is not a count of +/// pixels surviving a per-pixel CID, pattern, alpha, or logic test. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) struct Coverage { + pub(crate) x0: u32, + pub(crate) y0: u32, + pub(crate) x1: u32, + pub(crate) y1: u32, +} + +impl Coverage { + pub(crate) fn new(x0: i32, y0: i32, x1: i32, y1: i32) -> Option { + let x0 = x0.max(0); + let y0 = y0.max(0); + let x1 = x1.min(crate::dev::ng1::rex3::REX3_SCREEN_WIDTH - 1); + let y1 = y1.min(crate::dev::ng1::rex3::REX3_SCREEN_HEIGHT - 1); + (x0 <= x1 && y0 <= y1).then_some(Self { + x0: x0 as u32, y0: y0 as u32, x1: x1 as u32, y1: y1 as u32, + }) + } + + pub(crate) fn include_address(&mut self, address: u32) { + let stride = crate::disp::FB_STRIDE as u32; + let x = address % stride; + let y = address / stride; + self.x0 = self.x0.min(x); + self.y0 = self.y0.min(y); + self.x1 = self.x1.max(x); + self.y1 = self.y1.max(y); + } + + pub(crate) fn from_address(address: u32) -> Self { + let stride = crate::disp::FB_STRIDE as u32; + let x = address % stride; + let y = address / stride; + Self { x0: x, y0: y, x1: x, y1: y } + } + + fn area(self) -> u64 { + u64::from(self.x1 - self.x0 + 1) * u64::from(self.y1 - self.y0 + 1) + } + + fn tiles(self) -> u64 { + u64::from(self.x1 / TILE_W - self.x0 / TILE_W + 1) + * u64::from(self.y1 / TILE_H - self.y0 / TILE_H + 1) + } +} + +#[derive(Clone, Copy)] +pub(crate) struct DrawSample { + kind: PrimitiveKind, + address: AddressMode, + features: u32, +} + +impl DrawSample { + pub(crate) fn from_context(context: &Rex3Context) -> Self { + let dm0 = context.drawmode0; + let dm1 = context.drawmode1; + let kind = match dm0.opcode() { + DRAWMODE0_OPCODE_NOOP => PrimitiveKind::Noop, + DRAWMODE0_OPCODE_READ => PrimitiveKind::HostRead, + DRAWMODE0_OPCODE_SCR2SCR => PrimitiveKind::Copy, + DRAWMODE0_OPCODE_DRAW if dm0.colorhost() || dm0.alphahost() => PrimitiveKind::HostWrite, + _ => PrimitiveKind::Draw, + }; + let address = match dm0.adrmode() { + DRAWMODE0_ADRMODE_SPAN => AddressMode::Span, + DRAWMODE0_ADRMODE_BLOCK => AddressMode::Block, + DRAWMODE0_ADRMODE_I_LINE => AddressMode::IntegerLine, + DRAWMODE0_ADRMODE_F_LINE => AddressMode::FractionalLine, + DRAWMODE0_ADRMODE_A_LINE => AddressMode::AntialiasedLine, + _ => AddressMode::Other, + }; + let mut features = 0u32; + let mut set = |feature: ModeFeature, enabled: bool| { + if enabled { features |= 1 << feature as usize; } + }; + set(ModeFeature::Blend, dm1.blend()); + set(ModeFeature::Shade, dm0.shade()); + set(ModeFeature::ZPattern, dm0.enzpattern()); + set(ModeFeature::LinePattern, dm0.enlspattern()); + set(ModeFeature::Dither, dm1.dither()); + set(ModeFeature::FastClear, dm1.fastclear()); + set(ModeFeature::Scissor, context.clipmode & 0x1f != 0); + set(ModeFeature::CidTest, super::command::cid_test(context.clipmode)); + set(ModeFeature::Auxiliary, matches!(dm1.planes(), 4 | 5 | 6)); + set(ModeFeature::ColorHost, dm0.colorhost()); + set(ModeFeature::AlphaHost, dm0.alphahost()); + set(ModeFeature::PackedHost, dm1.rwpacked()); + set(ModeFeature::DoubleHost, dm1.rwdouble()); + set(ModeFeature::XyOffset, dm0.xyoffset()); + set(ModeFeature::YFlip, dm1.yflip()); + set(ModeFeature::AlphaCompare, dm1.compare() != 7); + Self { kind, address, features } + } + + pub(crate) fn is_host_read(self) -> bool { + self.kind == PrimitiveKind::HostRead + } + + pub(crate) fn is_host_write(self) -> bool { + self.kind == PrimitiveKind::HostWrite + } +} + +#[derive(Clone, Debug, Default, PartialEq, Eq)] +pub(crate) struct Workload { + pub(crate) profiling: bool, + pub(crate) accepted: u64, + pub(crate) noops: u64, + pub(crate) unsupported: u64, + pub(crate) primitive: [u64; PRIMITIVE_KINDS], + pub(crate) address: [u64; ADDRESS_MODES], + pub(crate) features: [u64; MODE_FEATURES], + pub(crate) clipped_area: Histogram, + pub(crate) clipped_tiles: Histogram, + pub(crate) host_write_qwords: Histogram, + pub(crate) host_read_requested_qwords: Histogram, + pub(crate) host_read_produced_qwords: Histogram, + pub(crate) host_write_words: u64, + pub(crate) host_read_words: u64, + pub(crate) batches: Histogram, + pub(crate) batch_tile_pairs: Histogram, + pub(crate) flush: [u64; FLUSH_REASONS], + pub(crate) set_waits: u64, + pub(crate) set_wait_ns: u64, + pub(crate) command_generation_ns: u64, + pub(crate) submission_ns: u64, + /// Profiling maps are never raw-plane or composed-frame observations. + pub(crate) profile_maps: u64, + pub(crate) profile_bytes: u64, + pub(crate) profile_samples: u64, + pub(crate) profile_busy_tiles: u64, + pub(crate) profile_tile_pairs: u64, + pub(crate) profile_bin_ns: u64, + pub(crate) profile_raster_ns: u64, +} + +impl Workload { + pub(crate) fn record_draw( + &mut self, + sample: DrawSample, + coverage: Option, + noop: bool, + host_qwords: u64, + generation_ns: u64, + ) -> Result<(), &'static str> { + add!(self.accepted, 1); + if noop { add!(self.noops, 1); } + add!(self.primitive[sample.kind as usize], 1); + add!(self.address[sample.address as usize], 1); + for (index, count) in self.features.iter_mut().enumerate() { + if sample.features & (1 << index) != 0 { add!(*count, 1); } + } + if let Some(coverage) = coverage { + self.clipped_area.record(coverage.area())?; + self.clipped_tiles.record(coverage.tiles())?; + } + if sample.is_host_write() { + self.host_write_qwords.record(host_qwords)?; + add!(self.host_write_words, host_qwords.checked_mul(2).ok_or("HostW word count overflow")?); + } else if sample.is_host_read() { + self.host_read_requested_qwords.record(host_qwords)?; + } + add!(self.command_generation_ns, generation_ns); + Ok(()) + } + + pub(crate) fn record_host_read(&mut self, produced: u64, generation_ns: u64) -> Result<(), &'static str> { + self.host_read_produced_qwords.record(produced)?; + add!(self.host_read_words, produced.checked_mul(2).ok_or("HostR word count overflow")?); + add!(self.command_generation_ns, generation_ns); + Ok(()) + } + + pub(crate) fn record_flush( + &mut self, reason: FlushReason, commands: usize, tile_pairs: u64, ns: u64, + waits: u64, wait_ns: u64, + ) -> Result<(), &'static str> { + if commands == 0 { + return Ok(()); + } + self.batches.record(commands as u64)?; + self.batch_tile_pairs.record(tile_pairs)?; + add!(self.flush[reason as usize], 1); + add!(self.submission_ns, ns); + add!(self.set_waits, waits); + add!(self.set_wait_ns, wait_ns); + Ok(()) + } + + pub(crate) fn delta_since(&self, before: &Self) -> Result { + if self.profiling != before.profiling { + return Err("resident profiling was toggled"); + } + let mut delta = self.clone(); + subtract!(delta, before, accepted, noops, unsupported, host_write_words, host_read_words, + set_waits, set_wait_ns, command_generation_ns, submission_ns, profile_maps, + profile_bytes, profile_samples, profile_busy_tiles, profile_tile_pairs, profile_bin_ns, + profile_raster_ns); + for (a, b) in delta.primitive.iter_mut().zip(before.primitive) { + *a = a.checked_sub(b).ok_or("primitive count decreased")?; + } + for (a, b) in delta.address.iter_mut().zip(before.address) { + *a = a.checked_sub(b).ok_or("address-mode count decreased")?; + } + for (a, b) in delta.features.iter_mut().zip(before.features) { + *a = a.checked_sub(b).ok_or("mode-feature count decreased")?; + } + for (a, b) in delta.flush.iter_mut().zip(before.flush) { + *a = a.checked_sub(b).ok_or("flush count decreased")?; + } + delta.clipped_area = self.clipped_area.delta_since(&before.clipped_area)?; + delta.clipped_tiles = self.clipped_tiles.delta_since(&before.clipped_tiles)?; + delta.host_write_qwords = self.host_write_qwords.delta_since(&before.host_write_qwords)?; + delta.host_read_requested_qwords = self.host_read_requested_qwords.delta_since(&before.host_read_requested_qwords)?; + delta.host_read_produced_qwords = self.host_read_produced_qwords.delta_since(&before.host_read_produced_qwords)?; + delta.batches = self.batches.delta_since(&before.batches)?; + delta.batch_tile_pairs = self.batch_tile_pairs.delta_since(&before.batch_tile_pairs)?; + Ok(delta) + } +} + +/// Occupancy samples can race the consumer; reservation failures themselves +/// distinguish a full published ring from producer lock contention exactly. +#[derive(Clone, Debug, Default, PartialEq, Eq)] +pub(crate) struct Board { + pub(crate) occupancy: Histogram, + pub(crate) reservations: u64, + pub(crate) backpressure: u64, + pub(crate) full: u64, + pub(crate) bus_busy: u64, + pub(crate) cut_latency_ns: Histogram, + /// Render entry to successful presenter submission; not display scanout. + pub(crate) presentation_latency_ns: Histogram, + pub(crate) errors: bool, +} + +impl Board { + pub(crate) fn reservation(&mut self, occupancy: usize, result: Result<(), crate::gfifo::PushFailure>) { + let result = (|| { + self.occupancy.record(occupancy as u64)?; + add!(self.reservations, 1); + if let Err(failure) = result { + add!(self.backpressure, 1); + if failure == crate::gfifo::PushFailure::Full { add!(self.full, 1); } + } + Ok::<(), &'static str>(()) + })(); + self.errors |= result.is_err(); + } + + pub(crate) fn bus_busy(&mut self) { + match self.bus_busy.checked_add(1) { + Some(count) => self.bus_busy = count, + None => self.errors = true, + } + } + + pub(crate) fn cut(&mut self, duration_ns: u64) { + self.errors |= self.cut_latency_ns.record(duration_ns).is_err(); + } + + pub(crate) fn presentation(&mut self, duration_ns: u64) { + self.errors |= self.presentation_latency_ns.record(duration_ns).is_err(); + } + + pub(crate) fn delta_since(&self, before: &Self) -> Result { + if self.errors || before.errors { return Err("board telemetry overflow"); } + let mut delta = self.clone(); + subtract!(delta, before, reservations, backpressure, full, bus_busy); + delta.occupancy = self.occupancy.delta_since(&before.occupancy)?; + delta.cut_latency_ns = self.cut_latency_ns.delta_since(&before.cut_latency_ns)?; + delta.presentation_latency_ns = self.presentation_latency_ns.delta_since(&before.presentation_latency_ns)?; + Ok(delta) + } +} + +#[derive(Clone, Debug, Default, PartialEq, Eq)] +pub(crate) struct Traffic { + pub(crate) raw_attempts: [u64; 6], + pub(crate) raw_bytes: [u64; 6], + pub(crate) raw_maps: u64, + /// Direct and readback-compatibility composed captures, respectively. + pub(crate) composed_attempts: [u64; 2], + pub(crate) composed_bytes: [u64; 2], +} + +impl Traffic { + fn delta_since(&self, before: &Self) -> Result { + let mut delta = self.clone(); + subtract!(delta, before, raw_maps); + for (a, b) in delta.raw_attempts.iter_mut().zip(before.raw_attempts) { + *a = a.checked_sub(b).ok_or("raw observation attempts decreased")?; + } + for (a, b) in delta.raw_bytes.iter_mut().zip(before.raw_bytes) { + *a = a.checked_sub(b).ok_or("raw observation bytes decreased")?; + } + for (a, b) in delta.composed_attempts.iter_mut().zip(before.composed_attempts) { + *a = a.checked_sub(b).ok_or("composed capture attempts decreased")?; + } + for (a, b) in delta.composed_bytes.iter_mut().zip(before.composed_bytes) { + *a = a.checked_sub(b).ok_or("composed capture bytes decreased")?; + } + Ok(delta) + } +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub(crate) struct Snapshot { + pub(crate) epoch: u64, + pub(crate) workload: Workload, + pub(crate) board: Board, + pub(crate) traffic: Traffic, +} + +impl Snapshot { + pub(crate) fn delta_since(&self, before: &Self) -> Result { + if self.epoch != before.epoch { + return Err("resident telemetry epoch changed"); + } + Ok(Self { + epoch: self.epoch, + workload: self.workload.delta_since(&before.workload)?, + board: self.board.delta_since(&before.board)?, + traffic: self.traffic.delta_since(&before.traffic)?, + }) + } +} + +impl fmt::Display for Snapshot { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + let w = &self.workload; + let b = &self.board; + write!(f, + "resident epoch={} accepted={} noop={} unsupported={} \ + kind(draw/copy/hostw/hostr/noop)={:?} address(span/block/i/f/a/other)={:?} \ + features(blend/shade/z/line/dither/clear/scissor/cid/aux/colorhost/alphahost/packed/double/xy/yflip/compare)={:?} \ + batches={} commands={} batch_p50/p95={:?}/{:?} \ + flush(capacity/host-transition/copy/idle/wait/fence/cut/observe/hostr/explicit/exit/latency/restore/unsupported)={:?} \ + area_p50/p95={:?}/{:?} tiles_p50/p95={:?}/{:?} \ + hostw(qwords/words)={}/{} hostw_qwords_p50/p95={:?}/{:?} \ + hostr(requested/produced_qwords/words)={}/{}/{} hostr_qwords_p50/p95={:?}/{:?} \ + gfifo_max={} gfifo_p50/p95={:?}/{:?} backpressure={} full={} busy={} \ + cut_p50/p95_ns={:?}/{:?} present_p50/p95_ns={:?}/{:?} generation_ns={} submit_ns={} waits={} wait_ns={} \ + raw_maps={} raw_bytes(hostr/snapshot/monitor/recording/validation/compat)={:?} \ + composed_attempts(direct/compat)={:?} composed_bytes(direct/compat)={:?} \ + profiling={} profile_maps={} profile_bytes={} profile_samples={} \ + profile_bin_ns={} profile_raster_ns={} busy_tiles={} tile_pairs={}", + self.epoch, w.accepted, w.noops, w.unsupported, w.primitive, w.address, w.features, + w.batches.samples, w.batches.sum, w.batches.percentile_upper(50), w.batches.percentile_upper(95), w.flush, + w.clipped_area.percentile_upper(50), w.clipped_area.percentile_upper(95), + w.clipped_tiles.percentile_upper(50), w.clipped_tiles.percentile_upper(95), + w.host_write_qwords.sum, w.host_write_words, + w.host_write_qwords.percentile_upper(50), w.host_write_qwords.percentile_upper(95), + w.host_read_requested_qwords.sum, w.host_read_produced_qwords.sum, w.host_read_words, + w.host_read_produced_qwords.percentile_upper(50), w.host_read_produced_qwords.percentile_upper(95), + b.occupancy.high_water, b.occupancy.percentile_upper(50), b.occupancy.percentile_upper(95), + b.backpressure, b.full, b.bus_busy, + b.cut_latency_ns.percentile_upper(50), b.cut_latency_ns.percentile_upper(95), + b.presentation_latency_ns.percentile_upper(50), b.presentation_latency_ns.percentile_upper(95), + w.command_generation_ns, w.submission_ns, w.set_waits, w.set_wait_ns, self.traffic.raw_maps, + self.traffic.raw_bytes, self.traffic.composed_attempts, self.traffic.composed_bytes, + w.profiling, w.profile_maps, w.profile_bytes, w.profile_samples, + w.profile_bin_ns, w.profile_raster_ns, w.profile_busy_tiles, w.profile_tile_pairs) + } +} + +pub(crate) fn duration_ns(duration: std::time::Duration) -> Result { + u64::try_from(duration.as_nanos()).map_err(|_| "resident telemetry duration overflow") +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::dev::ng1::rex3::{DrawMode0, DrawMode1, DRAWMODE0_ADRMODE_BLOCK_SH}; + use crate::gfifo::PushFailure; + + #[test] + fn resident_histogram_percentiles_and_overflow_are_checked() { + let mut h = Histogram::default(); + for value in [0, 1, 2, 3, 10, 4_000_000_000] { h.record(value).unwrap(); } + assert_eq!(h.samples, 6); + assert_eq!(h.percentile_upper(50), Some(3)); + assert_eq!(h.percentile_upper(95), Some(4_000_000_000)); + assert_eq!(h.percentile_upper(0), None); + let before = h.clone(); + h.sum = u64::MAX; + let exhausted = h.clone(); + assert!(h.record(1).is_err()); + assert_eq!(h, exhausted, "a failed histogram update must be atomic"); + assert!(before.delta_since(&h).is_err()); + } + + #[test] + fn resident_typed_workload_records_transfers_and_batch_reasons() { + std::thread::Builder::new().stack_size(64 * 1024 * 1024).spawn(|| { + let mut context = Rex3Context::power_on_default(); + context.drawmode0 = DrawMode0(DRAWMODE0_OPCODE_DRAW | DRAWMODE0_ADRMODE_BLOCK_SH | (1 << 6) | (1 << 18)); + context.drawmode1 = DrawMode1(4 | (1 << 7) | (1 << 10) | (1 << 18)); + let sample = DrawSample::from_context(&context); + assert_eq!(sample.kind, PrimitiveKind::HostWrite); + assert_eq!(sample.address, AddressMode::Block); + let mut workload = Workload::default(); + let bounds = Coverage::new(7, 7, 8, 8).unwrap(); + workload.record_draw(sample, Some(bounds), false, 3, 17).unwrap(); + assert_eq!(workload.accepted, 1); + assert_eq!(workload.primitive[PrimitiveKind::HostWrite as usize], 1); + assert_eq!(workload.address[AddressMode::Block as usize], 1); + for feature in [ModeFeature::Blend, ModeFeature::Shade, ModeFeature::Auxiliary, + ModeFeature::ColorHost, ModeFeature::PackedHost, ModeFeature::DoubleHost] { + assert_eq!(workload.features[feature as usize], 1, "{feature:?}"); + } + assert_eq!((workload.clipped_area.sum, workload.clipped_tiles.sum), (4, 4)); + assert_eq!((workload.host_write_qwords.sum, workload.host_write_words), (3, 6)); + + context.drawmode0.0 = DRAWMODE0_OPCODE_READ | DRAWMODE0_ADRMODE_BLOCK_SH; + let read = DrawSample::from_context(&context); + assert_eq!(read.kind, PrimitiveKind::HostRead); + workload.record_draw(read, None, true, 4, 11).unwrap(); + workload.record_host_read(2, 13).unwrap(); + workload.record_flush(FlushReason::Idle, 0, 0, 5, 0, 0).unwrap(); + workload.record_flush(FlushReason::Latency, 1, 4, 19, 1, 7).unwrap(); + assert_eq!((workload.host_read_requested_qwords.sum, workload.host_read_produced_qwords.sum), (4, 2)); + assert_eq!((workload.host_read_words, workload.noops, workload.command_generation_ns), (4, 1, 41)); + assert_eq!((workload.batches.sum, workload.batch_tile_pairs.sum, workload.submission_ns), (1, 4, 19)); + assert_eq!((workload.set_waits, workload.set_wait_ns), (1, 7)); + assert_eq!(workload.flush[FlushReason::Idle as usize], 0); + assert_eq!(workload.flush[FlushReason::Latency as usize], 1); + }).unwrap().join().unwrap(); + } + + #[test] + fn resident_coverage_is_screen_clipped_and_conservative() { + let bounds = Coverage::new(-8, -5, 15, 8).unwrap(); + assert_eq!(bounds.area(), 16 * 9); + assert_eq!(bounds.tiles(), 4); + assert!(Coverage::new(-20, -20, -1, -1).is_none()); + let mut pixels = Coverage::from_address(2048 + 5); + pixels.include_address(3 * 2048 + 9); + assert_eq!((pixels.x0, pixels.y0, pixels.x1, pixels.y1), (5, 1, 9, 3)); + } + + #[test] + fn resident_snapshot_deltas_detect_epoch_decrease_and_profile_changes() { + let mut before = Snapshot { epoch: 7, workload: Workload::default(), board: Board::default(), traffic: Traffic::default() }; + before.workload.accepted = 3; + before.board.occupancy.record(8).unwrap(); + before.traffic.raw_maps = 2; + let mut after = before.clone(); + after.workload.accepted = 5; + after.board.occupancy.record(2).unwrap(); + after.traffic.raw_maps = 3; + let delta = after.delta_since(&before).unwrap(); + assert_eq!((delta.workload.accepted, delta.board.occupancy.samples, delta.traffic.raw_maps), (2, 1, 1)); + assert_eq!(delta.board.occupancy.high_water, 8, "delta maxima remain lifetime gauges"); + let report = delta.to_string(); + assert!(report.contains("gfifo_p50/p95=")); + assert!(report.contains("hostw(qwords/words)=")); + assert!(report.contains("profile_bin_ns=")); + after.workload.accepted = 2; + assert!(after.delta_since(&before).is_err()); + after.workload.accepted = 5; + after.epoch += 1; + assert!(after.delta_since(&before).is_err()); + after.epoch = before.epoch; + after.workload.profiling = true; + assert!(after.delta_since(&before).is_err()); + after.workload.profiling = false; + after.traffic.raw_maps = 1; + assert!(after.delta_since(&before).is_err()); + } + + #[test] + fn resident_board_counts_full_contention_busy_and_latency() { + let mut board = Board::default(); + board.reservation(4, Ok(())); + board.reservation(7, Err(PushFailure::Full)); + board.reservation(1, Err(PushFailure::Contended)); + board.bus_busy(); + board.cut(40); + board.presentation(200); + assert_eq!((board.reservations, board.backpressure, board.full, board.bus_busy), (3, 2, 1, 1)); + assert_eq!(board.occupancy.high_water, 7); + assert_eq!(board.cut_latency_ns.percentile_upper(95), Some(63)); + assert_eq!(board.presentation_latency_ns.percentile_upper(50), Some(255)); + board.bus_busy = u64::MAX; + board.bus_busy(); + assert!(board.errors); + assert!(board.delta_since(&Board::default()).is_err()); + } +} diff --git a/src/gfifo.rs b/src/gfifo.rs index d29e9c539..916771bfe 100644 --- a/src/gfifo.rs +++ b/src/gfifo.rs @@ -16,6 +16,12 @@ use crossbeam_utils::CachePadded; /// Address marker for the payload entries that follow a `push_batch` token. pub const GFIFO_PAYLOAD: u32 = 0xFFFF_0003; +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) enum PushFailure { + Contended, + Full, +} + /// A GFIFO entry: plain addr + val, no synchronization fields. /// Ordering is guaranteed by the push spinlock (producers) and the /// tail Release store / head Acquire load (consumer). @@ -219,11 +225,22 @@ impl GFifo { /// anyway. #[inline] pub fn try_push(&self, addr: u32, val: u64) -> bool { + self.push_one_result(addr, val).is_ok() + } + + #[cfg(test)] + #[inline] + pub(crate) fn try_push_traced(&self, addr: u32, val: u64) -> Result<(), PushFailure> { + self.push_one_result(addr, val) + } + + #[inline] + fn push_one_result(&self, addr: u32, val: u64) -> Result<(), PushFailure> { // Acquire the spinlock — uncontested in the common case (one active // producer: IRIX only drives DMA for pixmap blits, never while the CPU // is writing REX3 registers), so a failure here is rare. if self.lock.compare_exchange(false, true, Ordering::Acquire, Ordering::Relaxed).is_err() { - return false; + return Err(PushFailure::Contended); } let tail = self.tail.load(Ordering::Relaxed); let next_tail = tail.wrapping_add(1) & Self::MASK; @@ -235,7 +252,7 @@ impl GFifo { // Full — release the lock and let the caller retry once the // consumer has drained something. self.lock.store(false, Ordering::Release); - return false; + return Err(PushFailure::Full); } } // SAFETY: we hold the lock; no other producer touches this slot. @@ -245,7 +262,7 @@ impl GFifo { // Release: consumer's Acquire on tail sees the slot write above. self.tail.store(next_tail, Ordering::Release); self.lock.store(false, Ordering::Release); - true + Ok(()) } /// Push two consecutive register writes as one atomic unit. @@ -266,8 +283,19 @@ impl GFifo { /// pair either lands completely or not at all. #[inline] pub fn try_push2(&self, addr0: u32, val0: u64, addr1: u32, val1: u64) -> bool { + self.push_pair_result(addr0, val0, addr1, val1).is_ok() + } + + #[cfg(test)] + #[inline] + pub(crate) fn try_push2_traced(&self, addr0: u32, val0: u64, addr1: u32, val1: u64) -> Result<(), PushFailure> { + self.push_pair_result(addr0, val0, addr1, val1) + } + + #[inline] + fn push_pair_result(&self, addr0: u32, val0: u64, addr1: u32, val1: u64) -> Result<(), PushFailure> { if self.lock.compare_exchange(false, true, Ordering::Acquire, Ordering::Relaxed).is_err() { - return false; + return Err(PushFailure::Contended); } let tail = self.tail.load(Ordering::Relaxed); let next = tail.wrapping_add(1) & Self::MASK; @@ -280,7 +308,7 @@ impl GFifo { self.shadow_head.set(cached_head); if next == cached_head || next2 == cached_head { self.lock.store(false, Ordering::Release); - return false; + return Err(PushFailure::Full); } } // SAFETY: we hold the lock; no other producer touches these slots. @@ -292,7 +320,7 @@ impl GFifo { // orders it after every write above. self.tail.store(next2, Ordering::Release); self.lock.store(false, Ordering::Release); - true + Ok(()) } /// Push `entries` (address, value) as one unit: all of them or, when the @@ -343,10 +371,20 @@ impl GFifo { /// has no EXEC_RETRY, and a partial commit followed by a retry would /// duplicate the prefix. pub fn try_push_batch(&self, token: u32, count_val: u64, vals: &[u64]) -> bool { + self.push_batch_result(token, count_val, vals).is_ok() + } + + #[cfg(test)] + pub(crate) fn try_push_batch_traced(&self, token: u32, count_val: u64, vals: &[u64]) -> Result<(), PushFailure> { + self.push_batch_result(token, count_val, vals) + } + + #[inline] + fn push_batch_result(&self, token: u32, count_val: u64, vals: &[u64]) -> Result<(), PushFailure> { let need = vals.len() + 1; assert!(need < DEPTH, "batch of {} exceeds GFIFO capacity", vals.len()); if self.lock.compare_exchange(false, true, Ordering::Acquire, Ordering::Relaxed).is_err() { - return false; + return Err(PushFailure::Contended); } let tail = self.tail.load(Ordering::Relaxed); let room = |head: usize| head.wrapping_sub(tail).wrapping_sub(1) & Self::MASK; @@ -355,7 +393,7 @@ impl GFifo { self.shadow_head.set(head); if room(head) < need { self.lock.store(false, Ordering::Release); - return false; + return Err(PushFailure::Full); } } // SAFETY: we hold the lock and have verified capacity for all `need` @@ -372,7 +410,7 @@ impl GFifo { // One Release publishes the token and every payload slot. self.tail.store(new_tail, Ordering::Release); self.lock.store(false, Ordering::Release); - true + Ok(()) } /// Push a batch token followed by its payload words, waiting until the @@ -681,6 +719,25 @@ and pixel data is being lost", slot.addr, GFIFO_PAYLOAD); mod tests { use super::*; + #[test] + fn resident_gfifo_failure_reason_distinguishes_full_and_contention() { + let f: Box> = Box::new(GFifo::new()); + for i in 0..6 { + assert_eq!(f.try_push_traced(i, i as u64), Ok(())); + } + assert_eq!(f.try_push2_traced(10, 10, 11, 11), Err(PushFailure::Full)); + assert_eq!(f.try_push_batch_traced(12, 1, &[13]), Err(PushFailure::Full)); + assert_eq!(f.len(), 6, "failed multi-entry reservations must be atomic"); + assert_eq!(f.try_push_traced(6, 6), Ok(())); + assert_eq!(f.try_push_traced(7, 7), Err(PushFailure::Full)); + f.lock.store(true, Ordering::Release); + assert_eq!(f.try_push_traced(8, 8), Err(PushFailure::Contended)); + assert_eq!(f.try_push2_traced(8, 8, 9, 9), Err(PushFailure::Contended)); + assert_eq!(f.try_push_batch_traced(8, 1, &[9]), Err(PushFailure::Contended)); + f.lock.store(false, Ordering::Release); + assert_eq!(f.len(), 7); + } + #[test] fn every_head_sees_every_entry() { let f: Box> = Box::new(GFifo::new()); diff --git a/src/machine.rs b/src/machine.rs index 1aca39ae8..bcc916e9a 100644 --- a/src/machine.rs +++ b/src/machine.rs @@ -173,6 +173,9 @@ pub struct Machine { lifecycle: GuestState, } +#[cfg(all(test, feature = "wgpu"))] +type ResidentCheckpoint = (toml::Value, (Vec, Vec)); + /// In-memory snapshot of the just-restored guest state. Populated at the end /// of `ci_restore`; consumed by `ci_rollback`. Trades ~270 MB of RSS for /// disk-IO-free rollback. @@ -193,6 +196,8 @@ struct RollbackCheckpoint { /// Framebuffer contents (RGB, aux). `None` when running headless. framebuffers: Option<(Vec, Vec)>, + #[cfg(all(test, feature = "wgpu"))] + resident_head1_framebuffers: Option<(Vec, Vec)>, /// Parsed device save_state TOMLs. Holding `toml::Value` directly skips /// the ~80 ms cpu.toml string-parse cost on every rollback. @@ -269,6 +274,8 @@ pub(crate) fn parse_cow_table(cow_toml: &toml::Value) -> Vec<(usize, Vec)> pub(crate) struct LiveCheckpoint { bank_words: [Vec; 4], framebuffers: Option<(Vec, Vec)>, + #[cfg(all(test, feature = "wgpu"))] + resident_head1_framebuffers: Option<(Vec, Vec)>, cpu: toml::Value, mc: toml::Value, ioc: toml::Value, @@ -1055,12 +1062,17 @@ impl Machine { } else { self.display_resolution }; - if let Some(rex3) = &self.board.rex3 { - rex3.apply_display_resolution(mode); - } - if let Some(rex3) = &self.board.rex3_head1 { + let apply = |rex3: &Rex3| { + #[cfg(all(test, feature = "wgpu"))] + if rex3.resident_active() { + rex3.apply_resident_display_resolution(mode, true) + .unwrap_or_else(|error| panic!("REX3 {error}")); + return; + } rex3.apply_display_resolution(mode); - } + }; + if let Some(rex3) = &self.board.rex3 { apply(rex3); } + if let Some(rex3) = &self.board.rex3_head1 { apply(rex3); } } /// Path of the configured scratch SCSI volume, if any. Used by the CI @@ -1516,10 +1528,53 @@ impl Machine { result } + #[cfg(all(test, feature = "wgpu"))] + fn capture_resident_checkpoint(rex: Option<&Rex3>) -> Result, String> { + rex.filter(|rex| rex.resident_active()) + .map(|rex| rex.snapshot_resident_checkpoint().map_err(|error| error.to_string())) + .transpose() + } + + #[cfg(all(test, feature = "wgpu"))] + fn validate_resident_primary_checkpoint(&self, registers: bool, planes: bool) -> Result<(), String> { + if self.board.rex3.as_ref().is_some_and(|rex| rex.resident_active()) && (!registers || !planes) { + return Err("resident REX3 checkpoint requires registers and both planes".into()); + } + Ok(()) + } + + #[cfg(all(test, feature = "wgpu"))] + fn validate_resident_head1_checkpoint(&self, registers: bool, planes: bool) -> Result<(), String> { + if self.board.rex3_head1.as_ref().is_some_and(|rex| rex.resident_active()) && (!registers || !planes) { + return Err("resident REX3 head 1 checkpoint requires registers and both planes".into()); + } + Ok(()) + } + + #[cfg(all(test, feature = "wgpu"))] + fn restore_resident_head1_framebuffers( + &self, + has_registers: bool, + planes: Option<&(Vec, Vec)>, + ) -> Result<(), String> { + if let Some(rex) = self.board.rex3_head1.as_ref().filter(|rex| rex.resident_active()) { + if !has_registers { + return Err("resident REX3 head 1 checkpoint has no registers".into()); + } + let (rgb, aux) = planes.ok_or("resident REX3 head 1 checkpoint has no planes")?; + rex.restore_framebuffers_inmem(rgb, aux); + } + Ok(()) + } + /// Capture in-memory state for fast rollback. Stops the CPU briefly. fn capture_rollback_checkpoint(&mut self, name: &str) -> Result { self.stop(); + #[cfg(all(test, feature = "wgpu"))] + let resident_primary = Self::capture_resident_checkpoint(self.board.rex3.as_deref())?; + #[cfg(all(test, feature = "wgpu"))] + let resident_head1 = Self::capture_resident_checkpoint(self.board.rex3_head1.as_deref())?; let cpu = self.cpu.save_state(); let mc = self.mc.save_state(); let ioc = self.hpc3.ioc().save_state(); @@ -1532,8 +1587,16 @@ impl Machine { let scsi1 = self.hpc3.scsi1().map(|dev| dev.save_state()); let seeq = self.hpc3.seeq().save_state(); let hpc3 = self.hpc3.save_state(); - let rex3 = self.board.rex3.as_ref().map(|r| r.save_state()); - let rex3_head1 = self.board.rex3_head1.as_ref().map(|r| r.save_state()); + let rex3 = self.board.rex3.as_ref().map(|r| { + #[cfg(all(test, feature = "wgpu"))] + if let Some((state, _)) = &resident_primary { return state.clone(); } + r.save_state() + }); + let rex3_head1 = self.board.rex3_head1.as_ref().map(|r| { + #[cfg(all(test, feature = "wgpu"))] + if let Some((state, _)) = &resident_head1 { return state.clone(); } + r.save_state() + }); let bank_words: [Vec; 4] = [ self.board.phys.snapshot_bank_inmem(0), @@ -1542,8 +1605,15 @@ impl Machine { self.board.phys.snapshot_bank_inmem(3), ]; - let framebuffers = self.board.rex3.as_ref() - .map(|r| r.snapshot_framebuffers_inmem()); + #[cfg(all(test, feature = "wgpu"))] + let framebuffers = match resident_primary { + Some((_, planes)) => Some(planes), + None => self.board.rex3.as_ref().map(|r| r.snapshot_framebuffers_inmem()), + }; + #[cfg(not(all(test, feature = "wgpu")))] + let framebuffers = self.board.rex3.as_ref().map(|r| r.snapshot_framebuffers_inmem()); + #[cfg(all(test, feature = "wgpu"))] + let resident_head1_framebuffers = resident_head1.map(|(_, planes)| planes); // Re-read cow.toml (and cow1.toml, for controller 1) so rollback // knows which dirty sectors to import back. The files were just @@ -1563,6 +1633,8 @@ impl Machine { overlay_sets1, bank_words, framebuffers, + #[cfg(all(test, feature = "wgpu"))] + resident_head1_framebuffers, cpu, mc, ioc, scc, pit, ps2, rtc, eeprom, scsi, scsi1, seeq, hpc3, rex3, rex3_head1, }) } @@ -1571,6 +1643,10 @@ impl Machine { /// the moment of capture. Skips disk IO and TOML string-parsing. fn apply_rollback_checkpoint(&mut self, cp: &RollbackCheckpoint) -> Result<(), String> { self.stop(); + #[cfg(all(test, feature = "wgpu"))] + self.validate_resident_primary_checkpoint(cp.rex3.is_some(), cp.framebuffers.is_some())?; + #[cfg(all(test, feature = "wgpu"))] + self.validate_resident_head1_checkpoint(cp.rex3_head1.is_some(), cp.resident_head1_framebuffers.is_some())?; self.power_on_devices(); self.cpu.load_state(&cp.cpu)?; @@ -1600,6 +1676,8 @@ impl Machine { if let (Some(rex3), Some((rgb, aux))) = (&self.board.rex3, &cp.framebuffers) { rex3.restore_framebuffers_inmem(rgb, aux); } + #[cfg(all(test, feature = "wgpu"))] + self.restore_resident_head1_framebuffers(cp.rex3_head1.is_some(), cp.resident_head1_framebuffers.as_ref())?; // Reflink the overlay back into place. saves//scsi*.overlay is // unchanged by guest writes (writes go to the live overlay), so this @@ -1631,6 +1709,12 @@ impl Machine { pub(crate) fn capture_live_checkpoint(&mut self) -> LiveCheckpoint { self.stop(); + #[cfg(all(test, feature = "wgpu"))] + let resident_primary = Self::capture_resident_checkpoint(self.board.rex3.as_deref()) + .unwrap_or_else(|error| panic!("REX3 {error}")); + #[cfg(all(test, feature = "wgpu"))] + let resident_head1 = Self::capture_resident_checkpoint(self.board.rex3_head1.as_deref()) + .unwrap_or_else(|error| panic!("REX3 {error}")); let cpu = self.cpu.save_state(); let mc = self.mc.save_state(); let ioc = self.hpc3.ioc().save_state(); @@ -1641,8 +1725,16 @@ impl Machine { let eeprom = self.hpc3.eeprom().lock().save_state_owned(); let seeq = self.hpc3.seeq().save_state(); let hpc3 = self.hpc3.save_state(); - let rex3 = self.board.rex3.as_ref().map(|r| r.save_state()); - let rex3_head1 = self.board.rex3_head1.as_ref().map(|r| r.save_state()); + let rex3 = self.board.rex3.as_ref().map(|r| { + #[cfg(all(test, feature = "wgpu"))] + if let Some((state, _)) = &resident_primary { return state.clone(); } + r.save_state() + }); + let rex3_head1 = self.board.rex3_head1.as_ref().map(|r| { + #[cfg(all(test, feature = "wgpu"))] + if let Some((state, _)) = &resident_head1 { return state.clone(); } + r.save_state() + }); let bank_words: [Vec; 4] = [ self.board.phys.snapshot_bank_inmem(0), @@ -1651,8 +1743,15 @@ impl Machine { self.board.phys.snapshot_bank_inmem(3), ]; - let framebuffers = self.board.rex3.as_ref() - .map(|r| r.snapshot_framebuffers_inmem()); + #[cfg(all(test, feature = "wgpu"))] + let framebuffers = match resident_primary { + Some((_, planes)) => Some(planes), + None => self.board.rex3.as_ref().map(|r| r.snapshot_framebuffers_inmem()), + }; + #[cfg(not(all(test, feature = "wgpu")))] + let framebuffers = self.board.rex3.as_ref().map(|r| r.snapshot_framebuffers_inmem()); + #[cfg(all(test, feature = "wgpu"))] + let resident_head1_framebuffers = resident_head1.map(|(_, planes)| planes); // Deliberately NOT restart_peripherals()/cpu.start() here — see this // function's own doc comment. @@ -1660,6 +1759,8 @@ impl Machine { LiveCheckpoint { bank_words, framebuffers, + #[cfg(all(test, feature = "wgpu"))] + resident_head1_framebuffers, cpu, mc, ioc, scc, pit, ps2, rtc, eeprom, seeq, hpc3, rex3, rex3_head1, } } @@ -1679,6 +1780,10 @@ impl Machine { /// stayed unresponsive after every `jitcheck` for no reason). pub(crate) fn restore_live_checkpoint(&mut self, cp: &LiveCheckpoint) -> Result<(), String> { self.stop(); + #[cfg(all(test, feature = "wgpu"))] + self.validate_resident_primary_checkpoint(cp.rex3.is_some(), cp.framebuffers.is_some())?; + #[cfg(all(test, feature = "wgpu"))] + self.validate_resident_head1_checkpoint(cp.rex3_head1.is_some(), cp.resident_head1_framebuffers.is_some())?; self.cpu.load_state(&cp.cpu)?; self.mc.load_state(&cp.mc)?; @@ -1703,6 +1808,8 @@ impl Machine { if let (Some(rex3), Some((rgb, aux))) = (&self.board.rex3, &cp.framebuffers) { rex3.restore_framebuffers_inmem(rgb, aux); } + #[cfg(all(test, feature = "wgpu"))] + self.restore_resident_head1_framebuffers(cp.rex3_head1.is_some(), cp.resident_head1_framebuffers.as_ref())?; // Peripherals back up; CPU deliberately left stopped — see this // function's own doc comment. @@ -1781,9 +1888,24 @@ impl Machine { pub fn save_snapshot(&mut self, name: &str) -> Result<(), String> { self.stop(); + #[cfg(all(test, feature = "wgpu"))] + let mut resident_primary = Self::capture_resident_checkpoint(self.board.rex3.as_deref())?; + #[cfg(all(test, feature = "wgpu"))] + let resident_head1 = Self::capture_resident_checkpoint(self.board.rex3_head1.as_deref())?; let dir = std::path::PathBuf::from("saves").join(name); let snap = Snapshot::new(&dir); snap.ensure_dir().map_err(|e| e.to_string())?; + #[cfg(all(test, feature = "wgpu"))] + let incomplete = dir.join("resident_snapshot_incomplete"); + #[cfg(all(test, feature = "wgpu"))] + let remove_incomplete = { + let resident = resident_primary.is_some() || resident_head1.is_some(); + if resident { + std::fs::write(&incomplete, b"resident machine snapshot is incomplete\n") + .map_err(|e| e.to_string())?; + } + resident || incomplete.try_exists().map_err(|e| e.to_string())? + }; // Write the manifest first so `read_manifest` succeeds even if a later // step crashes — the partial snapshot is at least diagnosable. @@ -1822,8 +1944,23 @@ impl Machine { // included in the chunks manifest below for v3+; v2 wrote them as // standalone .bin files. if let Some(rex3) = &self.board.rex3 { - snap.write_state("rex3", &rex3.save_state(), sv).map_err(|e| e.to_string())?; + #[cfg(all(test, feature = "wgpu"))] + let resident_state = resident_primary.as_ref().map(|(state, _)| state); + #[cfg(not(all(test, feature = "wgpu")))] + let resident_state: Option<&toml::Value> = None; + match resident_state { + Some(state) => snap.write_state("rex3", state, sv), + None => snap.write_state("rex3", &rex3.save_state(), sv), + }.map_err(|e| e.to_string())?; if sv < 3 { + #[cfg(all(test, feature = "wgpu"))] + if let Some((_, (rgb, aux))) = &resident_primary { + Rex3::write_captured_resident_framebuffers(&snap.dir, "rex3", rgb, aux) + .map_err(|e| e.to_string())?; + } else { + rex3.save_framebuffers(&snap.dir).map_err(|e| e.to_string())?; + } + #[cfg(not(all(test, feature = "wgpu")))] rex3.save_framebuffers(&snap.dir).map_err(|e| e.to_string())?; } } @@ -1831,9 +1968,30 @@ impl Machine { snap.write_state("testdev", &td.save_state(), sv).map_err(|e| e.to_string())?; } if let Some(rex3) = &self.board.rex3_head1 { - snap.write_state("rex3_head1", &rex3.save_state(), sv).map_err(|e| e.to_string())?; - if sv < 3 { + #[cfg(all(test, feature = "wgpu"))] + let resident_state = resident_head1.as_ref().map(|(state, _)| state); + #[cfg(not(all(test, feature = "wgpu")))] + let resident_state: Option<&toml::Value> = None; + match resident_state { + Some(state) => snap.write_state("rex3_head1", state, sv), + None => snap.write_state("rex3_head1", &rex3.save_state(), sv), + }.map_err(|e| e.to_string())?; + // The v3 chunk manifest covers only the primary head. Keep the + // gated second head in the existing named-plane format. + #[cfg(all(test, feature = "wgpu"))] + let save_planes = sv < 3 || rex3.resident_active(); + #[cfg(not(all(test, feature = "wgpu")))] + let save_planes = sv < 3; + if save_planes { let dir = &snap.dir; + #[cfg(all(test, feature = "wgpu"))] + if let Some((_, (rgb, aux))) = &resident_head1 { + Rex3::write_captured_resident_framebuffers(dir, "rex3_head1", rgb, aux) + .map_err(|e| e.to_string())?; + } else { + rex3.save_framebuffers_named(dir, "rex3_head1").map_err(|e| e.to_string())?; + } + #[cfg(not(all(test, feature = "wgpu")))] rex3.save_framebuffers_named(dir, "rex3_head1").map_err(|e| e.to_string())?; } } @@ -1857,7 +2015,11 @@ impl Machine { .map_err(|e| format!("CAS bank{} put: {}", i, e))?; } if let Some(rex3) = &self.board.rex3 { - let (rgb, aux) = rex3.snapshot_framebuffers_inmem(); + #[cfg(all(test, feature = "wgpu"))] + let planes = resident_primary.take().map(|(_, planes)| planes); + #[cfg(not(all(test, feature = "wgpu")))] + let planes: Option<(Vec, Vec)> = None; + let (rgb, aux) = planes.unwrap_or_else(|| rex3.snapshot_framebuffers_inmem()); let rgb_chunks = put_words_as_chunks(&store, &rgb) .map_err(|e| format!("CAS rex3 rgb put: {}", e))?; let aux_chunks = put_words_as_chunks(&store, &aux) @@ -1889,6 +2051,15 @@ impl Machine { .map_err(|e| e.to_string())?; } + #[cfg(all(test, feature = "wgpu"))] + if remove_incomplete { + for rex3 in [self.board.rex3.as_ref(), self.board.rex3_head1.as_ref()].into_iter().flatten() { + if rex3.resident_active() { + rex3.resident_status().map_err(|error| error.to_string())?; + } + } + std::fs::remove_file(incomplete).map_err(|e| e.to_string())?; + } self.restart_peripherals(); // Resume execution so the session feels like it never paused. // Without this the user sees JIT shutdown stats and a dead prompt @@ -1934,6 +2105,12 @@ impl Machine { fn load_snapshot_inner(&mut self, name: &str) -> Result<(), String> { self.stop(); + let dir = std::path::PathBuf::from("saves").join(name); + #[cfg(all(test, feature = "wgpu"))] + if dir.join("resident_snapshot_incomplete").try_exists().map_err(|e| e.to_string())? { + return Err("resident machine snapshot is incomplete".into()); + } + // Any prior in-memory rollback checkpoint is now stale (it described // a different snapshot). ci_restore will recapture if reached via // that path; the monitor `load` command leaves it cleared. @@ -1942,7 +2119,6 @@ impl Machine { // Reset to clean state before loading self.power_on_devices(); - let dir = std::path::PathBuf::from("saves").join(name); let snap = Snapshot::new(&dir); // Validate the manifest before reading anything else. Legacy snapshots @@ -2101,9 +2277,20 @@ impl Machine { } } if let Some(rex3) = &self.board.rex3_head1 { - if let Ok(rex3_v) = snap.read_state("rex3_head1", schema_version) { + let head1_state = snap.read_state("rex3_head1", schema_version); + #[cfg(all(test, feature = "wgpu"))] + if rex3.resident_active() { + if let Err(error) = &head1_state { + return Err(format!("resident REX3 head 1 state: {error}")); + } + } + if let Ok(rex3_v) = head1_state { rex3.load_state(&rex3_v)?; - if schema_version < 3 { + #[cfg(all(test, feature = "wgpu"))] + let load_planes = schema_version < 3 || rex3.resident_active(); + #[cfg(not(all(test, feature = "wgpu")))] + let load_planes = schema_version < 3; + if load_planes { rex3.load_framebuffers_named(&snap.dir, "rex3_head1").map_err(|e| e.to_string())?; } } @@ -2125,6 +2312,8 @@ impl Machine { let store = ChunkStore::new("saves"); let chunks = snap.read_chunks_manifest() .map_err(|e| format!("read chunks.bin: {}", e))?; + #[cfg(all(test, feature = "wgpu"))] + self.validate_resident_primary_checkpoint(true, chunks.framebuffer_chunks.is_some())?; for (i, hashes) in chunks.bank_chunks.iter().enumerate() { if hashes.is_empty() { continue; } let words = get_chunks_as_words(&store, hashes) diff --git a/src/wgpu/batch.rs b/src/wgpu/batch.rs index beedfdcaa..12b615c82 100644 --- a/src/wgpu/batch.rs +++ b/src/wgpu/batch.rs @@ -191,6 +191,7 @@ pub struct Timing { resolve: wgpu::Buffer, read: wgpu::Buffer, period_ns: f64, + inspect_dispatch: bool, pub batches: u64, pub prims: u64, pub bin_ns: f64, @@ -200,6 +201,8 @@ pub struct Timing { pub tiles_max: u32, /// Tile-primitive pairs the bin pass listed, summed over batches. pub pairs: u64, + /// A failed map, invalid timestamp or exhausted profiling counter. + pub failure: Option<&'static str>, /// Batches by raster time: under 0.1, 0.3, 1, 3, 10 ms and over. pub raster_hist: [u64; 6], /// `WGPU_BATCH_LOG=file`: one line a batch (prims, busy tiles, pairs, @@ -263,14 +266,22 @@ impl Clocks { impl Timing { fn new(gpu: &Gpu) -> Option { - if std::env::var("WGPU_BATCH_TIMING").is_err() || !gpu.device.features().contains(wgpu::Features::TIMESTAMP_QUERY) { + if std::env::var("WGPU_BATCH_TIMING").is_err() { + return None; + } + Self::new_explicit(gpu, false) + } + + fn new_explicit(gpu: &Gpu, inspect_dispatch: bool) -> Option { + if !gpu.device.features().contains(wgpu::Features::TIMESTAMP_QUERY) { return None; } let queries = gpu.device.create_query_set(&wgpu::QuerySetDescriptor { label: Some("batch timing"), ty: wgpu::QueryType::Timestamp, count: 4 }); let resolve = gpu.buffer("batch timing resolve", 32, wgpu::BufferUsages::QUERY_RESOLVE | wgpu::BufferUsages::COPY_SRC); - let read = gpu.buffer("batch timing read", 48, wgpu::BufferUsages::MAP_READ | wgpu::BufferUsages::COPY_DST); + let read = gpu.buffer("batch timing read", if inspect_dispatch { 64 } else { 48 }, wgpu::BufferUsages::MAP_READ | wgpu::BufferUsages::COPY_DST); let period_ns = gpu.queue.get_timestamp_period() as f64; - Some(Timing { queries, resolve, read, period_ns, batches: 0, prims: 0, bin_ns: 0.0, raster_ns: 0.0, tiles: 0, tiles_max: 0, pairs: 0, raster_hist: [0; 6], log: std::env::var("WGPU_BATCH_LOG").ok().and_then(|f| std::fs::File::create(f).ok()).map(std::io::LineWriter::new), clocks: Clocks::of(gpu), clock_ns: 0.0, gfx_ns_mhz: 0.0, mem_ns_mhz: 0.0 }) + let log = (!inspect_dispatch).then(|| std::env::var("WGPU_BATCH_LOG").ok().and_then(|f| std::fs::File::create(f).ok()).map(std::io::LineWriter::new)).flatten(); + Some(Timing { queries, resolve, read, period_ns, inspect_dispatch, batches: 0, prims: 0, bin_ns: 0.0, raster_ns: 0.0, tiles: 0, tiles_max: 0, pairs: 0, failure: None, raster_hist: [0; 6], log, clocks: Clocks::of(gpu), clock_ns: 0.0, gfx_ns_mhz: 0.0, mem_ns_mhz: 0.0 }) } fn writes(&self, first: u32) -> Option> { @@ -282,31 +293,79 @@ impl Timing { let slice = self.read.slice(..); slice.map_async(wgpu::MapMode::Read, |_| {}); if gpu.wait(Some(submission)).is_err() { + self.failure = Some("profiling submission wait failed"); + self.read.unmap(); return; } - let _ = gpu.wait(None); - let (t, busy) = { + if gpu.wait(None).is_err() { + self.failure = Some("profiling map wait failed"); + self.read.unmap(); + return; + } + let (t, busy, args) = { let Ok(v) = slice.get_mapped_range() else { + self.failure = Some("profiling map failed"); self.read.unmap(); return; }; let t: [u64; 4] = bytemuck::pod_read_unaligned(&v[0..32]); let head: BusyHead = bytemuck::pod_read_unaligned(&v[32..48]); - (t, head) + let args = self.inspect_dispatch.then(|| bytemuck::pod_read_unaligned::(&v[48..64])); + (t, head, args) }; self.read.unmap(); - let ns = |a: u64, b: u64| b.saturating_sub(a) as f64 * self.period_ns; + if let Some(args) = args { + let expected_x = busy.count.min(crate::wgpu::tiles::MAX_GROUPS); + let expected_y = if busy.count == 0 { 1 } else { + (busy.count - 1) / crate::wgpu::tiles::MAX_GROUPS + 1 + }; + if args.x != expected_x || args.y != expected_y || args.z != 1 { + self.failure = Some("GPU-generated indirect dispatch does not match busy tiles"); + return; + } + } + if t[1] < t[0] || t[3] < t[2] || !self.period_ns.is_finite() || self.period_ns <= 0.0 { + self.failure = Some("profiling timestamps decreased or have an invalid period"); + return; + } + let ns = |a: u64, b: u64| (b - a) as f64 * self.period_ns; let raster = ns(t[2], t[3]); - self.batches += 1; - self.prims += prims as u64; - self.bin_ns += ns(t[0], t[1]); - self.raster_ns += raster; - self.tiles += busy.count as u64; - self.tiles_max = self.tiles_max.max(busy.count); - self.pairs += busy.pairs as u64; let ms = raster / 1e6; let i = [0.1, 0.3, 1.0, 3.0, 10.0].iter().position(|&l| ms < l).unwrap_or(5); - self.raster_hist[i] += 1; + let Some(batches) = self.batches.checked_add(1) else { + self.failure = Some("profiling batch counter overflow"); + return; + }; + let Some(total_prims) = self.prims.checked_add(prims as u64) else { + self.failure = Some("profiling primitive counter overflow"); + return; + }; + let Some(tiles) = self.tiles.checked_add(busy.count as u64) else { + self.failure = Some("profiling tile counter overflow"); + return; + }; + let Some(pairs) = self.pairs.checked_add(busy.pairs as u64) else { + self.failure = Some("profiling pair counter overflow"); + return; + }; + let Some(bucket) = self.raster_hist[i].checked_add(1) else { + self.failure = Some("profiling histogram overflow"); + return; + }; + let bin_ns = self.bin_ns + ns(t[0], t[1]); + let raster_ns = self.raster_ns + raster; + if !bin_ns.is_finite() || !raster_ns.is_finite() { + self.failure = Some("profiling duration overflow"); + return; + } + self.batches = batches; + self.prims = total_prims; + self.bin_ns = bin_ns; + self.raster_ns = raster_ns; + self.tiles = tiles; + self.tiles_max = self.tiles_max.max(busy.count); + self.pairs = pairs; + self.raster_hist[i] = bucket; let (gfx, mem) = self.clocks.as_ref().and_then(|c| c.read()).unwrap_or((0.0, 0.0)); if gfx != 0.0 { self.clock_ns += raster; @@ -361,6 +420,13 @@ enum SetChoice { Wait(usize), } +#[cfg(test)] +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub(crate) struct BatchWaitStats { + pub(crate) waits: u64, + pub(crate) nanoseconds: u64, +} + /// Choose a free set in rotation, grow if permitted, otherwise wait for /// the oldest submission. `state` returns (available, submission order). fn choose_set( @@ -484,13 +550,17 @@ pub struct Batch { pub prims_total: u64, /// GPU times, with WGPU_BATCH_TIMING. pub timing: Option, + #[cfg(test)] + wait_stats: BatchWaitStats, + #[cfg(test)] + wait_probe: Option>, _p: PhantomData<(R, S)>, } impl Batch { /// A recorder over `grid` with room for `max_slots` slots a batch. pub fn new(gpu: &Arc, pipelines: &BatchPipelines, grid: TileGrid, max_slots: u32) -> Batch { - Self::with_set_policy(gpu, pipelines, grid, max_slots, SetPolicy::Grow) + Self::with_set_policy(gpu, pipelines, grid, max_slots, SetPolicy::Grow, true) } /// Like `new`, but cap the reusable sets. Once all are in flight, the @@ -503,7 +573,30 @@ impl Batch { max_slots: u32, max_sets: NonZeroUsize, ) -> Batch { - Self::with_set_policy(gpu, pipelines, grid, max_slots, SetPolicy::Bounded(max_sets)) + Self::with_set_policy(gpu, pipelines, grid, max_slots, SetPolicy::Bounded(max_sets), true) + } + + /// Resident work must not inherit the shared batch profiler from the + /// environment: enabling its map-and-wait path requires an explicit opt-in. + #[cfg(test)] + pub(crate) fn new_bounded_unprofiled( + gpu: &Arc, + pipelines: &BatchPipelines, + grid: TileGrid, + max_slots: u32, + max_sets: NonZeroUsize, + ) -> Batch { + Self::with_set_policy(gpu, pipelines, grid, max_slots, SetPolicy::Bounded(max_sets), false) + } + + #[cfg(test)] + pub(crate) fn enable_profiling(&mut self) -> Result<(), String> { + if !self.is_empty() || self.timing.is_some() { + return Err("batch profiling must be enabled once, between batches".into()); + } + self.timing = Some(Timing::new_explicit(&self.gpu, true) + .ok_or("batch profiling requires timestamp queries")?); + Ok(()) } fn with_set_policy( @@ -512,6 +605,7 @@ impl Batch { grid: TileGrid, max_slots: u32, set_policy: SetPolicy, + environment_timing: bool, ) -> Batch { let initial_sets = match set_policy { SetPolicy::Grow => BATCH_SETS, @@ -521,7 +615,7 @@ impl Batch { let storage = wgpu::BufferUsages::STORAGE | wgpu::BufferUsages::COPY_DST; let list_buf = gpu.buffer("batch tile lists", LIST_CAP as u64 * 4, storage); let busy_buf = gpu.buffer("batch busy", 16 + tiles as u64 * BUSY_ENTRY, storage | wgpu::BufferUsages::COPY_SRC); - let args_buf = gpu.buffer("batch dispatch args", 16, storage | wgpu::BufferUsages::INDIRECT); + let args_buf = gpu.buffer("batch dispatch args", 16, storage | wgpu::BufferUsages::INDIRECT | wgpu::BufferUsages::COPY_SRC); let mut b = Batch { gpu: Arc::clone(gpu), grid, @@ -536,7 +630,11 @@ impl Batch { last_set: 0, flushes: 0, prims_total: 0, - timing: Timing::new(gpu), + timing: if environment_timing { Timing::new(gpu) } else { None }, + #[cfg(test)] + wait_stats: BatchWaitStats::default(), + #[cfg(test)] + wait_probe: None, _p: PhantomData, }; for _ in 0..initial_sets { @@ -620,17 +718,35 @@ impl Batch { } SetChoice::Wait(k) => { let start = Instant::now(); - loop { + #[cfg(test)] + { + self.wait_stats.waits = self.wait_stats.waits.checked_add(1) + .ok_or("wgpu bounded set wait counter overflow")?; + if let Some(probe) = self.wait_probe.take() { + probe.send(()).map_err(|_| "wgpu bounded set wait test receiver closed")?; + } + } + let result = loop { self.gpu.instance.poll_all(false); - self.gpu.check()?; + if let Err(error) = self.gpu.check() { + break Err(error); + } if self.sets[k].done.load(Ordering::Acquire) { - break; + break Ok(()); } if start.elapsed() >= SET_WAIT_TIMEOUT { - return Err("timed out waiting for the oldest wgpu batch set".into()); + break Err("timed out waiting for the oldest wgpu batch set".into()); } std::thread::sleep(Duration::from_micros(100)); + }; + #[cfg(test)] + { + let elapsed = u64::try_from(start.elapsed().as_nanos()) + .map_err(|_| "wgpu bounded set wait duration overflow")?; + self.wait_stats.nanoseconds = self.wait_stats.nanoseconds.checked_add(elapsed) + .ok_or("wgpu bounded set wait duration overflow")?; } + result?; k } }; @@ -672,6 +788,29 @@ impl Batch { self.recording.slots.len() } + #[cfg(test)] + pub(crate) fn tile_pair_bound(&self) -> u64 { + u64::from(self.recording.pairs_bound) + } + + #[cfg(test)] + pub(crate) fn wait_stats(&self) -> BatchWaitStats { + self.wait_stats + } + + #[cfg(test)] + pub(crate) fn force_wait_for_test( + &mut self, + probe: std::sync::mpsc::Sender<()>, + ) -> Result, &'static str> { + if self.sets.len() != 1 || self.claimed.is_some() || !self.sets[0].done.load(Ordering::Acquire) { + return Err("test requires one completed, unclaimed batch set"); + } + self.sets[0].done.store(false, Ordering::Release); + self.wait_probe = Some(probe); + Ok(Arc::clone(&self.sets[0].done)) + } + /// Append a state slot; its index for records that follow. pub fn slot(&mut self, s: S) -> Result { self.recording.slot(s) @@ -777,6 +916,9 @@ impl Batch { enc.resolve_query_set(&t.queries, 0..4, &t.resolve, 0); enc.copy_buffer_to_buffer(&t.resolve, 0, &t.read, 0, 32); enc.copy_buffer_to_buffer(&self.busy_buf, 0, &t.read, 32, 16); + if t.inspect_dispatch { + enc.copy_buffer_to_buffer(&self.args_buf, 0, &t.read, 48, 16); + } } set.done.store(false, Ordering::Release); let submission = self.gpu.submit(enc);