diff --git a/HACKING.md b/HACKING.md index ffa1e90f2..c35613fa5 100644 --- a/HACKING.md +++ b/HACKING.md @@ -68,7 +68,8 @@ the monitor, the stored lifecycle state); its provided methods are either shared logic every machine wants the same way (`status`, and the idempotent `start_guest`/`stop_guest` behind `machine-start`/`machine-stop`, see `src/machine_state.rs`) or optional facilities (snapshots, CHD folding, the -scratch volume, disc changes, the NAT controls) that default to "not +scratch volume, disc changes, the NAT controls, the `disk_activity` and +`audio_frames` counters behind a front-end's activity lights) that default to "not supported on this machine". `Machine` implements all of it. A machine built around a different chipset implements the same trait and gets the window, iris-ci, iris-gui and the monitor's `reset`/`save`/`load`/`machine-status` @@ -171,7 +172,9 @@ pub trait Resettable { fn power_on(&self); } internals 32-bit; the GFIFO still carries a 32-bit register address. - **Device** is the lifecycle: devices with threads start and join them here. `register_commands`/`execute_command` give a device its own debug commands; - [debug.md](debug.md) documents them. + [debug.md](debug.md) documents them. A sound card also answers + `audio_report` (the monitor's `audio status`) and `audio_frames`, a count of + frames moved that only goes up while sound plays or records. - **Saveable** is snapshot state. A device that implements it gets a save → load → save round-trip test. State too big for a TOML table goes to the chunk store instead: the machine calls the device's own `save_chunks` and diff --git a/clanker/INDEX.md b/clanker/INDEX.md index 542d173de..bc022395f 100644 --- a/clanker/INDEX.md +++ b/clanker/INDEX.md @@ -539,6 +539,7 @@ Plans, task briefs and reviews written for agents. Each states its own status; m | [cow-chd-sync-plan.md](plans/cow-chd-sync-plan.md) | plan | Plan: copy-on-write protection + "Syncing CHD file…" apply-on-shutdown | | [GPUTodo.md](plans/GPUTodo.md) | plan | REX3 proper GPU mode: merged gated foundation, unmerged M2-M10 candidate stack and planned upstream PRs, with presenter/CI coordination | | [gui-benchmark-plan.md](plans/gui-benchmark-plan.md) | plan | Benchmark in iris-gui | +| [kona-tile-handoff-race.md](plans/kona-tile-handoff-race.md) | open | KONA tile scheduler strands a job when a tile changes hands (tail store `Release` / load `Acquire` in a store-then-load hand-over): `a_tiled_run_is_bit_exact_with_the_inline_one` hangs on Apple silicon and is ignored; stuck-state dump, cause, candidate two-line `SeqCst` fix with measurements | | [iris-daynaport-target.md](plans/iris-daynaport-target.md) | plan | Task: add a DaynaPort SCSI/Link target to the IRIS emulator | | [origin-octane-o2-landing-review.md](plans/origin-octane-o2-landing-review.md) | plan | Review of James Braid's SGI series (547 commits to 2026-11-11, two passes): Origin/Onyx2 two-node, Octane, O2, Fuel/Tezro (IP35), InfiniteReality (KONA); cache-as-RAM, DMA and interrupt paths per machine; ranked findings with status (KONA texture-bind race, Origin node-1 code never JIT-compiled, cross-node IRQs land on node 0, byte-wise DMA on Origin/IP35, BCM5701 lost coalescing wake) | | [irix7-fork-review.md](plans/irix7-fork-review.md) | plan | The `irix7/iris` fork — change inventory and triage | diff --git a/clanker/plans/kona-tile-handoff-race.md b/clanker/plans/kona-tile-handoff-race.md new file mode 100644 index 000000000..607468ffa --- /dev/null +++ b/clanker/plans/kona-tile-handoff-race.md @@ -0,0 +1,89 @@ +# KONA tile scheduler: a tile's hand-over can strand a job + +Status: **open, not fixed** (found 2026-11-20). The test that shows it, +`dev::kona::back::tiles::tests::a_tiled_run_is_bit_exact_with_the_inline_one`, +is `#[ignore]`d until the fix lands. A candidate fix is below; it was tried +locally and not committed. + +Code: `src/dev/kona/back/tiles.rs`, added 2026-11-10 in `30fc7236` ("kona: add +the pixel store, raster stage, tile scheduler and display"). The lines involved +are unchanged since. + +## Symptom + +On an Apple M1 (macOS, debug build, `cargo test`), the test never finishes: one +thread at 100% CPU, no output. Run alone it hangs every time (0 of 6 runs +passed; it hung for 2.5 hours inside a full `cargo test --workspace`). CI is +green on every platform, macOS arm64 included, so it depends on timing the +runners do not produce. + +It hangs in the first threaded configuration (one worker, ring of 4). Stacks: + +- the test thread spins in `Tiles::submit`'s wait loop (`yield_now`, `retire` + breaking on `pending != 0`); +- `kona-tile-0` sleeps in `Shared::worker` on the condition variable. + +State dumped from the stuck producer: + +``` +head=271 tail=267 inflight=3385 budget=16384 ring=4 workers=1 sleepers=1 ready_len=0 +slot 3 pending=1 (slots 0,1,2 pending=0, waiting behind it to retire in order) +tile 60 head=30 tail=31 scheduled=false +``` + +Tile 60 has a job in its queue, is not marked scheduled and is not on the +ready queue. Nothing will ever run it, so slot 3 never retires, the ring stays +full and the producer waits for ever. `Drop for Tiles` calls `drain`, so a +panic in that state hangs too. + +## Cause + +The hand-over of a tile between its owner and the producer is a +store-then-load on each side (Dekker's pattern): + +| | writes | then reads | +|---|---|---| +| owner, `run_tile` | `scheduled.store(false, SeqCst)` | `is_empty()`: `tail.load(Acquire)` | +| producer, `submit` | `Tile::push`: `tail.store(.., Release)` | `scheduled.swap(true, SeqCst)` | + +It is correct only if at least one side sees the other's write. That needs all +four operations to be `SeqCst`. With the tail store `Release` and the tail +load `Acquire`, both may miss: the owner reads the old tail, takes the queue +for empty and returns with `scheduled == false`; the producer's swap reads +`true` and leaves the job to the owner. The final state above is exactly that, +and no sequentially consistent interleaving reaches it. + +Not established: why the generated code differs in practice on aarch64 (both +orderings were expected to use the same instructions), and whether a running +Onyx2 guest hits it. It is the live InfiniteReality raster path, so it would +look like a frozen display with `kona-hip` spinning. + +## Candidate fix + +Two orderings in `Tile`: + +```rust +// push +self.tail.store(t.wrapping_add(1), SeqCst); // was Release +// is_empty +self.head.load(Relaxed) == self.tail.load(SeqCst) // was Acquire +``` + +Measured on the same machine, same build: + +| Variant | Runs passed | +|---|---| +| original | 0 of 6 | +| `fence(SeqCst)` after `scheduled.store(false)` in `run_tile` | 6 of 6 | +| the two `SeqCst` orderings above, no fence | 12 of 12 (about 0.05 s each) | + +`pop`'s `tail.load(Acquire)` is not part of the hand-over and can stay. + +## Before closing this + +- Apply the fix, remove the `#[ignore]`, and run the test in a loop (a few + hundred runs) on Apple silicon; run the rest of `dev::kona::back`. +- The sleep path has the same shape (`sleepers` against the ready queue) and + is already `SeqCst` on both sides; `loom` would check both hand-overs + properly (`TESTING.md`, "Loom models"). +- Only this one test was run against the candidate fix. diff --git a/src/dev/hal2.rs b/src/dev/hal2.rs index 92b735049..8c5cdbf31 100644 --- a/src/dev/hal2.rs +++ b/src/dev/hal2.rs @@ -1143,6 +1143,10 @@ impl Device for Hal2 { /// `audio status`: codec A is the only output counted. Its rate is the /// guest's while its timer runs. + fn audio_frames(&self) -> Option { + Some(self.meter.total()) + } + fn audio_report(&self) -> Option { let ca = self.ca_state.lock(); let rate = ca.timer_id.is_some().then(|| { diff --git a/src/dev/ice1724.rs b/src/dev/ice1724.rs index 86f4d3b98..06aa10da9 100644 --- a/src/dev/ice1724.rs +++ b/src/dev/ice1724.rs @@ -1885,6 +1885,12 @@ impl Device for Ice1724 { 0 } + fn audio_frames(&self) -> Option { + // The engines keep their own counts; the meter only mirrors them for a report. + let st = self.inner.state.lock().stats; + Some(st.capture_frames + st.analog_frames + st.spdif_frames) + } + fn audio_report(&self) -> Option { let i = &self.inner; let (rates, st) = { diff --git a/src/dev/kona/back/tiles.rs b/src/dev/kona/back/tiles.rs index e3c3fe8db..ca4be5205 100644 --- a/src/dev/kona/back/tiles.rs +++ b/src/dev/kona/back/tiles.rs @@ -572,7 +572,11 @@ mod tests { /// A tiled run is bit-exact with the inline one at every worker count, tile /// size and ring size, and the scene is order sensitive, so a lost ordering shows. + /// + /// Ignored: on Apple silicon it hangs. A tile's hand-over can strand a job + /// (see `clanker/plans/kona-tile-handoff-race.md`); remove this once that is fixed. #[test] + #[ignore = "hangs on Apple silicon: the tile hand-over strands a job (clanker/plans/kona-tile-handoff-race.md)"] fn a_tiled_run_is_bit_exact_with_the_inline_one() { let s = scene(400); let (inline, st) = run(&s, 0, 32, 64, 1 << 20); diff --git a/src/dev/mace_audio.rs b/src/dev/mace_audio.rs index afb527adc..46ec1fba5 100644 --- a/src/dev/mace_audio.rs +++ b/src/dev/mace_audio.rs @@ -757,6 +757,10 @@ impl crate::traits::Device for MaceAudio { fn is_running(&self) -> bool { true } fn get_clock(&self) -> u64 { 0 } + fn audio_frames(&self) -> Option { + Some(self.meter.total()) + } + fn audio_report(&self) -> Option { let st = self.st.lock(); let rate = |n: usize| st.dac_rate(n).filter(|_| st.ch[n].running()); diff --git a/src/dev/rad1.rs b/src/dev/rad1.rs index 34e958140..de037bed0 100644 --- a/src/dev/rad1.rs +++ b/src/dev/rad1.rs @@ -1452,6 +1452,10 @@ impl Device for Rad1 { fn is_running(&self) -> bool { self.inner.armed.load(Ordering::SeqCst) } fn get_clock(&self) -> u64 { 0 } + fn audio_frames(&self) -> Option { + Some(self.inner.meter.total()) + } + fn audio_report(&self) -> Option { let i = &self.inner; let rates = { diff --git a/src/host_audio.rs b/src/host_audio.rs index a52da0b32..7c8c6bc8b 100644 --- a/src/host_audio.rs +++ b/src/host_audio.rs @@ -131,6 +131,11 @@ impl Meter { self.moved[output].load(Ordering::Relaxed) } + /// Frames moved by every input and output together, for an activity light. + pub fn total(&self) -> u64 { + self.moved.iter().map(|m| m.load(Ordering::Relaxed)).sum() + } + /// The monitor's `audio status` text for one card. pub fn report(&self, rates: &[Option], outs: &[&Output]) -> String { let mut s = format!("audio: {}\n", self.card); diff --git a/src/ip30_machine.rs b/src/ip30_machine.rs index 5c1f299cc..ab22d23fe 100644 --- a/src/ip30_machine.rs +++ b/src/ip30_machine.rs @@ -1519,6 +1519,9 @@ impl Workstation for OctaneMachine { fn pending_chd_sync_count(&self) -> usize { self.disk_buses().iter().map(|b| b.pending_chd_sync_count()).sum() } + fn disk_activity(&self) -> Vec> { + self.disk_buses().iter().map(|b| b.activity()).collect() + } fn flush_disks(&self) -> std::io::Result<()> { self.disk_buses().iter().try_for_each(|b| b.flush()) } diff --git a/src/ip32_machine.rs b/src/ip32_machine.rs index 757f1e64a..a28441c0c 100644 --- a/src/ip32_machine.rs +++ b/src/ip32_machine.rs @@ -872,6 +872,13 @@ impl Workstation for O2Machine { self.bus.scsi.lock().unwrap().iter().map(|c| c.bus().pending_chd_sync_count()).sum::() + self.bus.nvme.get().map_or(0, |n| n.scsi_bus().pending_chd_sync_count()) } + fn disk_activity(&self) -> Vec> { + use crate::scsi_monitor::ScsiTargets; + let controllers = self.bus.scsi.lock().unwrap().clone(); + controllers.iter().map(|c| c.bus().activity()) + .chain(self.bus.nvme.get().map(|n| n.scsi_bus().activity())) + .collect() + } fn flush_disks(&self) -> std::io::Result<()> { self.bus.flush_disks() } diff --git a/src/machine.rs b/src/machine.rs index 68d6e484c..5b494af7e 100644 --- a/src/machine.rs +++ b/src/machine.rs @@ -2619,6 +2619,9 @@ impl Workstation for Machine { } fn remount_cdrom_guest(&self, scsi_id: u8) { Machine::remount_cdrom_guest(self, scsi_id) } fn pending_chd_sync_count(&self) -> usize { Machine::pending_chd_sync_count(self) } + fn disk_activity(&self) -> Vec> { + self.scsi_buses().iter().map(|b| b.activity()).collect() + } fn flush_disks(&self) -> std::io::Result<()> { self.scsi_buses().iter().try_for_each(|b| b.flush()) } diff --git a/src/monitor.rs b/src/monitor.rs index aef123616..cb1caa403 100644 --- a/src/monitor.rs +++ b/src/monitor.rs @@ -68,6 +68,12 @@ impl Monitor { audio_status(&self.devices.lock()) } + /// Frames every sound card has moved (see [`Device::audio_frames`]), or + /// `None` on a machine with no sound card. + pub fn audio_frames(&self) -> Option { + self.devices.lock().iter().filter_map(|d| d.audio_frames()).reduce(|a, b| a + b) + } + /// Bind `addr` and serve it on a new thread. The bind happens before /// this returns, so `bound_addr` answers at once whether this monitor /// has its port. @@ -323,6 +329,7 @@ mod tests { fn is_running(&self) -> bool { true } fn get_clock(&self) -> u64 { 0 } fn audio_report(&self) -> Option { Some(format!("audio: {}\n", self.0)) } + fn audio_frames(&self) -> Option { Some(self.0.len() as u64) } fn register_commands(&self) -> Vec<(String, String)> { if self.1 { vec![("audio".into(), "own".into())] } else { vec![] } } @@ -352,6 +359,9 @@ mod tests { monitor.register_device(Arc::new(Card("one", false))); monitor.register_device(Arc::new(Card("two", false))); assert_eq!(monitor.audio_status(), "audio: one\naudio: two\n"); + // The activity count sums the cards, and is absent with none. + assert_eq!(monitor.audio_frames(), Some(6)); + assert_eq!(Monitor::new().audio_frames(), None); } /// `quit` ends the connection even though the machine's log sink keeps a diff --git a/src/scsi.rs b/src/scsi.rs index 9810b78c4..e4f183b4e 100644 --- a/src/scsi.rs +++ b/src/scsi.rs @@ -17,7 +17,7 @@ mod attach; mod bus; pub use attach::{attach_failed, attach_from_config, attach_table, Attached, DaynaportRef, TableRef}; -pub use bus::{ScsiBus, ScsiIdentities, ScsiTarget, Targets}; +pub use bus::{DiskActivity, DiskCounts, ScsiBus, ScsiIdentities, ScsiTarget, Targets}; use crate::block_node::{Background, ChangerNode, Removable, BlockNode}; @@ -215,6 +215,8 @@ pub struct ScsiDevice { /// The disk's media while a CHD sync has it closed and the tray answers /// empty in its place; see [`ScsiTarget::take_pending_chd_sync`]. released: Option, + /// The bus's counters for this id, from the moment it is attached. + activity: Option>, } impl Drop for ScsiDevice { @@ -268,6 +270,7 @@ impl ScsiDevice { power_on_write_cache: false, background, released: None, + activity: None, } } @@ -288,6 +291,7 @@ impl ScsiDevice { power_on_write_cache: false, background: None, released: None, + activity: None, } } @@ -518,6 +522,9 @@ impl ScsiDevice { lba, count, self.logical_block_size, expected, data.len() ); } + if let Some(a) = &self.activity { + a.read(data.len() as u64); + } Ok(ScsiResponse { status: 0x00, data, @@ -581,6 +588,9 @@ impl ScsiDevice { } } + if let Some(a) = &self.activity { + a.write(data.len() as u64); + } Ok(ScsiResponse { status: 0x00, data: vec![], @@ -1273,6 +1283,10 @@ impl ScsiTarget for ScsiDevice { Ok(response) } + fn set_activity(&mut self, activity: std::sync::Arc) { + self.activity = Some(activity); + } + fn flush(&mut self) -> io::Result<()> { if self.is_cdrom() { return Ok(()); diff --git a/src/scsi/bus.rs b/src/scsi/bus.rs index d433dff02..47d4f4057 100644 --- a/src/scsi/bus.rs +++ b/src/scsi/bus.rs @@ -8,6 +8,8 @@ //! controller owns one of and reaches its targets through. use std::io; +use std::sync::atomic::{AtomicU64, Ordering}; +use std::sync::Arc; use parking_lot::{Mutex, MutexGuard}; @@ -38,6 +40,10 @@ pub trait ScsiTarget: Send { /// Stop it, with no command part way through. fn stop(&mut self) {} + /// Take the bus's counters for this target's id, to count the reads and + /// writes it carries out. A target that moves no blocks ignores them. + fn set_activity(&mut self, _activity: Arc) {} + /// Whether READ(6) and WRITE(6) move packets, not blocks, so a /// controller must not read their count as a number of sectors. fn moves_packets(&self) -> bool { false } @@ -140,6 +146,50 @@ pub struct ScsiIdentities { pub cdrom: ScsiIdentity, } +/// Reads and writes a target has completed since it was attached, for a +/// front-end's activity light. The counters only go up; whoever shows them +/// compares two samples. Reading takes no lock, so a sample never waits for +/// a command that holds the bus. +#[derive(Default)] +pub struct DiskActivity { + reads: AtomicU64, + read_bytes: AtomicU64, + writes: AtomicU64, + write_bytes: AtomicU64, +} + +/// One sample of a [`DiskActivity`]. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct DiskCounts { + /// Completed read commands and the bytes they returned. + pub reads: u64, + pub read_bytes: u64, + /// Completed write commands and the bytes they stored. + pub writes: u64, + pub write_bytes: u64, +} + +impl DiskActivity { + pub fn read(&self, bytes: u64) { + self.reads.fetch_add(1, Ordering::Relaxed); + self.read_bytes.fetch_add(bytes, Ordering::Relaxed); + } + + pub fn write(&self, bytes: u64) { + self.writes.fetch_add(1, Ordering::Relaxed); + self.write_bytes.fetch_add(bytes, Ordering::Relaxed); + } + + pub fn counts(&self) -> DiskCounts { + DiskCounts { + reads: self.reads.load(Ordering::Relaxed), + read_bytes: self.read_bytes.load(Ordering::Relaxed), + writes: self.writes.load(Ordering::Relaxed), + write_bytes: self.write_bytes.load(Ordering::Relaxed), + } + } +} + /// The targets on a bus by SCSI id, `None` where nothing answers. pub type Targets = Vec>>; @@ -150,6 +200,9 @@ pub type Targets = Vec>>; pub struct ScsiBus { identities: ScsiIdentities, targets: Mutex, + /// Per-id counters, outside the table's lock; a target keeps the ones + /// for its id from [`ScsiBus::attach`] on. + activity: Box<[Arc]>, } impl ScsiBus { @@ -158,6 +211,7 @@ impl ScsiBus { Self { identities, targets: Mutex::new(std::iter::repeat_with(|| None).take(ids).collect()), + activity: std::iter::repeat_with(Arc::default).take(ids).collect(), } } @@ -184,11 +238,12 @@ impl ScsiBus { } /// Put `target` at `id`, replacing what was there. - pub fn attach(&self, id: u8, target: Box) -> io::Result<()> { + pub fn attach(&self, id: u8, mut target: Box) -> io::Result<()> { let mut targets = self.targets.lock(); let slot = targets.get_mut(id as usize).ok_or_else(|| { io::Error::new(io::ErrorKind::InvalidInput, format!("SCSI id {id} out of range")) })?; + target.set_activity(self.activity[id as usize].clone()); *slot = Some(target); Ok(()) } @@ -271,6 +326,13 @@ impl ScsiBus { } /// Whether something answers at `id`. + /// Every id's read and write counters, by id. Takes no lock: safe from a + /// status poll while a command holds the bus. An id with nothing on it + /// stays at zero. + pub fn activity(&self) -> Vec { + self.activity.iter().map(|a| a.counts()).collect() + } + pub fn has_target(&self, id: u8) -> bool { self.targets.lock().get(id as usize).is_some_and(|t| t.is_some()) } @@ -490,6 +552,30 @@ mod tests { } } + /// A front-end's disk light: completed reads and writes count on the + /// id's own counters, a refused command and other ids do not, and a + /// sample needs no lock, so it works while a command holds the bus. + #[test] + fn activity_counts_completed_reads_and_writes_per_id() { + let dir = TempPath::dir("scsiact"); + let b = bus(8); + attach(&b, 1, &image(&dir, 8), false).unwrap(); + assert_eq!(b.activity(), vec![DiskCounts::default(); 8]); + + b.request(1, &write_block(2, 0x55)).unwrap().unwrap(); + b.request(1, &read_block(2)).unwrap().unwrap(); + b.request(1, &read_block(3)).unwrap().unwrap(); + // Past the end: CHECK CONDITION, nothing moved. + assert_ne!(b.request(1, &read_block(99)).unwrap().unwrap().status, 0); + b.request(1, &inquiry()).unwrap().unwrap(); + + let held = b.lock(); + let counts = b.activity(); + drop(held); + assert_eq!(counts[1], DiskCounts { reads: 2, read_bytes: 1024, writes: 1, write_bytes: 512 }); + assert!(counts.iter().enumerate().all(|(id, c)| id == 1 || *c == DiskCounts::default())); + } + /// The machine can exit without dropping the bus (`process::exit` after /// a guest power-off), so [`ScsiBus::flush`] is what the next run's /// overlay depends on. `mem::forget` stands in for that exit. diff --git a/src/sn.rs b/src/sn.rs index e87ad695d..82f6093de 100644 --- a/src/sn.rs +++ b/src/sn.rs @@ -989,6 +989,9 @@ impl Worksta fn pending_chd_sync_count(&self) -> usize { self.with_disks(|buses| buses.iter().map(|b| b.pending_chd_sync_count()).sum()) } + fn disk_activity(&self) -> Vec> { + self.with_disks(|buses| buses.iter().map(|b| b.activity()).collect()) + } fn flush_disks(&self) -> std::io::Result<()> { self.with_disks(|buses| buses.iter().try_for_each(|b| b.flush())) } diff --git a/src/traits.rs b/src/traits.rs index 391f721eb..26ae1aef0 100644 --- a/src/traits.rs +++ b/src/traits.rs @@ -497,6 +497,14 @@ pub trait Device: Send + Sync { fn audio_report(&self) -> Option { None } + + /// Frames a sound card has moved between the guest and the host, inputs + /// and outputs together, since it was built; other devices have none. + /// It only goes up while sound plays or records, so a front-end compares + /// two samples for an activity light. + fn audio_frames(&self) -> Option { + None + } } // Compile-time asserts: BUS_* constants must equal the corresponding ExecStatus values. diff --git a/src/workstation.rs b/src/workstation.rs index 5589f4a65..a53acf89d 100644 --- a/src/workstation.rs +++ b/src/workstation.rs @@ -316,6 +316,19 @@ pub trait Workstation: Send { fn pending_chd_sync_count(&self) -> usize { 0 } + /// Read and write counters of every disk and CD-ROM, for a front-end's + /// activity lights: one entry per disk bus, in the order the machine + /// numbers its SCSI controllers (then NVMe and Fibre Channel where it has + /// them), each a list by SCSI id. The counters only go up; compare two + /// samples. Takes no bus lock. + fn disk_activity(&self) -> Vec> { + Vec::new() + } + /// Frames the machine's sound cards have moved, for a front-end's audio + /// light; `None` with no sound card. Compare two samples. + fn audio_frames(&self) -> Option { + self.monitor().audio_frames() + } /// Make every disk's accepted writes survive the process, including a /// copy-on-write overlay's record of the sectors it holds. Run it before /// the process exits without dropping the machine.