Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions HACKING.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,8 @@ the monitor, the stored lifecycle state); its provided methods are either
shared logic every machine wants the same way (`status`, and the idempotent
`start_guest`/`stop_guest` behind `machine-start`/`machine-stop`, see
`src/machine_state.rs`) or optional facilities (snapshots, CHD folding, the
scratch volume, disc changes, the NAT controls) that default to "not
scratch volume, disc changes, the NAT controls, the `disk_activity` and
`audio_frames` counters behind a front-end's activity lights) that default to "not
supported on this machine". `Machine` implements all of it. A machine built
around a different chipset implements the same trait and gets the window,
iris-ci, iris-gui and the monitor's `reset`/`save`/`load`/`machine-status`
Expand Down Expand Up @@ -171,7 +172,9 @@ pub trait Resettable { fn power_on(&self); }
internals 32-bit; the GFIFO still carries a 32-bit register address.
- **Device** is the lifecycle: devices with threads start and join them here.
`register_commands`/`execute_command` give a device its own debug commands;
[debug.md](debug.md) documents them.
[debug.md](debug.md) documents them. A sound card also answers
`audio_report` (the monitor's `audio status`) and `audio_frames`, a count of
frames moved that only goes up while sound plays or records.
- **Saveable** is snapshot state. A device that implements it gets a
save → load → save round-trip test. State too big for a TOML table goes to the
chunk store instead: the machine calls the device's own `save_chunks` and
Expand Down
1 change: 1 addition & 0 deletions clanker/INDEX.md
Original file line number Diff line number Diff line change
Expand Up @@ -539,6 +539,7 @@ Plans, task briefs and reviews written for agents. Each states its own status; m
| [cow-chd-sync-plan.md](plans/cow-chd-sync-plan.md) | plan | Plan: copy-on-write protection + "Syncing CHD file…" apply-on-shutdown |
| [GPUTodo.md](plans/GPUTodo.md) | plan | REX3 proper GPU mode: merged gated foundation, unmerged M2-M10 candidate stack and planned upstream PRs, with presenter/CI coordination |
| [gui-benchmark-plan.md](plans/gui-benchmark-plan.md) | plan | Benchmark in iris-gui |
| [kona-tile-handoff-race.md](plans/kona-tile-handoff-race.md) | open | KONA tile scheduler strands a job when a tile changes hands (tail store `Release` / load `Acquire` in a store-then-load hand-over): `a_tiled_run_is_bit_exact_with_the_inline_one` hangs on Apple silicon and is ignored; stuck-state dump, cause, candidate two-line `SeqCst` fix with measurements |
| [iris-daynaport-target.md](plans/iris-daynaport-target.md) | plan | Task: add a DaynaPort SCSI/Link target to the IRIS emulator |
| [origin-octane-o2-landing-review.md](plans/origin-octane-o2-landing-review.md) | plan | Review of James Braid's SGI series (547 commits to 2026-11-11, two passes): Origin/Onyx2 two-node, Octane, O2, Fuel/Tezro (IP35), InfiniteReality (KONA); cache-as-RAM, DMA and interrupt paths per machine; ranked findings with status (KONA texture-bind race, Origin node-1 code never JIT-compiled, cross-node IRQs land on node 0, byte-wise DMA on Origin/IP35, BCM5701 lost coalescing wake) |
| [irix7-fork-review.md](plans/irix7-fork-review.md) | plan | The `irix7/iris` fork — change inventory and triage |
Expand Down
89 changes: 89 additions & 0 deletions clanker/plans/kona-tile-handoff-race.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
# KONA tile scheduler: a tile's hand-over can strand a job

Status: **open, not fixed** (found 2026-11-20). The test that shows it,
`dev::kona::back::tiles::tests::a_tiled_run_is_bit_exact_with_the_inline_one`,
is `#[ignore]`d until the fix lands. A candidate fix is below; it was tried
locally and not committed.

Code: `src/dev/kona/back/tiles.rs`, added 2026-11-10 in `30fc7236` ("kona: add
the pixel store, raster stage, tile scheduler and display"). The lines involved
are unchanged since.

## Symptom

On an Apple M1 (macOS, debug build, `cargo test`), the test never finishes: one
thread at 100% CPU, no output. Run alone it hangs every time (0 of 6 runs
passed; it hung for 2.5 hours inside a full `cargo test --workspace`). CI is
green on every platform, macOS arm64 included, so it depends on timing the
runners do not produce.

It hangs in the first threaded configuration (one worker, ring of 4). Stacks:

- the test thread spins in `Tiles::submit`'s wait loop (`yield_now`, `retire`
breaking on `pending != 0`);
- `kona-tile-0` sleeps in `Shared::worker` on the condition variable.

State dumped from the stuck producer:

```
head=271 tail=267 inflight=3385 budget=16384 ring=4 workers=1 sleepers=1 ready_len=0
slot 3 pending=1 (slots 0,1,2 pending=0, waiting behind it to retire in order)
tile 60 head=30 tail=31 scheduled=false
```

Tile 60 has a job in its queue, is not marked scheduled and is not on the
ready queue. Nothing will ever run it, so slot 3 never retires, the ring stays
full and the producer waits for ever. `Drop for Tiles` calls `drain`, so a
panic in that state hangs too.

## Cause

The hand-over of a tile between its owner and the producer is a
store-then-load on each side (Dekker's pattern):

| | writes | then reads |
|---|---|---|
| owner, `run_tile` | `scheduled.store(false, SeqCst)` | `is_empty()`: `tail.load(Acquire)` |
| producer, `submit` | `Tile::push`: `tail.store(.., Release)` | `scheduled.swap(true, SeqCst)` |

It is correct only if at least one side sees the other's write. That needs all
four operations to be `SeqCst`. With the tail store `Release` and the tail
load `Acquire`, both may miss: the owner reads the old tail, takes the queue
for empty and returns with `scheduled == false`; the producer's swap reads
`true` and leaves the job to the owner. The final state above is exactly that,
and no sequentially consistent interleaving reaches it.

Not established: why the generated code differs in practice on aarch64 (both
orderings were expected to use the same instructions), and whether a running
Onyx2 guest hits it. It is the live InfiniteReality raster path, so it would
look like a frozen display with `kona-hip` spinning.

## Candidate fix

Two orderings in `Tile`:

```rust
// push
self.tail.store(t.wrapping_add(1), SeqCst); // was Release
// is_empty
self.head.load(Relaxed) == self.tail.load(SeqCst) // was Acquire
```

Measured on the same machine, same build:

| Variant | Runs passed |
|---|---|
| original | 0 of 6 |
| `fence(SeqCst)` after `scheduled.store(false)` in `run_tile` | 6 of 6 |
| the two `SeqCst` orderings above, no fence | 12 of 12 (about 0.05 s each) |

`pop`'s `tail.load(Acquire)` is not part of the hand-over and can stay.

## Before closing this

- Apply the fix, remove the `#[ignore]`, and run the test in a loop (a few
hundred runs) on Apple silicon; run the rest of `dev::kona::back`.
- The sleep path has the same shape (`sleepers` against the ready queue) and
is already `SeqCst` on both sides; `loom` would check both hand-overs
properly (`TESTING.md`, "Loom models").
- Only this one test was run against the candidate fix.
4 changes: 4 additions & 0 deletions src/dev/hal2.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1143,6 +1143,10 @@ impl Device for Hal2 {

/// `audio status`: codec A is the only output counted. Its rate is the
/// guest's while its timer runs.
fn audio_frames(&self) -> Option<u64> {
Some(self.meter.total())
}

fn audio_report(&self) -> Option<String> {
let ca = self.ca_state.lock();
let rate = ca.timer_id.is_some().then(|| {
Expand Down
6 changes: 6 additions & 0 deletions src/dev/ice1724.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1885,6 +1885,12 @@ impl Device for Ice1724 {
0
}

fn audio_frames(&self) -> Option<u64> {
// The engines keep their own counts; the meter only mirrors them for a report.
let st = self.inner.state.lock().stats;
Some(st.capture_frames + st.analog_frames + st.spdif_frames)
}

fn audio_report(&self) -> Option<String> {
let i = &self.inner;
let (rates, st) = {
Expand Down
4 changes: 4 additions & 0 deletions src/dev/kona/back/tiles.rs
Original file line number Diff line number Diff line change
Expand Up @@ -572,7 +572,11 @@ mod tests {

/// A tiled run is bit-exact with the inline one at every worker count, tile
/// size and ring size, and the scene is order sensitive, so a lost ordering shows.
///
/// Ignored: on Apple silicon it hangs. A tile's hand-over can strand a job
/// (see `clanker/plans/kona-tile-handoff-race.md`); remove this once that is fixed.
#[test]
#[ignore = "hangs on Apple silicon: the tile hand-over strands a job (clanker/plans/kona-tile-handoff-race.md)"]
fn a_tiled_run_is_bit_exact_with_the_inline_one() {
let s = scene(400);
let (inline, st) = run(&s, 0, 32, 64, 1 << 20);
Expand Down
4 changes: 4 additions & 0 deletions src/dev/mace_audio.rs
Original file line number Diff line number Diff line change
Expand Up @@ -757,6 +757,10 @@ impl crate::traits::Device for MaceAudio {
fn is_running(&self) -> bool { true }
fn get_clock(&self) -> u64 { 0 }

fn audio_frames(&self) -> Option<u64> {
Some(self.meter.total())
}

fn audio_report(&self) -> Option<String> {
let st = self.st.lock();
let rate = |n: usize| st.dac_rate(n).filter(|_| st.ch[n].running());
Expand Down
4 changes: 4 additions & 0 deletions src/dev/rad1.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1452,6 +1452,10 @@ impl Device for Rad1 {
fn is_running(&self) -> bool { self.inner.armed.load(Ordering::SeqCst) }
fn get_clock(&self) -> u64 { 0 }

fn audio_frames(&self) -> Option<u64> {
Some(self.inner.meter.total())
}

fn audio_report(&self) -> Option<String> {
let i = &self.inner;
let rates = {
Expand Down
5 changes: 5 additions & 0 deletions src/host_audio.rs
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,11 @@ impl Meter {
self.moved[output].load(Ordering::Relaxed)
}

/// Frames moved by every input and output together, for an activity light.
pub fn total(&self) -> u64 {
self.moved.iter().map(|m| m.load(Ordering::Relaxed)).sum()
}

/// The monitor's `audio status` text for one card.
pub fn report(&self, rates: &[Option<u32>], outs: &[&Output]) -> String {
let mut s = format!("audio: {}\n", self.card);
Expand Down
3 changes: 3 additions & 0 deletions src/ip30_machine.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1519,6 +1519,9 @@ impl Workstation for OctaneMachine {
fn pending_chd_sync_count(&self) -> usize {
self.disk_buses().iter().map(|b| b.pending_chd_sync_count()).sum()
}
fn disk_activity(&self) -> Vec<Vec<crate::scsi::DiskCounts>> {
self.disk_buses().iter().map(|b| b.activity()).collect()
}
fn flush_disks(&self) -> std::io::Result<()> {
self.disk_buses().iter().try_for_each(|b| b.flush())
}
Expand Down
7 changes: 7 additions & 0 deletions src/ip32_machine.rs
Original file line number Diff line number Diff line change
Expand Up @@ -872,6 +872,13 @@ impl Workstation for O2Machine {
self.bus.scsi.lock().unwrap().iter().map(|c| c.bus().pending_chd_sync_count()).sum::<usize>()
+ self.bus.nvme.get().map_or(0, |n| n.scsi_bus().pending_chd_sync_count())
}
fn disk_activity(&self) -> Vec<Vec<crate::scsi::DiskCounts>> {
use crate::scsi_monitor::ScsiTargets;
let controllers = self.bus.scsi.lock().unwrap().clone();
controllers.iter().map(|c| c.bus().activity())
.chain(self.bus.nvme.get().map(|n| n.scsi_bus().activity()))
.collect()
}
fn flush_disks(&self) -> std::io::Result<()> {
self.bus.flush_disks()
}
Expand Down
3 changes: 3 additions & 0 deletions src/machine.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2619,6 +2619,9 @@ impl Workstation for Machine {
}
fn remount_cdrom_guest(&self, scsi_id: u8) { Machine::remount_cdrom_guest(self, scsi_id) }
fn pending_chd_sync_count(&self) -> usize { Machine::pending_chd_sync_count(self) }
fn disk_activity(&self) -> Vec<Vec<crate::scsi::DiskCounts>> {
self.scsi_buses().iter().map(|b| b.activity()).collect()
}
fn flush_disks(&self) -> std::io::Result<()> {
self.scsi_buses().iter().try_for_each(|b| b.flush())
}
Expand Down
10 changes: 10 additions & 0 deletions src/monitor.rs
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,12 @@ impl Monitor {
audio_status(&self.devices.lock())
}

/// Frames every sound card has moved (see [`Device::audio_frames`]), or
/// `None` on a machine with no sound card.
pub fn audio_frames(&self) -> Option<u64> {
self.devices.lock().iter().filter_map(|d| d.audio_frames()).reduce(|a, b| a + b)
}

/// Bind `addr` and serve it on a new thread. The bind happens before
/// this returns, so `bound_addr` answers at once whether this monitor
/// has its port.
Expand Down Expand Up @@ -323,6 +329,7 @@ mod tests {
fn is_running(&self) -> bool { true }
fn get_clock(&self) -> u64 { 0 }
fn audio_report(&self) -> Option<String> { Some(format!("audio: {}\n", self.0)) }
fn audio_frames(&self) -> Option<u64> { Some(self.0.len() as u64) }
fn register_commands(&self) -> Vec<(String, String)> {
if self.1 { vec![("audio".into(), "own".into())] } else { vec![] }
}
Expand Down Expand Up @@ -352,6 +359,9 @@ mod tests {
monitor.register_device(Arc::new(Card("one", false)));
monitor.register_device(Arc::new(Card("two", false)));
assert_eq!(monitor.audio_status(), "audio: one\naudio: two\n");
// The activity count sums the cards, and is absent with none.
assert_eq!(monitor.audio_frames(), Some(6));
assert_eq!(Monitor::new().audio_frames(), None);
}

/// `quit` ends the connection even though the machine's log sink keeps a
Expand Down
16 changes: 15 additions & 1 deletion src/scsi.rs
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ mod attach;
mod bus;

pub use attach::{attach_failed, attach_from_config, attach_table, Attached, DaynaportRef, TableRef};
pub use bus::{ScsiBus, ScsiIdentities, ScsiTarget, Targets};
pub use bus::{DiskActivity, DiskCounts, ScsiBus, ScsiIdentities, ScsiTarget, Targets};

use crate::block_node::{Background, ChangerNode, Removable, BlockNode};

Expand Down Expand Up @@ -215,6 +215,8 @@ pub struct ScsiDevice {
/// The disk's media while a CHD sync has it closed and the tray answers
/// empty in its place; see [`ScsiTarget::take_pending_chd_sync`].
released: Option<Media>,
/// The bus's counters for this id, from the moment it is attached.
activity: Option<std::sync::Arc<bus::DiskActivity>>,
}

impl Drop for ScsiDevice {
Expand Down Expand Up @@ -268,6 +270,7 @@ impl ScsiDevice {
power_on_write_cache: false,
background,
released: None,
activity: None,
}
}

Expand All @@ -288,6 +291,7 @@ impl ScsiDevice {
power_on_write_cache: false,
background: None,
released: None,
activity: None,
}
}

Expand Down Expand Up @@ -518,6 +522,9 @@ impl ScsiDevice {
lba, count, self.logical_block_size, expected, data.len()
);
}
if let Some(a) = &self.activity {
a.read(data.len() as u64);
}
Ok(ScsiResponse {
status: 0x00,
data,
Expand Down Expand Up @@ -581,6 +588,9 @@ impl ScsiDevice {
}
}

if let Some(a) = &self.activity {
a.write(data.len() as u64);
}
Ok(ScsiResponse {
status: 0x00,
data: vec![],
Expand Down Expand Up @@ -1273,6 +1283,10 @@ impl ScsiTarget for ScsiDevice {
Ok(response)
}

fn set_activity(&mut self, activity: std::sync::Arc<DiskActivity>) {
self.activity = Some(activity);
}

fn flush(&mut self) -> io::Result<()> {
if self.is_cdrom() {
return Ok(());
Expand Down
Loading