Testing Strategy

Testing Strategy

Specification: docs/testing-strategy.md. Current results: docs/STATUS.md. Known approximations: docs/accuracy-ledger.md.

The project's working definition of "cycle-accurate" is the public test-ROM suites: when a document and a passing test ROM disagree, the ROM wins and the document is corrected. The accuracy claims are meant to be checked by running those suites, not taken on trust.

The layers

1. Unit tests in each crate: opcodes and flags, PPU register rules and sprite evaluation, APU sequencing and mixer tables, per-mapper banking and IRQ counters.
2. Golden CPU log: nestest compared line by line against the Nintendulator log, with zero diff as the target.
3. Test-ROM corpus: blargg, kevtris and community suites (CC0 / public-domain ROMs in tests/roms/), and the AccuracyCoin battery, read from RAM rather than from the screen. Tests that are known to fail are kept as #[ignore] probes that fail loudly if their behaviour changes.
4. Golden framebuffers and audio for freely distributable ROMs.
5. Commercial-ROM oracle (local only): SHA-256-pinned dumps you supply under the git-ignored tests/roms/external/, with framebuffer, cycle-count and audio hashes asserted against committed snapshots. CI never depends on these files; only the snapshots, not the ROMs, are committed.
6. Co-simulation against an independently written SystemVerilog core (see Architecture-Overview and docs/mister.md).
7. Fuzzing of every untrusted-input parser (cartridge, save state, movie, movie import, netplay messages) and the register interfaces, run in CI with a fixed seed.
8. CI gates: format, clippy with warnings as errors across the enumerated feature sets, rustdoc, the test-ROM suites, a line-coverage floor, the no_std cross-build, the libretro targets and a frame-time regression gate.

Headline results at v3.2.0

At v3.2.0 the README reports AccuracyCoin 146/146 at upstream f5f41dc2, nestest 0-diff, and blargg's CPU interrupt, APU (NTSC and PAL) and MMC3 suites with the exceptions it lists. The 144/144 reported before v3.1.0 was overstated (an older ROM's fail path fell through to a pass); the v3.1.0 re-sync found and fixed the hidden failure. For anything more specific, read docs/STATUS.md, which is updated with each release.

Running the suites

cargo test --workspace                          # unit + integration
cargo test --workspace --features test-roms     # + the test-ROM suites
cargo test -p rustynes-cpu nestest              # one crate, one test name

The accuracy-ledger and STATUS documents explain which failures are expected and why.