Skip to content

feat: lifecycle coverage matrix + parity.json witness convention (#442 PR 6) - #451

Merged
Jammy2211 merged 13 commits into
mainfrom
feature/likelihood-lifecycle-p0-profiling-coverage
Oct 24, 2026
Merged

Jammy2211 merged 13 commits into
mainfrom
feature/likelihood-lifecycle-p0-profiling-coverage

Conversation

@Jammy2211

Copy link
Copy Markdown
Contributor

Summary

PR 6 of #442, the last PR of likelihood-lifecycle phase 0B: a coverage matrix and a parity-witness convention. Stacked on #450 (PR 5), which is stacked on #449 (PR 3b). Merge in order: #449 → #450 → this. This PR's own commits are 3f40ce1e, 1f3d336d and the merge 6f733073.

  • scripts/misc/tooling/coverage_matrix.py writes dashboard/coverage.{json,md}. For likelihood × class × measurement it reports current, stale, superseded or missing, read from the v3 dashboard/catalogue.json and the §3.3 provisional table. A new lint --check step stops it drifting, and it also fails if a planned cell's class disagrees with §3.3.
    • Today: of 246 planned cells, 2 are current, 6 stale, 0 superseded and 238 missing. Of the missing, 32 have only a nearest-size row; under the human size ruling (Delaunay 1250, rectangular 35² = 1225) they count as missing. The other 206 have no record. Parity is missing for all 123 planned setups, because no witnesses exist yet.
    • The 40 current records on planned cells all join their cell (2 + 6 + 32). The other 86 current records are compile/memory measurements or light_fixed variants, which no planned cell covers.
    • Stale rule: reused from the Brain profiling conductor (_profiling.py). A current record is stale when its release_pin is behind the catalogue's newest pin (currently 2026.10.17.1) or has no pin.
  • catalogue/provisional_designations.json is a machine copy of the epic report's §3.3 table, checked cell by cell, with notes on the readings that took judgment. A test keeps it in step with catalogue_lifecycle.py.
  • Parity witness convention: one parity.json per cell, documented in results/README.md and validated by parity_witness.py --check. It holds log L, the reference log L, a tolerance, status match / diverged / not_assessed, the reference source and date, and provenance. The builder emits a measurement: parity v3 record with metric abs_delta_log_likelihood and values {abs_delta_log_likelihood, tolerance}, because Pulse v3 requires non-negative dimensionless values. The signed values go in a parity block on the record. A not_assessed witness produces no record but is listed in parity_witnesses. No parity values were made up: there are 0 parity.json files, and the tests use fixtures only.

Scripts Changed

  • scripts/misc/tooling/coverage_matrix.py (new): coverage matrix and --check
  • scripts/misc/tooling/parity_witness.py (new): parity.json v1 schema, validator and --check
  • scripts/misc/tooling/build_catalogue.py: reads parity.json into v3 parity records (history copies become superseded) and adds parity_witnesses
  • catalogue/provisional_designations.json (new): §3.3 copy
  • .github/workflows/lint.yml: parity_witness and coverage_matrix --check steps
  • Tests: new test_coverage_matrix.py and test_parity_witness.py; parity cases added to test_build_catalogue.py
  • Docs: results/README.md ("Parity witness"), catalogue/README.md, scripts/misc/tooling/README.md; regenerated dashboard/catalogue.json and dashboard/coverage.*

Follow-ups (non-blocking review notes)

  • coverage_matrix.py:221: once real parity witnesses exist, their current records will be counted in current_records_unplanned. Exclude measurement == "parity" there.
  • The §3.3 reading exists twice, in the JSON and hard-coded in catalogue_lifecycle.py. A test keeps them in step; later, have catalogue_lifecycle read the JSON.

Test Plan

  • ruff 0.17.2 check and format --check
  • every lint --check step, including the two new ones and build_catalogue --check --validate-with ../PyAutoPulse (valid at @3)
  • pytest scripts/misc/*/test: 1600 passed, 6 skipped, 1 failed. The failure is the known test_hazards_prior_exit, which is environment-only and passes in CI.
  • smoke 9/9
  • browser test and lychee (CI)

Validation checklist (--auto run — plan was not pre-approved)

Refs #442

🤖 Generated with Claude Code

https://claude.ai/code/session_01SeDHwsR2dnwCk8Btfhej2L

Jammy2211 and others added 13 commits October 24, 2026 12:48
…y; drop results_layout (#442 PR 3b)

build_dashboard.py reads the mirrored tree natively: a series is
<measurement>:<dataset_type>/<variant>:<instrument>:<setup_id>:<host_class>:<model_variant>,
a cell's comparison.json entries join the setup row they summarise, and
results/lens keeps its versioned grammar. catalogue/series_id_map.json,
generated from catalogue/migration_day_zero.json by build_series_id_map.py,
maps every pre-move series id to its new id (151 ids: 107 to v3 ids, 44 with
no mirrored member kept behind legacy:) and attaches the pre-move points
outside the mirrored tree, so all 144 published series keep their 158
releases; a test checks the continuity and that the map is current.

build_readme.py reads breakdown rows and sweep comparisons from the mirrored
tree (unresolved legacy rows listed as setup not recorded), the misc groups
and their archived history (labelled archived), and matches archived
baselines to the current cell through the map's cell table.

scripts/misc/tooling/results_layout.py and its imports are gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SeDHwsR2dnwCk8Btfhej2L
…irrored tree; guard test (#442 PR 3b)

Writers that composed old-layout results paths outside resolve_output_paths
now place through place_output_path / resolve_output_paths, or write to the
results/misc/<group>/ landing folder where a file must keep its name:
likelihood_runtime vmap probes (+ resolve_vmap_batch), point-source
*_summary_v* writers, fixed_light witnesses and numba scaling/draws/memo
defaults, nautilus_batch_capture, separation_rule, oracle_figure, bakeoff,
memo_after_summary, misc/latent, likelihood_runtime sweep.py/aggregate.py
(comparisons to the mirrored cell) and 15 HPC submits. Their readers
(aggregate, size_sweep_table, fixed_light_scaling_table,
gradient_mode_library_ab, lens/solver/_corpus) resolve through the new
_profile_cli helpers (legacy_landing_dir, legacy_read_dirs,
locate_legacy_result, ...). Committed tables and 13/14 comparisons
re-render byte-identical.

test_results_write_destinations.py fails any script under scripts/,
_profile_cli.py or hpc/ that names a results path outside the mirrored
tree (heuristic and exclusions in its docstring).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SeDHwsR2dnwCk8Btfhej2L
…enames only (#442 PR 5)

Pure git mv: each test moves to the test/ dir of the scripts/misc group whose
code it exercises (repo-wide contract tests to tooling/test/, tests of cells
outside scripts/misc to cells/test/). The empty scripts/misc/test/__init__.py
is removed. Path and import fixups follow in the next commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SeDHwsR2dnwCk8Btfhej2L
…isons, writer and reader (#442 PR 3b review)

sweep.py now places every row itself with sweep_layout.placed_row (the day-zero
classification of <class>/<model>/<instrument>/<model>_<config>[_sparse].json
fed the instrument, precision and dense-route transformer the matrix fixes)
and passes the exact path as the new --output-path; aggregate.py reads the
same paths (rows a run wrote win over committed rows) instead of rebuilding
locations from instrument/precision hints. All 308 matrix rows get distinct
mirrored paths under their instrument; the 13 committed comparisons rebuild
byte-equal, including interferometer/{delaunay,pixelization}/sma and
datacube/delaunay/sma.

Comparisons are placed from the instrument their rows record, so the
two-level imaging/delaunay_nn cell goes to results/imaging/delaunay_nn/hst/.
Standalone leaf runs name a two-level runtime folder under --instrument and
the interferometric leaves record the transformer they hard-code, so a
config-named row cannot collide across instruments.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SeDHwsR2dnwCk8Btfhej2L
…ed (#442 PR 3b review)

The breakdown table lists, after the current v3 cells, every published row
with no v3 cell, latest per pre-move group, read where the day-zero move put
it under the name it kept: unresolved rows in results/breakdown/, the current
results/misc/likelihood_breakdown/ (2 datacube/inversion rows) and
archive/*/*/<slug>/breakdown/ (labelled archived). A group whose newest
release the map moved into a mirrored cell is shown by that cell's v3 row.
106 rows before the move-era re-key, 110 now; every old row is present.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SeDHwsR2dnwCk8Btfhej2L
…path, CI (#442 PR 5)

- moved tests: fixed Path(__file__).parents[N] indices +1, the browser
  check's dashboard root, and the sibling test_fixed_light_cell.py reads;
- scripts/misc/conftest.py puts scripts/misc on sys.path (pytest used to do
  it via the old test/__init__.py); a bare root pytest.ini makes the repo root
  pytest's rootdir so the conftest loads even for a single test file;
- lint.yml runs pytest scripts/misc/*/test and the moved browser check;
- build_catalogue, apply_migration_map, the write-destination guard and
  test_headline_completion skip every group's test/ dir (and the conftest);
- scripts/misc/test/<file> references in code comments, hpc submit comments,
  dashboard/README and the CI-enforced timing-noise inventory key list
  rewritten mechanically to the new paths (historical wiki ledgers untouched).

Same 1544 tests collected before and after (identical node ids modulo dir).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SeDHwsR2dnwCk8Btfhej2L
#442 PR 5)

A short index of every scripts/misc group (what it holds, where it writes,
how to run its tests), READMEs for the two groups that had none, the tooling
README's title and test pointer, and AGENTS.md's layout line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SeDHwsR2dnwCk8Btfhej2L
…iases and variant names (#442 PR 3b review)

test_results_write_destinations.py flattens each path expression into
segments (/ joins, .joinpath, os.path.join, Path/PurePosixPath args, split
constants, and names bound by a simple same-module NAME = <path>
assignment), so R = ROOT / "results"; R / "runtime" is caught. A
results/<dataset_type>/ path must name a variant scripts/<dataset_type>/ has
(results/imaging/pixelization fails). Self-checks cover each new form; what
it cannot follow (other modules, parameters, non-trivial bindings) is
documented. build_readme's pre-move map prefix becomes a LEGACY_ constant.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SeDHwsR2dnwCk8Btfhej2L
…ep_layout.py (#442 PR 3b review)

resolve_output_paths reads the instrument with getattr (callers pass a
minimal namespace) and only re-folders legacy results paths; the catalogue's
script navigation lists the new scripts/misc/likelihood_runtime/sweep_layout.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SeDHwsR2dnwCk8Btfhej2L
…ity records (#442 PR 6)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SeDHwsR2dnwCk8Btfhej2L
…ions copy; lint --check steps (#442 PR 6)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SeDHwsR2dnwCk8Btfhej2L
…untime/test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SeDHwsR2dnwCk8Btfhej2L
…+ coverage

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SeDHwsR2dnwCk8Btfhej2L
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant