Repository navigation
feat: lifecycle coverage matrix + parity.json witness convention (#442 PR 6) - #451
Merged
Jammy2211 merged 13 commits intoOct 24, 2026
Merged
Conversation
…y; drop results_layout (#442 PR 3b) build_dashboard.py reads the mirrored tree natively: a series is <measurement>:<dataset_type>/<variant>:<instrument>:<setup_id>:<host_class>:<model_variant>, a cell's comparison.json entries join the setup row they summarise, and results/lens keeps its versioned grammar. catalogue/series_id_map.json, generated from catalogue/migration_day_zero.json by build_series_id_map.py, maps every pre-move series id to its new id (151 ids: 107 to v3 ids, 44 with no mirrored member kept behind legacy:) and attaches the pre-move points outside the mirrored tree, so all 144 published series keep their 158 releases; a test checks the continuity and that the map is current. build_readme.py reads breakdown rows and sweep comparisons from the mirrored tree (unresolved legacy rows listed as setup not recorded), the misc groups and their archived history (labelled archived), and matches archived baselines to the current cell through the map's cell table. scripts/misc/tooling/results_layout.py and its imports are gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SeDHwsR2dnwCk8Btfhej2L
…irrored tree; guard test (#442 PR 3b) Writers that composed old-layout results paths outside resolve_output_paths now place through place_output_path / resolve_output_paths, or write to the results/misc/<group>/ landing folder where a file must keep its name: likelihood_runtime vmap probes (+ resolve_vmap_batch), point-source *_summary_v* writers, fixed_light witnesses and numba scaling/draws/memo defaults, nautilus_batch_capture, separation_rule, oracle_figure, bakeoff, memo_after_summary, misc/latent, likelihood_runtime sweep.py/aggregate.py (comparisons to the mirrored cell) and 15 HPC submits. Their readers (aggregate, size_sweep_table, fixed_light_scaling_table, gradient_mode_library_ab, lens/solver/_corpus) resolve through the new _profile_cli helpers (legacy_landing_dir, legacy_read_dirs, locate_legacy_result, ...). Committed tables and 13/14 comparisons re-render byte-identical. test_results_write_destinations.py fails any script under scripts/, _profile_cli.py or hpc/ that names a results path outside the mirrored tree (heuristic and exclusions in its docstring). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SeDHwsR2dnwCk8Btfhej2L
…enames only (#442 PR 5) Pure git mv: each test moves to the test/ dir of the scripts/misc group whose code it exercises (repo-wide contract tests to tooling/test/, tests of cells outside scripts/misc to cells/test/). The empty scripts/misc/test/__init__.py is removed. Path and import fixups follow in the next commit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SeDHwsR2dnwCk8Btfhej2L
…isons, writer and reader (#442 PR 3b review) sweep.py now places every row itself with sweep_layout.placed_row (the day-zero classification of <class>/<model>/<instrument>/<model>_<config>[_sparse].json fed the instrument, precision and dense-route transformer the matrix fixes) and passes the exact path as the new --output-path; aggregate.py reads the same paths (rows a run wrote win over committed rows) instead of rebuilding locations from instrument/precision hints. All 308 matrix rows get distinct mirrored paths under their instrument; the 13 committed comparisons rebuild byte-equal, including interferometer/{delaunay,pixelization}/sma and datacube/delaunay/sma. Comparisons are placed from the instrument their rows record, so the two-level imaging/delaunay_nn cell goes to results/imaging/delaunay_nn/hst/. Standalone leaf runs name a two-level runtime folder under --instrument and the interferometric leaves record the transformer they hard-code, so a config-named row cannot collide across instruments. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SeDHwsR2dnwCk8Btfhej2L
…ed (#442 PR 3b review) The breakdown table lists, after the current v3 cells, every published row with no v3 cell, latest per pre-move group, read where the day-zero move put it under the name it kept: unresolved rows in results/breakdown/, the current results/misc/likelihood_breakdown/ (2 datacube/inversion rows) and archive/*/*/<slug>/breakdown/ (labelled archived). A group whose newest release the map moved into a mirrored cell is shown by that cell's v3 row. 106 rows before the move-era re-key, 110 now; every old row is present. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SeDHwsR2dnwCk8Btfhej2L
…path, CI (#442 PR 5) - moved tests: fixed Path(__file__).parents[N] indices +1, the browser check's dashboard root, and the sibling test_fixed_light_cell.py reads; - scripts/misc/conftest.py puts scripts/misc on sys.path (pytest used to do it via the old test/__init__.py); a bare root pytest.ini makes the repo root pytest's rootdir so the conftest loads even for a single test file; - lint.yml runs pytest scripts/misc/*/test and the moved browser check; - build_catalogue, apply_migration_map, the write-destination guard and test_headline_completion skip every group's test/ dir (and the conftest); - scripts/misc/test/<file> references in code comments, hpc submit comments, dashboard/README and the CI-enforced timing-noise inventory key list rewritten mechanically to the new paths (historical wiki ledgers untouched). Same 1544 tests collected before and after (identical node ids modulo dir). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SeDHwsR2dnwCk8Btfhej2L
#442 PR 5) A short index of every scripts/misc group (what it holds, where it writes, how to run its tests), READMEs for the two groups that had none, the tooling README's title and test pointer, and AGENTS.md's layout line. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SeDHwsR2dnwCk8Btfhej2L
…iases and variant names (#442 PR 3b review) test_results_write_destinations.py flattens each path expression into segments (/ joins, .joinpath, os.path.join, Path/PurePosixPath args, split constants, and names bound by a simple same-module NAME = <path> assignment), so R = ROOT / "results"; R / "runtime" is caught. A results/<dataset_type>/ path must name a variant scripts/<dataset_type>/ has (results/imaging/pixelization fails). Self-checks cover each new form; what it cannot follow (other modules, parameters, non-trivial bindings) is documented. build_readme's pre-move map prefix becomes a LEGACY_ constant. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SeDHwsR2dnwCk8Btfhej2L
…ep_layout.py (#442 PR 3b review) resolve_output_paths reads the instrument with getattr (callers pass a minimal namespace) and only re-folders legacy results paths; the catalogue's script navigation lists the new scripts/misc/likelihood_runtime/sweep_layout.py. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SeDHwsR2dnwCk8Btfhej2L
…ity records (#442 PR 6) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SeDHwsR2dnwCk8Btfhej2L
…ions copy; lint --check steps (#442 PR 6) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SeDHwsR2dnwCk8Btfhej2L
…untime/test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SeDHwsR2dnwCk8Btfhej2L
…+ coverage Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SeDHwsR2dnwCk8Btfhej2L
Jammy2211
deleted the
feature/likelihood-lifecycle-p0-profiling-coverage
branch
October 24, 2026 13:51
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PR 6 of #442, the last PR of likelihood-lifecycle phase 0B: a coverage matrix and a parity-witness convention. Stacked on #450 (PR 5), which is stacked on #449 (PR 3b). Merge in order: #449 → #450 → this. This PR's own commits are
3f40ce1e,1f3d336dand the merge6f733073.scripts/misc/tooling/coverage_matrix.pywritesdashboard/coverage.{json,md}. For likelihood × class × measurement it reports current, stale, superseded or missing, read from the v3dashboard/catalogue.jsonand the §3.3 provisional table. A new lint--checkstep stops it drifting, and it also fails if a planned cell's class disagrees with §3.3._profiling.py). A current record is stale when itsrelease_pinis behind the catalogue's newest pin (currently 2026.10.17.1) or has no pin.catalogue/provisional_designations.jsonis a machine copy of the epic report's §3.3 table, checked cell by cell, with notes on the readings that took judgment. A test keeps it in step withcatalogue_lifecycle.py.parity.jsonper cell, documented inresults/README.mdand validated byparity_witness.py --check. It holds log L, the reference log L, a tolerance, statusmatch/diverged/not_assessed, the reference source and date, and provenance. The builder emits ameasurement: parityv3 record with metricabs_delta_log_likelihoodand values{abs_delta_log_likelihood, tolerance}, because Pulse v3 requires non-negative dimensionless values. The signed values go in aparityblock on the record. Anot_assessedwitness produces no record but is listed inparity_witnesses. No parity values were made up: there are 0parity.jsonfiles, and the tests use fixtures only.Scripts Changed
scripts/misc/tooling/coverage_matrix.py(new): coverage matrix and--checkscripts/misc/tooling/parity_witness.py(new): parity.json v1 schema, validator and--checkscripts/misc/tooling/build_catalogue.py: reads parity.json into v3 parity records (history copies become superseded) and addsparity_witnessescatalogue/provisional_designations.json(new): §3.3 copy.github/workflows/lint.yml: parity_witness and coverage_matrix--checkstepstest_coverage_matrix.pyandtest_parity_witness.py; parity cases added totest_build_catalogue.pyresults/README.md("Parity witness"),catalogue/README.md,scripts/misc/tooling/README.md; regenerateddashboard/catalogue.jsonanddashboard/coverage.*Follow-ups (non-blocking review notes)
coverage_matrix.py:221: once real parity witnesses exist, their current records will be counted incurrent_records_unplanned. Excludemeasurement == "parity"there.catalogue_lifecycle.py. A test keeps them in step; later, havecatalogue_lifecycleread the JSON.Test Plan
--checkstep, including the two new ones andbuild_catalogue --check --validate-with ../PyAutoPulse(valid at @3)scripts/misc/*/test: 1600 passed, 6 skipped, 1 failed. The failure is the knowntest_hazards_prior_exit, which is environment-only and passes in CI.Validation checklist (--auto run — plan was not pre-approved)
/prm, after feat: drop the old results layout — v3-keyed trends, writers into the mirrored tree (#442 PR 3b) #449 and refactor: group scripts/misc tests by folder, READMEs per group (#442 PR 5) #450.Refs #442
🤖 Generated with Claude Code
https://claude.ai/code/session_01SeDHwsR2dnwCk8Btfhej2L