Skip to content

feat: likelihood-lifecycle 0B — mirrored results tree + profiling-summary@3 catalogue #442

Closed

Description

@Jammy2211

Overview

Phase 0 sub-issue 0B of the likelihood-lifecycle epic (ledger PyAutoMind/draft/research/autolens_profiling/likelihood_lifecycle_epic.md). Restructures autolens_profiling's results/ into the §3.7 mirrored tree from a human-reviewed day-zero migration map, then upgrades the existing catalogue builder to emit profiling-summary@3 (contract shipped in PyAutoPulse#70) so Pulse can re-pin lens to @3. This is the last open phase-0 sub-issue (0A, 0C, 0D, 0E merged). Rulings R1/R2/R3/R7 are decided; nothing is re-measured.

Plan

  1. Migration-map PR — catalogue/migration_day_zero.json: old → new path + sha256 for all 1,481 tracked results/ files, results/notes/ ledgers → wiki/campaigns/<dataset_type>/<variant>/, dead experiments → archive/YYYY/MM/<slug>/. Nothing moves; the human reviews the map first.
  2. Move-only PR — git mv into the §3.7 layout + archive/ + README pointers + results/README.md redirect table. Held until docs(cluster): record the human decision — keep the 1.0″ starting grid (#439) #441 merges and no live RAL job runs on a RAL checkout of this repo.
  3. Catalogue builder PR — extend scripts/misc/tooling/build_catalogue.py to emit profiling-summary@3 (class/status/superseded_by, planned_cells, figures[], archive_record_count), mirror test, --check current.
  4. Pulse PR — registry.yaml lens pin @2 → @3, receipt refetched (unblocks 0C's predicted_ms_per_eval).
  5. scripts/misc/ grouping PR — misc/test/ split by group, misc/searches/ orphan removed, one-offs archived, README per group.
  6. Coverage matrix + parity convention PR — coverage_matrix.py (likelihood × class × measurement → current/superseded/missing/stale); parity.json per cell read by the builder as measurement: parity.

Tier: judge — merge mode: human /prm for every PR, in order. Autonomy: supervised.

Detailed implementation plan

Affected Repositories

  • autolens_profiling (primary)
  • PyAutoPulse (one-line registry re-pin after the builder merges)

Branch Survey

Repository Current Branch Dirty?
lens/autolens_profiling main (03d36c8) untracked dataset/abell_1201/ only
organs/PyAutoPulse main clean (unregistered worktree checkin/2026-10-17, unrelated)

Claim guard worktree_check_conflict likelihood-lifecycle-p0-profiling autolens_profiling PyAutoPulse → exit 0.

Suggested branch: feature/likelihood-lifecycle-p0-profiling (PRs 1–6 as stacked/sequential branches off it, e.g. -map, -move, -builder, -misc, -coverage).

Decisions settled in the plan

  • R3 provisional Optimal rows — Pulse REFERENCE.md: class: optimal requires a decision_id; none exists before phase 3. Emit class: secondary with "provisional_optimal: true" in limitations; phase 3 flips them when designations are written.
  • Mirror test location — repo tests live under scripts/misc/test/; the mirror test (test_results_mirror_scripts.py: every results/<dataset_type>/<variant>/ has scripts/<dataset_type>/<variant>/ and vice versa, archive/ excluded) goes there; PR 5 re-homes it with its group.
  • Move PR vs. path indexes — the move breaks path lists in results/provenance_grandfathered.txt, result destinations in catalogue/script_routes.json / wiki_bindings.json, and ALLOWED_NOTES_DIRS in check_results_layout.py. Regenerate these mechanically from the map inside the move PR (paths only, no measurement content; listed in the PR body) so main never goes red between PRs.

Implementation Steps

  1. Map generator (script under scripts/misc/tooling/), emitting catalogue/migration_day_zero.json; retire legacy names in paths (pixelization → rectangular, *_numba → R1 setup_id, host/tier → host_class). Archive classes: dated suffixes, _lever, _probe, pre-2026-09 series, baselines/PreOptimizationTimes, frozen packs. sha256 recomputed after docs(cluster): record the human decision — keep the 1.0″ starting grid (#439) #441 merges.
  2. Move PR applies the map with git mv only (plus map-derived path-index rewrites); check_results_layout.py --check green.
  3. build_catalogue.py → v3 shape per PyAutoPulse REFERENCE.md "profiling-summary v3" (metric/unit/values; no setups[]/selections[]); §3.6 status rule (29 planned_baseline + 3 reference_candidate → current where the host rule holds; others on mirrored keys → superseded + superseded_by; archived → counted); R7 local_* never current; dashboard/catalogue.json regenerated; test_build_catalogue.py extended.
  4. PyAutoPulse registry.yaml lens → profiling-summary@3; pyauto-pulse check accepts it; receipt refetched.
  5. scripts/misc/ polish.
  6. coverage_matrix.py + parity.json convention documented in results/README.md.

Witness

test_results_mirror_scripts.py passes; build_catalogue --check current, every record carries class/status, exactly one current per key; pyauto-pulse check accepts the re-pinned lens at @3 and its receipt refetches; the coverage matrix lists current/superseded/missing/stale cells per likelihood × class × measurement.

Key Files

  • scripts/misc/tooling/build_catalogue.py — v2 emitter → v3
  • scripts/misc/tooling/check_results_layout.py, results/provenance_grandfathered.txt — layout gate
  • catalogue/registry.json, catalogue/script_routes.json, catalogue/wiki_bindings.json, _script_routes.py
  • results/README.md — artefact policy + redirect table
  • PyAutoPulse registry.yaml, REFERENCE.md

Original Prompt

Click to expand starting prompt

Likelihood lifecycle phase 0B: autolens_profiling mirrored tree, day-zero migration map, profiling-summary@3 catalogue builder, misc grouping, coverage matrix, parity convention

Type: feature
Target: autolens_profiling
Repos:

  • autolens_profiling
  • PyAutoPulse
    Themes:
  • profiling
  • dashboard
    Difficulty: large
    Autonomy: supervised
    Priority: high
    Status: formalised
    Consequence: judge
    Witness: tests/test_results_mirror_scripts.py passes (every results/<dataset_type>/<variant>/ has a scripts/<dataset_type>/<variant>/ and vice versa, archive/ excluded); build_catalogue --check is current and emits profiling-summary@3 with every record carrying class/status and exactly one current per key; pyauto-pulse check accepts the re-pinned lens instance at @3 and its receipt refetches; the coverage matrix tool lists current / superseded / missing / stale cells per likelihood × class × measurement
    Review-minutes: 40
    Unattended: never
    Epic: likelihood-lifecycle
    Phase: 0
    Parent: draft/research/autolens_profiling/likelihood_lifecycle_epic.md
    Filed: 2026-10-21

Phase 0 sub-issue 0B of the likelihood-lifecycle epic (ledger draft/research/autolens_profiling/likelihood_lifecycle_epic.md; report §3.1, §3.6, §3.7, §4 phase 0; survey likelihood_lifecycle_epic_surveys/01_profiling_inference_layout.md). 0A (PyAutoPulse#70, merged 2026-10-21) defined the contract this issue emits: v3 records carry likelihood, instrument, regime, per-record setup + setup_id, host_class, model_variant, measurement, metric/unit/values, class, status, superseded_by, day_zero, figures[], decision_id, release_pin, measured_at; there are no setups[]/selections[] — planned_cells (with class) replace selections; archive_record_count is required (PyAutoPulse REFERENCE.md "profiling-summary v3").

Rulings applied (confirmed 2026-10-21)

R1 setup_id = <backend>_<device>_<operator>_<precision>[_<transformer>], host/tier moves to host_class; R2 label-only history cut — nothing deleted, nothing re-run; R3 the provisional designations of report §3.3 are the phase-0 class labels (provisional, confirmed by phases 2–3; no decision_id exists yet, so every provisional Optimal row is emitted class: optimal only once a designation decision exists — until then class: secondary with a provisional_optimal: true note in limitations, or the builder refuses; decide in the plan); R7 local_* rows are never current.

What, as separate PRs in this order (data guard: moves and content never in one PR)

  1. Commit nothing new first; migration map — catalogue/migration_day_zero.json: old path → new path (+ sha256) for every tracked file under results/ (1,456 files, 326 MB, results/lens/solver/corpus 165 MB) and results/notes/ ledgers → wiki/campaigns/<dataset_type>/<variant>/; dead experiments (dated suffixes, _lever, _probe, pre-2026-09 series, baselines/PreOptimizationTimes, frozen packs) → archive/YYYY/MM/<slug>/. Legacy model names retired in paths (pixelization → rectangular, *_numba → the setup_id). The map is reviewed by the human before any move.
  2. Move-only PR (git mv only, reviewed as renames): the §3.7 layout results/<dataset_type>/<variant>/<instrument>/<setup_id>/<host_class>/{runtime,breakdown,gradient,fit,parity,provenance}.* with history/<release_pin>_<date>/ for superseded files; results/lens/<component>/, results/misc/<group>/; wiki/campaigns/<dataset_type>/<variant>/; archive/YYYY/MM/<slug>/ with a README pointer at each old parent; results/README.md redirect table. Never under a live RAL job on a RAL checkout of this repo.
  3. Catalogue builder (scripts/misc/tooling/build_catalogue.py, catalogue/registry.json, _script_routes.py): reads the mirrored tree, emits profiling-summary@3, converts the config grammar, assigns class/status by the §3.6 rule (the 29 planned_baseline + 3 reference_candidate → current provisional where the host rule holds; other records on mirrored keys → superseded with superseded_by; archived → counted), planned_cells with class, figures[] where fit.png exists; build_catalogue --check; dashboard/catalogue.json regenerated; Pulse registry.yaml lens pin → profiling-summary@3 (one-line PyAutoPulse PR after the builder merges) and the Pulse receipt refetched.
  4. scripts/misc/ grouping polish: misc/test/ split by group, misc/searches/ orphan removed, one-off scripts to archive/, a README per group.
  5. Coverage matrix tool (extend the profiling conductor's campaign grid diff or add scripts/misc/tooling/coverage_matrix.py): likelihood × class × measurement → current / superseded / missing / stale, reading the v3 catalogue and the §3.3 provisional table.
  6. Parity witness convention: parity.json per cell (reference log-likelihood, tolerance, status ∈ match / diverged / not_assessed) documented in results/README.md and read by the builder into measurement: parity records.

Out of scope

Re-measuring anything (phase 3); designation decisions (phase 3); fit.png rendering in _profile_cli.py (phase 1B); runtime_estimates.yaml (phase 5E); autolens_inference (0C); Brain conductor grid (0E).

Blocked-by (operational, not a header gate)

(Resolved 2026-10-22: that task merged as autolens_profiling#438 and the claim is released.) autolens_profiling was claimed by cluster-pointsolver-robust-settings (issue #436, branch feature/cluster-pointsolver-robust-settings, 5 unmerged commits, RAL sweeps live on 2026-10-21). The move-only PR cannot land under that branch or under live RAL jobs on a RAL checkout; start after that task ships (or the human releases the claim) — /start_dev registers this in planned.md until then.

Activity

  1. added a commit that references this issue on Oct 23, 2026
  2. Jammy2211 commented on Oct 23, 2026

    @Jammy2211
    ContributorAuthor

    PR 1 of 6 (day-zero migration map) MERGED via /prm: #444 → aeef275. Map: 1,509 files, 23 unresolved (setup axis not recorded), human §3.7 rulings applied (model_variant segment, scans/experiments, hpc_ral_cpu for CPU-on-GPU-node, _wtilde). Next: PR 2 move-only — regenerate the map first if #441 has merged; never under live RAL jobs on a RAL checkout. Issue stays open.

  3. added 2 commits that reference this issue on Oct 23, 2026
  4. Jammy2211 commented on Oct 24, 2026

    @Jammy2211
    ContributorAuthor

    PR 2 merged; PR 3 and PR 4 opened (2026-10-24)

    Remaining after that: PR 3b, PR 5 (misc grouping), PR 6 (coverage matrix and parity).

  5. added a commit that references this issue on Oct 24, 2026
  6. Jammy2211 commented on Oct 24, 2026

    @Jammy2211
    ContributorAuthor

    PR 3 + PR 4 merged (2026-10-24, /prm)

    Remaining: PR 3b (shim, legacy writers, trend re-key; Mind draft), PR 5 (scripts/misc/ grouping), PR 6 (coverage matrix and parity convention).

  7. Jammy2211 commented on Oct 24, 2026

    @Jammy2211
    ContributorAuthor

    Auto run (human launch 2026-10-24): PR 3b, PR 5, PR 6 through to open PRs

    Under AUTONOMY.md, an explicit auto launch on a supervised, judge-tier task opens each PR and flags it. Every merge is a human /prm. Heart is YELLOW, with the reason set the human acknowledged at launch: manifest drift and stale PyAutoLens release validation. Any new reason, or RED, stops the run. Each PR is gated on the full pytest suite, every lint --check step, an independent review verdict of CLEAN, and Heart. No profiling is run and no measurement artefact is moved or edited.

    PRs are stacked in this order, each branched from the one before. Merge them in order: 3b → 5 → 6.

    1. PR 3b, legacy layout: re-key the build_dashboard.py and build_readme.py trend series and README tables on the v3 identity, with an explicit old-id → v3-id map so published history continues. Route the ~30 legacy-name writers (versioned summaries, vmap probes, sweep.py/aggregate.py, HPC submits) through resolve_output_paths together with their readers, verified only by path-builder unit tests or dry runs. Delete results_layout.py, and add a test that no script writes outside the mirrored tree. Anything that only a real run can verify is listed for the next RAL batch, which needs a human go.
    2. PR 5, scripts/misc/ grouping: split misc/test/ by group (git mv plus import and conftest fixes only), remove the misc/searches/ orphan, move one-off scripts to archive/, and add a README per group.
    3. PR 6, coverage and parity:
      • scripts/misc/tooling/coverage_matrix.py: likelihood × class × measurement → current, superseded, missing or stale, read from the v3 catalogue and the §3.3 provisional table. Standard-size cells covered only by a size fallback count as missing.
      • A parity.json convention per cell (reference log-likelihood, tolerance, status match, diverged or not_assessed), documented in results/README.md and read by the builder into measurement: parity records.
      • Re-running build_catalogue --check --validate-with ../PyAutoPulse must stay valid at @3.

    The 3b Mind draft is folded into this task, so it no longer appears as separate backlog.

  8. Jammy2211 commented on Oct 24, 2026

    @Jammy2211
    ContributorAuthor

    Auto run finished: PR 3b, PR 5 and PR 6 are open, stacked. Merge in this order with /prm: #449 → #450 → #451.

    PR What Independent review
    #449 (3b) Trend series and README tables keyed on the v3 identity, with the committed catalogue/series_id_map.json (144 series and 158 points before, 145 and 159 after; nothing lost). ~40 writer sites and 15 HPC submits now write into the mirrored tree through one placement rule. results_layout.py is deleted. A guard test stops writes outside the mirrored tree. FINDINGS: 1 blocking (interferometer/datacube sweep rows colliding across instruments, with aggregate reading stale rows) plus 3 non-blocking. All fixed; re-review CLEAN
    #450 (5) scripts/misc/test/ split into scripts/misc/<group>/test/. The move commit is renames only, and the same 1554 test ids are collected before and after. A README per group. misc/searches/ was already gone, and nothing was archived. CLEAN
    #451 (6) coverage_matrix.py → dashboard/coverage.{json,md}, with a lint check. Of 246 planned cells: 2 current, 6 stale, 238 missing (32 of them size fallbacks, per the size ruling). The parity.json convention, read into measurement: parity v3 records; no witnesses exist yet and none were made up. CLEAN (2 non-blocking follow-ups in the PR body)

    Heart was YELLOW throughout, within the human-acknowledged reasons. A mid-run RED (local PyAutoLens behind origin) was cleared by a fast-forward the human approved. Profiling items that only a real run can verify are listed in #449. Once #451 merges, phase 0B is complete.

  9. added 3 commits that reference this issue on Oct 24, 2026
  10. Jammy2211 commented on Oct 24, 2026

    @Jammy2211
    ContributorAuthor

    Shipped: phase 0B complete (2026-10-24)

    All seven PRs are merged:

    PR What Merge
    #444 Day-zero migration map (human-reviewed; layout rulings; size standard Delaunay 1250, rectangular 35² = 1225) aeef275
    #447 Move-only: results/ restructured to mirror scripts/, plus archive/ 0c6be19
    #448 Catalogue builder emitting profiling-summary@3 (126 current, 319 superseded, archive 99) 81183fd
    PyAutoLabs/PyAutoPulse#74 Pulse lens moved to @3, receipt refetched (pyauto-pulse check OK) 5f3ec2c8
    #449 No old layout left: v3-keyed trends (history kept through series_id_map.json), writers into the mirrored tree, results_layout.py deleted, write guard 572d32f
    #450 scripts/misc/ tests grouped by folder (same 1554 tests), a README per group 73b9385
    #451 Coverage matrix (dashboard/coverage.*) and the parity.json convention 9bc9eb3

    Witness:

    • The mirror test passes.
    • build_catalogue --check is current at @3, every record carries class and status, and each key has exactly one current record.
    • pyauto-pulse check accepts lens at @3 and its receipt refetched.
    • The coverage matrix lists current, superseded, missing and stale cells. Of 246 planned cells, 2 are current, 6 stale and 238 missing (32 of those via the size fallback).

    Carried forward:

    This completes phase 0 of the likelihood-lifecycle epic (0A–0E all merged).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions