From c2dfa0d8e8c2de747df99a916bab55885610a957 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 12 Sep 2026 05:28:03 +0000 Subject: [PATCH 1/4] =?UTF-8?q?cortex:=20one=20ledger=20per=20project=20?= =?UTF-8?q?=E2=80=94=20tasks,=20rulings=20and=20gates=20retire=20to=20arch?= =?UTF-8?q?ive/?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Cortex is no longer a task/ruling state machine. Each science project now has one file, projects/.md, holding `## Now` (where the human is, what next), `## Runs` (jobs on the cluster, open | running) and `## Log` (dated entries newest-first: run | result | lesson | note). Nothing in a ledger is inferred from results: the script records cluster facts, and a result or lesson is written only in the human's words. - scripts/cortex.py: check · new · run · running · done · log · now · issue · retire, over the new schema; move/rule/gates are gone. - projects/: every task, run, ruling and note of 2026-08/09 migrated into its project's log in the words already on file; tasks/, rulings/ and batches/ moved under archive/, frozen. - scripts/ledger_merge.py: ledger = projects/, checkin.yaml, the two pages; any change under archive/ is code. - workflows watch projects/ and archive/; README is one opening paragraph plus the dashboard pointer; AGENTS.md, REFERENCE.md rewritten; schema decision 60 records why. - tests: new skeleton fixture (four rows, three ledgers) and contract tests for every verb and every check rule. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_011UbmGoiKHHsNmMWp9GH4se --- .github/workflows/cortex_check.yml | 10 +- .github/workflows/dashboard_refresh.yml | 17 +- .github/workflows/ledger_merge.yml | 27 +- .github/workflows/pages_dashboard.yml | 2 +- AGENTS.md | 224 +-- README.md | 95 +- REFERENCE.md | 842 ++------- archive/AGENTS.md | 17 + {batches => archive/batches}/2026-08-31-am.md | 0 {batches => archive/batches}/2026-08-31-pm.md | 0 {batches => archive/batches}/2026-09-02-pm.md | 0 {batches => archive/batches}/AGENTS.md | 2 + .../batches}/reviews/2026-08-31-am.md | 0 .../batches}/reviews/2026-08-31-pm.md | 0 .../batches}/reviews/2026-09-02-pm.md | 0 .../batches}/reviews/AGENTS.md | 2 + .../rulings}/2026/08/R-20260831-01.md | 0 .../rulings}/2026/08/R-20260831-02.md | 0 .../rulings}/2026/08/R-20260831-03.md | 0 .../rulings}/2026/08/R-20260831-04.md | 0 .../rulings}/2026/08/R-20260831-05.md | 0 .../rulings}/2026/08/R-20260831-06.md | 0 .../rulings}/2026/08/R-20260831-07.md | 0 .../rulings}/2026/08/R-20260831-08.md | 0 .../rulings}/2026/09/R-20260901-01.md | 0 .../rulings}/2026/09/R-20260901-02.md | 0 .../rulings}/2026/09/R-20260901-03.md | 0 .../rulings}/2026/09/R-20260901-04.md | 0 .../rulings}/2026/09/R-20260902-01.md | 0 .../rulings}/2026/09/R-20260902-02.md | 0 .../rulings}/2026/09/R-20260902-03.md | 0 .../rulings}/2026/09/R-20260902-04.md | 0 .../rulings}/2026/09/R-20260902-05.md | 0 .../rulings}/2026/09/R-20260902-06.md | 0 .../rulings}/2026/09/R-20260902-07.md | 0 .../rulings}/2026/09/R-20260902-08.md | 0 .../rulings}/2026/09/R-20260902-09.md | 0 .../rulings}/2026/09/R-20260902-10.md | 0 .../rulings}/2026/09/R-20260904-01.md | 0 .../rulings}/2026/09/R-20260904-02.md | 0 .../rulings}/2026/09/R-20260904-03.md | 0 .../rulings}/2026/09/R-20260904-04.md | 0 .../rulings}/2026/09/R-20260904-05.md | 0 .../rulings}/2026/09/R-20260907-01.md | 0 .../rulings}/2026/09/R-20260907-02.md | 0 .../rulings}/2026/09/R-20260907-03.md | 0 .../rulings}/2026/09/R-20260907-04.md | 0 .../rulings}/2026/09/R-20260907-05.md | 0 .../rulings}/2026/09/R-20260907-06.md | 0 .../rulings}/2026/09/R-20260910-01.md | 0 .../rulings}/2026/09/R-20260910-02.md | 0 .../rulings}/2026/09/R-20260910-03.md | 0 .../rulings}/2026/09/R-20260910-04.md | 0 {rulings => archive/rulings}/AGENTS.md | 2 + .../analytic_gaussian/ensemble_parity.md | 0 .../minimal_ep_legb_mu_threshold.md | 0 .../autolens_inference/slam_hst_base.md | 0 .../ep_toy_gaussian/nuts_vs_ep_x3_n50.md | 0 .../tasks}/euclid/magnification_robustness.md | 0 .../resimulate_fitted_lens_simulator.md | 0 .../tasks}/euclid/sersic_index_recovery.md | 0 .../dr1_prelim_10_lens_science_run.md | 0 ...ordered_mge_control_unordered_102005065.md | 0 .../ordered_mge_witness_102005065.md | 0 .../ordered_rerun_catalogue_parity.md | 0 .../tasks}/ic50_workspace/ep_scale_up.md | 0 .../cluster_extended_source_inference.md | 0 .../cluster_gradient_search_benchmark.md | 0 .../delaunay_fp64_retro_baseline.md | 0 .../failed_submissions_342008_10.md | 0 .../inference_refs_v1_redo.md | 0 .../inference_programme/knn_rerun_342016_7.md | 0 .../legacy_point_output_sweep.md | 0 .../likelihood_term_levers.md | 0 .../mass_pix_gradient_cost_probe.md | 0 .../mge_fp64_retro_baseline.md | 0 .../inference_programme/mge_pos_ref_reuse.md | 0 .../multiband_compile_census_completion.md | 0 .../nautilus_mass_pix_baseline.md | 0 .../phase2_nss_mainline_gate_a_reuse.md | 0 ...hase3_prodigy_reliability_gate_b1_reuse.md | 0 .../phase4_positions_gate_b2_reuse.md | 0 .../phase5_mesh_gradient_positions_on.md | 0 .../inference_programme/prodigy_mass_pix.md | 0 .../inference_programme/refs_5_6_342016_56.md | 0 .../refs_v1_positions_on_completion.md | 0 .../inference_programme/rewind_2026_08_31.md | 0 .../inference_programme/slogdet_ab_342017.md | 0 .../inference_programme/smc_probe_342018.md | 0 .../tasks}/slope_hierarchy/methods_writeup.md | 0 .../slope_hierarchy_scale/n25_scale_up.md | 0 .../delaunay_adapt_split_pl_eff_0.md | 0 .../delaunay_adapt_split_pl_eff_1_outer.md | 0 .../delaunay_adapt_split_pl_sersic_0.md | 0 .../rectangular_adapt_pl_eff_0.md | 0 .../rectangular_adapt_pl_eff_1_outer.md | 0 .../rectangular_adapt_pl_sersic_0.md | 0 docs/schema_decisions.md | 37 + projects/AGENTS.md | 26 + projects/analytic_gaussian.md | 20 + projects/autolens_inference.md | 15 + projects/ep_toy_gaussian.md | 20 + projects/euclid.md | 16 + projects/euclid_dr1_prelim.md | 93 + projects/ic50_workspace.md | 24 + projects/inference_programme.md | 182 ++ projects/slope_hierarchy.md | 14 + projects/slope_hierarchy_scale.md | 24 + projects/subhalo_validation.md | 103 ++ scripts/cortex.py | 1636 ++++++----------- scripts/ledger_merge.py | 50 +- tests/fixtures/empty/projects.yaml | 2 +- .../{rulings/.gitkeep => projects/.keep} | 0 tests/fixtures/empty/tasks/.gitkeep | 0 .../skeleton/batches/2026-09-01-pm.md | 10 - tests/fixtures/skeleton/checkin.yaml | 1 + tests/fixtures/skeleton/projects.yaml | 62 +- tests/fixtures/skeleton/projects/example.md | 26 + tests/fixtures/skeleton/projects/single.md | 14 + .../fixtures/skeleton/projects/wound_down.md | 16 + .../skeleton/rulings/2026/09/R-20260901-01.md | 18 - .../skeleton/rulings/2026/09/R-20260901-02.md | 22 - .../skeleton/rulings/2026/09/R-20260901-03.md | 18 - .../skeleton/rulings/2026/09/R-20260901-04.md | 19 - .../skeleton/rulings/2026/09/R-20260901-05.md | 19 - .../skeleton/tasks/example/01_scope.md | 29 - .../skeleton/tasks/example/02_gated_on_dev.md | 30 - .../tasks/example/03_ready_cleared.md | 29 - .../tasks/example/04_submitted_override.md | 33 - .../tasks/example/05_running_array.md | 37 - .../skeleton/tasks/example/06_pulled.md | 32 - .../tasks/example/07_awaiting_ruling.md | 34 - .../skeleton/tasks/example/08_accepted.md | 35 - .../skeleton/tasks/example/09_rerun.md | 34 - .../skeleton/tasks/example/10_dropped.md | 34 - tests/test_cortex.py | 1232 +++---------- tests/test_ledger_merge.py | 284 ++- 137 files changed, 1969 insertions(+), 3601 deletions(-) create mode 100644 archive/AGENTS.md rename {batches => archive/batches}/2026-08-31-am.md (100%) rename {batches => archive/batches}/2026-08-31-pm.md (100%) rename {batches => archive/batches}/2026-09-02-pm.md (100%) rename {batches => archive/batches}/AGENTS.md (75%) rename {batches => archive/batches}/reviews/2026-08-31-am.md (100%) rename {batches => archive/batches}/reviews/2026-08-31-pm.md (100%) rename {batches => archive/batches}/reviews/2026-09-02-pm.md (100%) rename {batches => archive/batches}/reviews/AGENTS.md (79%) rename {rulings => archive/rulings}/2026/08/R-20260831-01.md (100%) rename {rulings => archive/rulings}/2026/08/R-20260831-02.md (100%) rename {rulings => archive/rulings}/2026/08/R-20260831-03.md (100%) rename {rulings => archive/rulings}/2026/08/R-20260831-04.md (100%) rename {rulings => archive/rulings}/2026/08/R-20260831-05.md (100%) rename {rulings => archive/rulings}/2026/08/R-20260831-06.md (100%) rename {rulings => archive/rulings}/2026/08/R-20260831-07.md (100%) rename {rulings => archive/rulings}/2026/08/R-20260831-08.md (100%) rename {rulings => archive/rulings}/2026/09/R-20260901-01.md (100%) rename {rulings => archive/rulings}/2026/09/R-20260901-02.md (100%) rename {rulings => archive/rulings}/2026/09/R-20260901-03.md (100%) rename {rulings => archive/rulings}/2026/09/R-20260901-04.md (100%) rename {rulings => archive/rulings}/2026/09/R-20260902-01.md (100%) rename {rulings => archive/rulings}/2026/09/R-20260902-02.md (100%) rename {rulings => archive/rulings}/2026/09/R-20260902-03.md (100%) rename {rulings => archive/rulings}/2026/09/R-20260902-04.md (100%) rename {rulings => archive/rulings}/2026/09/R-20260902-05.md (100%) rename {rulings => archive/rulings}/2026/09/R-20260902-06.md (100%) rename {rulings => archive/rulings}/2026/09/R-20260902-07.md (100%) rename {rulings => archive/rulings}/2026/09/R-20260902-08.md (100%) rename {rulings => archive/rulings}/2026/09/R-20260902-09.md (100%) rename {rulings => archive/rulings}/2026/09/R-20260902-10.md (100%) rename {rulings => archive/rulings}/2026/09/R-20260904-01.md (100%) rename {rulings => archive/rulings}/2026/09/R-20260904-02.md (100%) rename {rulings => archive/rulings}/2026/09/R-20260904-03.md (100%) rename {rulings => archive/rulings}/2026/09/R-20260904-04.md (100%) rename {rulings => archive/rulings}/2026/09/R-20260904-05.md (100%) rename {rulings => archive/rulings}/2026/09/R-20260907-01.md (100%) rename {rulings => archive/rulings}/2026/09/R-20260907-02.md (100%) rename {rulings => archive/rulings}/2026/09/R-20260907-03.md (100%) rename {rulings => archive/rulings}/2026/09/R-20260907-04.md (100%) rename {rulings => archive/rulings}/2026/09/R-20260907-05.md (100%) rename {rulings => archive/rulings}/2026/09/R-20260907-06.md (100%) rename {rulings => archive/rulings}/2026/09/R-20260910-01.md (100%) rename {rulings => archive/rulings}/2026/09/R-20260910-02.md (100%) rename {rulings => archive/rulings}/2026/09/R-20260910-03.md (100%) rename {rulings => archive/rulings}/2026/09/R-20260910-04.md (100%) rename {rulings => archive/rulings}/AGENTS.md (93%) rename {tasks => archive/tasks}/analytic_gaussian/ensemble_parity.md (100%) rename {tasks => archive/tasks}/analytic_gaussian/minimal_ep_legb_mu_threshold.md (100%) rename {tasks => archive/tasks}/autolens_inference/slam_hst_base.md (100%) rename {tasks => archive/tasks}/ep_toy_gaussian/nuts_vs_ep_x3_n50.md (100%) rename {tasks => archive/tasks}/euclid/magnification_robustness.md (100%) rename {tasks => archive/tasks}/euclid/resimulate_fitted_lens_simulator.md (100%) rename {tasks => archive/tasks}/euclid/sersic_index_recovery.md (100%) rename {tasks => archive/tasks}/euclid_dr1_prelim/dr1_prelim_10_lens_science_run.md (100%) rename {tasks => archive/tasks}/euclid_dr1_prelim/ordered_mge_control_unordered_102005065.md (100%) rename {tasks => archive/tasks}/euclid_dr1_prelim/ordered_mge_witness_102005065.md (100%) rename {tasks => archive/tasks}/euclid_dr1_prelim/ordered_rerun_catalogue_parity.md (100%) rename {tasks => archive/tasks}/ic50_workspace/ep_scale_up.md (100%) rename {tasks => archive/tasks}/inference_programme/cluster_extended_source_inference.md (100%) rename {tasks => archive/tasks}/inference_programme/cluster_gradient_search_benchmark.md (100%) rename {tasks => archive/tasks}/inference_programme/delaunay_fp64_retro_baseline.md (100%) rename {tasks => archive/tasks}/inference_programme/failed_submissions_342008_10.md (100%) rename {tasks => archive/tasks}/inference_programme/inference_refs_v1_redo.md (100%) rename {tasks => archive/tasks}/inference_programme/knn_rerun_342016_7.md (100%) rename {tasks => archive/tasks}/inference_programme/legacy_point_output_sweep.md (100%) rename {tasks => archive/tasks}/inference_programme/likelihood_term_levers.md (100%) rename {tasks => archive/tasks}/inference_programme/mass_pix_gradient_cost_probe.md (100%) rename {tasks => archive/tasks}/inference_programme/mge_fp64_retro_baseline.md (100%) rename {tasks => archive/tasks}/inference_programme/mge_pos_ref_reuse.md (100%) rename {tasks => archive/tasks}/inference_programme/multiband_compile_census_completion.md (100%) rename {tasks => archive/tasks}/inference_programme/nautilus_mass_pix_baseline.md (100%) rename {tasks => archive/tasks}/inference_programme/phase2_nss_mainline_gate_a_reuse.md (100%) rename {tasks => archive/tasks}/inference_programme/phase3_prodigy_reliability_gate_b1_reuse.md (100%) rename {tasks => archive/tasks}/inference_programme/phase4_positions_gate_b2_reuse.md (100%) rename {tasks => archive/tasks}/inference_programme/phase5_mesh_gradient_positions_on.md (100%) rename {tasks => archive/tasks}/inference_programme/prodigy_mass_pix.md (100%) rename {tasks => archive/tasks}/inference_programme/refs_5_6_342016_56.md (100%) rename {tasks => archive/tasks}/inference_programme/refs_v1_positions_on_completion.md (100%) rename {tasks => archive/tasks}/inference_programme/rewind_2026_08_31.md (100%) rename {tasks => archive/tasks}/inference_programme/slogdet_ab_342017.md (100%) rename {tasks => archive/tasks}/inference_programme/smc_probe_342018.md (100%) rename {tasks => archive/tasks}/slope_hierarchy/methods_writeup.md (100%) rename {tasks => archive/tasks}/slope_hierarchy_scale/n25_scale_up.md (100%) rename {tasks => archive/tasks}/subhalo_validation/delaunay_adapt_split_pl_eff_0.md (100%) rename {tasks => archive/tasks}/subhalo_validation/delaunay_adapt_split_pl_eff_1_outer.md (100%) rename {tasks => archive/tasks}/subhalo_validation/delaunay_adapt_split_pl_sersic_0.md (100%) rename {tasks => archive/tasks}/subhalo_validation/rectangular_adapt_pl_eff_0.md (100%) rename {tasks => archive/tasks}/subhalo_validation/rectangular_adapt_pl_eff_1_outer.md (100%) rename {tasks => archive/tasks}/subhalo_validation/rectangular_adapt_pl_sersic_0.md (100%) create mode 100644 projects/AGENTS.md create mode 100644 projects/analytic_gaussian.md create mode 100644 projects/autolens_inference.md create mode 100644 projects/ep_toy_gaussian.md create mode 100644 projects/euclid.md create mode 100644 projects/euclid_dr1_prelim.md create mode 100644 projects/ic50_workspace.md create mode 100644 projects/inference_programme.md create mode 100644 projects/slope_hierarchy.md create mode 100644 projects/slope_hierarchy_scale.md create mode 100644 projects/subhalo_validation.md rename tests/fixtures/empty/{rulings/.gitkeep => projects/.keep} (100%) delete mode 100644 tests/fixtures/empty/tasks/.gitkeep delete mode 100644 tests/fixtures/skeleton/batches/2026-09-01-pm.md create mode 100644 tests/fixtures/skeleton/checkin.yaml create mode 100644 tests/fixtures/skeleton/projects/example.md create mode 100644 tests/fixtures/skeleton/projects/single.md create mode 100644 tests/fixtures/skeleton/projects/wound_down.md delete mode 100644 tests/fixtures/skeleton/rulings/2026/09/R-20260901-01.md delete mode 100644 tests/fixtures/skeleton/rulings/2026/09/R-20260901-02.md delete mode 100644 tests/fixtures/skeleton/rulings/2026/09/R-20260901-03.md delete mode 100644 tests/fixtures/skeleton/rulings/2026/09/R-20260901-04.md delete mode 100644 tests/fixtures/skeleton/rulings/2026/09/R-20260901-05.md delete mode 100644 tests/fixtures/skeleton/tasks/example/01_scope.md delete mode 100644 tests/fixtures/skeleton/tasks/example/02_gated_on_dev.md delete mode 100644 tests/fixtures/skeleton/tasks/example/03_ready_cleared.md delete mode 100644 tests/fixtures/skeleton/tasks/example/04_submitted_override.md delete mode 100644 tests/fixtures/skeleton/tasks/example/05_running_array.md delete mode 100644 tests/fixtures/skeleton/tasks/example/06_pulled.md delete mode 100644 tests/fixtures/skeleton/tasks/example/07_awaiting_ruling.md delete mode 100644 tests/fixtures/skeleton/tasks/example/08_accepted.md delete mode 100644 tests/fixtures/skeleton/tasks/example/09_rerun.md delete mode 100644 tests/fixtures/skeleton/tasks/example/10_dropped.md diff --git a/.github/workflows/cortex_check.yml b/.github/workflows/cortex_check.yml index 3d49597..27422f3 100644 --- a/.github/workflows/cortex_check.yml +++ b/.github/workflows/cortex_check.yml @@ -13,9 +13,8 @@ on: push: branches: [main] paths: - - "tasks/**" - - "rulings/**" - - "batches/**" + - "projects/**" + - "archive/**" - "projects.yaml" - "checkin.yaml" - "dashboard.md" @@ -25,9 +24,8 @@ on: - ".github/workflows/cortex_check.yml" pull_request: paths: - - "tasks/**" - - "rulings/**" - - "batches/**" + - "projects/**" + - "archive/**" - "projects.yaml" - "checkin.yaml" - "dashboard.md" diff --git a/.github/workflows/dashboard_refresh.yml b/.github/workflows/dashboard_refresh.yml index 340278a..82264cd 100644 --- a/.github/workflows/dashboard_refresh.yml +++ b/.github/workflows/dashboard_refresh.yml @@ -2,10 +2,9 @@ name: Dashboard Refresh # Keeps the generated Cortex pages (`dashboard.md`, linked from the README, and # `dashboard.html`, the Pages index) in step with the registry they describe. -# The pages are rendered by PyAutoBrain's cortex conductor from `tasks/`, -# `rulings/`, `batches/`, `projects.yaml` and `checkin.yaml`, so any push that -# files a task, moves it, records a run, lands a ruling or stamps a check-in -# makes them stale. +# The pages are rendered by PyAutoBrain's cortex conductor from `projects/`, +# `projects.yaml` and `checkin.yaml`, so any push that records a run, logs an +# entry, rewrites a `## Now` or stamps a check-in makes them stale. # # Same shape as the Mind's dashboard_refresh.yml, and for the same reason: on # pull requests a stale page is an error the author fixes on the branch; on @@ -30,18 +29,16 @@ on: push: branches: [main] paths: - - "tasks/**" - - "rulings/**" - - "batches/**" + - "projects/**" + - "archive/**" - "projects.yaml" - "checkin.yaml" - "dashboard.md" - "dashboard.html" pull_request: paths: - - "tasks/**" - - "rulings/**" - - "batches/**" + - "projects/**" + - "archive/**" - "projects.yaml" - "checkin.yaml" - "dashboard.md" diff --git a/.github/workflows/ledger_merge.yml b/.github/workflows/ledger_merge.yml index 0a50911..034bd6b 100644 --- a/.github/workflows/ledger_merge.yml +++ b/.github/workflows/ledger_merge.yml @@ -3,7 +3,7 @@ name: Cortex Ledger Merge # Lands PyAutoCortex's own ledger work without anyone having to ask for it. # # THE PROBLEM. A branch-scoped session — the phone, claude.ai/code, any -# `claude/**` flow — pushes a task move, a ruling, a batch record to a feature +# `claude/**` flow — pushes a run recorded, an entry logged, a check-in to a feature # branch and it stops there: nothing downstream looks at a `claude/**` push. # The Mind learned this first (mind_ledger_merge.yml); the Cortex is born with # the answer. @@ -15,21 +15,20 @@ name: Cortex Ledger Merge # pushed it ever comes back. # # WHAT IT WILL AND WILL NOT MERGE. Only a branch whose whole diff is *ledger*: -# `tasks/`, `rulings/`, `batches/`, `checkin.yaml`. The line is drawn by +# `projects/`, `checkin.yaml`, the two generated pages. The line is drawn by # scripts/ledger_merge.py — a script, so it is testable and a session can # predict the verdict — and it is DEFAULT DENY: anything under `scripts/`, # `tests/`, `.github/`, `policy/`, `docs/`, `projects.yaml`, the prose pages, # or any path nobody has classified, stops the merge and leaves the branch for -# a human. So does a MODIFIED or DELETED file under `rulings/` — the ledger of -# record is append-only, and the classifier reads `git diff --name-status` to -# hold that line. This workflow's own file is on the code side, which is the +# a human. So does ANY change under `archive/` — the frozen 2026-08/09 task, +# ruling and batch ledgers, which nothing writes any more. This workflow's own file is on the code side, which is the # intended self-consistency: it cannot merge a change to itself. # # THE TRIAL-MERGE CHECK. `cortex.py check` is run on the branch tip (so audit # mode reports something) and AGAIN on the merged tree, after `git merge # --no-ff` and before `git push`. That second run is what catches the -# ruling-id race: two branches that both assign `R--03` each pass alone -# and fail merged. A failing check resets the trial merge and leaves the +# same-ledger race: two branches that each edit one project's ledger pass +# alone and can fail merged. A failing check resets the trial merge and leaves the # branch for a human. on: @@ -148,14 +147,14 @@ jobs: if [ "$RC" -ne 0 ]; then echo "verdict=code" >> "$GITHUB_OUTPUT" echo "" >> "$GITHUB_STEP_SUMMARY" - echo "**Left for a human** — this branch changes code, or edits a ruling, not just the ledger." >> "$GITHUB_STEP_SUMMARY" + echo "**Left for a human** — this branch changes code, or touches the archive, not just the ledger." >> "$GITHUB_STEP_SUMMARY" else echo "verdict=ledger" >> "$GITHUB_OUTPUT" fi - # Structural drift on the branch tip BLOCKS. A task in a state its - # header contradicts, a dangling Supersedes:, a run line that does not - # parse — nothing heals that automatically, and merging it would put the + # Structural drift on the branch tip BLOCKS. A run line that does not + # parse, a log that is not newest-first, an active project with no + # ledger — nothing heals that automatically, and merging it would put the # contradiction on main. Also the only check audit mode gets to report. # `cortex.py` reads `projects.yaml` with PyYAML — its one dependency. - name: dependencies @@ -205,12 +204,12 @@ jobs: exit 1 fi # THE TRIAL-MERGE CHECK. The branch passed alone; does it pass - # merged? This is where two branches that both assigned the same - # ruling id, or both moved the same task, are caught. + # merged? This is where two branches that both wrote the same + # project's ledger are caught. if ! python3 scripts/cortex.py check; then git reset --hard origin/main echo "::error::'$BRANCH' merges cleanly but 'cortex.py check' fails on the merged tree — left for a human" - echo "**Blocked** — \`$BRANCH\` fails \`cortex.py check\` once merged with main (a ruling-id or task-state race?); left for a human." >> "$GITHUB_STEP_SUMMARY" + echo "**Blocked** — \`$BRANCH\` fails \`cortex.py check\` once merged with main (two branches wrote the same ledger?); left for a human." >> "$GITHUB_STEP_SUMMARY" exit 1 fi if [ -n "$PR" ]; then diff --git a/.github/workflows/pages_dashboard.yml b/.github/workflows/pages_dashboard.yml index 6a05ef4..a79fd77 100644 --- a/.github/workflows/pages_dashboard.yml +++ b/.github/workflows/pages_dashboard.yml @@ -15,7 +15,7 @@ name: Pages Dashboard # `dashboard.md` (one census, two pages) and kept fresh by the same self-heal in # `dashboard_refresh.yml`; this workflow only publishes it. # Only the uploaded files reach Pages — nothing else in the repo, and in -# particular not `tasks/`, `rulings/` or `projects.yaml`, whose absolute laptop +# particular not `projects/` or `projects.yaml`, whose absolute laptop # paths are not for a public site. on: diff --git a/AGENTS.md b/AGENTS.md index 3d14104..94061ef 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -2,7 +2,7 @@ This file is for AI coding agents (Claude Code, Codex, Cursor, etc.) and humans discovering this repository. PyAutoCortex is the **Cortex** organ of the PyAuto -organism — where the organism learns what is true. +organism — where the organism keeps track of what is true. **You are one organ of the PyAuto organism** — an agentic ecosystem for @@ -15,7 +15,7 @@ Canonical boundaries live in `PyAutoBrain/ORGANISM.md`; the full body map |-------|------|------| | **Brain** | PyAutoBrain | Reasoning/orchestration layer; how work is decomposed and routed; the specialist agents. | | **Mind** | PyAutoMind | Intent, goals, priorities, workflow state; every task starts as a markdown prompt here. | -| **Cortex** | PyAutoCortex | The Cortex — where the organism learns what is true: the science body map (`projects.yaml`) and the rulings of record for every science run; the science mirror of the Mind (runs and rulings, not prompts and PRs). | +| **Cortex** | PyAutoCortex | The Cortex — where the organism keeps track of what is true: the science body map (`projects.yaml`) and one ledger per science project (what was run, what came back, what was learned, where to pick up); the science mirror of the Mind (runs and a dated log, not prompts and PRs). | | **Memory** | PyAutoMemory | Long-term scientific/software/project knowledge (see science pointer below). | | **Heart** | PyAutoHeart | Health/readiness — the authoritative "is it safe to release?" verdict. | | **Hands** | PyAutoHands | Packaging, tagging, notebook generation, PyPI release execution. | @@ -26,48 +26,36 @@ Call chain (always this order): **Brain → Heart (gate) → Build (execute)**. Generated from `PyAutoMind/repos.yaml` + `PyAutoBrain/ORGANISM.md`; edit there, then run `python3 PyAutoMind/scripts/repos_sync.py --write`. - ## What this repo is **The Mind decides what to build, the Brain routes the work and executes -nothing, the Cortex learns what is true.** The Cortex is a **run-and-ruling -registry**: `project → task → runs → rulings`. It owns two pieces of state no +nothing, the Cortex keeps track of what is true.** The Cortex is a +**project ledger**: `project → runs → log`. It owns two pieces of state no other organ holds — the **science body map** (`projects.yaml`: every science project, where it lives on the laptop and on RAL, how it syncs, where its own -commentary ledger is) and the **rulings ledger of record** (`rulings/`: every -verdict a human has passed on a science run, append-only). - -It is **not a second PyAutoMind.** The Mind is PR-shaped all the way down — -prompts, issues, branches, completion records, "delivered = a PR with a diff -and checks". The Cortex's unit is a **task** of a **project**, which spawns -**runs** (SLURM job ids) and ends in a **ruling**. Development work a science -task waits on stays in the Mind and is named here only as **gates** — GitHub -issue/PR refs in a task's `Gates:` header, one grammar, one direction. The -Mind learns nothing about the Cortex beyond a render-time badge. - -**Tasks, not phases** (2026-09-07). A project holds *unordered* tasks — ideas -the human runs in whatever order the results and the priorities dictate — so -there is no number and no sequence. A task's identity is its **slug**, unique -per project by the path itself, and every task carries a required `Summary:` -of at most ten words: the *question* it answers, which is the one line the -board shows. Mind **epics** are still numbered phases; that is development -work, and it really is sequential. - -The schemas — task files, the run-line grammar, rulings, batches, the -restricted `projects.yaml` subset — and the `scripts/cortex.py` verbs are in -[REFERENCE.md](REFERENCE.md). Every choice the birth epic did not fix is dated -in [docs/schema_decisions.md](docs/schema_decisions.md); read decisions there, -do not re-derive them. - -## The ruling of record - -**A verdict recorded only outside the Cortex does not exist.** `rulings/` is -canonical. A project's own ledger (`DECISIONS.md`, `state.md`, `RESULTS.md`) -remains as scientific commentary — evidence, reasoning, consequences — and -cites the ruling id. Rulings are **append-only**: a wrong ruling is superseded -by a new one (`Supersedes: R-…`) and never edited; `scripts/ledger_merge.py` -treats any modification or deletion under `rulings/` as code, which is a -human's turn. Full rules in [rulings/AGENTS.md](rulings/AGENTS.md). +commentary ledger is) and **one ledger per project** (`projects/.md`: +where the human is with it, what is on the cluster, and a dated history of +what they set off, saw and learned). + +It is **not a second PyAutoMind**, and it is **not a task tracker** +(2026-09-12, schema decision 60). Science is not worked as a queue of +pre-registered questions that get done and retired: the human does runs, looks +at results, thinks, and organically decides where to go next. So a ledger +holds no state machine, no witness, no verdict — it holds a **Now**, the +**Runs** on the cluster, and a **Log** that only ever gets longer. The board is +a live view of what has been going through the human's head on each project, +and the thing that makes a project easy to pick up after time away. + +**Nothing in a ledger is inferred from results.** A run's submission, start +and end are cluster facts and the script records them. A `result` or a +`lesson` is the human's own words, written on their ask. An agent that reads a +results file tells the human what it sees; the human says what to log. + +The schema — the ledger file, the run and log line grammars, the +`projects.yaml` subset — and the `scripts/cortex.py` verbs are in +[REFERENCE.md](REFERENCE.md). Every choice is dated in +[docs/schema_decisions.md](docs/schema_decisions.md); read decisions there, do +not re-derive them. ## Layout (operational) @@ -75,50 +63,48 @@ human's turn. Full rules in [rulings/AGENTS.md](rulings/AGENTS.md). code, not ledger**: `sync_cli` and `local_path` are paths a conductor will execute under, so a change to it is always a human's turn. `cortex.py` reads it with PyYAML and validates the fields (REFERENCE.md "projects.yaml"). -- **`tasks//.md`** — one file per task, with the Mind's light - header (`Key: value` lines, no YAML frontmatter — `Project:`, `Summary:`, - `State:`, …) and a `## Runs` body in the SLURM run-line grammar. There is no - `Phase:` number: the slug is the identity, and `check` names `Phase:` as a - retired key. `State:` is one of `planned | gated | ready | - submitted | running | pulled | awaiting-ruling | accepted | rerun | dropped`. - `legacy` and `legacy_wrong` are states of a **run**, never of a task. -- **`rulings///R--.md`** — the ledger of record. -- **`batches/`** — closed history. Three 2026-08/09 batch records and the - human's verbatim reviews under `reviews/`, kept because 13 rulings cite them - and append-only for the merge gate: never modified, only added. Nothing - writes new ones (the review-slot apparatus was retired 2026-09-03). +- **`projects/.md`** — one ledger per project (REFERENCE.md "The + ledger file"). Title `# — `; header `Project:` and + `Issue:` (`Repo#N`, an issue URL, or `none`); then exactly three sections: + `## Now`, `## Runs` (`open | running`), `## Log` (newest first, kind + `run | result | lesson | note`). Every `status: active` row must have one. + `projects/AGENTS.md` is the doctrine for the folder. +- **`archive/`** — frozen: the 2026-08/09 task, ruling and batch ledgers, + kept because the science repos cite their ids. Nothing writes there; any + change under it is code for the merge gate. - **`checkin.yaml`** — one key, `refreshed: `: when the science state was last actually checked in, written by `pyauto-brain cortex checkin --apply` and read back by the board. It means *last check-in*, never last - render, so a doc-only push cannot fake freshness; the HTML board reddens it - on the reader's own clock once it is more than an hour old. -- **`scripts/cortex.py`** — the one lifecycle script (stdlib only): + render; the HTML board reddens it on the reader's own clock once it is more + than an hour old. +- **`scripts/cortex.py`** — the one lifecycle script (PyYAML + stdlib): - `check` — every structural rule, hermetic; `cortex check: OK` or `DRIFT` with one ` - …` line per finding, exit 1. - - `gates` — read-only and offline: every gated task, its refs and their - URLs. Nothing polls GitHub and nothing flips a state; a human reads the - listing, opens the refs and types `move ready`. - - `rule --body …` — assigns the next ruling id, writes - the ruling file(s), updates the task's `Ruling:` and `State:`. The only - door to `accepted`, `rerun`, `dropped` and `leave-to-finish`. - - `move …` — every other transition, per the table in - REFERENCE.md; refuses the ruling edges. - - `new --summary "<≤10 words>" …` — writes a task file - from the template; `--summary` is required and capped, as `check` is. + - `new --summary "" [--issue Repo#N]` — opens a ledger + for a row of `projects.yaml`. + - `run "" [--partition P]` — records a submission: + the run goes under `## Runs` as `open` and the log gets a `run` entry. + - `running ` / `done [--failed] [--wall + H:MM] [--note …]` — what the cluster said; `done` moves the run into the + log. + - `log "" [--kind note|result|lesson]` — the human's words, + dated today, at the head of the log. + - `now ""` — rewrites `## Now`. + - `issue ` — prints the concise block that sits at the top of the + project's GitHub issue (Now, Runs, the last five entries) between two + markers; the Brain's `cortex issue --apply` writes it there. - `retire --why ""` — the only verb that writes - `projects.yaml`: the row's `status:` becomes `retired` and its - `note:` records `retired : `. The row, its tasks and its - rulings all stay; it refuses while any task of that project is - outside `accepted | rerun | dropped | planned`. + `projects.yaml`: the row's `status:` becomes `retired`, its `note:` + records why, and the ledger logs it. It refuses while the ledger still + lists a run. - **`dashboard.md` / `dashboard.html`** — GENERATED, never hand-edited: the - board, rendered by the Brain's cortex conductor and self-healed on `main` - (see "Driving the Cortex" below). They are ledger for the merge gate. + board, rendered by the Brain's cortex conductor and self-healed on `main`. + They are ledger for the merge gate. - **`scripts/ledger_merge.py`** — the default-deny classifier behind `.github/workflows/ledger_merge.yml`: a `claude/**` push whose whole diff is - ledger (`tasks/`, `rulings/`, `batches/`, `checkin.yaml`, the two generated - dashboards) lands on `main` by itself; anything else waits for a human — and - that "anything else" includes an `AGENTS.md` or `TEMPLATE.md` *inside* a - ledger dir, which is doctrine, not an entry. `python3 + ledger (`projects/`, `checkin.yaml`, the two generated dashboards) lands on + `main` by itself; anything else waits for a human — `projects/AGENTS.md` + included (doctrine, not an entry), and anything under `archive/`. `python3 scripts/ledger_merge.py classify --base origin/main` predicts the verdict. ## Driving the Cortex @@ -126,33 +112,29 @@ human's turn. Full rules in [rulings/AGENTS.md](rulings/AGENTS.md). The Cortex holds state and checks itself; it does not reason. The reasoning lives in the Brain's **cortex conductor** — `pyauto-brain cortex `, or `python3 PyAutoBrain/agents/conductors/cortex/_cortex.py --cortex -` with no Brain install. It is read-mostly: it renders, it plans and -it scores. The only bytes it writes of its own are the two generated pages; -every change to a *task* goes through `scripts/cortex.py`, which owns the -state table. +` with no Brain install. It is read-mostly: it pulls, it shows, it +renders. The only bytes it writes of its own are the two generated pages and +`checkin.yaml`; every change to a *ledger* goes through `scripts/cortex.py`. | Verb | What it does | |------|--------------| -| `checkin [--dry-run \| --apply] [--push \| --no-push] [--project KEY] [--skip-pull]` | **the check-in** — the one door: pull every active project through its own `sync_cli`, score every live task, move what came back, re-render the board, push the ledger where the rule allows, and summarise **by project**. `--dry-run` is the default and reaches nothing | -| `census [--json]` | what the Cortex is holding, by state — the one-screen answer | +| `checkin [--dry-run \| --apply] [--push \| --no-push] [--project KEY] [--skip-pull]` | **the check-in** — the one door: pull every active project through its own `sync_cli`, run its `jobs` verb where it has one and show the output verbatim, re-render the board, push the ledger where the rule allows, and summarise **by project** (Now, runs, the last five entries, the commands you are likely to type next). `--dry-run` is the default and reaches nothing | +| `census [--json]` | what the Cortex is holding, by project — the one-screen answer | | `dashboard --check` \| `--apply` | render `dashboard.md` + `dashboard.html`; `--check` exits **1 on drift**, **2 on bad args**, anything else = the renderer could not run | -| `gates` | every gated task, its refs and their URLs — read-only, offline | -| `collect [--task REL] [--pull] [--refreshed ISO] [--apply]` | score what came back; with no `--task` it scopes to **every** `submitted \| running` task, and `--pull` runs the *project's own* sync CLI | +| `issue [--project KEY] [--apply]` | the concise ledger block for each project's GitHub issue; `--apply` writes it into the issue body between the markers (needs `gh`) | **`--apply` here, `--write` there.** The conductor's verbs spell the writing -flag `--apply` (the Brain's house spelling, as intake does); `scripts/cortex.py` -spells it `--write`. +flag `--apply` (the Brain's house spelling); `scripts/cortex.py` verbs write +directly and take `--today` for tests. **Nothing here submits a job by itself.** A submission is made only on the -human's ask — by the human, or by the agent in the session — with the project's -own `sync_cli submit` line, followed at once by `cortex.py move -submitted --run ` (ruled 2026-09-05; before that the human typed the -submit). -`checkin` and `collect --pull` are the only legs that shell out, and only to -the project's own CLI — the conductor adds no SSH of its own. Each project -`checkin` pulls gets a `/.cortex/pull.json` manifest -(REFERENCE.md "Check-in"), merged into whatever that project's CLI already -wrote there. +human's ask — by the human, or by the agent in the session — with the +project's own `sync_cli submit` line, followed at once by `cortex.py run + ""`. `running` and `done` may be typed by the agent +from what `jobs` printed, because those are cluster facts; `log --kind result` +and `--kind lesson` are typed only for words the human said. `checkin` is the +only leg that shells out, and only to the project's own CLI — the conductor +adds no SSH of its own. ### What runs by itself @@ -160,20 +142,14 @@ Three workflows, and only these may write: | Workflow | Trigger | May write | |---|---|---| -| `cortex_check.yml` | push/PR on ledger, scripts, tests, the dashboards | **nothing** — `cortex.py check` + pytest | +| `cortex_check.yml` | push/PR on the ledger, scripts, tests, the dashboards | **nothing** — `cortex.py check` + pytest | | `dashboard_refresh.yml` | push to main, PR, nightly 03:35 UTC, dispatch | `dashboard.md`, `dashboard.html` (self-heal on main; a PR run errors instead of healing) | | `pages_dashboard.yml` | push to `dashboard.html`, dispatch | nothing in the repo — it publishes to Pages | | `ledger_merge.yml` | push to `claude/**`, dispatch | merges a ledger-only branch to main | -**No scheduled job mutates the ledger.** A daily gate-grading cron did, until -gate grading was retired on 2026-09-03 (2 gated refs, 0 flips in its -lifetime); the only self-healing writer left renders the two generated pages. - -The main-writers share `concurrency: group: cortex-main-writers`, so two -bots never race for the tip of main. And because a `GITHUB_TOKEN` push triggers -no workflow at all, each of them re-dispatches by name what its push should have -woken (`pages_dashboard.yml` after a heal; `cortex_check.yml` + -`dashboard_refresh.yml` after a grading commit). +**No scheduled job mutates the ledger.** The main-writers share `concurrency: +group: cortex-main-writers`, so two bots never race for the tip of main, and +each re-dispatches by name what its `GITHUB_TOKEN` push cannot wake. The board is published at **** — `dashboard.html` as the index. @@ -185,50 +161,43 @@ exception, and it is confined to `projects.yaml`: a project's `local_path` and `mirror` are absolute laptop paths **outside** the workspace (the Science folder under `/mnt/c/…`), because that is where the datasets, the `output/` trees and the pulled results live and the review happens at the laptop. The -exception is stated in the file's header and here, and nowhere else — a task -file or a ruling points into a project through its `projects.yaml` row, never -with a bare absolute path of its own. +exception is stated in the file's header and here, and nowhere else — a ledger +points into a project through its `projects.yaml` row, never with a bare +absolute path of its own. ## What is out of scope, and why Quoted verbatim from `PyAutoMind/complete/archive/shelved/batch_science_lane.md` so nobody re-derives these: -> ## What is now out of scope, and why -> > - **RAL as canonical home for the science project.** Refused above. The datasets > and `output/` stay under `/mnt/c/…/Science/`. > - **A git-courier cron on the RAL login node.** Its value collapses once the > laptop is canonical: the laptop has to be on to hold and push the data anyway, -> so a courier saves almost nothing. It was the right answer to a question no -> longer being asked. +> so a courier saves almost nothing. > - **Globus Compute endpoint / self-hosted GitHub runner on the login node.** > Same reasoning, plus both are persistent login-node processes needing an -> operator conversation. Not worth it for a lane the human is happy to drive. +> operator conversation. > - Recorded so nobody re-derives them: SSH from a Claude container is a > non-starter in every variant (HTTPS-only proxied egress, no keys); Open > OnDemand is an admin-installed inbound portal; Cirun cannot reach someone -> else's SLURM. And phases 4, 6a and 6b of `euclid-dr1-prep` say in their own -> prompts that they are human-driven and supervised with a judged verdict as the -> deliverable — **no transport was ever going to make those unattended.** +> else's SLURM. -Every Cortex task therefore runs on the laptop. That was once written into -each task file as `Lane: local-dev` — 32 of 32 said the same thing, so the -header was deleted on 2026-09-03 and the fact lives here instead. (The quote -above is the Mind's, verbatim: the "phases 4, 6a and 6b" it names are phases of -the `euclid-dr1-prep` **Mind epic**, which is development work and genuinely -sequential — not Cortex tasks.) +Every Cortex project therefore runs from the laptop. ## Hard rules 1. **Never rewrite history on any branch with a remote** (block below). 2. **Pull before edit.** `git fetch && git status` first, every time. -3. **Never edit a ruling.** Supersede it. `ledger_merge.py` refuses to - auto-merge a modified or deleted file under `rulings/`. -4. **Run `python3 scripts/cortex.py check` before you push** a task, ruling - or batch change (it is what refuses a missing or over-long `Summary:`); `ledger_merge.yml` runs it on the trial-merge tree and a +3. **Never write a result or a lesson the human did not say.** Cluster facts + (`run`, `running`, `done`) are the script's to record; meaning is the + human's. +4. **Never edit a ledger by hand in a session.** Every write is a + `scripts/cortex.py` verb, and `python3 scripts/cortex.py check` runs + before you push; `ledger_merge.yml` runs it on the trial-merge tree and a failing check leaves the branch for a human. -5. **No hand-written HTML.** `dashboard.html` is rendered by the cortex +5. **Never touch `archive/`.** It is frozen; the merge gate refuses it. +6. **No hand-written HTML.** `dashboard.html` is rendered by the cortex conductor; a hand edit is drift that `dashboard_refresh.yml` will overwrite on the next push to `main`. @@ -278,11 +247,11 @@ several, which is exactly the session no hook fires in. *parent*, so a bare `skills/…` reads as a missing file rather than a missing repo prefix. - ## When in doubt Read [README.md](README.md) and [REFERENCE.md](REFERENCE.md). The birth epic's -ledger is `PyAutoMind/complete/archive/epics/cortex_birth_epic.md`. +ledger is `PyAutoMind/complete/archive/epics/cortex_birth_epic.md`; the +2026-09-12 redesign is schema decision 60. ## Never rewrite history @@ -292,7 +261,6 @@ tracked repo, no force-push to `main`, no fresh-start "Initial commit", no `filter-repo` / `filter-branch` / `rebase -i` on pushed branches. To get a clean tree: `git fetch origin && git reset --hard origin/main && git clean -fd`. - ## Sessions end at their deliverable diff --git a/README.md b/README.md index bb35e4b..70acfbd 100644 --- a/README.md +++ b/README.md @@ -1,61 +1,60 @@ # PyAutoCortex -**PyAutoCortex is the Cortex of the PyAutoScientist** — where the organism -learns what is true. The Mind decides what to build, the Brain routes the work -and executes nothing, the Cortex learns what is true: it holds the **science -body map** (every science project, where it lives, how it syncs) and the -**rulings of record** — every verdict a human has passed on a science run, -append-only, superseded but never edited. +[![PyAutoScientist GitHub](https://img.shields.io/badge/%F0%9F%94%AD%20PyAutoScientist-GitHub-181717?style=flat-square)](https://github.com/PyAutoLabs/PyAutoScientist) [![PyAutoScientist ReadTheDocs](https://img.shields.io/badge/%F0%9F%93%96%20PyAutoScientist-ReadTheDocs-8CA1AF?style=flat-square)](https://pyautoscientist.readthedocs.io) -**The board** — every task the Cortex is holding, on one page, with the next -command a tap away: (markdown twin: -[dashboard.md](dashboard.md)). Both pages are generated by the Brain's cortex -conductor and refreshed on every push; never hand-edit them. +**PyAutoCortex is the Cortex of the PyAutoScientist** — where the organism +keeps track of what is true. It holds the **science body map** (every science +project, where it lives, how it syncs) and one **ledger per project**: what was +set off and when, what came back, what you learned, and where you meant to go +next — a history of how each project unfolds, written in your own words, never +a queue of things to finish. + +See the **[PyAutoCortex Dashboard](https://pyautolabs.github.io/PyAutoCortex/)** +for how this is used in practice: every active project on one page — where you +are with it, what is on the cluster right now, and the last five things you did +or learned — so a project you have been away from for a while is picked up +where you left it. The markdown twin is [dashboard.md](dashboard.md); both are +generated and never hand-edited. ## How PyAutoCortex works -The unit is a **task** of a **project**, which spawns **runs** (SLURM job ids) -and ends in a **ruling**: - -1. **Declare the task.** A markdown file under `tasks//` states the - question, the pre-registered witness, the wall budget and the GitHub issues - it waits on (`Gates:`). It starts `planned` or `gated`. -2. **Run it.** When the gates clear the task is `ready`; a submission makes - it `submitted` and then `running`, with every job id recorded in the file's - `## Runs` block in SLURM notation. -3. **Pull and review.** When the results are pulled to the laptop the task - becomes `pulled` and then `awaiting-ruling` — it is waiting on the human's - eyes, and on nothing else. -4. **Rule.** The human's verdict — `accept`, `rerun`, `drop` or - `leave-to-finish` — is written into `rulings/` with a permanent id. *A - verdict recorded only outside the Cortex does not exist.* - -In full those four steps are ten states — `planned`, `gated`, `ready`, -`submitted`, `running`, `pulled`, `awaiting-ruling`, then `accepted`, `rerun` -or `dropped` — of which only the last three are reachable, and only through -`scripts/cortex.py rule` (`legacy` and `legacy_wrong` are states of a *run*, -never of a task). Checking in is one command from the Brain's cortex -conductor — **`pyauto-brain cortex checkin --apply`** (see `--dry-run` first) -pulls every active project through its own sync CLI, scores every submitted or -running task against its pre-registered witness, moves what came back to -`awaiting-ruling`, re-renders the board, pushes the ledger where it is allowed -to, and prints a summary **by project** with the next prompt for each task. `batches/` is closed history: the three 2026-08/09 batch -records and the human's verbatim reviews stay readable because 13 rulings cite -them, and nothing writes new ones. - -The Cortex is **not a second PyAutoMind**: development tasks stay in the Mind -and are named here only as gates. Its one script, `scripts/cortex.py`, checks -the tree and moves tasks between states; the ledger folders auto-merge from -any surface that can push. +A project's ledger is one markdown file, `projects/.md`, with three +sections and nothing else: + +1. **Now.** Two or three lines, rewritten whenever you touch the project: what + is running, and what you meant to do next. +2. **Runs.** The jobs on the cluster right now, each `open` (submitted) or + `running`, with its date and one line saying what it is. A finished run + leaves this list and becomes a log entry. +3. **Log.** Dated entries, newest first — a run set off or finished, a + `result` you read off, a `lesson` you want kept, a `note`. Nothing here is + ever done; it only gets older. The board shows the last five. + +Every write goes through `scripts/cortex.py`: `run` records a submission, +`running` and `done` record what the cluster says, `log` and `now` record what +*you* say. Nothing in a ledger is inferred from results — a result or a lesson +is written only when you state it. Checking in is one command from the Brain's +cortex conductor, **`pyauto-brain cortex checkin --apply`** (`/cortex` in +Claude Code): it pulls every active project through that project's own sync +CLI, shows where each run stands, re-renders the board, pushes the ledger where +it is allowed to, and reads the projects back to you one by one. + +Each project links to a GitHub issue (`Issue:` in its header) that carries the +detailed run-through; the top of that issue is the same concise ledger the +board shows, regenerated from the project file with `pyauto-brain cortex issue`. + +The Cortex is **not a second PyAutoMind**: development work stays in the Mind. +The retired task-and-ruling ledgers of 2026-08/09 are frozen under +[`archive/`](archive/) because the science repos cite their ids. `scripts/cortex.py` needs Python 3.10+ and **PyYAML** (`python3 -m pip install pyyaml`), which is how it reads `projects.yaml`; the tests additionally need `pytest`. Nothing else. -The schemas — task files, the run-line grammar, rulings, the `projects.yaml` -fields — are in [REFERENCE.md](REFERENCE.md); how agents should -operate this repo is in [AGENTS.md](AGENTS.md); every design choice the birth -epic did not fix is dated in [docs/schema_decisions.md](docs/schema_decisions.md). -The organism this repo is the Cortex of is described once in +The schema — the ledger file, the run and log line grammars, the `projects.yaml` +fields, how the ledger auto-merges — is in [REFERENCE.md](REFERENCE.md); how +agents should operate this repo is in [AGENTS.md](AGENTS.md); every design +choice is dated in [docs/schema_decisions.md](docs/schema_decisions.md). The +organism this repo is the Cortex of is described once in [PyAutoBrain/ORGANISM.md](https://github.com/PyAutoLabs/PyAutoBrain/blob/main/ORGANISM.md) and documented in full at . diff --git a/REFERENCE.md b/REFERENCE.md index 16daf7b..e6e4641 100644 --- a/REFERENCE.md +++ b/REFERENCE.md @@ -1,754 +1,206 @@ -# PyAutoCortex reference +# PyAutoCortex — Reference -The schemas, grammars and conventions for this repo — the file `scripts/cortex.py` -implements and `scripts/cortex.py check` enforces. Agent docs that point at a -schema ("the transition table", "the run-line grammar", "the ruling schema") -resolve here, one link from [AGENTS.md](AGENTS.md). +The schema of the Cortex, in full. Everything `scripts/cortex.py check` +enforces is written here; nothing it enforces is not. Design choices are dated +in [docs/schema_decisions.md](docs/schema_decisions.md). ---- +## The ledger file -## What a task looks like +One file per project, `projects/.md`, where `` is a row of +`projects.yaml` (`^[a-z][a-z0-9_]*$`). Every `status: active` row must have +one; a `dormant`, `planned` or `retired` row may. -Here is a task file — `tasks/example/05_running_array.md` from the test -fixture — of a project whose GPU array is most of the way through: +```markdown +# — -````markdown -# Example — the nine-lens array +Project: +Issue: //issues/N | none> -Project: example -Summary: Does Delaunay reach the same theta_E basin on ten lenses -State: running -Gates: PyAutoArray#431 -Witness: nine of ten array tasks write a sane checkpoint.hdf5 within 8:00 wall -Budget: 8:00 -Runs: 342091, 342102 -Ruling: R-20260901-03 -Review-minutes: 8 -Epic: example-programme -Filed: 2026-08-29 +## Now -## Question +Two or three lines, rewritten whenever the project is touched: what is on the +cluster and what the human meant to do next. May not be empty on an active +project. -Does the Delaunay pipeline reach the same theta_E basin on all ten lenses? +## Runs -## Witness +- — — — — + an optional continuation line, indented two spaces -`output/task_05/*/checkpoint.hdf5` present for nine lenses, `.err` clean. +## Log -## Where to look +- — — + an optional continuation line, indented two spaces +``` -- `/mnt/c/Users/Jammy/Science/example/output/task_05/` +The header holds exactly `Project:` and `Issue:`; `Project:` must equal the +key. The three sections appear in that order and no other section exists. +` -- ` is read as ` — ` everywhere (phone keyboards). -## Runs +### Run lines (`## Runs`) -- 342091_[0-8,10]: done — gpu — submitted 2026-08-30 — wall 6:12 - pulled_to: /mnt/c/Users/Jammy/Science/example/output/task_05 -- 342091_9: failed — gpu — submitted 2026-08-30 — wall 0:00 — OOM before the first step -- 342102: running — gpu — submitted 2026-09-01 — wall 0:00 — task 9 resubmitted alone - after: 342091_9 - -## Ruling - -R-20260901-03 — leave-to-finish -```` - -The header is the Mind's **light header**: line 1 is `# `; `Key: value` -lines within the first 30 lines — the block starts at the first `Key: value` -line after the title and ends at the next blank line; the first occurrence of a -key wins; a list-valued key is a bare `Key:` line followed by `- item` lines; a -key with no value (`Gates:`, `Runs:`, `Ruling:`) may be left empty or omitted. -No YAML frontmatter. The body -sections are fixed: `## Question`, `## Witness`, `## Where to look`, `## Runs`, -`## Ruling`. - -**`## Where to look` is rendered, not just parsed.** Its `- ` bullets are the -answer to *which folder do I open* — the dashboard's `## By project` view and -`pyauto-brain cortex checkin` print them verbatim under the task, and -`collect` resolves the ones that are absolute paths inside the project's roots -to the artefacts it scores. `new` seeds the section with the placeholder -`- (the output path, once there is one)`, which is honest on a `planned` task -and a hole on any other, so **`check` requires a task whose `State:` is past -`planned` to carry at least one bullet that is not that placeholder**. A -bullet is free text: a path, a project-row-relative location, a Mind prompt, a -GitHub ref — whatever a human would open. - -A task has **no number**. Its identity is its **slug** — the file name, unique -per project by the path itself — because a science project is a set of ideas -run in whatever order the results dictate, not a sequence (decision, 2026-09-07). - -### Task header keys - -| Key | Value | Notes | +| Field | Grammar | Meaning | |---|---|---| -| `Project:` | project key | must equal the directory name **and** a `projects.yaml` key | -| `Summary:` | at most ten words | **required** — the QUESTION the task answers, not its method or its run ids; the board's one line per task. No project name, no number | -| `State:` | `planned \| gated \| ready \| submitted \| running \| pulled \| awaiting-ruling \| accepted \| rerun \| dropped` | | -| `Gates:` | comma-separated GitHub refs | `Repo#N` (owner `PyAutoLabs`) or an issue/PR URL; **no `owner/Repo#N`** form | -| `Reset:` | reason | written by `move ready --reason "<reason>"` when a `submitted \| running` task goes back to `ready` | -| `Witness:` | free text | **mandatory before `submitted`** — the pre-registered checkable claim | -| `Budget:` | `H+:MM` | wall budget per run | -| `Runs:` | comma-separated job **stems** | the index of the `## Runs` body; equal to the set of body stems | -| `Ruling:` | ruling id | the chain head (see "Rulings") | -| `Review-minutes:` | integer | a seed, not a measurement | -| `Epic:` | slug | OPTIONAL join key to a **Mind** epic; the Cortex keeps no epics file of its own | -| `Filed:` | `YYYY-MM-DD` | | -| `Migrated-from:` | source path or ledger anchor | the Mind prompt, review file or project-ledger entry this task was transcribed from (phase 4 of the birth epic) | - -Gate refs are matched by `GATE_REF_RE`, copied verbatim from -`PyAutoMind/scripts/lifecycle.py`: - -```python -GATE_REF_RE = re.compile( - r"https://github\.com/([\w.-]+)/([\w.-]+)/(?:issues|pull)/(\d+)" - r"|(?<![\w/])([A-Za-z_][\w.]*)#(\d+)\b" -) -DEFAULT_GATE_OWNER = "PyAutoLabs" -``` +| jobid | `342301`, `342301_3`, `342301_[0-9]`, `342301_[0,2,4-9]` | the SLURM id, array task or task set | +| state | `open` \| `running` | `open` = submitted and not yet seen running. There is no third state: a run that has finished is removed by `done` and becomes a log entry | +| partition | a bare word (`ral`, `gpu`) | must agree with the row's `partition:` unless that is `both` | +| date | `YYYY-MM-DD` | the submission date | +| what | free text, one line | what the run is — the human's description at submission | -The lookbehind `(?<![\w/])` is what rejects `owner/Repo#N`: the shorthand is -`Repo#N` with the default owner, and any other owner is spelled as a URL. +A job id appears once per file. A retired project lists no runs. ---- - -## How a task flows - -``` - planned ──(Gates: non-empty)──► gated ──(move ready: the human judged the refs)──► ready - │ │ - └──────(Gates: empty)───────────────────────────────────────────────────────────────┤ - ▼ - ready ──(Witness: + --run)──► submitted ──► running ──(no live run)──► pulled ──► awaiting-ruling - ▲ │ │ │ - │ (no live run, ≥1 failed|timeout|void, --reason) │ rule - └──────────────────────────────┴──────────────┘ ▼ - accepted │ rerun ──► ready │ dropped - │ rule --supersedes - └──► rerun | dropped -``` +### Log lines (`## Log`) -`cortex.py move` owns every edge except the ruling edges, which `cortex.py rule` -owns. The full table: +Newest first — `check` refuses a date that is later than the entry above it. +Four kinds: -| from | to | condition | +| Kind | Written by | What it is | |---|---|---| -| planned | gated / ready | `Gates:` non-empty / empty | -| gated | ready | `move <task> ready` — a human read `gates`, opened the refs and judged them cleared. No flag, and nothing written but `State:` | -| ready | gated | not an edge: re-gating a `ready` task is a judgement, so it is a hand edit of the header | -| ready | submitted | `Witness:` non-empty AND `--run <id>` supplied | -| ready | pulled | legacy-born task: every run line is `legacy\|legacy_wrong` (`new --legacy-run` / `move pulled`); `Witness:` still mandatory; every `legacy` run gets a `pulled_to:` (`--pulled-to`, else its own `where:`); refused when no run is `legacy` (nothing to review — `rule drop`) | -| submitted / running | same | `--run <id>` appends a wave, a chained job or a checkpoint resubmit (`resumes:`); state unchanged | -| submitted | running | — | -| submitted / running | ready | no run line in `submitted\|running` AND ≥1 `failed\|timeout\|void`; `--reason` required (→ writes `Reset:`) | -| running | pulled | no run line live; or `--partial` (a partial array), which `check` expects closed by a `leave-to-finish` ruling; `--pulled-to <path>` writes `pulled_to:` on every `done` run lacking one, and the move is refused when no `done` run would carry one | -| pulled | awaiting-ruling | — (the task joins the rolling board) | -| awaiting-ruling | accepted / rerun / dropped | **`rule` only** | -| running / pulled / awaiting-ruling | same | `rule leave-to-finish` (state unchanged) | -| accepted | rerun / dropped | `rule --supersedes <current Ruling:>` only (the REWIND case) | -| rerun | ready | — (the witness may be re-registered; run history is kept) | -| any non-terminal | dropped | `rule drop` only | -| dropped | — | terminal (revival = a new slug) | - -`accepted` is **not** terminal: a later ruling may supersede the acceptance -(2026-08-31's REWIND superseded accepted gates). `dropped` is. - -Offline invariants `check` enforces on top of the table: - -- `State ∈ submitted..accepted` (and `rerun`) ⇒ `Witness:` non-empty. -- `State ∈ accepted | rerun | dropped` ⇒ `Ruling:` present (those states are - reachable only through `rule`). -- `State = pulled` AND any run line `submitted | running` ⇒ `Ruling:` present - and its head verb is `leave-to-finish` (the `--partial` case). -- `State = gated` ⇒ `Gates:` non-empty (a gate that does not exist cannot - clear). -- A header key outside the table is drift (`Gate:` must not pass as a silent - typo for `Gates:`); the five body sections are present; `Summary:` is - present, non-empty and at most ten words; `Review-minutes:` is an integer, - `Budget:` is `H+:MM` and `Filed:` is `YYYY-MM-DD`. A key with an empty value - is read as absent. `Phase:` is named as a **retired** key rather than passing - as a generic unknown one — the number went on 2026-09-07. - ---- +| `run` | `cortex.py run` / `done` | `<jobid> submitted: <what>` · `<jobid> finished — wall H:MM: <what>` · `<jobid> failed: <what>` | +| `result` | `cortex.py log --kind result` | what the human read off the results, in their words | +| `lesson` | `cortex.py log --kind lesson` | something the human wants kept — a trap, a rule, a fact about the setup | +| `note` | `cortex.py log` (default), `new`, `retire` | anything else: a thought, a plan, `ledger opened`, `retired: <why>` | -## Repository layout - -``` -PyAutoCortex/ -├── README.md ← short front page -├── AGENTS.md ← agent guidance (generated blocks: organism map, remote sessions, history) -├── CLAUDE.md ← GENERATED pointer (@AGENTS.md) — repos_sync --write -├── REFERENCE.md ← this file (schemas + grammars) -├── LICENSE .gitignore -│ -├── projects.yaml ← the science body map — CODE, not ledger (read with PyYAML) -├── checkin.yaml ← `refreshed: <UTC ISO>` — the last check-in stamp; LEDGER -├── dashboard.md ← GENERATED board — cortex conductor (`dashboard --apply`); LEDGER -├── dashboard.html ← GENERATED board, the Pages index; LEDGER -│ -├── tasks/<project>/<slug>.md ← one file per task (LEDGER) -├── rulings/AGENTS.md ← the append-only rule -├── rulings/<YYYY>/<MM>/R-<YYYYMMDD>-<nn>.md ← the ledger of record (LEDGER, append-only) -├── batches/AGENTS.md ← "closed history: never modified, only added" -├── batches/<YYYY-MM-DD>-<slot>.md ← the 2026-08/09 records (LEDGER, append-only) -├── batches/reviews/{AGENTS.md,<slot>.md} ← the human's verbatim reviews (LEDGER, append-only) -│ -├── docs/schema_decisions.md ← every dated choice the epic did not fix -├── policy/never_rewrite_history.md policy/remote_sessions.md ← copies of the Mind's; spliced into AGENTS.md -│ -├── scripts/cortex.py ← check · gates · rule · move · new · retire (stdlib + PyYAML) -├── scripts/ledger_merge.py ← the default-deny ledger classifier (+ append-only on rulings/, batches/) -├── tests/test_cortex.py tests/test_ledger_merge.py -├── tests/fixtures/skeleton/ ← one project, one task per state, five rulings — the witness -├── tests/fixtures/empty/ ← an empty map passes `check` -│ -├── .github/workflows/cortex_check.yml ← check + pytest on push/PR -├── .github/workflows/ledger_merge.yml ← lands ledger-only `claude/**` branches -├── .github/workflows/dashboard_refresh.yml ← renders the board; self-heals main -├── .github/workflows/pages_dashboard.yml ← publishes dashboard.html to Pages -└── .claude/hooks/session-start.sh .claude/settings.json ← GENERATED — repos_sync --write -``` +The board and the issue block show the newest `LOG_WINDOW = 5` entries; the +file keeps them all. An entry is never edited or removed: a wrong entry is +answered by a newer one. -No `skills/` — the Cortex exposes no commands of its own; the conductor that -drives it lives in the Brain (`pyauto-brain cortex …`, AGENTS.md "Driving the -Cortex"), and the workflows above are the only things that run it unattended. +### The issue block ---- - -## Run lines (`## Runs`) - -SLURM notation, strict grammar. One line per job or per task set of an array; -structured facts on **indented continuation lines**, never in the note. +`cortex.py issue <project> [-n 5]` prints the concise ledger that sits at the +top of the project's GitHub issue: the title line and status, `**Now**`, the +runs on the cluster, and the last `n` entries, fenced by ``` -- <stem>[_<task>|_[<a>-<b>,<c>]]: <run-state> — <partition> — submitted <YYYY-MM-DD> — wall <H+:MM>[ — <note>] - pulled_to: <path> | after: <run> | resumes: <run> | where: <path> | ruled: <id> -``` - -- `<stem>` — the SLURM job id, digits. `_<task>` — one array task. - `_[<set>]` — an array task set: comma-separated integers and `a-b` ranges, - ascending, no spaces (`342091_[0-8,10]`). -- `<run-state>` ∈ `submitted | running | done | failed | timeout | void | - legacy | legacy_wrong`. `void` = cancelled or never produced a step. - `legacy` / `legacy_wrong` = quarantine (reusable / not) — **a run state, - never a task state**. -- `<partition>` — the SLURM partition the job went to, a bare word - (`^[a-z][a-z0-9_-]*$`, e.g. `gpu`). The project's `partition:` row says - which it may use. -- `submitted <YYYY-MM-DD>` — the submission date. `wall <H+:MM>` — wall time - used so far (`0:00` for a job that never ran). -- `<note>` — free text after a fourth ` — `; never carries a structured fact. -- Continuation lines — exactly four spaces then `<key>: <value>`, keys: - `pulled_to:` (laptop path the results were pulled to), `after:` (SLURM - `afterok` dependency), `resumes:` (a checkpoint resubmit of that run), - `where:` (the quarantine path of a `legacy*` run), `ruled:` (a ruling id - naming this run). -- The em dash `—` is canonical; `--` is accepted for it (phone keyboards). - `check` reads both as the same separator; the writing verbs emit `—`. - -The regex, as `check` applies it (after `--` → `—` normalisation): - -```python -RUN_LINE_RE = re.compile( - r"^- (?P<stem>\d+)" - r"(?:_(?P<task>\d+)|_\[(?P<tasks>\d+(?:-\d+)?(?:,\d+(?:-\d+)?)*)\])?" - r": (?P<state>submitted|running|done|failed|timeout|void|legacy|legacy_wrong)" - r" — (?P<partition>[a-z][a-z0-9_-]*)" - r" — submitted (?P<date>\d{4}-\d{2}-\d{2})" - r" — wall (?P<wall>\d+:\d{2})" - r"(?: — (?P<note>.+))?$" -) -RUN_CONT_RE = re.compile( - r"^ (?P<key>pulled_to|after|resumes|where|ruled): (?P<value>\S.*)$" -) +<!-- cortex:ledger begin — regenerated from projects/<key>.md; edit there --> +… +<!-- cortex:ledger end --> ``` -`check` enforces: +The Brain's `cortex issue --apply` replaces exactly that span in the issue +body (or prepends it when the markers are absent) and leaves everything under +it — the detailed, agent-friendly run-through — untouched. The Cortex never +creates an issue; `Issue:` is set by hand (`new --issue`, or an edit that +`check` validates). -- every non-blank line under `## Runs` matches `RUN_LINE_RE` or `RUN_CONT_RE`; - a continuation line follows a run line; -- task sets on one stem are **disjoint within a task** (job ids are unique - per task, not globally — one array may feed two tasks); -- the `Runs:` header equals the set of body stems (both empty when the task - has no runs); -- `State: pulled` ⇒ at least one `done | legacy` run carries `pulled_to:`; -- a `legacy | legacy_wrong` run carries `where:`; -- an `after:` / `resumes:` target is the identifier (the text before the - colon — `342091`, `342091_9` or `342091_[0-8,10]`) of another run line of - the **same task**; -- a `ruled:` value resolves to a ruling file. - ---- - -## Rulings - -`rulings/<YYYY>/<MM>/R-<YYYYMMDD>-<nn>.md`. The id is `R-YYYYMMDD-nn` — a -two-digit per-day sequence, global across projects; `rule` assigns it; the -filename equals the id and so does the title line (`# <id>` or -`# <id> — <one-line summary>`); `rule` generates -`# <id> — <verb> <project> <slug>`. - -````markdown -# R-20260901-02 — re-accept example 08_accepted with the corrected evidence pointer - -Project: example -Task: tasks/example/08_accepted.md -Runs: 342050 -Ruling: accept -Supersedes: R-20260901-01 -Batch: 2026-09-01-pm -Reviewed-at: 2026-09-01T17:20Z -Review-minutes-actual: 6 -Follow-ups: PyAutoLens#901 - -## Ruling - -The human's words, verbatim. - -## Evidence - -- pointers into the pulled results, the project ledger, figures -```` +## `scripts/cortex.py` — the verb reference -| Key | Value | Notes | +| Verb | Writes | Refuses when | |---|---|---| -| `Project:` | project key | | -| `Task:` | one task path | repo-relative, `tasks/<project>/<slug>.md` | -| `Runs:` | comma-separated stems, or empty | ⊆ the task's `Runs:` | -| `Ruling:` | `accept \| rerun \| drop \| leave-to-finish` | the verb | -| `Supersedes:` | one ruling id | optional; see the chain rules | -| `Batch:` | `<YYYY-MM-DD>-<slot>` | optional-**historical**; the 2026-08/09 rulings' join, nothing writes new ones | -| `Reviewed-at:` | timestamp | | -| `Review-minutes-actual:` | integer | | -| `Follow-ups:` | comma-separated GitHub refs | `GATE_REF_RE`; **the issue is created before `rule` runs** | -| `Migrated-from:` | source path or ledger anchor | the review file or project-ledger entry a backfilled ruling was transcribed from | - -Body: `## Ruling` — the human's words verbatim; `## Evidence` — pointers. - -**One ruling file per task.** A multi-task ruling (a REWIND) is N `rule` -invocations with the same body — one per task. - -**Chain rules** (`check`): - -- the id is unique, equals the filename stem and the title, and its date is - the file's `<YYYY>/<MM>` directory; the body has `## Ruling` and - `## Evidence`; -- `Supersedes:` resolves to an existing ruling, is not the ruling itself, is - lexically smaller (earlier), and names the same project **and** task; -- **at most one successor per ruling** — a chain, not a tree: supersede the - head; -- the task's `Ruling:` is a chain head (no ruling supersedes it) whose - `Task:` is that task; -- the head's verb matches the task's state: `accept ⇒ accepted`, - `drop ⇒ dropped`, `rerun ⇒ rerun | ready | submitted | running | pulled | - awaiting-ruling` (the task has moved on), `leave-to-finish ⇒ any - non-terminal state`; -- the ruling's `Runs:` ⊆ the task's runs; -- `Batch:`, when present, names an existing `batches/<slot>.md`. - -A task may hold rulings that are not chained to each other — a -`leave-to-finish` followed by an `accept` is two chains of one. `Supersedes:` -is for replacing a verdict, not for sequencing. - -The rest of the rule — append-only, supersede-never-edit, *a verdict recorded -only outside the Cortex does not exist* — is [rulings/AGENTS.md](rulings/AGENTS.md). - ---- - -## Batches — closed history - -`batches/<YYYY-MM-DD>-<slot>.md` and the human's verbatim reviews under -`batches/reviews/` are read-only history: three records from 2026-08/09, cited -by 13 rulings, kept because *a verdict recorded only outside the Cortex does -not exist*. They are **append-only** for the merge gate — never modified, only -added — and `check` no longer has a schema for them: the review-slot apparatus -that produced them (the packet page, the scored members, the review-minute -budget, the `-r<N>` sittings) was retired on 2026-09-03, and nothing writes a -new record. - -A ruling's `Batch:` field is therefore **optional-historical**: where present -it must still name an existing `batches/<slot>.md`, and nothing writes a new -one. - ---- +| `check` | nothing | — (exit 1 on any finding) | +| `new <project> --summary "…" [--issue REF] [--now "…"]` | a fresh `projects/<key>.md` from the template, with one `note — ledger opened` | the key is not a row; the file exists; the summary is over 15 words; the ref is malformed | +| `run <project> <jobid> "<what>" [--partition P]` | a run line (`open`) + a `run` entry | the id is not SLURM-shaped; it is already listed; the row is `both` and no `--partition` | +| `running <project> <jobid>` | the run line's state | the id is not under `## Runs` | +| `done <project> <jobid> [--failed] [--wall H:MM] [--note "…"]` | removes the run line, prepends a `run` entry | the id is not under `## Runs`; `--wall` is not `H:MM` | +| `log <project> "<text>" [--kind KIND]` | one entry at the head of the log | empty text; a `--today` earlier than the head entry | +| `now <project> "<text>"` | the body of `## Now` | empty text | +| `issue <project> [-n N]` | nothing (prints) | — | +| `retire <project> --why "…"` | `projects.yaml` (`status: retired`, `note:`), one `note` entry | the row is already retired; the ledger still lists a run; `--why` holds `"` | + +Every verb takes `--root <dir>` (default: this checkout) and every dated verb +`--today YYYY-MM-DD`, so the tests run against a copy of the fixture with a +fixed clock. A verb that would leave a file `check` cannot read refuses and +writes nothing. ## `projects.yaml` -Ordinary YAML, read with `yaml.safe_load` (PyYAML is `cortex.py`'s one -dependency; the hand-rolled restricted-subset parser it replaced was retired -on 2026-09-03, its own parity test having proved it equivalent). The document -is a mapping of project key → field mapping: +The science body map. **Code, not ledger** — `sync_cli` and `local_path` are +paths a conductor executes under. Real YAML read with `yaml.safe_load`; the +shape is validated by `cortex.py`: -```yaml -<key>: # ^[a-z][a-z0-9_]*$ - <field>: <scalar> # fixed field set, unknown field = error - sync_verbs: [pull, push] # a list of bare words - note: "free text" # the one optional field -``` - -What `check` validates is the **fields**, not the syntax: the document is a -mapping of rows, every key matches the key grammar, every field is in the set -below, `sync_verbs` is a list of bare words, and each value passes the rules -listed here. A document PyYAML cannot read is reported as one problem. - -- Fields (required): `remote` (`owner/repo` | `none` — PyAutoLabs for active - projects; a personal remote is recorded as a fact with a `note:`), - `local_path` (absolute laptop path — the Cortex-only exception to the - workspace-paths rule), `ral_root`, `mirror` (path | `none`), `sync_cli`, - `sync_verbs`, `ledger`, `assistant`, `witness_file` (glob), `partition` - (`gpu | ral | both`), `status` - (`active | dormant | planned | retired`). -- `assistant` is the domain assistant this project's work enters through — a - workspace-relative repo name such as `autolens_assistant`, or `none`; the - Cortex names it, never reads it. `check` accepts `none` or a bare directory - name (`^[A-Za-z0-9_.-]+$`: no `/`, never absolute) and stats nothing, so the - render does not need an assistant checkout. -- `note` (free text) is the **one optional field**: a row may omit it, an - empty `note:` is drift, and any other field is still an unknown-field - error. A note holding `:` or `#` is quoted like any other scalar. -- **Retiring a project.** `cortex.py retire <key> --why "<one line>"` is - the one verb that writes this file: it flips the row's `status:` to - `retired` and rewrites (or appends) its `note:` as - `"retired <today>: <why>"`. The **row stays** — it is the only record of - where that project's data lives — and no task or ruling is touched; - `rulings/` is append-only and a change of status is not a rewrite of - history. It refuses while the project holds a task in any state - outside `accepted | rerun | dropped | planned`, naming each one, so the - live questions are ruled or dropped first; `planned` tasks are unasked - questions and may stay. The edit is exactly those two lines — every - other byte of the file is preserved — and the result is re-parsed - before it is kept, the original bytes restored if it does not read - back clean. -- Science repos are **not** added to `PyAutoMind/repos.yaml` — that map is the - workspace, this one is the science. -- A file with no rows parses to an empty map (`{}`); PyYAML returns `None` - for a comment-only document, so the `yaml.safe_load(text) == parse(text)` - parity test runs on the fixture, which has a row. - ---- - -## How the ledger lands (`ledger_merge.yml`) +| Field | Value | Notes | +|---|---|---| +| `remote` | `owner/repo` or `none` | PyAutoLabs for active projects; a personal remote is a fact with a `note:` | +| `local_path` | absolute path | the laptop checkout — the workspace-paths exception (AGENTS.md) | +| `ral_root` | absolute path | the project root on RAL | +| `mirror` | absolute path or `none` | the laptop pull root the sync CLI fills | +| `sync_cli` | path relative to `local_path` | the project's own sync CLI — the only thing that reaches a cluster | +| `sync_verbs` | flow list of bare words | the verbs that CLI has; `pull` is what `checkin` runs, `jobs` what it shows | +| `ledger` | path relative to `local_path` | the project's own commentary ledger (a `wiki/project/state.md`, a `NOTES.md`) | +| `assistant` | a workspace-relative repo name or `none` | the domain assistant the project's work enters through; named, never read | +| `witness_file` | glob relative to the mirror | kept for the projects' own tooling; the Cortex no longer scores it | +| `partition` | `gpu` \| `ral` \| `both` | which SLURM partition(s) the project runs on; `run` needs `--partition` on `both` | +| `status` | `active` \| `dormant` \| `planned` \| `retired` | `retire` writes the last | +| `note` | free text, **optional** | the only optional field; an empty `note:` is drift | -A push to a `claude/**` branch whose whole diff is **ledger** is merged into -`main` by `.github/workflows/ledger_merge.yml` and the branch deleted — no PR, -no session step. The line is drawn by `scripts/ledger_merge.py`, **default -deny**: - -| Ledger — merged automatically | Code — always a human | -|---|---| -| `tasks/**`, `rulings/**`, `batches/**` | `scripts/`, `tests/`, `.github/`, `policy/`, `docs/` | -| `checkin.yaml` | `projects.yaml`, `README.md`, `AGENTS.md`, `REFERENCE.md`, … | -| `dashboard.md`, `dashboard.html` (generated) | **`AGENTS.md` / `TEMPLATE.md` inside a ledger dir** | -| | anything unclassified — a new root file, a new top-level folder | -| | **any modification or deletion under `rulings/` or `batches/`** (append-only) | - -Three exceptions inside the ledger dirs. Two are the Mind's: a **dot-path** -anywhere, and a file pytest would **collect** (`conftest.py`, `test_*.py`, -`*_test.py`). The third is the Cortex's own **doctrine carve-out** — -`rulings/AGENTS.md`, `batches/AGENTS.md` and `batches/reviews/AGENTS.md` are -ledger by location but instructional by content: they say what an entry may be and what -every future entry is stamped from, so a change to one is a change to -behaviour. Auto-merging a rewrite of `rulings/AGENTS.md` would let a branch -edit the rule that governs its own merge. - -Pulling the other way, the two **generated** board pages are ledger: a branch -that moves a task re-renders them in the same push, and -`dashboard_refresh.yml`'s self-heal commit has to land without a human. - -And a fourth Cortex rule on *kind* rather than path: `ledger_merge.py` -classifies via `git diff --name-status`, and an `M`, `D` or `R` entry under -`rulings/**` or `batches/**` is code. `rulings/` because a ruling is -superseded, never edited; `batches/` since 2026-09-03, when the review-slot -apparatus was retired and its records became history — never modified, only -added. - -The blocking check is `python3 scripts/cortex.py check` run on the **trial-merge -tree** — after `git merge --no-ff`, before `git push` — which is what catches -the ruling-id race (two branches both assign `R-…-03`: each passes alone, the -merge fails). A failing check resets the trial merge and leaves the branch for -a human. Predict the verdict before you push: +## Repository layout -```bash -python3 scripts/ledger_merge.py classify --base origin/main # exit 0 = will auto-merge -python3 scripts/cortex.py check # exit 0 = will not block +``` +projects.yaml the body map (code) +projects/<key>.md one ledger per project (ledger) +projects/AGENTS.md doctrine for the folder (code) +checkin.yaml the last-check-in stamp (ledger) +dashboard.md, .html generated board (ledger; never hand-edited) +archive/ frozen 2026-08/09 tasks, rulings, batches (code) +scripts/cortex.py the lifecycle script +scripts/ledger_merge.py the auto-merge classifier +tests/ pytest; fixtures/skeleton and fixtures/empty +docs/schema_decisions.md every dated choice ``` ---- - -## `scripts/cortex.py` — the verb reference +## How the ledger lands (`ledger_merge.yml`) -Stdlib only; `main(argv)`; no import-time side effects; every verb takes -`--root <dir>` (default: the repo root the script lives in). Every leg of -`check` takes `root: Path`, so tests run it against a `tmp_path` copy. - -- **`check`** — every rule in this file: task headers and states, the witness - invariant, `## Where to look` naming somewhere past `planned`, run lines, - ruling ids and chains, the verb↔state agreement, every project named by a - task path is a `projects.yaml` key, and `projects.yaml`'s own fields. Hermetic (no network, no git). Output in `lifecycle.py`'s shape — - `cortex check: OK` or `cortex check: DRIFT` followed by one ` - …` line per - finding; exit 1 on drift. -- **`gates`** — read-only and offline: every `gated` task, its refs and the - URL each ref resolves to (`GATE_REF_RE` → `gate_url`). Nothing polls GitHub - and nothing flips a state. Grading was retired on 2026-09-03 — in its whole - life it saw 2 gated refs and flipped 0, while schema decision 54 routes - sequencing through prose `Ready when:` lines. A human reads the listing, - opens the refs and types `move <task> ready`. -- **`rule <task> <verb> --body <file> [--supersedes <id>] [--batch <slot>] - [--minutes n] [--follow-up <ref>]...`** — assigns the next id for today, - writes the ruling file, updates the task's `Ruling:` and `State:` per the - table (and appends `<id> — <verb>` to the task's - `## Ruling`); the ruling's `Runs:` is the task's, its `## Evidence` is the - task's `## Where to look`; refuses to touch an existing ruling; refuses a - verb the table does not allow from the task's state; validates everything - before writing anything. -- **`move <task> <state> [--run <id>] [--reason ..] - [--partial] [--pulled-to <path>] [--partition ..] [--after <run>] - [--resumes <run>] [--note ..]`** — the table; refuses every ruling edge - with a message naming `rule`. `--run` on `submitted`/`running` appends a - run line and keeps the state; the appended line is - `- <id>: submitted — <partition> — submitted <today> — wall 0:00[ — <note>]` - with `after:` / `resumes:` continuations from the flags, and its - partition is the project's `partition:` row unless that is `both`, when - `--partition` is required. Every edit is an in-place header edit or an - appended run line; every other byte of the file is preserved. -- **`new <project> <slug> --summary "<≤10 words>" [--gates ..] [--epic ..] - [--legacy-run <id>]... [--legacy-wrong <id>]... [--where <path>] - [--partition ..] [--witness ..] [--budget ..] [--minutes n] [--title ..]`** - — writes `tasks/<project>/<slug>.md` from the template in `planned` (or - `ready` when `--legacy-run` / `--legacy-wrong` is given and every run is - legacy); each legacy run line is written with today's date, `wall 0:00` - and the note `pre-Cortex run, migrated` — the human corrects the date and - wall by hand — and `--where` (required) as its `where:`; a legacy-born - task refuses `--gates`. `--summary` is required and held to the same ten - words `check` holds it to. Refuses an existing file (the slug is the - identity — there is nothing else to collide on) or an unknown project. -- **`retire <project> --why "<one line>"`** — the only verb that writes - `projects.yaml`: `status: retired` plus a `note:` reading - `"retired <today>: <why>"`, two lines edited in place and every other - byte preserved. Refuses an unknown key, an already-retired row, and any - project still holding a task outside `accepted | rerun | dropped | - planned` (the message names each `<task> — <state>`). The row, its - tasks and its rulings all stay; see `projects.yaml` above. - ---- +A push to `claude/**` whose whole diff is ledger — `projects/*.md` (not +`AGENTS.md` / `TEMPLATE.md`), `checkin.yaml`, `dashboard.md`, +`dashboard.html` — is merged into `main` with `--no-ff` and the branch +deleted. Anything else waits for a human: `scripts/`, `tests/`, `.github/`, +`policy/`, `docs/`, `projects.yaml`, the prose pages, anything under +`archive/`, and anything unclassified (default deny). `cortex.py check` runs +on the branch tip and again on the trial-merge tree, so two branches that +each edited the same ledger are caught before either lands. `python3 +scripts/ledger_merge.py classify --base origin/main` predicts the verdict +(exit 0 ledger · 1 code · 2 could not classify). ## Check-in -**One command.** `pyauto-brain cortex checkin` is the door the human types when -they want to know where their science is: +`pyauto-brain cortex checkin` (the Brain's cortex conductor) is the one door: ```bash -pyauto-brain cortex checkin --dry-run # what it would pull and score -pyauto-brain cortex checkin --apply # pull, score, move, render, push -pyauto-brain cortex checkin --apply --no-push # ... without the push +pyauto-brain cortex checkin --dry-run # what it would pull; reaches nothing +pyauto-brain cortex checkin --apply # pull, show jobs, render, push +pyauto-brain cortex checkin --apply --no-push pyauto-brain cortex checkin --apply --project subhalo_validation -pyauto-brain cortex checkin --apply --skip-pull # re-score what is already here +pyauto-brain cortex checkin --apply --skip-pull ``` -It sweeps every `status: active` row of `projects.yaml`, plus any project that -owns a task in `submitted | running`, and runs **that project's own** -`<local_path>/<sync_cli> pull` — the verb all seven implement. Nothing else -reaches a cluster; the conductor adds no SSH. A pull that exits non-zero is -recorded against its project and the sweep continues. Then it scores every live -task (the six legs below), moves `running → pulled → awaiting-ruling`, -re-renders `dashboard.md` + `dashboard.html`, and prints a summary keyed **by -project** — the last thing on screen, so a chat sees it first. - -### `checkin.yaml` — the last-check-in stamp +It sweeps every `status: active` row plus any project whose ledger lists a +run, runs **that project's own** `<local_path>/<sync_cli> pull` (a failing +pull is recorded against its project and the sweep continues), runs its `jobs` +verb where the row has one and prints the output verbatim, writes +`checkin.yaml`, re-renders the two pages, pushes the ledger on +`claude/checkin-<date>` when `gh` is logged in and the checkout is clean on +`main`, and prints last a summary keyed by project: Now, the runs, the last +five entries, and the `cortex.py` lines the human is likely to type next. It +flips no state and writes no entry of its own — the jobs output is read by +the human or the agent in the session and recorded with `running` / `done`. -One key, one line, committed and pushed with the ledger: +### `checkin.yaml` ```yaml -refreshed: 2026-09-04T14:03:00Z +refreshed: 2026-09-10T19:41Z ``` -It records the last **check-in**, not the last render: `checkin --apply` (and -`collect --apply`) writes it from the same stamp it scored against, before the -pages are rendered, so a doc-only push that re-renders the board cannot fake -freshness. The board reads it back as "Last check-in"; the HTML twin computes -the age on the *viewer's* clock and reddens it past sixty minutes, because a -static page cannot know when it is being read. Absent, the board says "never -checked in". The file is stable between renders, so `dashboard --check` needs -no rule for it. - -### The by-project view - -The summary is a tree, and it is the same tree in three places: the -`## By project` section of `dashboard.md` / `dashboard.html`, the text -`pyauto-brain cortex census --by-project` prints from the checkout alone, and -the closing summary of a real `checkin` (which is the only one of the three -carrying pull results and six-leg health). One builder, three renderings — a -prompt added to a state appears in all of them. - -Per project: the three folders it lives in (`local_path`, `mirror` when it has -one, `ral_root`), what came of its pull, its task counts by state, then every -task that is still **open** — awaiting a ruling, still out there, ready to -submit, gated, planned — each with its state and health, its `## Where to look` -bullets verbatim, and a copy chip per prompt its state carries. `accepted`, -`rerun` and `dropped` tasks are history and appear only in the counts. A -project gets a block when its `projects.yaml` row is `status: active`, or when -it still holds an open task; every other project folds into one line. - -On the board `## By project` is the **first** section, above the state lists: -those are cross-project, so every row has to be read to find out whose it is, -while this one answers "where is everything" a screen per project. The counts -table sits above it and keeps the urgency sections one tap away; the counts -strip itself is unchanged (the Brain board reads it). - -### The five starting prompts - -Every task row hands the reader a command rather than a decision. Which -prompts a task carries is its **state**, and nothing else: - -| Prompt | States | What it says | -|---|---|---| -| rule on it | `pulled`, `awaiting-ruling` | read the witness and the pulled evidence, draft the ruling body, `rule <task> <verb> --body <file>` | -| the results are good — accept and open the next task | `pulled`, `awaiting-ruling` | `rule <task> accept --body <file>`, then open the next task (below) | -| run it again | `pulled`, `awaiting-ruling`, `running` | `rule <task> rerun --body <file>`, `move <task> ready`, the project's own submit, `move <task> submitted --run <jobid>` | -| where the jobs stand | `submitted`, `running` | the project's own `<sync_cli> jobs` | -| submit it | `ready` | the task, the project's own submit verb, `move <task> submitted --run <jobid>` | - -(`gated` and `planned` tasks carry the one-line `gates` / `move … ready` -command they always had.) - -**Accept and open the next task** is two commands in one prompt because they -are one decision: an accept that opens nothing leaves the programme where it -was. With the number gone there is no "N+1" to prefill — the tasks a project -holds are unordered ideas, so the second half offers the project's `planned` -and `gated` tasks to open (`move … ready` + the launch) and otherwise a `new` -line for a task the human names with a slug and a `Summary:`. The `new` line -carries the task's own `Epic:` and spells out the title rule: **`new` writes -`# <Project> — ` in front of `--title` itself**, so `--title` takes the tail -alone. - -**Run it again** is a ruling first: `rerun` is the only verb that puts a task -back on the board, so the verdict on what came back is what makes the next -submission a rerun rather than a repeat. Its `move … submitted --run` line -carries the reminder that it is one run per call (`--after <run>` chains the -next). - -The payloads are short on purpose. They are pasted into a fresh chat, not run -as scripts: every one of them ends in a human's own reading, and the `<file>`, -`<script>`, `<slug>` and `<jobid>` placeholders are where that reading lands. - -### `.cortex/pull.json` — the pull manifest - -After a successful pull the door writes the manifest at the top of that -project's pull root (its `mirror` if it has one, else its `local_path`) — -the same file `collect`'s checkpoint leg reads: - -| Key | Written by | Meaning | -|---|---|---| -| `project` | the check-in | the `projects.yaml` key this tree belongs to | -| `pulled_at` | the check-in | UTC `YYYY-MM-DDTHH:MMZ` of the pull that just finished | -| `cmd` | the check-in | the pull as a human would type it (`cd <local_path> && <sync_cli> pull`) | -| `rc` | the check-in | that pull's exit code — `0`, or the manifest is not written | -| `tasks_live` | the check-in | the task paths that were `submitted \| running` when it ran | -| `schema` | a project's own CLI | `1` where the CLI writes the richer shape; absent = the older `runs`-only one | -| `checkpoints` | a project's own CLI | `{"<run dir, relative to the pull root>": {"bytes": N, "mtime": ISO}}` | -| `runs` | a project's own CLI | `{"<jobid|jobid_task>": {"checkpoint_bytes": N, "checkpoint_mtime": ISO}}` | - -**Merged, never clobbered.** `autolens_profiling`'s own `hpc/sync` already -writes `checkpoints` / `runs` here — the only window onto the RAL-only -`search_internal/checkpoint.hdf5`, which no pull mirrors. The check-in adds its -five keys and leaves every other key exactly as it found it, so the richer -manifest keeps making the checkpoint leg scorable. Without a manifest that leg -is `UNOBSERVABLE`, not `FAIL`. - -The manifests live **outside** this repository, in the science trees, so -`cortex.py check` says nothing about them; the check-in's own note line -(`<project>: pulled → <path>`) is the record that one was written. - -### The push - -`--push` is allowed only when **`gh auth status` succeeds** *and* **this -checkout is clean on `main`** — checked before anything is written, since -"clean" stops being true the moment the tasks move. That is the cloud/laptop -split and it is also the default: a laptop pushes without asking, a cloud -session cannot and says so. The push cuts `claude/checkin-<YYYY-MM-DD>` from a -fresh `origin/main`, commits the changed paths explicitly and pushes; -`ledger_merge.yml` then merges it into `main` and deletes the branch (below). -`scripts/ledger_merge.py classify` is asked **first** and a code-classified -diff — `projects.yaml`, `scripts/`, the prose pages — is refused before the -branch is cut. Never `main` directly, never `--force`. - -## Driving the Cortex — the conductor and the workflows - -The Cortex is state plus `cortex.py`. The reasoning over it — the board and the -check-in — is the Brain's **cortex conductor**: - -```bash -pyauto-brain cortex checkin [--dry-run|--apply] [--push|--no-push] [--project KEY] [--skip-pull] [--refreshed ISO] -pyauto-brain cortex census [--json] # what is held, by state -pyauto-brain cortex census --by-project # ... by project: folders, open tasks, prompts -pyauto-brain cortex dashboard --check | --apply # render the two pages -pyauto-brain cortex gates # the gated tasks and their refs -pyauto-brain cortex collect [--task REL] [--pull] [--refreshed ISO] [--apply] [--out F] -``` - -`checkin` composes the others (above). `collect` with no `--task` scopes to -**every** task in `submitted | running` — the scorer on its own. +Written by `checkin --apply` before the pages are rendered, so a doc-only +push that re-renders the board cannot fake freshness. The HTML board reddens +the stamp on the viewer's clock once it is an hour old. -With no Brain install, or from a workflow: +## The board -```bash -python3 ../PyAutoBrain/agents/conductors/cortex/_cortex.py dashboard --cortex . --check -``` - -`--cortex` is a flag of the **subcommand**, not a global one: it follows the -verb. (`--cortex . dashboard --check` exits 2.) The root is resolved -`--cortex` → `$PYAUTO_CORTEX` → `PyAutoCortex` beside the Brain checkout. - -**Two spellings of the same flag.** The conductor writes with `--apply` (the -Brain's house spelling); `scripts/cortex.py` writes with `--write`. Every edit -to a task is `cortex.py`'s either way. - -**The `--check` exit-code contract** (`dashboard_refresh.yml` depends on it): - -| Code | Meaning | What the caller does | -|---|---|---| -| 0 | the committed pages match a fresh render | nothing | -| 1 | **drift** — and nothing else | error on a PR, self-heal on main | -| 2 | bad args — this Brain has no such verb/flag | report a renderer failure | -| other | the renderer could not run (a Brain/Cortex skew) | report a renderer failure | - -Treating 2 as drift sends whoever reads the log to fix `dashboard.md` when the -broken thing is the Brain: `dashboard_refresh.yml` wraps the call in `check()` -for exactly that reason. - -### What each workflow may write - -| Workflow | Trigger | Writes | -|---|---|---| -| `cortex_check.yml` | push/PR on `tasks/ rulings/ batches/ projects.yaml checkin.yaml dashboard.* scripts/ tests/` | nothing (`cortex.py check` + pytest) | -| `dashboard_refresh.yml` | push to main + PR on the ledger paths and the two pages, nightly **03:35 UTC**, dispatch | `dashboard.md`, `dashboard.html` on main (3-attempt fetch/reset/render/commit/push); a PR run errors instead of healing | -| `pages_dashboard.yml` | push to `dashboard.html`, dispatch | nothing in git — publishes `dashboard.html` as the Pages index | -| `ledger_merge.yml` | push to `claude/**`, dispatch | merges a ledger-only branch into main | - -**No scheduled job mutates the ledger.** A daily gate-grading cron did until -2026-09-03; retiring gate grading retired it with no replacement. - -Every workflow that can push to main shares `concurrency: group: -cortex-main-writers, cancel-in-progress: false` — `dashboard_refresh.yml` and -`ledger_merge.yml` — so two bot writers never race for the tip. -`pages_dashboard.yml` keeps its own `group: pages`. - -A push made with `GITHUB_TOKEN` triggers **no** workflow, so each writer -re-dispatches by name what its push should have woken: `dashboard_refresh.yml` -asks for `pages_dashboard.yml` (on both paths — fresh *and* healed; folding the -fresh one away is the bug that stranded the Mind's published board), and -`ledger_merge.yml` asks for `cortex_check.yml`. - -The board is published at **<https://pyautolabs.github.io/PyAutoCortex/>**. - ---- +`dashboard.md` and `dashboard.html`, rendered by the conductor: a counts table +(**Running**, **Open**, **Projects** — the Brain board reads these three +rows), the check-in chip and stamp, a four-column **Summary** (`Project · +Running · Open · Last update` — deliberately no free text, so it fits a +phone), then one card per project with a ledger: the facts line, **Now**, the +**Runs**, the **Last 5** entries, and a single 📋 **resume** chip for an active +project — the paste that has an agent read the ledger and the project's own +state file and say where you left off. Retired projects fold into one line +each; dormant rows with no ledger are a small table. No chip on the page +submits, scores or decides anything. ## Bootstrap ```bash -cd ~/Code/PyAutoLabs -git clone https://github.com/PyAutoLabs/PyAutoCortex.git -python3 PyAutoMind/scripts/repos_sync.py --write # CLAUDE.md, .claude/, the AGENTS.md blocks -python3 PyAutoCortex/scripts/cortex.py check # OK on an empty tree +python3 -m pip install pyyaml pytest +python3 scripts/cortex.py check +python3 -m pytest -q ``` - -The checkout directory must be named `PyAutoCortex`; the organism's sync and -drift scripts address it by that name. diff --git a/archive/AGENTS.md b/archive/AGENTS.md new file mode 100644 index 0000000..701d212 --- /dev/null +++ b/archive/AGENTS.md @@ -0,0 +1,17 @@ +# archive/ — frozen + +The task, ruling and batch ledgers the Cortex kept between 2026-08-31 and +2026-09-11, retired on 2026-09-12 (schema decision 60) when the Cortex became +one ledger per project. Nothing writes here any more, nothing reads it at +render time, and `scripts/ledger_merge.py` classifies **any** change under +this directory — an addition included — as code, which is a human's turn. + +It stays because the science repos' own ledgers cite these files by id +(`R-20260910-04`, `tasks/euclid_dr1_prelim/ordered_rerun_catalogue_parity.md`). +Every entry that mattered was carried into `projects/<key>.md` on the day of +the migration, in the words already on file, with a `[archive/tasks/…]` link +back to the question it came from. + +- `tasks/<project>/<slug>.md` — the pre-registered questions and their run lines +- `rulings/<YYYY>/<MM>/R-<YYYYMMDD>-<nn>.md` — the 37 verdicts, the human's words verbatim +- `batches/` — the three 2026-08/09 review-slot records and their reviews diff --git a/batches/2026-08-31-am.md b/archive/batches/2026-08-31-am.md similarity index 100% rename from batches/2026-08-31-am.md rename to archive/batches/2026-08-31-am.md diff --git a/batches/2026-08-31-pm.md b/archive/batches/2026-08-31-pm.md similarity index 100% rename from batches/2026-08-31-pm.md rename to archive/batches/2026-08-31-pm.md diff --git a/batches/2026-09-02-pm.md b/archive/batches/2026-09-02-pm.md similarity index 100% rename from batches/2026-09-02-pm.md rename to archive/batches/2026-09-02-pm.md diff --git a/batches/AGENTS.md b/archive/batches/AGENTS.md similarity index 75% rename from batches/AGENTS.md rename to archive/batches/AGENTS.md index dd6699f..b121b4a 100644 --- a/batches/AGENTS.md +++ b/archive/batches/AGENTS.md @@ -1,3 +1,5 @@ +> **Frozen 2026-09-12** — this folder is history under `archive/` (schema decision 60). Nothing writes here any more; the rules below describe how it was kept. + # Batch records — closed history Read-only history: never modified, only added. Three 2026-08/09 batch records diff --git a/batches/reviews/2026-08-31-am.md b/archive/batches/reviews/2026-08-31-am.md similarity index 100% rename from batches/reviews/2026-08-31-am.md rename to archive/batches/reviews/2026-08-31-am.md diff --git a/batches/reviews/2026-08-31-pm.md b/archive/batches/reviews/2026-08-31-pm.md similarity index 100% rename from batches/reviews/2026-08-31-pm.md rename to archive/batches/reviews/2026-08-31-pm.md diff --git a/batches/reviews/2026-09-02-pm.md b/archive/batches/reviews/2026-09-02-pm.md similarity index 100% rename from batches/reviews/2026-09-02-pm.md rename to archive/batches/reviews/2026-09-02-pm.md diff --git a/batches/reviews/AGENTS.md b/archive/batches/reviews/AGENTS.md similarity index 79% rename from batches/reviews/AGENTS.md rename to archive/batches/reviews/AGENTS.md index 9f4ef4e..f7336bd 100644 --- a/batches/reviews/AGENTS.md +++ b/archive/batches/reviews/AGENTS.md @@ -1,3 +1,5 @@ +> **Frozen 2026-09-12** — this folder is history under `archive/` (schema decision 60). Nothing writes here any more; the rules below describe how it was kept. + # Batch reviews — what the human ruled, verbatim Read-only history, kept for the same reason the records beside them are: 13 diff --git a/rulings/2026/08/R-20260831-01.md b/archive/rulings/2026/08/R-20260831-01.md similarity index 100% rename from rulings/2026/08/R-20260831-01.md rename to archive/rulings/2026/08/R-20260831-01.md diff --git a/rulings/2026/08/R-20260831-02.md b/archive/rulings/2026/08/R-20260831-02.md similarity index 100% rename from rulings/2026/08/R-20260831-02.md rename to archive/rulings/2026/08/R-20260831-02.md diff --git a/rulings/2026/08/R-20260831-03.md b/archive/rulings/2026/08/R-20260831-03.md similarity index 100% rename from rulings/2026/08/R-20260831-03.md rename to archive/rulings/2026/08/R-20260831-03.md diff --git a/rulings/2026/08/R-20260831-04.md b/archive/rulings/2026/08/R-20260831-04.md similarity index 100% rename from rulings/2026/08/R-20260831-04.md rename to archive/rulings/2026/08/R-20260831-04.md diff --git a/rulings/2026/08/R-20260831-05.md b/archive/rulings/2026/08/R-20260831-05.md similarity index 100% rename from rulings/2026/08/R-20260831-05.md rename to archive/rulings/2026/08/R-20260831-05.md diff --git a/rulings/2026/08/R-20260831-06.md b/archive/rulings/2026/08/R-20260831-06.md similarity index 100% rename from rulings/2026/08/R-20260831-06.md rename to archive/rulings/2026/08/R-20260831-06.md diff --git a/rulings/2026/08/R-20260831-07.md b/archive/rulings/2026/08/R-20260831-07.md similarity index 100% rename from rulings/2026/08/R-20260831-07.md rename to archive/rulings/2026/08/R-20260831-07.md diff --git a/rulings/2026/08/R-20260831-08.md b/archive/rulings/2026/08/R-20260831-08.md similarity index 100% rename from rulings/2026/08/R-20260831-08.md rename to archive/rulings/2026/08/R-20260831-08.md diff --git a/rulings/2026/09/R-20260901-01.md b/archive/rulings/2026/09/R-20260901-01.md similarity index 100% rename from rulings/2026/09/R-20260901-01.md rename to archive/rulings/2026/09/R-20260901-01.md diff --git a/rulings/2026/09/R-20260901-02.md b/archive/rulings/2026/09/R-20260901-02.md similarity index 100% rename from rulings/2026/09/R-20260901-02.md rename to archive/rulings/2026/09/R-20260901-02.md diff --git a/rulings/2026/09/R-20260901-03.md b/archive/rulings/2026/09/R-20260901-03.md similarity index 100% rename from rulings/2026/09/R-20260901-03.md rename to archive/rulings/2026/09/R-20260901-03.md diff --git a/rulings/2026/09/R-20260901-04.md b/archive/rulings/2026/09/R-20260901-04.md similarity index 100% rename from rulings/2026/09/R-20260901-04.md rename to archive/rulings/2026/09/R-20260901-04.md diff --git a/rulings/2026/09/R-20260902-01.md b/archive/rulings/2026/09/R-20260902-01.md similarity index 100% rename from rulings/2026/09/R-20260902-01.md rename to archive/rulings/2026/09/R-20260902-01.md diff --git a/rulings/2026/09/R-20260902-02.md b/archive/rulings/2026/09/R-20260902-02.md similarity index 100% rename from rulings/2026/09/R-20260902-02.md rename to archive/rulings/2026/09/R-20260902-02.md diff --git a/rulings/2026/09/R-20260902-03.md b/archive/rulings/2026/09/R-20260902-03.md similarity index 100% rename from rulings/2026/09/R-20260902-03.md rename to archive/rulings/2026/09/R-20260902-03.md diff --git a/rulings/2026/09/R-20260902-04.md b/archive/rulings/2026/09/R-20260902-04.md similarity index 100% rename from rulings/2026/09/R-20260902-04.md rename to archive/rulings/2026/09/R-20260902-04.md diff --git a/rulings/2026/09/R-20260902-05.md b/archive/rulings/2026/09/R-20260902-05.md similarity index 100% rename from rulings/2026/09/R-20260902-05.md rename to archive/rulings/2026/09/R-20260902-05.md diff --git a/rulings/2026/09/R-20260902-06.md b/archive/rulings/2026/09/R-20260902-06.md similarity index 100% rename from rulings/2026/09/R-20260902-06.md rename to archive/rulings/2026/09/R-20260902-06.md diff --git a/rulings/2026/09/R-20260902-07.md b/archive/rulings/2026/09/R-20260902-07.md similarity index 100% rename from rulings/2026/09/R-20260902-07.md rename to archive/rulings/2026/09/R-20260902-07.md diff --git a/rulings/2026/09/R-20260902-08.md b/archive/rulings/2026/09/R-20260902-08.md similarity index 100% rename from rulings/2026/09/R-20260902-08.md rename to archive/rulings/2026/09/R-20260902-08.md diff --git a/rulings/2026/09/R-20260902-09.md b/archive/rulings/2026/09/R-20260902-09.md similarity index 100% rename from rulings/2026/09/R-20260902-09.md rename to archive/rulings/2026/09/R-20260902-09.md diff --git a/rulings/2026/09/R-20260902-10.md b/archive/rulings/2026/09/R-20260902-10.md similarity index 100% rename from rulings/2026/09/R-20260902-10.md rename to archive/rulings/2026/09/R-20260902-10.md diff --git a/rulings/2026/09/R-20260904-01.md b/archive/rulings/2026/09/R-20260904-01.md similarity index 100% rename from rulings/2026/09/R-20260904-01.md rename to archive/rulings/2026/09/R-20260904-01.md diff --git a/rulings/2026/09/R-20260904-02.md b/archive/rulings/2026/09/R-20260904-02.md similarity index 100% rename from rulings/2026/09/R-20260904-02.md rename to archive/rulings/2026/09/R-20260904-02.md diff --git a/rulings/2026/09/R-20260904-03.md b/archive/rulings/2026/09/R-20260904-03.md similarity index 100% rename from rulings/2026/09/R-20260904-03.md rename to archive/rulings/2026/09/R-20260904-03.md diff --git a/rulings/2026/09/R-20260904-04.md b/archive/rulings/2026/09/R-20260904-04.md similarity index 100% rename from rulings/2026/09/R-20260904-04.md rename to archive/rulings/2026/09/R-20260904-04.md diff --git a/rulings/2026/09/R-20260904-05.md b/archive/rulings/2026/09/R-20260904-05.md similarity index 100% rename from rulings/2026/09/R-20260904-05.md rename to archive/rulings/2026/09/R-20260904-05.md diff --git a/rulings/2026/09/R-20260907-01.md b/archive/rulings/2026/09/R-20260907-01.md similarity index 100% rename from rulings/2026/09/R-20260907-01.md rename to archive/rulings/2026/09/R-20260907-01.md diff --git a/rulings/2026/09/R-20260907-02.md b/archive/rulings/2026/09/R-20260907-02.md similarity index 100% rename from rulings/2026/09/R-20260907-02.md rename to archive/rulings/2026/09/R-20260907-02.md diff --git a/rulings/2026/09/R-20260907-03.md b/archive/rulings/2026/09/R-20260907-03.md similarity index 100% rename from rulings/2026/09/R-20260907-03.md rename to archive/rulings/2026/09/R-20260907-03.md diff --git a/rulings/2026/09/R-20260907-04.md b/archive/rulings/2026/09/R-20260907-04.md similarity index 100% rename from rulings/2026/09/R-20260907-04.md rename to archive/rulings/2026/09/R-20260907-04.md diff --git a/rulings/2026/09/R-20260907-05.md b/archive/rulings/2026/09/R-20260907-05.md similarity index 100% rename from rulings/2026/09/R-20260907-05.md rename to archive/rulings/2026/09/R-20260907-05.md diff --git a/rulings/2026/09/R-20260907-06.md b/archive/rulings/2026/09/R-20260907-06.md similarity index 100% rename from rulings/2026/09/R-20260907-06.md rename to archive/rulings/2026/09/R-20260907-06.md diff --git a/rulings/2026/09/R-20260910-01.md b/archive/rulings/2026/09/R-20260910-01.md similarity index 100% rename from rulings/2026/09/R-20260910-01.md rename to archive/rulings/2026/09/R-20260910-01.md diff --git a/rulings/2026/09/R-20260910-02.md b/archive/rulings/2026/09/R-20260910-02.md similarity index 100% rename from rulings/2026/09/R-20260910-02.md rename to archive/rulings/2026/09/R-20260910-02.md diff --git a/rulings/2026/09/R-20260910-03.md b/archive/rulings/2026/09/R-20260910-03.md similarity index 100% rename from rulings/2026/09/R-20260910-03.md rename to archive/rulings/2026/09/R-20260910-03.md diff --git a/rulings/2026/09/R-20260910-04.md b/archive/rulings/2026/09/R-20260910-04.md similarity index 100% rename from rulings/2026/09/R-20260910-04.md rename to archive/rulings/2026/09/R-20260910-04.md diff --git a/rulings/AGENTS.md b/archive/rulings/AGENTS.md similarity index 93% rename from rulings/AGENTS.md rename to archive/rulings/AGENTS.md index a89dfaa..ad7abee 100644 --- a/rulings/AGENTS.md +++ b/archive/rulings/AGENTS.md @@ -1,3 +1,5 @@ +> **Frozen 2026-09-12** — this folder is history under `archive/` (schema decision 60). Nothing writes here any more; the rules below describe how it was kept. + # Rulings — the ledger of record `rulings/<YYYY>/<MM>/R-<YYYYMMDD>-<nn>.md`, one file per verdict per task. diff --git a/tasks/analytic_gaussian/ensemble_parity.md b/archive/tasks/analytic_gaussian/ensemble_parity.md similarity index 100% rename from tasks/analytic_gaussian/ensemble_parity.md rename to archive/tasks/analytic_gaussian/ensemble_parity.md diff --git a/tasks/analytic_gaussian/minimal_ep_legb_mu_threshold.md b/archive/tasks/analytic_gaussian/minimal_ep_legb_mu_threshold.md similarity index 100% rename from tasks/analytic_gaussian/minimal_ep_legb_mu_threshold.md rename to archive/tasks/analytic_gaussian/minimal_ep_legb_mu_threshold.md diff --git a/tasks/autolens_inference/slam_hst_base.md b/archive/tasks/autolens_inference/slam_hst_base.md similarity index 100% rename from tasks/autolens_inference/slam_hst_base.md rename to archive/tasks/autolens_inference/slam_hst_base.md diff --git a/tasks/ep_toy_gaussian/nuts_vs_ep_x3_n50.md b/archive/tasks/ep_toy_gaussian/nuts_vs_ep_x3_n50.md similarity index 100% rename from tasks/ep_toy_gaussian/nuts_vs_ep_x3_n50.md rename to archive/tasks/ep_toy_gaussian/nuts_vs_ep_x3_n50.md diff --git a/tasks/euclid/magnification_robustness.md b/archive/tasks/euclid/magnification_robustness.md similarity index 100% rename from tasks/euclid/magnification_robustness.md rename to archive/tasks/euclid/magnification_robustness.md diff --git a/tasks/euclid/resimulate_fitted_lens_simulator.md b/archive/tasks/euclid/resimulate_fitted_lens_simulator.md similarity index 100% rename from tasks/euclid/resimulate_fitted_lens_simulator.md rename to archive/tasks/euclid/resimulate_fitted_lens_simulator.md diff --git a/tasks/euclid/sersic_index_recovery.md b/archive/tasks/euclid/sersic_index_recovery.md similarity index 100% rename from tasks/euclid/sersic_index_recovery.md rename to archive/tasks/euclid/sersic_index_recovery.md diff --git a/tasks/euclid_dr1_prelim/dr1_prelim_10_lens_science_run.md b/archive/tasks/euclid_dr1_prelim/dr1_prelim_10_lens_science_run.md similarity index 100% rename from tasks/euclid_dr1_prelim/dr1_prelim_10_lens_science_run.md rename to archive/tasks/euclid_dr1_prelim/dr1_prelim_10_lens_science_run.md diff --git a/tasks/euclid_dr1_prelim/ordered_mge_control_unordered_102005065.md b/archive/tasks/euclid_dr1_prelim/ordered_mge_control_unordered_102005065.md similarity index 100% rename from tasks/euclid_dr1_prelim/ordered_mge_control_unordered_102005065.md rename to archive/tasks/euclid_dr1_prelim/ordered_mge_control_unordered_102005065.md diff --git a/tasks/euclid_dr1_prelim/ordered_mge_witness_102005065.md b/archive/tasks/euclid_dr1_prelim/ordered_mge_witness_102005065.md similarity index 100% rename from tasks/euclid_dr1_prelim/ordered_mge_witness_102005065.md rename to archive/tasks/euclid_dr1_prelim/ordered_mge_witness_102005065.md diff --git a/tasks/euclid_dr1_prelim/ordered_rerun_catalogue_parity.md b/archive/tasks/euclid_dr1_prelim/ordered_rerun_catalogue_parity.md similarity index 100% rename from tasks/euclid_dr1_prelim/ordered_rerun_catalogue_parity.md rename to archive/tasks/euclid_dr1_prelim/ordered_rerun_catalogue_parity.md diff --git a/tasks/ic50_workspace/ep_scale_up.md b/archive/tasks/ic50_workspace/ep_scale_up.md similarity index 100% rename from tasks/ic50_workspace/ep_scale_up.md rename to archive/tasks/ic50_workspace/ep_scale_up.md diff --git a/tasks/inference_programme/cluster_extended_source_inference.md b/archive/tasks/inference_programme/cluster_extended_source_inference.md similarity index 100% rename from tasks/inference_programme/cluster_extended_source_inference.md rename to archive/tasks/inference_programme/cluster_extended_source_inference.md diff --git a/tasks/inference_programme/cluster_gradient_search_benchmark.md b/archive/tasks/inference_programme/cluster_gradient_search_benchmark.md similarity index 100% rename from tasks/inference_programme/cluster_gradient_search_benchmark.md rename to archive/tasks/inference_programme/cluster_gradient_search_benchmark.md diff --git a/tasks/inference_programme/delaunay_fp64_retro_baseline.md b/archive/tasks/inference_programme/delaunay_fp64_retro_baseline.md similarity index 100% rename from tasks/inference_programme/delaunay_fp64_retro_baseline.md rename to archive/tasks/inference_programme/delaunay_fp64_retro_baseline.md diff --git a/tasks/inference_programme/failed_submissions_342008_10.md b/archive/tasks/inference_programme/failed_submissions_342008_10.md similarity index 100% rename from tasks/inference_programme/failed_submissions_342008_10.md rename to archive/tasks/inference_programme/failed_submissions_342008_10.md diff --git a/tasks/inference_programme/inference_refs_v1_redo.md b/archive/tasks/inference_programme/inference_refs_v1_redo.md similarity index 100% rename from tasks/inference_programme/inference_refs_v1_redo.md rename to archive/tasks/inference_programme/inference_refs_v1_redo.md diff --git a/tasks/inference_programme/knn_rerun_342016_7.md b/archive/tasks/inference_programme/knn_rerun_342016_7.md similarity index 100% rename from tasks/inference_programme/knn_rerun_342016_7.md rename to archive/tasks/inference_programme/knn_rerun_342016_7.md diff --git a/tasks/inference_programme/legacy_point_output_sweep.md b/archive/tasks/inference_programme/legacy_point_output_sweep.md similarity index 100% rename from tasks/inference_programme/legacy_point_output_sweep.md rename to archive/tasks/inference_programme/legacy_point_output_sweep.md diff --git a/tasks/inference_programme/likelihood_term_levers.md b/archive/tasks/inference_programme/likelihood_term_levers.md similarity index 100% rename from tasks/inference_programme/likelihood_term_levers.md rename to archive/tasks/inference_programme/likelihood_term_levers.md diff --git a/tasks/inference_programme/mass_pix_gradient_cost_probe.md b/archive/tasks/inference_programme/mass_pix_gradient_cost_probe.md similarity index 100% rename from tasks/inference_programme/mass_pix_gradient_cost_probe.md rename to archive/tasks/inference_programme/mass_pix_gradient_cost_probe.md diff --git a/tasks/inference_programme/mge_fp64_retro_baseline.md b/archive/tasks/inference_programme/mge_fp64_retro_baseline.md similarity index 100% rename from tasks/inference_programme/mge_fp64_retro_baseline.md rename to archive/tasks/inference_programme/mge_fp64_retro_baseline.md diff --git a/tasks/inference_programme/mge_pos_ref_reuse.md b/archive/tasks/inference_programme/mge_pos_ref_reuse.md similarity index 100% rename from tasks/inference_programme/mge_pos_ref_reuse.md rename to archive/tasks/inference_programme/mge_pos_ref_reuse.md diff --git a/tasks/inference_programme/multiband_compile_census_completion.md b/archive/tasks/inference_programme/multiband_compile_census_completion.md similarity index 100% rename from tasks/inference_programme/multiband_compile_census_completion.md rename to archive/tasks/inference_programme/multiband_compile_census_completion.md diff --git a/tasks/inference_programme/nautilus_mass_pix_baseline.md b/archive/tasks/inference_programme/nautilus_mass_pix_baseline.md similarity index 100% rename from tasks/inference_programme/nautilus_mass_pix_baseline.md rename to archive/tasks/inference_programme/nautilus_mass_pix_baseline.md diff --git a/tasks/inference_programme/phase2_nss_mainline_gate_a_reuse.md b/archive/tasks/inference_programme/phase2_nss_mainline_gate_a_reuse.md similarity index 100% rename from tasks/inference_programme/phase2_nss_mainline_gate_a_reuse.md rename to archive/tasks/inference_programme/phase2_nss_mainline_gate_a_reuse.md diff --git a/tasks/inference_programme/phase3_prodigy_reliability_gate_b1_reuse.md b/archive/tasks/inference_programme/phase3_prodigy_reliability_gate_b1_reuse.md similarity index 100% rename from tasks/inference_programme/phase3_prodigy_reliability_gate_b1_reuse.md rename to archive/tasks/inference_programme/phase3_prodigy_reliability_gate_b1_reuse.md diff --git a/tasks/inference_programme/phase4_positions_gate_b2_reuse.md b/archive/tasks/inference_programme/phase4_positions_gate_b2_reuse.md similarity index 100% rename from tasks/inference_programme/phase4_positions_gate_b2_reuse.md rename to archive/tasks/inference_programme/phase4_positions_gate_b2_reuse.md diff --git a/tasks/inference_programme/phase5_mesh_gradient_positions_on.md b/archive/tasks/inference_programme/phase5_mesh_gradient_positions_on.md similarity index 100% rename from tasks/inference_programme/phase5_mesh_gradient_positions_on.md rename to archive/tasks/inference_programme/phase5_mesh_gradient_positions_on.md diff --git a/tasks/inference_programme/prodigy_mass_pix.md b/archive/tasks/inference_programme/prodigy_mass_pix.md similarity index 100% rename from tasks/inference_programme/prodigy_mass_pix.md rename to archive/tasks/inference_programme/prodigy_mass_pix.md diff --git a/tasks/inference_programme/refs_5_6_342016_56.md b/archive/tasks/inference_programme/refs_5_6_342016_56.md similarity index 100% rename from tasks/inference_programme/refs_5_6_342016_56.md rename to archive/tasks/inference_programme/refs_5_6_342016_56.md diff --git a/tasks/inference_programme/refs_v1_positions_on_completion.md b/archive/tasks/inference_programme/refs_v1_positions_on_completion.md similarity index 100% rename from tasks/inference_programme/refs_v1_positions_on_completion.md rename to archive/tasks/inference_programme/refs_v1_positions_on_completion.md diff --git a/tasks/inference_programme/rewind_2026_08_31.md b/archive/tasks/inference_programme/rewind_2026_08_31.md similarity index 100% rename from tasks/inference_programme/rewind_2026_08_31.md rename to archive/tasks/inference_programme/rewind_2026_08_31.md diff --git a/tasks/inference_programme/slogdet_ab_342017.md b/archive/tasks/inference_programme/slogdet_ab_342017.md similarity index 100% rename from tasks/inference_programme/slogdet_ab_342017.md rename to archive/tasks/inference_programme/slogdet_ab_342017.md diff --git a/tasks/inference_programme/smc_probe_342018.md b/archive/tasks/inference_programme/smc_probe_342018.md similarity index 100% rename from tasks/inference_programme/smc_probe_342018.md rename to archive/tasks/inference_programme/smc_probe_342018.md diff --git a/tasks/slope_hierarchy/methods_writeup.md b/archive/tasks/slope_hierarchy/methods_writeup.md similarity index 100% rename from tasks/slope_hierarchy/methods_writeup.md rename to archive/tasks/slope_hierarchy/methods_writeup.md diff --git a/tasks/slope_hierarchy_scale/n25_scale_up.md b/archive/tasks/slope_hierarchy_scale/n25_scale_up.md similarity index 100% rename from tasks/slope_hierarchy_scale/n25_scale_up.md rename to archive/tasks/slope_hierarchy_scale/n25_scale_up.md diff --git a/tasks/subhalo_validation/delaunay_adapt_split_pl_eff_0.md b/archive/tasks/subhalo_validation/delaunay_adapt_split_pl_eff_0.md similarity index 100% rename from tasks/subhalo_validation/delaunay_adapt_split_pl_eff_0.md rename to archive/tasks/subhalo_validation/delaunay_adapt_split_pl_eff_0.md diff --git a/tasks/subhalo_validation/delaunay_adapt_split_pl_eff_1_outer.md b/archive/tasks/subhalo_validation/delaunay_adapt_split_pl_eff_1_outer.md similarity index 100% rename from tasks/subhalo_validation/delaunay_adapt_split_pl_eff_1_outer.md rename to archive/tasks/subhalo_validation/delaunay_adapt_split_pl_eff_1_outer.md diff --git a/tasks/subhalo_validation/delaunay_adapt_split_pl_sersic_0.md b/archive/tasks/subhalo_validation/delaunay_adapt_split_pl_sersic_0.md similarity index 100% rename from tasks/subhalo_validation/delaunay_adapt_split_pl_sersic_0.md rename to archive/tasks/subhalo_validation/delaunay_adapt_split_pl_sersic_0.md diff --git a/tasks/subhalo_validation/rectangular_adapt_pl_eff_0.md b/archive/tasks/subhalo_validation/rectangular_adapt_pl_eff_0.md similarity index 100% rename from tasks/subhalo_validation/rectangular_adapt_pl_eff_0.md rename to archive/tasks/subhalo_validation/rectangular_adapt_pl_eff_0.md diff --git a/tasks/subhalo_validation/rectangular_adapt_pl_eff_1_outer.md b/archive/tasks/subhalo_validation/rectangular_adapt_pl_eff_1_outer.md similarity index 100% rename from tasks/subhalo_validation/rectangular_adapt_pl_eff_1_outer.md rename to archive/tasks/subhalo_validation/rectangular_adapt_pl_eff_1_outer.md diff --git a/tasks/subhalo_validation/rectangular_adapt_pl_sersic_0.md b/archive/tasks/subhalo_validation/rectangular_adapt_pl_sersic_0.md similarity index 100% rename from tasks/subhalo_validation/rectangular_adapt_pl_sersic_0.md rename to archive/tasks/subhalo_validation/rectangular_adapt_pl_sersic_0.md diff --git a/docs/schema_decisions.md b/docs/schema_decisions.md index b127044..53f7e03 100644 --- a/docs/schema_decisions.md +++ b/docs/schema_decisions.md @@ -426,3 +426,40 @@ Choices made in phase 2 — the conductor, the board and the workflows inference — gradient-based fitting building on JAX knowledge") was not it. Mind **epics** keep their numbered phases: development work really is sequential, and `Epic:` on a task file is unchanged. + +60. **One ledger per project; tasks, witnesses, rulings and gates are + retired to `archive/`** (2026-09-12). The tree is `projects/<key>.md`, + one file per `projects.yaml` row that is doing anything, holding exactly + `## Now` (rewritten), `## Runs` (`open | running`; a finished run leaves + the list) and `## Log` (dated, newest-first, `run | result | lesson | + note`). The header is `Project:` and `Issue:`; the title carries a + ≤15-word summary. `scripts/cortex.py` keeps `check` and `retire`, gains + `new` / `run` / `running` / `done` / `log` / `now` / `issue`, and loses + `move`, `rule`, `gates`. The conductor keeps `checkin`, `census`, + `dashboard`, gains `issue`, and loses `collect`, `gates`, the six scoring + legs, the pull manifest, the work briefs and every task chip; the board + keeps one chip per active project (**resume**) and the check-in chip. + `tasks/`, `rulings/` and `batches/` moved under `archive/`, frozen: the + merge gate classifies any change there as code. Every task, run, ruling + and note was carried into its project's log on migration day, in the + words already on file, each question linking back to its archived task. + The organism table's Cortex role and the Brain's organ page say + "keeps track of what is true". + Why: the human's own account of how they do science, and the ledger's + evidence for it. In twelve days the task model produced 43 tasks of which + 16 were dropped by ruling; 37 rulings were written around one- or + two-line human verdicts; the pre-registered witnesses were restated + after the data came in (euclid_dr1_prelim's parity clause "cannot be met + by any pair of these runs and needs restating"; analytic_gaussian was + accepted with 4 of 11 criteria missed). The one thing the human kept + coming back to was the dated `## Notes` journal that grew inside a task + file — what was run, what was found, what next — which is exactly the + shape the ledger now has. A task is a thing to finish and retire; + science is done by running, looking, thinking and organically deciding + where to go next. So the Cortex is a history of how each project unfolds + and a way to pick one up after time away, not a queue. Two consequences + are rules rather than schema: **nothing in a ledger is inferred from + results** (a `result` or `lesson` is the human's words on the human's + ask; the agent records only cluster facts), and the top of each project's + GitHub issue is the same concise ledger the board shows (the `issue` + block), with the detailed run-through underneath it. diff --git a/projects/AGENTS.md b/projects/AGENTS.md new file mode 100644 index 0000000..aeb586e --- /dev/null +++ b/projects/AGENTS.md @@ -0,0 +1,26 @@ +# projects/ — one ledger per science project + +`projects/<key>.md`, one file per row of `projects.yaml` that is doing +anything. The file is a **history of how the project unfolds**, not a task: +nothing in it is ever done, completed or retired — it only gets older. + +## The rules + +1. **Every write is a `scripts/cortex.py` verb.** `run` / `running` / `done` + record what the cluster did; `log` and `now` record what the human said. + Never edit a ledger by hand in a session, and never write an entry the + human did not ask for. +2. **Nothing is inferred from results.** A `result` or `lesson` entry is the + human's words, on the human's ask. An agent reading a results file writes + nothing here about what it means; it tells the human, and the human says + what to log. +3. **`## Now` is rewritten, `## Log` is prepended.** Now is the head pointer + (where am I, what next); the log is the record. The board shows Now, the + runs on the cluster, and the last five entries. +4. **One line per entry.** `- YYYY-MM-DD — kind — text`, newest first; + continuation lines are indented two spaces. `--` is read as the dash. +5. **Run `python3 scripts/cortex.py check` before you push.** The + auto-merge gate runs it on the merged tree and leaves a failing branch for + a human. + +`AGENTS.md` here is doctrine, not an entry: the merge gate treats it as code. diff --git a/projects/analytic_gaussian.md b/projects/analytic_gaussian.md new file mode 100644 index 0000000..02d377e --- /dev/null +++ b/projects/analytic_gaussian.md @@ -0,0 +1,20 @@ +# analytic_gaussian — Graphical and EP against a closed-form Gaussian posterior, at ensemble scale + +Project: analytic_gaussian +Issue: none + +## Now + +Wave 1 (342413, 200 seeds at N=5) is accepted as the baseline: autofit EP is exact on the Gaussian leg, leg B sigma misses as pre-registered (78/200), collapse rate 0/200. Follow-ups filed through /intake. +Next: settle whether criterion 2's mu threshold or the minimal-EP control is wrong before any rerun; the N=25 rung is written but not submitted. + +## Runs + +## Log + +- 2026-09-10 — result — accepted ensemble_parity (R-20260910-04): ok accept and intake the things to address. we can do another run down the line so also achieve the results in the analytic_gaussian project for future comparison +- 2026-09-10 — note — question: Is criterion 2's mu threshold or the minimal-EP control wrong — planned, never run [archive/tasks/analytic_gaussian/minimal_ep_legb_mu_threshold.md] +- 2026-09-09 — run — 342413_[0-199] submitted (done, wall 0:05): ensemble_parity — the N=5 seed ensemble, 200 seeds, 50 concurrent, `hpc/batch_cpu/submit_ensemble_n5`, sample `ens_n5`. + RAL PyAuto mirror verified at PyAutoFit `66f9f8d5d` before submission, which **contains** `08207bad0` (#1580) — unlike phase 3's EP arm, this wave carries the full D1–D6 wave plus the review bundle; sacct 200/200 COMPLETED exit 0:0, per-seed wall 63–303 s; pulled + aggregated 2026-09-10 → analytic_gaussian b44390d (WITNESS NOT MET: 7 met / 4 miss) pulled_to: + /mnt/c/Users/Jammy/Science/analytic_gaussian/hpc/batch_cpu/output +- 2026-09-09 — note — question: Do graphical and EP recover closed-form means and errors [archive/tasks/analytic_gaussian/ensemble_parity.md] diff --git a/projects/autolens_inference.md b/projects/autolens_inference.md new file mode 100644 index 0000000..27a8bcc --- /dev/null +++ b/projects/autolens_inference.md @@ -0,0 +1,15 @@ +# autolens_inference — Do the PyAutoLens inference backends agree on the HST SLaM posterior + +Project: autolens_inference +Issue: none + +## Now + +Born 2026-09-10 as the from-scratch restart of inference_programme; slam_hst_base is the first question and is ready to submit — nothing has run yet. +Next: hpc/sync submit the six backend legs on the HST SLaM dataset and record the job ids here. + +## Runs + +## Log + +- 2026-09-11 — note — question: Do six backend legs agree on the HST SLaM posterior? — ready, never run [archive/tasks/autolens_inference/slam_hst_base.md] diff --git a/projects/ep_toy_gaussian.md b/projects/ep_toy_gaussian.md new file mode 100644 index 0000000..41be53b --- /dev/null +++ b/projects/ep_toy_gaussian.md @@ -0,0 +1,20 @@ +# ep_toy_gaussian — NUTS versus three EP fits on the #1405 collapse toy at N=50 + +Project: ep_toy_gaussian +Issue: none + +## Now + +n50_seed42 is on RAL: NUTS + one-by-one as 342639, EP x3 on identical data as 342640_[0-2]. The one-by-one leg finished (50/50 in 319 s); the EP repeats are turning per-factor Dynesty fits. +Next: pull and run scripts/compare.py --sample n50_seed42; settle first whether a prior factor with only NO_CHANGE rows counts as zero SUCCESS updates. + +## Runs + +- 342639 — open — ral — 2026-09-10 — nuts_vs_ep_x3_n50: one_by_one + BlackJAX NUTS, partition ral +- 342640 — open — ral — 2026-09-10 — nuts_vs_ep_x3_n50: EP x3 on identical data, array 0-2 + +## Log + +- 2026-09-10 — run — 342640 submitted: nuts_vs_ep_x3_n50 — EP x3 on identical data, array 0-2 +- 2026-09-10 — run — 342639 submitted: nuts_vs_ep_x3_n50 — one_by_one + BlackJAX NUTS, partition ral +- 2026-09-10 — note — question: Do three EP fits agree with NUTS at N=50 [archive/tasks/ep_toy_gaussian/nuts_vs_ep_x3_n50.md] diff --git a/projects/euclid.md b/projects/euclid.md new file mode 100644 index 0000000..2f08901 --- /dev/null +++ b/projects/euclid.md @@ -0,0 +1,16 @@ +# euclid — The original Euclid DR1 grade-AB catalogue project, superseded by euclid_dr1_prelim + +Project: euclid +Issue: none + +## Now + +Dormant since 2026-09-07: superseded by euclid_dr1_prelim. Three questions were planned and never run (see the archive). + +## Runs + +## Log + +- 2026-08-28 — note — question: Do we recover Sersic indices or is prior pile-up artefact — planned, never run [archive/tasks/euclid/sersic_index_recovery.md] +- 2026-08-28 — note — question: Can a fitted Euclid lens be resimulated and recovered — planned, never run [archive/tasks/euclid/resimulate_fitted_lens_simulator.md] +- 2026-08-28 — note — question: How robust are magnification estimates under model match and mismatch — planned, never run [archive/tasks/euclid/magnification_robustness.md] diff --git a/projects/euclid_dr1_prelim.md b/projects/euclid_dr1_prelim.md new file mode 100644 index 0000000..9e4e055 --- /dev/null +++ b/projects/euclid_dr1_prelim.md @@ -0,0 +1,93 @@ +# euclid_dr1_prelim — Refit the ten DR1-prelim tiles and rebuild the catalogue on RAL + +Project: euclid_dr1_prelim +Issue: none + +## Now + +342629 reran the ten tiles on refreshed libraries; eight finished and confirmed all four 09-10 defects fixed, tiles 1 and 3 are still in vis_lp. The SED chain ran on GPU for the eight (342648) and the CPU verification on tile 9 (342668_9) passed in 9:24, faster than the GPU — the CPU route is the one the main experiment uses. +Next: pull, bundle with output_sed, compare magnitudes (µJy fluxes) against the 20260623 catalogue; then tiles 1/3 SED on the CPU script. The parity clause needs restating: rerun scatter, not ordering, is what fails it. + +## Runs + +- 342629_[0-9] — running — ral — 2026-09-10 — ordered_rerun_catalogue_parity: two-stage vis_lp(n_live=750)+vis_pix(n_live=300), array 0-9, ten dr1_prelim tiles; libraries refreshed BEFORE submit (PyAutoNerves 0e7163bc2, PyAutoFit e354dbb6b, PyAutoArray 667deed3a, PyAutoGalaxy 6640a7494, PyAutoLens 0da06de63) carrying PyAutoFit#1598/#1600/#1602 and PyAutoLens#734; pipeline main 2a8b4db (PR#65 + #67); + 342398 output archived to output_v2_pre_refresh on laptop and RAL; purpose: confirmation rerun — vis_lp latents written (12 keys on 102005065), catalogue rebuild with 0 blank cells, same-library rerun scatter vs 342398 for the restated parity witness +- 342648_[0,2,4-9] — open — gpu — 2026-09-11 — ordered_rerun_catalogue_parity: SED chain `hpc/batch_gpu/submit_sersic_waveband_8tiles` (a copy of `submit_sersic_waveband` differing only in `--array=0,2,4-9`, because `hpc/sync submit` passes no sbatch arguments), `PYAUTO_OUTPUT_DIR=output_sed`: Sersic lens model on VIS then every non-VIS waveband fit (nir_y/j/h on all eight tiles, plus decam_g/r/i/z on the seven non-102005065 tiles), for the eight tiles 342629 completed; tiles 1 and 3 omitted because 342629_1/_3 are still fitting `vis_lp` and have no result to seed from. + `output_sed/<sample>/<tile>/initial_lens_model/vis_lp/<hash>.zip` seeded by `cp -p` from `output/` for all eight tiles before submit so the upstream stage short-circuits; `output/` untouched (16 zips before and after). + No library refresh on RAL, no push. Purpose: produce `sersic_lens_model/{vis,<band>}` so `catalogue/scripts/magnitudes.py` and `multi_wavelength.py` can run and the per-band magnitudes can be compared against `dr1_prelim_grade_ab_catalogue_csvs_20260623`. + Progress at 15:40 on 2026-09-11: tasks 0, 2, 4, 5, 6 COMPLETED on `euclid-ral-gpu-2` in 13:34–20:15 each (13:43, 15:08, 15:42, 20:15, 13:34), task 7 RUNNING, task 8 PENDING. **Task `_9` (`Tile102008848RA0601376380877DECNEG0634605061157`) was cancelled while still PENDING on 2026-09-11**, on the human's authorisation, so that tile could be run instead on the CPU route as the verification run + recorded below; it is the only array index removed and the other seven are unaffected +- 342668_9 — open — ral — 2026-09-11 — ordered_rerun_catalogue_parity: CPU verification of the SED chain via hpc/batch_cpu/submit_sersic_waveband (pipeline PR #70, sbatch --array=9): tile 102008848, JAX on the CPU backend, 8 CPU; same seeded vis_lp zip as the GPU tiles; purpose: verify the CPU setup the main experiment will use, and give a GPU-vs-CPU wall-time and result comparison. + Started 15:37:49 BST on `euclid-ral-compute-10-2`, 8 CPUs, partition `ral`. First-log evidence, all four pre-registered PASS criteria met inside 30 s: `JAX backend: cpu` (the script's cpu-backend guard); + `15:37:59 … Starting non-linear search with JAX (CPU: cpu)`; `15:38:02 vis_lp … output path … initial_lens_model/vis_lp/ac98229c0d67998215f1885157f8c5f4` — the seeded hash exactly — followed by `Fit Already Completed: skipping non-linear search`; then the Sersic VIS search `15:38:14 vis … Starting new Nautilus non-linear search` and `15:38:18 … JAX jit compilation of vectorized (vmap) likelihood function complete in 1.5 seconds`. + The submitted script is PR #70's byte-for-byte apart from the four site lines this clone's other `batch_cpu/` scripts already carry (`PROJECT_PATH` / `PYAUTO_HPC_BASE` exports, `sample=dr1_prelim_grade_ab`, the ten-tile dataset list, the real `--mail-user`); the PR file is the generic pipeline version (`sample=q1_walsmley`, one-entry dataset list, no exports), so a byte-identical copy would have resolved array index 9 to an empty dataset. + Every element under test — the `#SBATCH` resource directives, `PYAUTO_OUTPUT_DIR=output_sed`, the cpu-backend guard, the `env` line and `--number_of_cores=$THREADS` — is unchanged from PR #70 + +## Log + +- 2026-09-11 — note — **CPU verification PASSED and was faster than the GPU.** `342668_9` (ral, 8 CPU, JAX on the CPU backend) COMPLETED in **9:24** wall, exit 0, all eight `sersic_lens_model/{vis,nir_y,nir_j,nir_h,decam_g,decam_r,decam_i,decam_z}` searches written for `Tile102008848…`; the GPU tiles of `342648` took 13:34–20:56 each (`_0,_2,_4-7` COMPLETED). + The CPU route is the one the main experiment uses. Anomaly to read next: `342648_8` (`Tile102008532…`, GPU) was still RUNNING at 1:36 — five times its siblings. `342629_1/_3` still in `vis_lp` at 19:33. Next: pull, bundle with `output_sed`, magnitudes (µJy fluxes) vs `dr1_prelim_grade_ab_catalogue_csvs_20260623`; then tiles 1/3 SED on the CPU script. +- 2026-09-11 — note — **342629 scored on the eight tiles that finished; all four 09-10 defects confirmed fixed; parity fails on rerun scatter, not ordering.** `sacct`: tasks 0,2,4,5,6,7,8,9 COMPLETED (walls 5:45–9:42, MaxRSS 7.6–10.9 GB); tasks 1 (`Tile102007299…`) and 3 (`Tile102007903…`) still RUNNING in **stage 1 `vis_lp`** at 15:52 of 36:00 — alive, not hung (`checkpoint.hdf5` 72.5/76.9 MB written minutes ago; first checkpoint only at 7.6 h / 9.9 h in). + Those are two of the three positions-penalty-pinned tiles. Fixes verified: (1) every one of the 16 zips carries `.completed` and a **12-key** `files/latent/latent_summary.json` in *both* stages, all finite, and no `search.log` says "latent function raised" or "no finite latent samples" (PyAutoLens#734 + PyAutoFit#1600); + (2) `lens_mass.csv` has **0 blank cells across all 42 columns** including the `effective_einstein_radius` family (pipeline#65); (3) **0 of 64** (row, quantity) cells have 3σ exactly equal to 1σ, 62/64 strictly outside (PyAutoFit#1598); + (4) `output/` holds **0** sibling `<hash>/` directories beside a zip and **0** `samples.csv`, the aggregator found 16/16 searches and the bundle has **8 rows** where 09-10 got 5 of 10 (PyAutoFit#1602). + The #1602 fix was also re-run against the tree that broke it: `output_v2_pre_refresh`, with its 15 residue dirs, now builds a **10-row, 0-blank** `lens_mass.csv` while emitting 60 warnings "…has no `.completed` file but ….zip does; the zip was used", and writing nothing into that tree (verified byte-identical). + **Parity** (reports under `inspect/`): vs `20260623`, tile identity 8/8 exact, coverage 100 %, `effective_einstein_radius` 2/8 within combined 3σ (median z 4.94, max 27.3); vs **342398, the same code on the same data**, 2/8 with median z **15.4**, max **41.9** — three times worse, so the disagreement is Nautilus rerun scatter and the pre-registered "within combined 3σ" clause **cannot be met by any pair of these runs and needs restating**. "Astrometry exact" is likewise untestable today: `lens_mass.csv` has no astrometry column, and `magnitudes.csv` needs the SED chain, which has never run. + **MGE ordering is clean**: key `A.ell_comps_1 > B.ell_comps_1` holds **8/8**, and the two lens-light bases agree with 342398 as a set **8/8** (7 same order, 1 swap on `Tile102008165…`, whose key separation 0.0130 sits below its marginal width 0.0389 — the undetermined case the script documents). + Note the comparator applies the basis-swap symmetry to `lens_mass.csv`'s `ell_comps`, which are the `Isothermal` **mass** ellipticity, so its `ell_comps` FAIL is a scatter row, not a labelling finding. + **Disk**: 16 zips = **47.44 MB** (tree 49 MB) against the `output_v1` baseline 130.05 MB / 17 zips / 503 MB — **2.97 MB per search vs 7.65 = 38.8 %**, which clears the <40 % clause per search, while the eight-tile payload projected to ten is ≈59 MB = **45.6 %**, which does not; the clause needs a reading. + New: latent 1σ **collapses** on some stages (the latent PDF is drawn from 100 samples, so both 1σ percentiles can land on one sample — `Tile102008532…` `vis_lp` is degenerate on 12 of 12). `Tile102008475…` `vis_lp` is still pinned (log Z −5.07e6 → max source-plane separation ≈0.2507" against `threshold` 0.2); its `vis_pix` is healthy. + Science clone merged to pipeline main (`a8529cb`, PR #68) and committed locally at `d481264`; nothing pushed, nothing submitted. +- 2026-09-11 — run — 342668_9 submitted: ordered_rerun_catalogue_parity — CPU verification of the SED chain via hpc/batch_cpu/submit_sersic_waveband (pipeline PR #70, sbatch --array=9): tile 102008848, JAX on the CPU backend, 8 CPU; same seeded vis_lp zip as the GPU tiles; purpose: verify the CPU setup the main experiment will use, and give a GPU-vs-CPU wall-time and result comparison. + Started 15:37:49 BST on `euclid-ral-compute-10-2`, 8 CPUs, partition `ral`. First-log evidence, all four pre-registered PASS criteria met inside 30 s: `JAX backend: cpu` (the script's cpu-backend guard); + `15:37:59 … Starting non-linear search with JAX (CPU: cpu)`; `15:38:02 vis_lp … output path … initial_lens_model/vis_lp/ac98229c0d67998215f1885157f8c5f4` — the seeded hash exactly — followed by `Fit Already Completed: skipping non-linear search`; then the Sersic VIS search `15:38:14 vis … Starting new Nautilus non-linear search` and `15:38:18 … JAX jit compilation of vectorized (vmap) likelihood function complete in 1.5 seconds`. + The submitted script is PR #70's byte-for-byte apart from the four site lines this clone's other `batch_cpu/` scripts already carry (`PROJECT_PATH` / `PYAUTO_HPC_BASE` exports, `sample=dr1_prelim_grade_ab`, the ten-tile dataset list, the real `--mail-user`); the PR file is the generic pipeline version (`sample=q1_walsmley`, one-entry dataset list, no exports), so a byte-identical copy would have resolved array index 9 to an empty dataset. + Every element under test — the `#SBATCH` resource directives, `PYAUTO_OUTPUT_DIR=output_sed`, the cpu-backend guard, the `env` line and `--number_of_cores=$THREADS` — is unchanged from PR #70 +- 2026-09-11 — run — 342648_[0,2,4-9] submitted: ordered_rerun_catalogue_parity — SED chain `hpc/batch_gpu/submit_sersic_waveband_8tiles` (a copy of `submit_sersic_waveband` differing only in `--array=0,2,4-9`, because `hpc/sync submit` passes no sbatch arguments), `PYAUTO_OUTPUT_DIR=output_sed`: Sersic lens model on VIS then every non-VIS waveband fit (nir_y/j/h on all eight tiles, plus decam_g/r/i/z on the seven non-102005065 tiles), for the eight tiles 342629 completed; tiles 1 and 3 omitted because 342629_1/_3 are still fitting `vis_lp` and have no result to seed from. + `output_sed/<sample>/<tile>/initial_lens_model/vis_lp/<hash>.zip` seeded by `cp -p` from `output/` for all eight tiles before submit so the upstream stage short-circuits; `output/` untouched (16 zips before and after). + No library refresh on RAL, no push. Purpose: produce `sersic_lens_model/{vis,<band>}` so `catalogue/scripts/magnitudes.py` and `multi_wavelength.py` can run and the per-band magnitudes can be compared against `dr1_prelim_grade_ab_catalogue_csvs_20260623`. + Progress at 15:40 on 2026-09-11: tasks 0, 2, 4, 5, 6 COMPLETED on `euclid-ral-gpu-2` in 13:34–20:15 each (13:43, 15:08, 15:42, 20:15, 13:34), task 7 RUNNING, task 8 PENDING. **Task `_9` (`Tile102008848RA0601376380877DECNEG0634605061157`) was cancelled while still PENDING on 2026-09-11**, on the human's authorisation, so that tile could be run instead on the CPU route as the verification run + recorded below; it is the only array index removed and the other seven are unaffected +- 2026-09-10 — result — accepted dr1_prelim_10_lens_science_run (R-20260910-03): For euclid_dr1_prelim, I am content these 3 are success so wrap them up and then do the 4th task on the catalogue stuff +- 2026-09-10 — result — accepted ordered_mge_control_unordered_102005065 (R-20260910-02): For euclid_dr1_prelim, I am content these 3 are success so wrap them up and then do the 4th task on the catalogue stuff +- 2026-09-10 — result — accepted ordered_mge_witness_102005065 (R-20260910-01): For euclid_dr1_prelim, I am content these 3 are success so wrap them up and then do the 4th task on the catalogue stuff +- 2026-09-10 — note — **library and pipeline fixes shipped, confirmation rerun 342629 submitted.** pipeline#65 fixed the `latent.<name>` prefix so `effective_einstein_radius` is no longer a silent `None`; + PyAutoFit#1598 corrected the 3σ / `max_lh` aggregate-CSV paths; PyAutoLens#734 + PyAutoFit#1600 + pipeline#67 trace `LatentEuclid.variables` under the latent engine's `jax.jit` so `vis_lp` writes latents (12 keys on 102005065). + PyAutoFit#1602 found the aggregator sibling-dir root cause: post-completion caches recreate `<hash>/` via `_files_path` mkdir plus a `preserve_in_zip` loose copy — fixed, and aggregator output order is now path-sorted. The 14-file laptop residue dirs were rsync-without-`--delete` mid-run pulls, not a library bug. +- 2026-09-10 — note — **laptop-side scoring of 342398; RAL unreachable** (the SSH jump host refuses the key, so the SED chain, the RAL catalogue build and the RAL `output_v1` check did not run). + All ten tiles finished both stages, task walls 6:41–13:12, `.err` clean. Findings: (1) `vis_lp` on 102007299 / 102007903 / 102008475 is pinned by the positions penalty — log Z −7.9e7 / −5.1e7 / −5.1e6, max source-plane separation 0.99 / 0.71 / 0.25 against `positions_threshold` 0.2 — completed searches, not usable fits, and the same tiles that failed in 342301. + (2) 15 unzipped `<hash>/` directories sit beside their zips (19.1 MB, RAL mtimes inside the job window: `hpc_mode` removed the tree and the pre-fit file set was re-emitted at the stage boundary; no `.completed`). + A sibling directory makes `Aggregator.from_directory(completed_only=True)` take the directory, never open the zip, and drop the search silently (controlled test: 1 vs 4 search outputs) — `lens_mass.csv` came out with 5 of 10 rows and the bundle build aborted at stage 4. + (3) `effective_einstein_radius` is blank on every row: the `latent.<name>` argument path no longer matches the unprefixed keys the latent summary merges in, a silent `None`. (4) Parity against `dr1_prelim_grade_ab_catalogue_csvs_20260623` on a zip-only rebuild: tile identity 10/10; astrometry exact 0/10; + `einstein_radius` within combined 3σ 6/10; mass `ell_comps` 3/10 and 6/10; shear 4/10 and 7/10. A three-way control on `samples_summary.json` medians shows the 2026-09-07 `output_v1` runs miss the June reference by the same or larger margins, so the disagreement is rerun scatter, not the ordering. + (5) MGE ordering key `A_ec1 > B_ec1` holds 10/10; sets agree with `output_v1` up to a swap on 9/10 (clean swaps on 102005065, 102007903, 102008532); 102008165 is undetermined (key separation 0.0125, 1σ overlap) and 102007299 marginal. + (6) Disk: zips 61.54 MB = 47.3 % of the 130.05 MB `output_v1` zip baseline (fails the <40 % clause); whole tree 81 MB vs 526 MB (15.4 %); `samples.csv` 0 loose, 0 in-zip. (7) `output_v1` no longer loads under the installed stack (`autoarray/inversion/mesh/mesh/delaunay.py:97` `zeroed_pixels` array-truth error on the 09-07 `model.json`). + Prepared, not submitted: `hpc/batch_gpu/submit_sersic_waveband` in the science clone (uncommitted; ten tiles, array 0-9, `output_sed`), with the RAL seed-copy sequence in the session report. Laptop bundle: `inspect/dr1_prelim_grade_ab_ordered_v2/` (stage 1 PNGs for all ten, 5-row `lens_mass.csv`). +- 2026-09-10 — run — 342629_[0-9] submitted: ordered_rerun_catalogue_parity — two-stage vis_lp(n_live=750)+vis_pix(n_live=300), array 0-9, ten dr1_prelim tiles; libraries refreshed BEFORE submit (PyAutoNerves 0e7163bc2, PyAutoFit e354dbb6b, PyAutoArray 667deed3a, PyAutoGalaxy 6640a7494, PyAutoLens 0da06de63) carrying PyAutoFit#1598/#1600/#1602 and PyAutoLens#734; pipeline main 2a8b4db (PR#65 + #67); + 342398 output archived to output_v2_pre_refresh on laptop and RAL; purpose: confirmation rerun — vis_lp latents written (12 keys on 102005065), catalogue rebuild with 0 blank cells, same-library rerun scatter vs 342398 for the restated parity witness +- 2026-09-09 — run — 342398_[0-9] submitted (done, wall 13:12): ordered_rerun_catalogue_parity — two-stage vis_lp(n_live=750)+vis_pix(n_live=300), array 0-9, ten dr1_prelim tiles; output/ archived to output_v1 on laptop and RAL; + RAL libs refreshed BEFORE submit (PyAutoFit 66f9f8d5d, Array 35aa681f, Lens 7d1b04de8, Galaxy 99cf7429, Nerves 0e7163b) and must not be touched until it finishes - mid-run library drift is what broke 342301_1/_7/_8; pipeline main 93a389e (PR#61); all ten tiles finished both stages, .err files carry only the standard warnings pulled_to: + /mnt/c/Users/Jammy/Science/euclid_dr1_prelim/output/dr1_prelim_grade_ab +- 2026-09-09 — note — question: Do ordered Nautilus reruns reproduce the euclid catalogue values? [archive/tasks/euclid_dr1_prelim/ordered_rerun_catalogue_parity.md] +- 2026-09-08 — run — 342375_[0-1] submitted (done, wall 0:46): ordered_mge_witness_102005065 — two independent unseeded runs, own output roots output_ordered_witness/run_0,1; + RAL PyAutoFit 84512bffa, PyAutoGalaxy 99cf7429, science clone 152d086; both finished vis_lp clean, log Z 9501.04 in each, walls 0:45 and 0:46 pulled_to: /mnt/c/Users/Jammy/Science/euclid_dr1_prelim/output_ordered_witness +- 2026-09-08 — note — question: Does ordering the MGE bases fix labelling on one tile [archive/tasks/euclid_dr1_prelim/ordered_mge_witness_102005065.md] +- 2026-09-08 — run — 342377_0 submitted (done, wall 1:18): ordered_mge_control_unordered_102005065 — unordered control on today's stack; own output root output_ordered_witness/control_unordered; finished vis_lp clean, log Z 9501.12 — within 0.08 of the ordered runs, so the 8-nat + gain over the phase-4 baseline is library drift pulled_to: /mnt/c/Users/Jammy/Science/euclid_dr1_prelim/output_ordered_witness/control_unordered +- 2026-09-08 — note — question: Is the ordered run's 8-nat gain ordering or library drift [archive/tasks/euclid_dr1_prelim/ordered_mge_control_unordered_102005065.md] +- 2026-09-07 — note — **task 3 resubmitted; the other nine run on under the pre-fix code.** `342301_3` (`Tile102007903RA0668831429074DECNEG0648901814905`) died at 03:52 when the first quick update built the max-likelihood instance from a source MGE `ell_comps` of magnitude 1.009 — outside the unit disk, so `ModelParameterException`. + Two merged fixes answer it: PyAutoFit `f6a991504` (PR #1568) makes the quick update tolerate an invalid instance, and pipeline `d78468b` (PR #53) bounds the source MGE `ell_comps` to [-0.7, 0.7] per component so the corner of the unit box is unreachable. + Only PyAutoFit was pulled on RAL (`cdda28b5f` → `f6a991504`); PyAutoArray `e36a5af4`, PyAutoGalaxy `6d216c15`, PyAutoLens `146a3d725` and PyAutoNerves `fc9c474` were deliberately left alone so the nine live tasks' `vis_pix` stage — a fresh interpreter — picks up no unrelated library change mid-run. + The nine were 1:20 in and were not cancelled, so this phase's evidence is provenance-split: `342301_[0-9]` minus task 3 under the pre-fix code, `342314_3` under the fix. That PyAutoFit fast-forward also carried two unrelated commits (`86cc1e182` deferring `scipy.special`, `e27eb8cbb` a recursion-walk skip), which the nine tasks' `vis_pix` interpreters will therefore also see. +- 2026-09-07 — note — launched: sort key `sorted()` (plain Python, byte-wise ascending) over the directory names under `/mnt/c/Users/Jammy/Science/euclid/dataset/dr1_prelim_grade_ab/` (3097 entries, all directories), first 10 taken; datasets: `Tile102005065RA0135279431487DECNEG0701599765928`, `Tile102007299RA0702283866574DECNEG0660415308762`, `Tile102007899RA0631694872236DECNEG0650584220817`, `Tile102007903RA0668831429074DECNEG0648901814905`, `Tile102008165RA0109664211519DECNEG0642902327064`, `Tile102008219RA0727851454839DECNEG0644382776514`, `Tile102008468RA3567390985250DECNEG0647172046261`, `Tile102008475RA0039777627054DECNEG0637715426503`, `Tile102008532RA0683495100072DECNEG0642073858939`, `Tile102008848RA0601376380877DECNEG0634605061157`; jobs 342301_[0-9]; project born at `/mnt/c/Users/Jammy/Science/euclid_dr1_prelim`, RAL `/mnt/ral/jnightin/euclid_dr1_prelim`. + Born by cloning `euclid_strong_lens_modeling_pipeline` (`dbdbe1d`) and nothing else; the only local adaptations are `hpc/sync.conf` and the project's own copy of `hpc/batch_cpu/submit_initial_lens_model_two_stage` (sample, the 10-entry `datasets` list, `--array=0-9`, `--partition=ral`, the mail address, and `PROJECT_PATH` / `PYAUTO_HPC_BASE` exported in-script because `hpc/sync submit` carries none). + `config/` is exactly as committed — the configuration the 2026-09-03 RAL acceptance run used. +- 2026-09-07 — run — 342314_3 submitted (done, wall 7:28): dr1_prelim_10_lens_science_run — task 3 (Tile102007903RA0668831429074DECNEG0648901814905) resubmitted under the ell_comps fix — PyAutoFit f6a991504 (#1568) + pipeline d78468b (#53); both stages finished (the other nine 342301 tasks ran on under the pre-fix code: RAL PyAutoFit was cdda28b5f; + PyAutoArray e36a5af4 / PyAutoGalaxy 6d216c15 / PyAutoLens 146a3d725 / PyAutoNerves fc9c474 left untouched) pulled_to: /mnt/c/Users/Jammy/Science/euclid_dr1_prelim/output_v1/dr1_prelim_grade_ab +- 2026-09-07 — run — 342301_3 submitted (failed): dr1_prelim_10_lens_science_run — died at the first quick update on a source MGE ell_comps magnitude 1.009 (ModelParameterException), see ## Notes; resubmitted alone as 342314_3 +- 2026-09-07 — run — 342301_[1,7-8] submitted (failed): dr1_prelim_10_lens_science_run — vis_lp finished, then the vis_pix stage aborted with "Run the light-profile stage first": the vis_lp result identifier changed when PyAutoFit was pulled on RAL mid-run; walls 1:39, 1:59, 1:56 +- 2026-09-07 — run — 342301_[0,2,4-6,9] submitted (done, wall 6:54): dr1_prelim_10_lens_science_run — six of the ten tiles finished both stages (vis_lp JAX on CPU then vis_pix numba+pool) under the pre-fix code; walls 5:35, 6:54, 4:52, 6:50, 5:12, 5:23; sort key and dataset list + in ## Notes pulled_to: /mnt/c/Users/Jammy/Science/euclid_dr1_prelim/output_v1/dr1_prelim_grade_ab +- 2026-09-05 — note — **`vis_pix` magnification column.** The archived `initial_lens_model/vis_pix` results (and the 20260623 reference this witness compares against) carry `magnification = 0.0` as a sentinel: the library latent was 0/0 for any pixelized source (PyAutoLens#726). + PyAutoLens PR #727 (merged) and #728 (correction: per-data-pixel convention) fix it; euclid PR #51 adds the test. Until the reference is re-derived under the fixed library, the numerics witness must exclude the `vis_pix` magnification column, and any comparison of it is against a sentinel, not a measurement. Mind ledger: `PyAutoMind/draft/feature/euclid/euclid_dr1_prep_epic.md` item 8. +- 2026-08-28 — note — question: Ten real DR1 lenses fitted from the pipeline repo alone [archive/tasks/euclid_dr1_prelim/dr1_prelim_10_lens_science_run.md] diff --git a/projects/ic50_workspace.md b/projects/ic50_workspace.md new file mode 100644 index 0000000..b0134e7 --- /dev/null +++ b/projects/ic50_workspace.md @@ -0,0 +1,24 @@ +# ic50_workspace — EP against the graphical joint fit on IC50 dose-response data, scaling up + +Project: ic50_workspace +Issue: none + +## Now + +Parity at N=5 achieved on RAL (33/33 within 3σ for both methods, runs 342408 / 342409); the scale ladder runs 342411 and 342412 are submitted. +Next: pull the ladder and see where EP's cost and hill_coef width go as N grows; ep_lbfgs_jax is the scale lever. + +## Runs + +- 342408 — open — ral — 2026-09-09 — ep_scale_up: EP sim, nlive 150, max_steps 12 +- 342409 — open — ral — 2026-09-09 — ep_scale_up: graphical sim joint Dynesty, nlive 150 +- 342411 — open — ral — 2026-09-09 — ep_scale_up: EP scale ladder, sim rungs 5/10/25/50, nlive 150 max_steps 12 +- 342412 — open — ral — 2026-09-09 — ep_scale_up: graphical scale ladder, sim rungs 5/10/25/50, nlive 150 + +## Log + +- 2026-09-09 — run — 342412 submitted: ep_scale_up — graphical scale ladder, sim rungs 5/10/25/50, nlive 150 +- 2026-09-09 — run — 342411 submitted: ep_scale_up — EP scale ladder, sim rungs 5/10/25/50, nlive 150 max_steps 12 +- 2026-09-09 — run — 342409 submitted: ep_scale_up — graphical sim joint Dynesty, nlive 150 +- 2026-09-09 — run — 342408 submitted: ep_scale_up — EP sim, nlive 150, max_steps 12 +- 2026-08-19 — note — question: Does EP match the graphical joint fit end to end [archive/tasks/ic50_workspace/ep_scale_up.md] diff --git a/projects/inference_programme.md b/projects/inference_programme.md new file mode 100644 index 0000000..d6200d0 --- /dev/null +++ b/projects/inference_programme.md @@ -0,0 +1,182 @@ +# inference_programme — The first inference programme, retired and restarted as autolens_inference + +Project: inference_programme +Issue: none + +## Now + +Retired 2026-09-07 and restarted from scratch as autolens_inference; every open question was dropped by ruling. + +## Runs + +## Log + +- 2026-09-07 — note — dropped legacy_point_output_sweep (R-20260907-06): I want to retire inference_programme fully, I will start it again from scratch, I am not happy with how i set up that science project and it has done lots to inform how Cortex manages things. +- 2026-09-07 — note — dropped cluster_extended_source_inference (R-20260907-05): I want to retire inference_programme fully, I will start it again from scratch, I am not happy with how i set up that science project and it has done lots to inform how Cortex manages things. +- 2026-09-07 — note — dropped likelihood_term_levers (R-20260907-04): I want to retire inference_programme fully, I will start it again from scratch, I am not happy with how i set up that science project and it has done lots to inform how Cortex manages things. +- 2026-09-07 — note — dropped prodigy_mass_pix (R-20260907-03): I want to retire inference_programme fully, I will start it again from scratch, I am not happy with how i set up that science project and it has done lots to inform how Cortex manages things. +- 2026-09-07 — note — dropped nautilus_mass_pix_baseline (R-20260907-02): I want to retire inference_programme fully, I will start it again from scratch, I am not happy with how i set up that science project and it has done lots to inform how Cortex manages things. +- 2026-09-04 — note — dropped multiband_compile_census_completion (R-20260904-05): Dropped 2026-09-04: the multi-band compile question was closed by the compile-axis arc (Mind record complete/2026/08/jax-compile-time-profiling-absorbed.md — settings suffice); the confirmatory + A100/hetero rows are not needed and the README verdict bullets in scripts/misc/jax_compile/README.md are the record. +- 2026-09-04 — note — dropped mass_pix_gradient_cost_probe (R-20260904-04): Dropped 2026-09-04 by human decision: the epic asks whether gradients reduce the number of inference steps, not what a gradient costs. + The cost probe is re-filed as an ordinary autolens_profiling prompt in PyAutoMind (`draft/feature/autolens_profiling/gradient_cost_probe.md`); the science phases start at phase 21. +- 2026-09-04 — note — dropped cluster_gradient_search_benchmark (R-20260904-03): Dropped 2026-09-04: absorbed into its Mind prompt per the phase's own note; nothing to run here. +- 2026-09-04 — note — dropped phase5_mesh_gradient_positions_on (R-20260904-02): Superseded 2026-09-04 by `gradient-slam-baseline`: the mesh gradient question restarts on a `mass[1]`-shaped target (fixed lens light, fixed source), not on the all-free SLaM targets. +- 2026-09-04 — result — accepted refs_v1_positions_on_completion (R-20260904-01): Accepted. Job 342241 tasks 11–13 delivered `pixelization_pos_fp64` (max logL 30,941.226, 48,592 evals, 43:03), `knn_pos_fp64` (30,923.144) and `delaunay_matern_pos_fp64` (31,405.425), all `.completed`, all with `positions.info` (threshold 0.2, penalty 0 at the best point), all physical (r_E 1.60). + They join `InferenceRefs_v1` as rows 11–13; the set is now 9 certified baselines. This is the last ruling of `jax-inference-profiling`; the programme is retired by the same check-in and continues as `gradient-slam-baseline`. +- 2026-09-04 — note — question: Does a gradient search beat Nautilus on mass_pix [archive/tasks/inference_programme/prodigy_mass_pix.md] +- 2026-09-04 — note — question: What Nautilus achieves on mass_pix at f1e8 and f1e5 [archive/tasks/inference_programme/nautilus_mass_pix_baseline.md] +- 2026-09-04 — note — question: Is a mass_pix gradient affordable on an A100 and correct [archive/tasks/inference_programme/mass_pix_gradient_cost_probe.md] +- 2026-09-04 — note — question: Are likelihood terms behind gradient search failure on mass_pix [archive/tasks/inference_programme/likelihood_term_levers.md] +- 2026-09-02 — result — accepted phase4_positions_gate_b2_reuse (R-20260902-10): **phase4-positions-gate-b2-reuse — ACCEPTED; GATE B part 2 moves from PROVISIONAL back to CALLED, with a design note for Phase 5.** Human, 2026-09-02, verbatim: *"continue"* — ratifying the packet recommendation (accept). + The human did not pin Phase 5's penalty factor in this ruling; the note below is the scribe's recommendation, to be decided when Phase 5 is filed. **What stands (recomputed from the committed rows):** PositionsLH is safe for gradient MAP on MGE at factor ≤ 1e5; factor 1e8 is rejected for gradient search. + Nautilus is inert on the posterior — Δ logZ ≤ **0.022 nats** on/off across 5 seeds (t0.3 f1e8), max logL lower with positions on in 5/5 seeds by a mean 0.126 nats (t = −3.45), wall −3 % to +7 %. Prodigy n=256 hits: positions-off **5/5**, t0.3 f1e5 **5/5**, t0.3 f1e8 **2/5**, tauto0.2 f1e8 **0/4** — stiffness, not tightness: a 1e8 slope against a 3×10⁴-scale log-likelihood inflates the step scale and pins lanes. + The six caveats of the 2026-08-27 call ride with it unchanged (idealised truth positions; one cell / five seeds; 1e5 shown safe, not calibrated; Nautilus unaffected; no `penalty_at_best` field; `target_id` positions block fixed only in the 2026-08-27 PR). + 341892_0 (a resumed run, 68 s) stays excluded as Phase 4 already excluded it. Stage 1 has no SLURM job — CPU transects, cited by artefact. **How this sits beside R-20260902-01:** the mesh positions rule and the `pos_tauto0.2_f1e8` reference rows are **nested-sampling** results — rejection sampling never walks the hinge, and this phase measures Nautilus inert at f1e8. + The "1e8 rejected" finding is about **fixed-step gradient MAP**. Different engines; no conflict. **Design note for Phase 5 (recommendation, not yet ruled):** gradient search on a mesh with positions on should run at factor **1e5** (threshold per the cell's convention), not the reference rows' f1e8, or the transit damage measured here on MGE returns. + Whether MGE's 1e5 transfers to a mesh likelihood scale is unmeasured and is Phase 5's first check. Laptop action: PROGRAMME.md "Gates A–F" row — GATE B pt 2 CALLED (re-ratified 2026-09-02, R-20260902-10); the runs stay in `output/legacy/` as cited, reusable evidence. +- 2026-09-02 — result — accepted phase3_prodigy_reliability_gate_b1_reuse (R-20260902-09): **phase3-prodigy-reliability-gate-b1-reuse — ACCEPTED; + GATE B part 1 moves from PROVISIONAL back to CALLED.** Human, 2026-09-02, verbatim: *"continue"* — ratifying the packet recommendation (accept). **What stands (recomputed from the committed rows):** Prodigy(n=256, prior_box, autoconv, positions-off) is a reliable global MAP searcher on MGE — n256 **15/15** cumulative (wave 1 5/5 + fresh seeds 10/10), every max logL in [31787.8807, 31787.9175], +1.1 nats above the Nautilus bar with no impostor basin; + Wilson-95 lower bound **0.796**; cost **3.1–4.3×** under the same-night Nautilus sampler wall (3.4–4.5× against the recorded 772.7 s); auto-convergence never hit the 3000-step ceiling. Caveat (a) **n=256 only** stands: n16 is 1/5, n64 5/5 with Wilson-95 lower 0.57. + **Not re-verified:** the lane-level layer — p̂_hit = 61/1280 = 0.048 [0.037, 0.061], tier overlap, pinned-lane fraction (~half), the `ell_comps` scan — is read from RESULTS.md and the adversarial review; only run-level success was recomputed. + The gate rests on run-level reliability, which was. Laptop action: PROGRAMME.md "Gates A–F" row — GATE B pt 1 CALLED (re-ratified 2026-09-02, R-20260902-09); the runs stay in `output/legacy/` as cited, reusable evidence. +- 2026-09-02 — result — accepted phase2_nss_mainline_gate_a_reuse (R-20260902-08): **phase2-nss-mainline-gate-a-reuse — ACCEPTED; GATE A moves from PROVISIONAL back to CALLED on the MGE evidence, with the Delaunay sentence struck.** Human, 2026-09-02, verbatim: *"continue"* — ratifying the packet recommendation (accept, one sentence struck) after the three members were presented with their recomputed witness numbers. + **What stands (recomputed from the committed rows, version `2026.8.17.1`):** H2.1 closed — the logZ bias is inner-kernel under-mixing (+8.36 / +0.71 / −0.46 nats at inner 5 / 30 / 45); operating point n200 / nd100 / inner30; + 5/5 seeds at **+1.02 ± 0.39 nats** above the Nautilus bar 31690.4965, max logL within 0.55 nats of truth every time; wall **4.99×** Nautilus on MGE; sample economy Kish ESS 4,121 vs 1,315 (~15 vs ~940 evals per effective sample). + Nautilus remains the nested baseline on every model family; `af.NSS` stays mainlined as a correct, tuned alternative, not default. **What is struck:** the "18.4× on Delaunay" leg (339069 ÷ 339071). Both runs are in `legacy_wrong`; + 339071 is the run R-20260901-03 dropped. Under R-20260901-03 and R-20260902-01 neither is citable. The per-family NSS-on-mesh cost is therefore **unmeasured** until an NSS positions-on mesh run exists; the gate does not rest on it. + **Not re-derived:** the Kish ESS figures are read from the write-up (eval counts and walls were verified from the rows). Recorded, not a blocker. Laptop action: PROGRAMME.md "Gates A–F" row — GATE A CALLED (re-ratified 2026-09-02, R-20260902-08), Delaunay leg struck; the twelve MGE runs stay in `output/legacy/` as cited, reusable evidence. +- 2026-09-02 — result — accepted inference_refs_v1_redo (R-20260902-01): **inference-refs-v1-redo — ACCEPTED, scoped to the four positions-on rows.** Human, verbatim: *"pos_tauto0.2_f1e8 gives a physical solution which is acceptable. + All folders with this name do, confirming that for a mesh source this is required to get a physical source and that gradient profiling can be done robustly. If a fit uses a mesh and does not have a positions.info file, it should be assumed its result is unreliable and it cannot be used."* **Binding rule (generalises R-20260901-03):** a mesh / pixelization run dir without a `positions.info` file is unreliable and cannot be used or cited — in `output/searches/` or anywhere else. + The active tree is to hold no such run. `pos_tauto0.2_f1e8` is the confirmed physical configuration for mesh sources; the fixed-threshold `pos_t0.3_f1e8` row (t2) also carries `positions.info` and is accepted under the same rule. + **Delivery.** All 10 tasks of 342091 delivered on the witness's own terms, checked from the mirror artefacts: sampler walls 0:35–1:23 (10–23 % of the 6:00 budget); every `.err` is the 222 B mask-padding UserWarning and nothing else; every `.out` ends "Finished." and every run dir carries `.completed`; zero "Fit Already Completed"; no overflow signature; the maxLL / logZ table is bit-identical to `model.results` / `files/samples_info.json` in all ten run dirs. + **One leg is unverified from the laptop:** the result-row leg (version `2026.8.17.1`, matching `target_id`). The ten rows were written to RAL `results/searches/nautilus/imaging/…` and have not been pulled or committed — `origin/main` still carries the 2026-08-26 rows (stale `priors_ref bf2c8742c334`) and the jump host is throttled. + Accepted on the artefact evidence; the rows are pulled and checked on the next RAL contact before any is adopted into `InferenceRefs_v1`. **Rows.** - **Citable** (`PositionsLH` on, `positions.info` present): t2 `delaunay_nn` pos-0.3 (maxLL 31,351.39 / logZ 31,275.23); t4 `slam_source_pix` pos-auto (31,547.24 / 31,452.10); t6 `slam_source_pix_nn` pos-auto (31,405.63 / 31,324.84); t10 `delaunay` pos-auto (31,338.43 / 31,264.83). + These four are the `InferenceRefs_v1` references for `delaunay_nn_pos_fp64`, `slam_source_pix_pos_fp64`, `slam_source_pix_nn_pos_fp64` and `delaunay_pos_fp64`. - **Spent** (positions off, no `positions.info`): t0 `pixelization`, t1 `delaunay_nn`, t3 `slam_source_pix`, t5 `slam_source_pix_nn`, t7 `knn`, t8 `delaunay_matern`. + Not citable; their run dirs move to `output/legacy_wrong/` on RAL and the mirror. The wave corroborates R-03 on its own: in every paired cell the positions-on run sits 520–700 nats **above** its positions-off twin (t2−t1 +700.6, t4−t3 +520.6, t6−t5 +654.6). + A penalty cannot raise a likelihood, so the positions-off searches ended in a lower basin — the demagnified solution. **Consequences.** - The positions-off mesh reference design is retired: SUBMIT_LIST rows 0/1/3/5/7/8 are struck. + `pixelization_fp64`, `knn_fp64` and `delaunay_matern_fp64` become positions-on targets and need a fresh wave — a new phase, not a rerun of this one. MGE rows are unaffected. - The stale-knn flag stays CLOSED as a delivery matter, but the "~494 nats below the Prodigy arm bar 30,557.03" reading is withdrawn: that bar is a Phase 8B arm in `legacy_wrong` and cannot serve as a comparison. knn's standing is judged on its positions-on rerun. - t4 at +25.50 above the truth likelihood is noted, not a finding — a pixelized source has more freedom than the truth model. - Laptop actions (human-directed 2026-09-02): pull and commit the ten result rows; + `mv` the six positions-off run dirs to `legacy_wrong` (RAL **and** mirror), and sweep the active tree for any other mesh run without `positions.info`; adopt the four citable rows into `results/baselines/InferenceRefs_v1/<key>/` and update `INDEX.json`/`INDEX.md`/`SUBMIT_LIST.md`. +- 2026-09-02 — run — 342241_[11-13] submitted (done, wall 0:55): refs_v1_positions_on_completion — three tasks serial on euclid-ral-gpu-2, 20:59 BST 2 Sep → 00:50 BST 3 Sep; wall shown is the longest single task (knn, sampler 3,257 s); sbatch --array=11-13 --requeue from + hpc/batch_gpu on RAL, 2026-09-02 ~16:50Z; pending (Resources) at submit pulled_to: output/searches/nautilus/imaging +- 2026-09-02 — note — question: Do positions-on replacements give clean references for three mesh cells [archive/tasks/inference_programme/refs_v1_positions_on_completion.md] +- 2026-09-02 — note — question: Does Prodigy reach the physical basin on mesh cells positions-on [archive/tasks/inference_programme/phase5_mesh_gradient_positions_on.md] +- 2026-09-02 — note — question: Does the legacy MGE PositionsLH evidence behind GATE B2 stand [archive/tasks/inference_programme/phase4_positions_gate_b2_reuse.md] +- 2026-09-02 — note — question: Does the legacy MGE Prodigy-reliability evidence behind GATE B1 stand [archive/tasks/inference_programme/phase3_prodigy_reliability_gate_b1_reuse.md] +- 2026-09-02 — note — question: Does the legacy MGE evidence behind GATE A stand [archive/tasks/inference_programme/phase2_nss_mainline_gate_a_reuse.md] +- 2026-09-01 — note — dropped failed_submissions_342008_10 (R-20260901-04): Carried, then superseded by the rewind. The three submissions were the one member of the 2026-08-31-am packet the human left untouched (`UNREVIEWED, carried to next packet`), and they were carried again past the pm slot. + There was never anything to rule on: each job died in about a second under the CWD trap, wrote no run directory and left no `.out` log on the mirror, so there is no evidence to accept and nothing to reuse. + The question they were dispatched to answer is in any case one the 2026-08-31 REWIND retired — the programme restarts at Phase 1. Dropped for the record, so the ids stop being carried forward. Revival is a new phase number. +- 2026-09-01 — note — dropped delaunay_fp64_retro_baseline (R-20260901-03): **delaunay-fp64-retro-baseline — REJECTED, with a programme-wide finding.** The human inspected `output/legacy_wrong/searches/nautilus/imaging/delaunay/hst/hpc_a100_fp64/` and identified the solution as a **demagnified-source unphysical solution** — the classic Inversion bias `PositionsLH` exists to remove (https://pyautolens.readthedocs.io/en/latest/general/demagnified_solutions.html). + Human, verbatim: *"This is why we basically need to rerun loads of JAX gradient runs (MGE sources are not effect by this so safe) wit the position penalty set up and used. The problem is it is not used in all these JAX Gradient runs!"* **Binding consequence for the redo (from the rejection):** every mesh / pixelization run — the refs array's mesh rows included — must carry a `PositionsLH` positions penalty (positions + threshold per the workspace convention, e.g. + 0.3"). MGE-source rows are unaffected and stay valid. The human also flagged that the "Inference Baselines v1" summary carries the same unphysical-solution issue. Mesh rows produced without the penalty are not citable as references. + The 342091 wave's mesh rows should be checked against this before certification — a mesh row without the penalty is spent evidence. **Output-tree consequence (human directive, verbatim intent):** clear the RAL active `output/` of all mesh results, the `image_plane`/`source_plane` point source fits, the `Cluster` folder and the `point_source` folder — moved (not deleted) to a new `output/legacy_point/` folder, so the active tree is clean and updates with each task. + Laptop action, queued as a local-dev prompt in the Mind. +- 2026-09-01 — result — accepted mge_fp64_retro_baseline (R-20260901-02): **mge-fp64-retro-baseline — ACCEPTED.** The retro-adopted `mge_fp64` baseline stands under the redo standard; no fresh `_ref` run needed. +- 2026-09-01 — result — accepted mge_pos_ref_reuse (R-20260901-01): **mge-pos-ref-reuse — ACCEPTED.** The quarantined-but-reusable `340210_9` run (2026-08-25) is adopted as the `mge_pos_fp64` InferenceRefs_v1 reference. + Per the MGE reuse rule: cite it and `mv` it from `output/legacy/searches/…` back into the active tree (RAL **and** mirror) — laptop action, queued for the next local-dev slot. Array task 9 stays excluded from the refs array. +- 2026-09-01 — note — question: Can spent point-source families be swept out of active output [archive/tasks/inference_programme/legacy_point_output_sweep.md] +- 2026-08-31 — note — dropped slogdet_ab_342017 (R-20260831-05): I have updated the JAX gradient epic to do alot of phases back from the start, so wee will basically discard this and start again. Dont change anything just let that recent intake that precendent. +- 2026-08-31 — note — dropped refs_5_6_342016_56 (R-20260831-04): I have updated the JAX gradient epic to do alot of phases back from the start, so wee will basically discard this and start again. Dont change anything just let that recent intake that precendent. +- 2026-08-31 — note — dropped knn_rerun_342016_7 (R-20260831-03): I have updated the JAX gradient epic to do alot of phases back from the start, so wee will basically discard this and start again. Dont change anything just let that recent intake that precendent. +- 2026-08-31 — note — dropped smc_probe_342018 (R-20260831-02): I have updated the JAX gradient epic to do alot of phases back from the start, so wee will basically discard this and start again. Dont change anything just let that recent intake that precendent. +- 2026-08-31 — note — dropped rewind_2026_08_31 (R-20260831-01): **Decision (human, 2026-08-31):** the inference programme **rewinds**. The epic ran largely without human review, and every mesh / pixelization result it produced goes to an unphysical solution. + This ledger already carries the three symptoms: the λ⁴ non-PD **overflow flood** (the regularization matrix goes non-PD from c~1e4, fp64 Cholesky returns finite garbage, and Nautilus accepts `log_l` up to **3e+303** as a best point — the 341908_5 entry); + **59 %** of Phase-8B best points (23 of 39) clustered at the **|e| = 1.41421** box corner of the pixelized cells' box-clipped `ell_comps` geometry (W10); and a `cond ~ 1e15–1e19` NaN population where the reconstruction solve NaNs despite finite log-det terms (CP-4/W7). + Beyond the failures, the human had lost track of what the programme had actually established. The rewind quarantines the bad evidence and restarts the review discipline. **What moved where (executed 2026-08-31, identically on RAL `/mnt/ral/jnightin/autolens_profiling/output/` and the laptop mirror `/mnt/c/Users/Jammy/Science/inference_programme/output/`, directory structure preserved):** | Tree | Contents | Run-config dirs | |---|---|---| | `output/legacy/searches/…` | **MGE** — reusable, pending batch review | **83**: nautilus 16, nss 13, nuts 2, smc 3, multi_start_prodigy 4, multi_start_prodigy_autoconv 45 | | `output/legacy_wrong/searches/…` | **mesh / pixelization** — unphysical, quarantined | **71**: nautilus 21 (delaunay, delaunay_matern, delaunay_nn, knn, pixelization, slam_source_pix, slam_source_pix_nn), nss 3, multi_start_prodigy 47 (delaunay_adapt_split, knn); plus RAL-only `slogdet_ab_harvest` and `_void_341874_24` | The active `output/searches/` now holds only the **point_source** and **cluster** trees. + `hpc/sync pull` scopes to `output/searches` and passes no `--delete`, so it can never reach into `legacy*/`. **Restart point: Phase 1 — the `InferenceRefs_v1` reference baselines**, the first phase whose design does RAL runs. + Phase 0 outcomes stand, and **all shipped source and registry code stands** — the PositionsLH fix (PyAutoLens#699/#700), blackjax 1.6.2 on cloud/local/RAL, `_targets.py`, benchmark schema v2, `af.SMC`, the NUTS plumbing. + Nothing is reverted. Only **run evidence** and **phase/gate state** rewind. Re-execution is human-driven and step-by-step under the batch-and-review workflow: human-declared `review-at:` slots, review packets at `PyAutoMind/batches/packets/` (spec `batches/packets/TEMPLATE.md`), with the laptop mirror as the review surface. + **Gates → PROVISIONAL.** Each call below stands as a *working assumption* and may continue to be used as one, but the legacy MGE evidence beneath it must be surfaced in batch review packets and ruled on by a human before the gate is re-confirmed: - **Gate A** (2026-08-24 — Nautilus stays the nested baseline; + `af.NSS` is the tuned alternative) → **provisional**. Re-confirmation needs the Phase-2 inner-steps / logZ-bias scan reviewed in packets; note its Delaunay comparison re-baselines now sit in `legacy_wrong` and cannot be cited. - **Gate B pt 1** (2026-08-23 — Prodigy n=256, prior_box, autoconv, positions-off ratified as global MAP searcher on MGE) → **provisional**. + Re-confirmation needs the Phase-3 reliability wave (p̂_hit, fresh-seed tiers) reviewed in packets. - **Gate B pt 2** (2026-08-27 — PositionsLH safe for gradient MAP at factor ≤ 1e5 on MGE, 1e8 rejected) → **provisional**. + Re-confirmation needs the Phase-4 Stage 2/3 arms reviewed in packets, with the six caveats that ride with the call restated rather than assumed. Gates C–F remain open and unstarted. **MGE reuse rule (binding for the redo).** Before submitting any MGE run during the redo, check `output/legacy/searches/…` (RAL **and** the mirror) for an existing result at the same target/config. + If one is present and has not yet been ruled on in a batch review packet, **do not resubmit** — surface it as a packet member (question → witness → health evidence → readout → where-to-look) for a human ruling. + Accepted → cite it and `mv` it back into the active tree; rejected → queue a fresh run. `legacy_wrong` results are **never** reused or cited except as failure-mode documentation. **Resume-trap note.** The quarantine doubles as the mandated resume-trap relocation. + With the old run directories off the active path, a redo run launched with identical knobs starts fresh instead of silently re-emitting the old fit — reuse means *reading from* `legacy/`, never resuming in place. + **Records:** autolens_profiling#200 (queue-of-record anchor for the redo — redo slices ship as autolens_profiling issues/PRs per the epics rule); `PROGRAMME.md` §"2026-08-31 REWIND" + the annotated phase/gate table; + `PyAutoMind/epics.md` (JAX profiling epic notes). --- Follow-up filed from the same finding (2026-09-01): the 59 % Phase-8B clustering at the |e| = 1.41421 box corner of the pixelized cells' box-clipped `ell_comps` geometry is PyAutoGalaxy#594 — the joint disk constraint on `ell_comps`, with the 8B crash channel PyAutoFit#1535 beside it. +- 2026-08-31 — note — question: Should the programme rewind, quarantine its runs and restart [archive/tasks/inference_programme/rewind_2026_08_31.md] +- 2026-08-31 — run — 342091_[0-8,10] submitted (legacy): inference_refs_v1_redo — pre-Cortex run, migrated; 10 tasks run serially on one A100, 22:00:39 BST 31 Aug → 08:08:54 BST 1 Sep (10 h 08 m end to end); wall shown is the longest single task. + Task 9 (mge_pos) held back under the MGE reuse rule — see phase 7 where: output/searches/nautilus/imaging pulled_to: output/searches/nautilus/imaging +- 2026-08-31 — note — question: Does the refs array deliver clean reference rows after rewind [archive/tasks/inference_programme/inference_refs_v1_redo.md] +- 2026-08-30 — run — 342018_2 submitted (legacy): smc_probe_342018 — pre-Cortex run, migrated where: output/legacy/searches/smc/imaging/mge/hst/n256_mala_m5_seed0_cold/hpc_a100_fp64_n256_mala_cold/1e3dea2f7a4d385eac93f5bbc80de7cf +- 2026-08-30 — run — 342018_1 submitted (legacy): smc_probe_342018 — pre-Cortex run, migrated where: output/legacy/searches/smc/imaging/mge/hst/n256_hmc_m5_i8_seed0_warmebe2e28d_massprior_scaled/hpc_a100_fp64_n256_hmc_warm/3b4ab5e2d6ff3b453680eee171bb6f4b +- 2026-08-30 — run — 342018_0 submitted (legacy): smc_probe_342018 — pre-Cortex run, migrated where: output/legacy/searches/smc/imaging/mge/hst/n256_mala_m5_seed0_warmebe2e28d_massprior_scaled/hpc_a100_fp64_n256_mala_warm/7fe1b8d8baee5edc4a5c485a8d48b45b +- 2026-08-30 — note — question: Does the SMC arm reach the Nautilus bar on MGE [archive/tasks/inference_programme/smc_probe_342018.md] +- 2026-08-30 — run — 342017_1 submitted (legacy, wrong settings): slogdet_ab_342017 — pre-Cortex run, migrated where: output/legacy_wrong/searches/nautilus/imaging/slam_source_pix_nn/hst/ld_slogdet/hpc_a100_fp64_slogdet_ab_slogdet/7eb80135de1db2112c5bb5383b32bc49 +- 2026-08-30 — run — 342017_0 submitted (legacy, wrong settings): slogdet_ab_342017 — pre-Cortex run, migrated where: output/legacy_wrong/searches/nautilus/imaging/slam_source_pix_nn/hst/ld_cholesky/hpc_a100_fp64_slogdet_ab_cholesky/ae454f54e14847ca62929c70033c11a5 +- 2026-08-30 — note — question: Does slogdet change the fit versus Cholesky on slam_source_pix_nn [archive/tasks/inference_programme/slogdet_ab_342017.md] +- 2026-08-30 — run — 342016_6 submitted (legacy, wrong settings): refs_5_6_342016_56 — pre-Cortex run, migrated where: output/legacy_wrong/searches/nautilus/imaging/slam_source_pix_nn/hst/pos_tauto0.2_f1e8/hpc_a100_fp64_ref_pos_tauto0.2_f1e8/35b83ae5f6eab98575809e7c5dc5f3af +- 2026-08-30 — note — question: Do slam_source_pix_nn rows 5 and 6 certify as references [archive/tasks/inference_programme/refs_5_6_342016_56.md] +- 2026-08-30 — run — 342016_7 submitted (legacy, wrong settings): knn_rerun_342016_7 — pre-Cortex run, migrated where: output/legacy_wrong/searches/nautilus/imaging/knn/hst/hpc_a100_fp64_ref/39b5ccc228ad4b4389b1129a7e5cf605 +- 2026-08-30 — note — question: Does the capped knn re-run close its evidence deficit [archive/tasks/inference_programme/knn_rerun_342016_7.md] +- 2026-08-29 — run — 342016_5 submitted (legacy, wrong settings): refs_5_6_342016_56 — pre-Cortex run, migrated where: output/legacy_wrong/searches/nautilus/imaging/slam_source_pix_nn/hst/hpc_a100_fp64_ref/7ced3989425354c2a53d4a79d7da2b11 +- 2026-08-29 — run — 342010 submitted (legacy, wrong settings): failed_submissions_342008_10 — pre-Cortex run, migrated; died in ~1 s under the CWD trap — no run directory, no output, no .out log on the mirror where: output/legacy_wrong/ +- 2026-08-29 — run — 342009 submitted (legacy, wrong settings): failed_submissions_342008_10 — pre-Cortex run, migrated; died in ~1 s under the CWD trap — no run directory, no output, no .out log on the mirror where: output/legacy_wrong/ +- 2026-08-29 — run — 342008 submitted (legacy, wrong settings): failed_submissions_342008_10 — pre-Cortex run, migrated; died in ~1 s under the CWD trap — no run directory, no output, no .out log on the mirror where: output/legacy_wrong/ +- 2026-08-29 — note — question: Why did three submissions die instantly and is anything recoverable [archive/tasks/inference_programme/failed_submissions_342008_10.md] +- 2026-08-28 — run — 341874_24 submitted (legacy, wrong settings): rewind_2026_08_31 — pre-Cortex run, migrated; the RAL-side copy is quarantined as _void_341874_24 where: output/legacy_wrong/searches/multi_start_prodigy/imaging/knn/hst/n16_s3000_seed4/knn_auto_none_seed4 +- 2026-08-27 — run — 341908_5 submitted (legacy, wrong settings): rewind_2026_08_31 — pre-Cortex run, migrated; killed by the likelihood-overflow flood — checkpoint.hdf5 is the only artifact it left where: + output/legacy_wrong/searches/nautilus/imaging/slam_source_pix_nn/hst/hpc_a100_fp64_ref/c16ff7b02fc168edd8eb9b4aaa4f4bc7 +- 2026-08-27 — run — 341892_[5-9] submitted (legacy): phase4_positions_gate_b2_reuse — Stage 3 arm B, t0.3 f1e5: the 5/5 arm the call rests on (longest task 297.4 s) where: + output/legacy/searches/multi_start_prodigy_autoconv/imaging/mge/hst/n256_s3000_seed{0-4}_pos_t0.3_f1e5/hpc_a100_fp64_n256_seed{0-4}_pos_t0.3_f1e5 pulled_to: output/legacy/searches/multi_start_prodigy_autoconv/imaging/mge/hst/n256_s3000_seed{0-4}_pos_t0.3_f1e5/hpc_a100_fp64_n256_seed{0-4}_pos_t0.3_f1e5 +- 2026-08-27 — run — 341892_[0-4] submitted (legacy): phase4_positions_gate_b2_reuse — Stage 3 arm A, tauto0.2 f1e8: 0/4, task 0 INVALID (silent resume, 68 s) + (longest valid task 336.4 s) where: output/legacy/searches/multi_start_prodigy_autoconv/imaging/mge/hst/n256_s3000_seed{0-4}_pos_tauto0.2_f1e8/hpc_a100_fp64_n256_seed{0-4}_pos_tauto0.2_f1e8 pulled_to: output/legacy/searches/multi_start_prodigy_autoconv/imaging/mge/hst/n256_s3000_seed{0-4}_pos_tauto0.2_f1e8/hpc_a100_fp64_n256_seed{0-4}_pos_tauto0.2_f1e8 +- 2026-08-25 — run — 340115_[0-4] submitted (legacy): phase4_positions_gate_b2_reuse — Stage 2 Prodigy n256, positions ON t0.3 f1e8: the 2/5 arm (longest task 553.4 s) where: + output/legacy/searches/multi_start_prodigy_autoconv/imaging/mge/hst/n256_s3000_seed{0-4}_pos_t0.3_f1e8/hpc_a100_fp64_n256_seed{0-4}_pos_t0.3_f1e8 pulled_to: output/legacy/searches/multi_start_prodigy_autoconv/imaging/mge/hst/n256_s3000_seed{0-4}_pos_t0.3_f1e8/hpc_a100_fp64_n256_seed{0-4}_pos_t0.3_f1e8 +- 2026-08-25 — run — 340114_[5-9] submitted (legacy): phase4_positions_gate_b2_reuse — Stage 2 Nautilus, positions ON t0.3 f1e8, seeds 0-4 (longest task 810.4 s) where: output/legacy/searches/nautilus/imaging/mge/hst/pos_t0.3_f1e8/hpc_a100_fp64_seed{0-4}_pos_t0.3_f1e8 pulled_to: + output/legacy/searches/nautilus/imaging/mge/hst/pos_t0.3_f1e8/hpc_a100_fp64_seed{0-4}_pos_t0.3_f1e8 +- 2026-08-25 — run — 340114_[0-4] submitted (legacy): phase4_positions_gate_b2_reuse — Stage 2 Nautilus, positions OFF, seeds 0-4 (longest task 773.5 s) where: output/legacy/searches/nautilus/imaging/mge/hst/hpc_a100_fp64_seed{0-4} pulled_to: + output/legacy/searches/nautilus/imaging/mge/hst/hpc_a100_fp64_seed{0-4} +- 2026-08-25 — run — 340210_9 submitted (legacy): mge_pos_ref_reuse — pre-Cortex run, migrated where: output/legacy/searches/nautilus/imaging/mge/hst/pos_tauto0.2_f1e8/hpc_a100_fp64_ref_pos_tauto0.2_f1e8/dc42087fb0524e78fd43eced7706b365 pulled_to: + output/legacy/searches/nautilus/imaging/mge/hst/pos_tauto0.2_f1e8/hpc_a100_fp64_ref_pos_tauto0.2_f1e8/dc42087fb0524e78fd43eced7706b365 +- 2026-08-25 — note — question: Is legacy run 340210_9 adopted as the mge_pos_fp64 reference [archive/tasks/inference_programme/mge_pos_ref_reuse.md] +- 2026-08-24 — run — 339070 submitted (legacy): phase3_prodigy_reliability_gate_b1_reuse — Nautilus MGE re-baseline; the wall denominator that retired the 523 s figure, already accepted where: + output/legacy/searches/nautilus/imaging/mge/hst/hpc_a100_fp64/181b13114ba3c2298191185ff74f90d8 pulled_to: output/legacy/searches/nautilus/imaging/mge/hst/hpc_a100_fp64/181b13114ba3c2298191185ff74f90d8 ruled: R-20260901-02 +- 2026-08-24 — run — 339071 submitted (legacy, wrong settings): phase2_nss_mainline_gate_a_reuse — Nautilus Delaunay re-baseline; the 18.4x denominator, dropped as a demagnified-source solution where: + output/legacy_wrong/searches/nautilus/imaging/delaunay/hst/hpc_a100_fp64/b29ffe390c18e070b3eaba60270cb502 ruled: R-20260901-03 +- 2026-08-24 — run — 339069 submitted (legacy, wrong settings): phase2_nss_mainline_gate_a_reuse — NSS mainline on imaging/delaunay/hst; the 18.4x numerator, quarantined mesh where: + output/legacy_wrong/searches/nss/imaging/delaunay/hst/hpc_a100_fp64_mainline/458f07dbe90f54a13492930e923e20a5 +- 2026-08-24 — run — 339070 submitted (legacy): phase2_nss_mainline_gate_a_reuse — Nautilus MGE re-baseline; the denominator of the 5.0x ratio, already accepted where: output/legacy/searches/nautilus/imaging/mge/hst/hpc_a100_fp64/181b13114ba3c2298191185ff74f90d8 pulled_to: + output/legacy/searches/nautilus/imaging/mge/hst/hpc_a100_fp64/181b13114ba3c2298191185ff74f90d8 ruled: R-20260901-02 +- 2026-08-24 — run — 339068 submitted (legacy): phase2_nss_mainline_gate_a_reuse — termination row, dlogz -10 where: output/legacy/searches/nss/imaging/mge/hst/hpc_a100_fp64_inner30_nd100_dlogz10/49ffc0b1a0f54213efc58055c0da5907 pulled_to: + output/legacy/searches/nss/imaging/mge/hst/hpc_a100_fp64_inner30_nd100_dlogz10/49ffc0b1a0f54213efc58055c0da5907 +- 2026-08-24 — run — 339070 submitted (legacy): mge_fp64_retro_baseline — pre-Cortex run, migrated where: output/legacy/searches/nautilus/imaging/mge/hst/hpc_a100_fp64/181b13114ba3c2298191185ff74f90d8 pulled_to: + output/legacy/searches/nautilus/imaging/mge/hst/hpc_a100_fp64/181b13114ba3c2298191185ff74f90d8 +- 2026-08-24 — note — question: Does the retro-adopted mge_fp64 reference baseline still stand [archive/tasks/inference_programme/mge_fp64_retro_baseline.md] +- 2026-08-24 — run — 339071 submitted (legacy, wrong settings): delaunay_fp64_retro_baseline — pre-Cortex run, migrated where: output/legacy_wrong/searches/nautilus/imaging/delaunay/hst/hpc_a100_fp64/b29ffe390c18e070b3eaba60270cb502 +- 2026-08-24 — note — question: Does the retro-adopted delaunay_fp64 reference baseline still stand [archive/tasks/inference_programme/delaunay_fp64_retro_baseline.md] +- 2026-08-23 — run — 338525_[0-4] submitted (legacy): phase4_positions_gate_b2_reuse — the CP-3 positions-OFF n256 comparator the Stage 3 table is scored against; also phase 14's tier where: + output/legacy/searches/multi_start_prodigy_autoconv/imaging/mge/hst/n256_s3000_seed{0-4}/hpc_a100_fp64_n256_seed{0-4} pulled_to: output/legacy/searches/multi_start_prodigy_autoconv/imaging/mge/hst/n256_s3000_seed{0-4}/hpc_a100_fp64_n256_seed{0-4} +- 2026-08-23 — run — 339066_[0-9] submitted (legacy): phase3_prodigy_reliability_gate_b1_reuse — fresh-seed tier, n256, seeds 105-114: the 10 that make 15/15 (longest task 184.1 s) where: + output/legacy/searches/multi_start_prodigy_autoconv/imaging/mge/hst/n256_s3000_seed{105-114}/hpc_a100_fp64_n256_seed{105-114} pulled_to: output/legacy/searches/multi_start_prodigy_autoconv/imaging/mge/hst/n256_s3000_seed{105-114}/hpc_a100_fp64_n256_seed{105-114} +- 2026-08-23 — run — 339065_[0-4] submitted (legacy): phase3_prodigy_reliability_gate_b1_reuse — fresh-seed tier, n128, seeds 100-104 (longest task 198.3 s) where: + output/legacy/searches/multi_start_prodigy_autoconv/imaging/mge/hst/n128_s3000_seed{100-104}/hpc_a100_fp64_n128_seed{100-104} pulled_to: output/legacy/searches/multi_start_prodigy_autoconv/imaging/mge/hst/n128_s3000_seed{100-104}/hpc_a100_fp64_n128_seed{100-104} +- 2026-08-23 — run — 338526_[0-4] submitted (legacy): phase3_prodigy_reliability_gate_b1_reuse — the theta_E ~ U(0.2,8) diagnostic arm, n16, seeds 0-4 (longest task 284.0 s) where: + output/legacy/searches/multi_start_prodigy_autoconv/imaging/mge/hst/n16_s3000_seed{0-4}/hpc_a100_fp64_n16_seed{0-4}_diag_theta_e pulled_to: output/legacy/searches/multi_start_prodigy_autoconv/imaging/mge/hst/n16_s3000_seed{0-4}/hpc_a100_fp64_n16_seed{0-4}_diag_theta_e +- 2026-08-23 — run — 338525_[0-4] submitted (legacy): phase3_prodigy_reliability_gate_b1_reuse — CP-3 wave 1, n_starts 256, seeds 0-4: the tier the gate was called at (longest task 224.8 s) where: + output/legacy/searches/multi_start_prodigy_autoconv/imaging/mge/hst/n256_s3000_seed{0-4}/hpc_a100_fp64_n256_seed{0-4} pulled_to: output/legacy/searches/multi_start_prodigy_autoconv/imaging/mge/hst/n256_s3000_seed{0-4}/hpc_a100_fp64_n256_seed{0-4} +- 2026-08-23 — run — 338524_[0-4] submitted (legacy): phase3_prodigy_reliability_gate_b1_reuse — CP-3 wave 1, n_starts 64, seeds 0-4 (longest task 280.0 s) where: output/legacy/searches/multi_start_prodigy_autoconv/imaging/mge/hst/n64_s3000_seed{0-4}/hpc_a100_fp64_n64_seed{0-4} + pulled_to: output/legacy/searches/multi_start_prodigy_autoconv/imaging/mge/hst/n64_s3000_seed{0-4}/hpc_a100_fp64_n64_seed{0-4} +- 2026-08-23 — run — 338523_[0-4] submitted (legacy): phase3_prodigy_reliability_gate_b1_reuse — CP-3 wave 1, n_starts 16, seeds 0-4 (longest task 245.8 s) where: output/legacy/searches/multi_start_prodigy_autoconv/imaging/mge/hst/n16_s3000_seed{0-4}/hpc_a100_fp64_n16_seed{0-4} + pulled_to: output/legacy/searches/multi_start_prodigy_autoconv/imaging/mge/hst/n16_s3000_seed{0-4}/hpc_a100_fp64_n16_seed{0-4} +- 2026-08-23 — run — 339067_[0-3] submitted (legacy): phase2_nss_mainline_gate_a_reuse — seeds 43-46 at the operating point (longest task) where: output/legacy/searches/nss/imaging/mge/hst/hpc_a100_fp64_inner30_nd100_seed{43,44,45,46} pulled_to: + output/legacy/searches/nss/imaging/mge/hst/hpc_a100_fp64_inner30_nd100_seed{43,44,45,46} +- 2026-08-23 — run — 338873 submitted (legacy): phase2_nss_mainline_gate_a_reuse — wave 2, n_live 1000 / nd250 where: output/legacy/searches/nss/imaging/mge/hst/hpc_a100_fp64_inner30_n1000/583b9edbb652eddd72d75e659726243b pulled_to: + output/legacy/searches/nss/imaging/mge/hst/hpc_a100_fp64_inner30_n1000/583b9edbb652eddd72d75e659726243b +- 2026-08-23 — run — 338872 submitted (legacy): phase2_nss_mainline_gate_a_reuse — wave 2, n_live 500 / nd125 where: output/legacy/searches/nss/imaging/mge/hst/hpc_a100_fp64_inner30_n500/f918b17003b3a4c547dc99c761dcb3aa pulled_to: + output/legacy/searches/nss/imaging/mge/hst/hpc_a100_fp64_inner30_n500/f918b17003b3a4c547dc99c761dcb3aa +- 2026-08-23 — run — 338871 submitted (legacy): phase2_nss_mainline_gate_a_reuse — wave 2, nd20 where: output/legacy/searches/nss/imaging/mge/hst/hpc_a100_fp64_inner30_nd20/f34e1b347cfb871be4f7d3ca0e4e7e50 pulled_to: + output/legacy/searches/nss/imaging/mge/hst/hpc_a100_fp64_inner30_nd20/f34e1b347cfb871be4f7d3ca0e4e7e50 +- 2026-08-23 — run — 338870 submitted (legacy): phase2_nss_mainline_gate_a_reuse — wave 2, nd100: the operating point where: output/legacy/searches/nss/imaging/mge/hst/hpc_a100_fp64_inner30_nd100/7d818ae1c5d5e96eb2e940229b411b4e pulled_to: + output/legacy/searches/nss/imaging/mge/hst/hpc_a100_fp64_inner30_nd100/7d818ae1c5d5e96eb2e940229b411b4e +- 2026-08-23 — run — 338493 submitted (legacy): phase2_nss_mainline_gate_a_reuse — wave 1, inner 45 (=3d) where: output/legacy/searches/nss/imaging/mge/hst/hpc_a100_fp64_inner45/0f93cae07b4b2e67f52cfd285c5e87a3 pulled_to: + output/legacy/searches/nss/imaging/mge/hst/hpc_a100_fp64_inner45/0f93cae07b4b2e67f52cfd285c5e87a3 +- 2026-08-23 — run — 338492 submitted (legacy): phase2_nss_mainline_gate_a_reuse — wave 1, inner 30 (=2d) where: output/legacy/searches/nss/imaging/mge/hst/hpc_a100_fp64_inner30/a3d0599d72a83731a0b3a88feb3f959e pulled_to: + output/legacy/searches/nss/imaging/mge/hst/hpc_a100_fp64_inner30/a3d0599d72a83731a0b3a88feb3f959e +- 2026-08-23 — run — 338491 submitted (legacy): phase2_nss_mainline_gate_a_reuse — NSS anchor at fork knobs (inner 5); node start 16:16 BST where: output/legacy/searches/nss/imaging/mge/hst/hpc_a100_fp64/b51405821b58b0c5b0d03cb2790446fa pulled_to: + output/legacy/searches/nss/imaging/mge/hst/hpc_a100_fp64/b51405821b58b0c5b0d03cb2790446fa +- 2026-08-19 — note — question: Cluster extended source inference testing gradient based fitting [archive/tasks/inference_programme/cluster_extended_source_inference.md] +- 2026-07-31 — note — question: Does Prodigy beat Nautilus at cluster scale for point sources [archive/tasks/inference_programme/cluster_gradient_search_benchmark.md] +- 2026-07-30 — note — question: Do the confirmatory multi-band compile census legs change anything [archive/tasks/inference_programme/multiband_compile_census_completion.md] diff --git a/projects/slope_hierarchy.md b/projects/slope_hierarchy.md new file mode 100644 index 0000000..7ecd16a --- /dev/null +++ b/projects/slope_hierarchy.md @@ -0,0 +1,14 @@ +# slope_hierarchy — Hierarchical slope recovery at N=5, wrapped up and succeeded by slope_hierarchy_scale + +Project: slope_hierarchy +Issue: none + +## Now + +Retired 2026-09-08; wrapped up at N=5 on 2026-07-22 and succeeded by slope_hierarchy_scale. + +## Runs + +## Log + +- 2026-07-22 — note — question: What does the finished N=5 slope-hierarchy result actually claim — planned, never run [archive/tasks/slope_hierarchy/methods_writeup.md] diff --git a/projects/slope_hierarchy_scale.md b/projects/slope_hierarchy_scale.md new file mode 100644 index 0000000..aa447c5 --- /dev/null +++ b/projects/slope_hierarchy_scale.md @@ -0,0 +1,24 @@ +# slope_hierarchy_scale — Hierarchical slope recovery with graphical and EP at N=25 to 50 + +Project: slope_hierarchy_scale +Issue: none + +## Now + +The N=25 scale-up is running: graphical array 342348_[0-24], joint fits 342350_0 / 342351_0, and the EP rerun 342410 on the mirror that carries #1580. +Next: pull wave 1 and compare EP against the NUTS parent sigma; then decide whether to go to N=50. + +## Runs + +- 342348_[0-24] — open — gpu — 2026-09-08 — n25_scale_up: one lens per array task, hpc/batch_gpu/submit_one_by_one, sample_n25_seed42 +- 342350_0 — open — gpu — 2026-09-08 — n25_scale_up: joint hierarchical NUTS fit, hpc/batch_gpu/submit_graphical +- 342351_0 — open — ral — 2026-09-08 — n25_scale_up: EP arm on the CPU partition, hpc/batch_cpu/submit_ep, max_steps 12; RAL PyAutoFit mirror 68ff9bd57 predates PyAutoFit#1580, so this arm runs without the stale-mask fixed-point fix +- 342410 — open — ral — 2026-09-09 — n25_scale_up: EP arm rerun on the JAX-on-CPU path (no multiprocessing pool); replaces 342351_0 + +## Log + +- 2026-09-09 — run — 342410 submitted: n25_scale_up — EP arm rerun on the JAX-on-CPU path (no multiprocessing pool); replaces 342351_0 +- 2026-09-08 — run — 342351_0 submitted: n25_scale_up — EP arm on the CPU partition, hpc/batch_cpu/submit_ep, max_steps 12; RAL PyAutoFit mirror 68ff9bd57 predates PyAutoFit#1580, so this arm runs without the stale-mask fixed-point fix +- 2026-09-08 — run — 342350_0 submitted: n25_scale_up — joint hierarchical NUTS fit, hpc/batch_gpu/submit_graphical +- 2026-09-08 — run — 342348_[0-24] submitted: n25_scale_up — one lens per array task, hpc/batch_gpu/submit_one_by_one, sample_n25_seed42 +- 2026-07-22 — note — question: Does hierarchical slope recovery hold at N=25 to 50 [archive/tasks/slope_hierarchy_scale/n25_scale_up.md] diff --git a/projects/subhalo_validation.md b/projects/subhalo_validation.md new file mode 100644 index 0000000..f594f8d --- /dev/null +++ b/projects/subhalo_validation.md @@ -0,0 +1,103 @@ +# subhalo_validation — Do the SLaM subhalo pipelines false-detect on lenses with no subhalo + +Project: subhalo_validation +Issue: none + +## Now + +Five chains on RAL under the corrected adapt settings: rectangular_adapt on pl_eff_0 / pl_eff_1_outer (job A done, job B 342311 submitted) and the delaunay_adapt_split reruns plus rectangular_adapt on pl_sersic_0. pl_sersic_0 on delaunay_adapt_split is accepted (no false detection, +2.557). +Next: pull when the numba chains land and read the evidence_increase of each; no more 1000-pixel runs. + +## Runs + +- 342299_1 — open — ral — 2026-09-07 — delaunay_adapt_split_pl_eff_0: job B, numba chain at the standard 1250-pixel Delaunay mesh (8c / 96gb / 48 h); reloads job A 342231_1; the 09-07 ruling: run all datasets at 1250, no more pix1000 +- 342299_2 — open — ral — 2026-09-07 — delaunay_adapt_split_pl_eff_1_outer: job B, numba chain at the standard 1250-pixel Delaunay mesh (8c / 96gb / 48 h); reloads job A 342231_2; the 09-07 ruling: run all datasets at 1250, no more pix1000 +- 342311_1 — open — ral — 2026-09-07 — rectangular_adapt_pl_eff_0: job B, numba chain (8c / 96gb / 48 h) at the standard 1250-pixel mesh under PIPELINE=rectangular_adapt; reloads job A 342237_1; the 09-07 ask: submit the rectangular runs too +- 342311_2 — open — ral — 2026-09-07 — rectangular_adapt_pl_eff_1_outer: job B, numba chain (8c / 96gb / 48 h) at the standard 1250-pixel mesh under PIPELINE=rectangular_adapt; reloads job A 342237_2; the 09-07 ask: submit the rectangular runs too +- 342311_0 — open — ral — 2026-09-07 — rectangular_adapt_pl_sersic_0: job B, numba chain (8c / 96gb / 48 h) at the standard 1250-pixel mesh under PIPELINE=rectangular_adapt; reloads job A 342237_0; the 09-07 ask: submit the rectangular runs too + +## Log + +- 2026-09-07 — result — accepted delaunay_adapt_split_pl_sersic_0 (R-20260907-01): Accepted. The numba delaunay_adapt_split chain on pl_sersic_0 (job 342273_0, 22 h 08 m, 1250 Hilbert / 1310 Delaunay pixels) lands evidence_increase +2.557 (< 5): no false detection. + The best cell parks a 1.6e6 Msun nuisance subhalo at (-1.43, -2.36)", outside the ring. The shift from the legacy -0.30 comes with the S/N-3 adapt cap, adaptive over-sampling and the numba pool, not a code fault. Results are good; run the remaining datasets at 1250 pixels; no more 1000-pixel runs. +- 2026-09-07 — run — 342311_0 submitted: rectangular_adapt_pl_sersic_0 — job B, numba chain (8c / 96gb / 48 h) at the standard 1250-pixel mesh under PIPELINE=rectangular_adapt; reloads job A 342237_0; the 09-07 ask: submit the rectangular runs too +- 2026-09-07 — run — 342311_2 submitted: rectangular_adapt_pl_eff_1_outer — job B, numba chain (8c / 96gb / 48 h) at the standard 1250-pixel mesh under PIPELINE=rectangular_adapt; reloads job A 342237_2; the 09-07 ask: submit the rectangular runs too +- 2026-09-07 — run — 342311_1 submitted: rectangular_adapt_pl_eff_0 — job B, numba chain (8c / 96gb / 48 h) at the standard 1250-pixel mesh under PIPELINE=rectangular_adapt; reloads job A 342237_1; the 09-07 ask: submit the rectangular runs too +- 2026-09-07 — run — 342299_2 submitted: delaunay_adapt_split_pl_eff_1_outer — job B, numba chain at the standard 1250-pixel Delaunay mesh (8c / 96gb / 48 h); reloads job A 342231_2; the 09-07 ruling: run all datasets at 1250, no more pix1000 +- 2026-09-07 — run — 342299_1 submitted: delaunay_adapt_split_pl_eff_0 — job B, numba chain at the standard 1250-pixel Delaunay mesh (8c / 96gb / 48 h); reloads job A 342231_1; the 09-07 ruling: run all datasets at 1250, no more pix1000 +- 2026-09-04 — run — 342273_0 submitted (done, wall 22:08): delaunay_adapt_split_pl_sersic_0 — job B, numba chain (8c / 96gb / 48 h) on the post-numba-speed-up mains (PyAutoFit cdda28b5f, PyAutoArray e36a5af4, PyAutoGalaxy 6d216c15, PyAutoLens 146a3d725); pl_sersic_0 alone + (ARRAY=0) to watch memory and runtime before the other lenses; reloads job A 342231_0 pulled_to: output/subhalo/detect/delaunay_adapt_split/pl_sersic_0_no_subhalo/ +- 2026-09-02 — result — rerun asked delaunay_adapt_split_pl_sersic_0 (R-20260902-07): The problem with the pl_eff_1_outer result, for the delaunay, is that the Hilbert mesh is not adapting to the fainter images of the outer source and thus still has too low resolution. + I think this is because the adapt_image does not have this scaling, which is used in other scritps, applied to it, which caps the max values to 3.0, meaning fainter sources are up weighted compared to the brightest soures. + For all runs, I can also see like it looks like the adaptive over sampling of the lensed source also is not right with most pixels still 4. Okay I think we will want to do a rerun of everything for self consistency [...] the rectangular should also used the capped adapt image as even there dont want to over weight bright galaxies, and the aapt images may also impact regularization and thus use this capped image. + Then, we should rerun all subhalo_validation with these updated settings, put old results to a folder titled "incorrect_adapt_sub" and clear them off RAL so tey dont gert in my way. --- Filed on all three ruled phases with the same body, 2026-09-02: both diagnoses were verified in the run outputs (the uncapped S/N adapt image gave pl_eff_1_outer's outer images 4 Hilbert points each against 79 on the core; the over-sampling call divided the S/N map by the noise map a second time, putting 76-90 % of the mask at sub-size 4 on every lens). + Fixed in subhalo_validation commits 9893b12 and 7165a0b; every pre-fix stage output from source_pix[1] onwards and every witness JSON moved to `incorrect_adapt_sub/` on the laptop and on RAL. The accepts R-20260902-02 (pl_eff_0) and R-20260902-03 (rectangular_adapt pl_sersic_0) are superseded because their witnesses were produced under the same two bugs, not because their numbers were wrong. + Resubmitted on RAL: delaunay_adapt_split 342231 (job A) → 342234 (job B), rectangular_adapt 342237 → 342240, arrays 0-2. +- 2026-09-02 — result — rerun asked rectangular_adapt_pl_sersic_0 (R-20260902-06): The problem with the pl_eff_1_outer result, for the delaunay, is that the Hilbert mesh is not adapting to the fainter images of the outer source and thus still has too low resolution. + I think this is because the adapt_image does not have this scaling, which is used in other scritps, applied to it, which caps the max values to 3.0, meaning fainter sources are up weighted compared to the brightest soures. + For all runs, I can also see like it looks like the adaptive over sampling of the lensed source also is not right with most pixels still 4. Okay I think we will want to do a rerun of everything for self consistency [...] the rectangular should also used the capped adapt image as even there dont want to over weight bright galaxies, and the aapt images may also impact regularization and thus use this capped image. + Then, we should rerun all subhalo_validation with these updated settings, put old results to a folder titled "incorrect_adapt_sub" and clear them off RAL so tey dont gert in my way. --- Filed on all three ruled phases with the same body, 2026-09-02: both diagnoses were verified in the run outputs (the uncapped S/N adapt image gave pl_eff_1_outer's outer images 4 Hilbert points each against 79 on the core; the over-sampling call divided the S/N map by the noise map a second time, putting 76-90 % of the mask at sub-size 4 on every lens). + Fixed in subhalo_validation commits 9893b12 and 7165a0b; every pre-fix stage output from source_pix[1] onwards and every witness JSON moved to `incorrect_adapt_sub/` on the laptop and on RAL. The accepts R-20260902-02 (pl_eff_0) and R-20260902-03 (rectangular_adapt pl_sersic_0) are superseded because their witnesses were produced under the same two bugs, not because their numbers were wrong. + Resubmitted on RAL: delaunay_adapt_split 342231 (job A) → 342234 (job B), rectangular_adapt 342237 → 342240, arrays 0-2. +- 2026-09-02 — result — rerun asked delaunay_adapt_split_pl_eff_0 (R-20260902-05): The problem with the pl_eff_1_outer result, for the delaunay, is that the Hilbert mesh is not adapting to the fainter images of the outer source and thus still has too low resolution. + I think this is because the adapt_image does not have this scaling, which is used in other scritps, applied to it, which caps the max values to 3.0, meaning fainter sources are up weighted compared to the brightest soures. + For all runs, I can also see like it looks like the adaptive over sampling of the lensed source also is not right with most pixels still 4. Okay I think we will want to do a rerun of everything for self consistency [...] the rectangular should also used the capped adapt image as even there dont want to over weight bright galaxies, and the aapt images may also impact regularization and thus use this capped image. + Then, we should rerun all subhalo_validation with these updated settings, put old results to a folder titled "incorrect_adapt_sub" and clear them off RAL so tey dont gert in my way. --- Filed on all three ruled phases with the same body, 2026-09-02: both diagnoses were verified in the run outputs (the uncapped S/N adapt image gave pl_eff_1_outer's outer images 4 Hilbert points each against 79 on the core; the over-sampling call divided the S/N map by the noise map a second time, putting 76-90 % of the mask at sub-size 4 on every lens). + Fixed in subhalo_validation commits 9893b12 and 7165a0b; every pre-fix stage output from source_pix[1] onwards and every witness JSON moved to `incorrect_adapt_sub/` on the laptop and on RAL. The accepts R-20260902-02 (pl_eff_0) and R-20260902-03 (rectangular_adapt pl_sersic_0) are superseded because their witnesses were produced under the same two bugs, not because their numbers were wrong. + Resubmitted on RAL: delaunay_adapt_split 342231 (job A) → 342234 (job B), rectangular_adapt 342237 → 342240, arrays 0-2. +- 2026-09-02 — result — rerun asked delaunay_adapt_split_pl_eff_1_outer (R-20260902-04): The problem with the pl_eff_1_outer result, for the delaunay, is that the Hilbert mesh is not adapting to the fainter images of the outer source and thus still has too low resolution. + I think this is because the adapt_image does not have this scaling, which is used in other scritps, applied to it, which caps the max values to 3.0, meaning fainter sources are up weighted compared to the brightest soures. + For all runs, I can also see like it looks like the adaptive over sampling of the lensed source also is not right with most pixels still 4. Okay I think we will want to do a rerun of everything for self consistency [...] the rectangular should also used the capped adapt image as even there dont want to over weight bright galaxies, and the aapt images may also impact regularization and thus use this capped image. + Then, we should rerun all subhalo_validation with these updated settings, put old results to a folder titled "incorrect_adapt_sub" and clear them off RAL so tey dont gert in my way. --- Filed on all three ruled phases with the same body, 2026-09-02: both diagnoses were verified in the run outputs (the uncapped S/N adapt image gave pl_eff_1_outer's outer images 4 Hilbert points each against 79 on the core; the over-sampling call divided the S/N map by the noise map a second time, putting 76-90 % of the mask at sub-size 4 on every lens). + Fixed in subhalo_validation commits 9893b12 and 7165a0b; every pre-fix stage output from source_pix[1] onwards and every witness JSON moved to `incorrect_adapt_sub/` on the laptop and on RAL. The accepts R-20260902-02 (pl_eff_0) and R-20260902-03 (rectangular_adapt pl_sersic_0) are superseded because their witnesses were produced under the same two bugs, not because their numbers were wrong. + Resubmitted on RAL: delaunay_adapt_split 342231 (job A) → 342234 (job B), rectangular_adapt 342237 → 342240, arrays 0-2. +- 2026-09-02 — result — accepted rectangular_adapt_pl_sersic_0 (R-20260902-03): rectangular_adapt_pl_sersic_0 is good — ACCEPTED. evidence_increase = +0.2538 (logL +4.1732 for 3 extra parameters), taken from the best subhalo[1] grid cell (the project's first grid-derived witness, subhalo[2] off), well under the < 5 null. + The four grid cells sit between -0.27 and +0.25 nats of the mass_total evidence (21402.320): no cell prefers a subhalo by more than a quarter of a nat. The nuisance subhalo is M200 2.4e10 Msun at (1.58, -0.04)", theta_E = 1.95". + RectangularBilinear passes on the same lens the Delaunay baseline passed on (R-20260831-06). Commentary for the wiki, not part of the verdict: rectangular_adapt's mass_total evidence is 21402 against 20326 for delaunay_adapt_split on identical data, a +1076 nat gap between the two meshes. +- 2026-09-02 — result — accepted delaunay_adapt_split_pl_eff_0 (R-20260902-02): pl_eff_0 / delaunay_adapt_split — ACCEPTED. The clumpy six-component source does not produce a false subhalo detection: evidence_increase = -0.2425 (logL +0.6093 for 3 extra parameters), and all four subhalo[1] grid cells sit 0.9-1.0 nats below the no-subhalo mass_total evidence, so the grid answer and the refine answer agree. + The nuisance subhalo parks outside the ring at (-1.30, -2.51)", M200 1.2e8 Msun, with theta_E = 1.93". Second clean null on this pipeline after pl_sersic_0. The witness came from subhalo[2] (pre subhalo_stage key), like pl_sersic_0; future rows report the subhalo[1] grid cell, which here gives the same verdict. + This is the second successful lens for the RectangularBilinear comparison asked for under R-20260831-06: go ahead and submit rectangular_adapt on pl_eff_0 (ARRAY=1) in the next local-dev slot. +- 2026-09-02 — run — 342240_0 submitted (void): rectangular_adapt_pl_sersic_0 — job B, cancelled 2026-09-02 while pending afterok, to wait for the numba likelihood speed-ups after: 342237_0 +- 2026-09-02 — run — 342237_0 submitted (done, wall 0:02): rectangular_adapt_pl_sersic_0 — job A, source_lp[1] on JAX; rerun with the S/N-3 adapt-image cap and fixed over-sampling (R-20260902-04..07) +- 2026-09-02 — run — 342240_2 submitted (void): rectangular_adapt_pl_eff_1_outer — job B, cancelled 2026-09-02 while pending afterok, to wait for the numba likelihood speed-ups after: 342237_2 +- 2026-09-02 — run — 342237_2 submitted (done, wall 1:52): rectangular_adapt_pl_eff_1_outer — job A, source_lp[1] on JAX +- 2026-09-02 — note — question: Does rectangular_adapt false-detect a subhalo on the outer-component lens [archive/tasks/subhalo_validation/rectangular_adapt_pl_eff_1_outer.md] +- 2026-09-02 — run — 342240_1 submitted (void): rectangular_adapt_pl_eff_0 — job B, cancelled 2026-09-02 while pending afterok, to wait for the numba likelihood speed-ups after: 342237_1 +- 2026-09-02 — run — 342237_1 submitted (done, wall 1:52): rectangular_adapt_pl_eff_0 — job A, source_lp[1] on JAX +- 2026-09-02 — note — question: Does rectangular_adapt false-detect a subhalo on the clumpy source [archive/tasks/subhalo_validation/rectangular_adapt_pl_eff_0.md] +- 2026-09-02 — run — 342234_0 submitted (void): delaunay_adapt_split_pl_sersic_0 — job B, cancelled 2026-09-02 at 12 min inside source_pix[1] to wait for the numba likelihood speed-ups; partial source_pix[1] output removed on RAL after: 342231_0 +- 2026-09-02 — run — 342231_0 submitted (done, wall 0:02): delaunay_adapt_split_pl_sersic_0 — job A, source_lp[1] on JAX; rerun with the S/N-3 adapt-image cap and fixed over-sampling (R-20260902-04..07) pulled_to: output/subhalo/detect/delaunay_adapt_split/pl_sersic_0_no_subhalo/ +- 2026-09-02 — run — 342234_2 submitted (void): delaunay_adapt_split_pl_eff_1_outer — job B, cancelled 2026-09-02 at 12 min inside source_pix[1] to wait for the numba likelihood speed-ups; partial source_pix[1] output removed on RAL after: 342231_2 +- 2026-09-02 — run — 342231_2 submitted (done, wall 0:02): delaunay_adapt_split_pl_eff_1_outer — job A, source_lp[1] on JAX; rerun with the S/N-3 adapt-image cap and fixed over-sampling (R-20260902-04..07) +- 2026-09-02 — run — 342234_1 submitted (void): delaunay_adapt_split_pl_eff_0 — job B, cancelled 2026-09-02 at 12 min inside source_pix[1] to wait for the numba likelihood speed-ups; partial source_pix[1] output removed on RAL after: 342231_1 +- 2026-09-02 — run — 342231_1 submitted (done, wall 0:02): delaunay_adapt_split_pl_eff_0 — job A, source_lp[1] on JAX; rerun with the S/N-3 adapt-image cap and fixed over-sampling (R-20260902-04..07) +- 2026-08-31 — result — left to finish delaunay_adapt_split_pl_eff_1_outer (R-20260831-08): Review on numba likelihood function stuff: 1) End to end runs are good in general, with numba speed up run times roughly x8 faster than old code, incredible. + Do an intake to investigate if the linear algebra approach taken for this is the same as the sparse operator mode alreasdy implemented for JAX, and if it is not ask whether it is amenable to GPU JAX speed up or if there is some sparsity exploitation that means this is suited to CPU. + 2) The pl_eff_1_outer_no_subhalo task did not get a good result, the outer source galaxy messed up the solution. I think the issue is that source_pix[1] used ConstantSplit, not AdaptSplit, so can you rerun source_pix[1] on this lens with a suffix "_adapt_split_fix" and I will inspect the result. + We dont need subhalo[2] for these currently, knowing if theres a detection in subhalo[1] is sufficent so dont run subhalo[2] for future runs. Lets also do a RectangularBilinear run for the two successful lenses in the next batch / task. --- Filed on both `pl_eff` phases with the same body: at the review both `342027_1` and `342027_2` were still inside `subhalo[2]` with about three hours of wall + left, and the human left them to finish rather than ruling on an unfinished chain. +- 2026-08-31 — result — left to finish delaunay_adapt_split_pl_eff_0 (R-20260831-07): Review on numba likelihood function stuff: 1) End to end runs are good in general, with numba speed up run times roughly x8 faster than old code, incredible. + Do an intake to investigate if the linear algebra approach taken for this is the same as the sparse operator mode alreasdy implemented for JAX, and if it is not ask whether it is amenable to GPU JAX speed up or if there is some sparsity exploitation that means this is suited to CPU. + 2) The pl_eff_1_outer_no_subhalo task did not get a good result, the outer source galaxy messed up the solution. I think the issue is that source_pix[1] used ConstantSplit, not AdaptSplit, so can you rerun source_pix[1] on this lens with a suffix "_adapt_split_fix" and I will inspect the result. + We dont need subhalo[2] for these currently, knowing if theres a detection in subhalo[1] is sufficent so dont run subhalo[2] for future runs. Lets also do a RectangularBilinear run for the two successful lenses in the next batch / task. --- Filed on both `pl_eff` phases with the same body: at the review both `342027_1` and `342027_2` were still inside `subhalo[2]` with about three hours of wall + left, and the human left them to finish rather than ruling on an unfinished chain. +- 2026-08-31 — result — accepted delaunay_adapt_split_pl_sersic_0 (R-20260831-06): Review on numba likelihood function stuff: 1) End to end runs are good in general, with numba speed up run times roughly x8 faster than old code, incredible. + Do an intake to investigate if the linear algebra approach taken for this is the same as the sparse operator mode alreasdy implemented for JAX, and if it is not ask whether it is amenable to GPU JAX speed up or if there is some sparsity exploitation that means this is suited to CPU. + 2) The pl_eff_1_outer_no_subhalo task did not get a good result, the outer source galaxy messed up the solution. I think the issue is that source_pix[1] used ConstantSplit, not AdaptSplit, so can you rerun source_pix[1] on this lens with a suffix "_adapt_split_fix" and I will inspect the result. + We dont need subhalo[2] for these currently, knowing if theres a detection in subhalo[1] is sufficent so dont run subhalo[2] for future runs. Lets also do a RectangularBilinear run for the two successful lenses in the next batch / task. +- 2026-08-31 — run — 342095_0 submitted (legacy): rectangular_adapt_pl_sersic_0 — pre-Cortex run, migrated; job B, the numba chain (8c / 96gb / 48 h) with subhalo[2] off — witness landed 2026-09-01 12:54 local (17:54 BST) where: + output/subhalo/detect/rectangular_adapt/pl_sersic_0_no_subhalo/ after: 342094_0 pulled_to: output/subhalo/detect/rectangular_adapt/pl_sersic_0_no_subhalo/ +- 2026-08-31 — run — 342094_0 submitted (legacy): rectangular_adapt_pl_sersic_0 — pre-Cortex run, migrated; job A, JAX source_lp[1] under the rectangular tree (4c / 32gb / 12 h) — COMPLETED 0:0 in 01:44:40, ended 23:52:17 BST where: + output/subhalo/detect/rectangular_adapt/pl_sersic_0_no_subhalo/ pulled_to: output/subhalo/detect/rectangular_adapt/pl_sersic_0_no_subhalo/ +- 2026-08-31 — note — question: Does rectangular_adapt behave like delaunay_adapt_split on the clean null [archive/tasks/subhalo_validation/rectangular_adapt_pl_sersic_0.md] +- 2026-08-31 — run — 342093_0 submitted (legacy): delaunay_adapt_split_pl_eff_1_outer — pre-Cortex run, migrated; the source_pix[1] AdaptSplit rerun the am review asked for — recipe delaunay_adapt_split_fix, --output_suffix=_adapt_split_fix --stop_after=source_pix_1, so it writes + no witness JSON by design; awaiting the human's judgement where: output/subhalo/detect/delaunay_adapt_split/pl_eff_1_outer_no_subhalo_adapt_split_fix/ pulled_to: output/subhalo/detect/delaunay_adapt_split/pl_eff_1_outer_no_subhalo_adapt_split_fix/ +- 2026-08-29 — run — 342027_0 submitted (legacy): delaunay_adapt_split_pl_sersic_0 — pre-Cortex run, migrated; MaxRSS 27.0 GB, autolens 2026.8.17.1 + PyAutoFit 56fb8b63b (the PR #1548 pool fix); witness predates the subhalo_stage key — read it as subhalo[2]_[single_plane_refine] + where: output/subhalo/detect/delaunay_adapt_split/pl_sersic_0_no_subhalo/ pulled_to: output/subhalo/detect/delaunay_adapt_split/pl_sersic_0_no_subhalo/ +- 2026-08-29 — note — question: Does delaunay_adapt_split false-detect a subhalo on the null lens [archive/tasks/subhalo_validation/delaunay_adapt_split_pl_sersic_0.md] +- 2026-08-29 — run — 342027_2 submitted (legacy): delaunay_adapt_split_pl_eff_1_outer — pre-Cortex run, migrated; COMPLETED 0:0 in 1-23:05:05, ended 01:59:36 BST 2026-09-01 with 55 min of wall to spare; witness predates the subhalo_stage key where: + output/subhalo/detect/delaunay_adapt_split/pl_eff_1_outer_no_subhalo/ pulled_to: output/subhalo/detect/delaunay_adapt_split/pl_eff_1_outer_no_subhalo/ +- 2026-08-29 — note — question: Does delaunay_adapt_split false-detect on the outer-component lens and AdaptSplit fix [archive/tasks/subhalo_validation/delaunay_adapt_split_pl_eff_1_outer.md] +- 2026-08-29 — run — 342027_1 submitted (legacy): delaunay_adapt_split_pl_eff_0 — pre-Cortex run, migrated; COMPLETED 0:0 in 1-21:10:24, ended 00:04:55 BST 2026-09-01 — inside the 2-day wall after all; witness predates the subhalo_stage key where: + output/subhalo/detect/delaunay_adapt_split/pl_eff_0_no_subhalo/ pulled_to: output/subhalo/detect/delaunay_adapt_split/pl_eff_0_no_subhalo/ +- 2026-08-29 — note — question: Does delaunay_adapt_split false-detect a subhalo on the clumpy source [archive/tasks/subhalo_validation/delaunay_adapt_split_pl_eff_0.md] diff --git a/scripts/cortex.py b/scripts/cortex.py index c83e823..35f7553 100644 --- a/scripts/cortex.py +++ b/scripts/cortex.py @@ -1,24 +1,36 @@ #!/usr/bin/env python3 -"""cortex.py — the Cortex's lifecycle script: check · gates · rule · move · new · retire. +"""cortex.py — the Cortex's lifecycle script over one ledger per project. -The run-and-ruling registry (project → task → runs → rulings) is a tree of -markdown files with light headers, and this script is the one thing that -writes their state and the one thing that checks it. Every rule it enforces is -written down in REFERENCE.md; the transition table there is `move`'s and the -chain rules are `rule`'s. Nothing here is not in that file. +The Cortex is a **history of how each science project unfolds**, not a queue +of work to finish. Every project in `projects.yaml` that is doing anything has +one file, `projects/<key>.md`, holding three things and nothing else: + + ## Now — two or three lines, rewritten: what is running and what the + human meant to do next. The "pick up where I left off" line. + ## Runs — the jobs on the cluster right now, `open` or `running`. A run + that has finished leaves this list and becomes a log entry. + ## Log — dated entries, newest first: a run set off, a result the human + noted, a lesson they stated, a thought. Nothing here is ever + "done"; it only gets older. The board shows the last five. + +Nothing in a ledger is inferred from results. A run's submission, start and +end are cluster facts and this script records them; a `result` or `lesson` +entry is the human's own words, written on their ask. PyYAML for `projects.yaml` and otherwise stdlib only, `main(argv)`, no -import-time side effects, `--root` on every verb -(default: the repo this script lives in), so every leg runs against a -`tmp_path` copy of the fixture in tests. The date is injectable (`--today`) -for the same reason. +import-time side effects, `--root` on every verb (default: the repo this +script lives in), so every leg runs against a `tmp_path` copy of the fixture +in tests. The date is injectable (`--today`) for the same reason. Usage: - python3 scripts/cortex.py check # OK or DRIFT (exit 1) - python3 scripts/cortex.py gates # gated tasks and their refs - python3 scripts/cortex.py rule <task> <verb> --body <file> [...] - python3 scripts/cortex.py move <task> <state> [...] - python3 scripts/cortex.py new <project> <slug> --summary "<≤10 words>" [...] + python3 scripts/cortex.py check # OK or DRIFT (exit 1) + python3 scripts/cortex.py new <project> --summary "<one line>" [--issue Repo#N] + python3 scripts/cortex.py run <project> <jobid> "<what it is>" [--partition P] + python3 scripts/cortex.py running <project> <jobid> + python3 scripts/cortex.py done <project> <jobid> [--failed] [--wall H:MM] [--note "..."] + python3 scripts/cortex.py log <project> "<text>" [--kind note|result|lesson] + python3 scripts/cortex.py now <project> "<text>" + python3 scripts/cortex.py issue <project> [-n 5] # the issue-top block python3 scripts/cortex.py retire <project> --why "<one line>" # a project's row Exit codes: 0 = done · 1 = drift or a refused edit · 2 = bad arguments. @@ -29,7 +41,7 @@ import argparse import re import sys -from datetime import date, datetime, timezone +from datetime import date from pathlib import Path import yaml @@ -39,118 +51,66 @@ # --------------------------------------------------------------------------- # # vocabulary (REFERENCE.md) # --------------------------------------------------------------------------- # -TASK_STATES = ( - "planned", "gated", "ready", "submitted", "running", "pulled", - "awaiting-ruling", "accepted", "rerun", "dropped", -) -TERMINAL_STATES = {"dropped"} -NON_TERMINAL_STATES = set(TASK_STATES) - TERMINAL_STATES -#: `submitted..accepted` (and `rerun`) — the states that need a `Witness:`. -WITNESS_STATES = {"submitted", "running", "pulled", "awaiting-ruling", - "accepted", "rerun"} -#: reachable only through `rule`, so they need a `Ruling:`. -RULED_STATES = {"accepted", "rerun", "dropped"} - -RUN_STATES = ("submitted", "running", "done", "failed", "timeout", "void", - "legacy", "legacy_wrong") -LIVE_RUN_STATES = {"submitted", "running"} -LEGACY_RUN_STATES = {"legacy", "legacy_wrong"} -RESET_RUN_STATES = {"failed", "timeout", "void"} - -#: What `new` writes into `## Where to look` before a task has an output -#: path. It is honest on a `planned` task and a hole on any other, because -#: the section is now rendered as the "which folder do I open" answer on the -#: dashboard's `## By project` view and in `pyauto-brain cortex checkin`. -WHERE_PLACEHOLDER = "(the output path, once there is one)" -#: The one state that may still be carrying the placeholder. -WHERE_EXEMPT_STATES = {"planned"} - -RULING_VERBS = ("accept", "rerun", "drop", "leave-to-finish") -#: the head verb ↔ task state agreement `check` enforces. -VERB_STATES = { - "accept": {"accepted"}, - "drop": {"dropped"}, - "rerun": {"rerun", "ready", "submitted", "running", "pulled", "awaiting-ruling"}, - "leave-to-finish": NON_TERMINAL_STATES, -} -#: `rule` writes the task into this state (None = unchanged). -VERB_TARGET = {"accept": "accepted", "rerun": "rerun", "drop": "dropped", - "leave-to-finish": None} - -# Canonical header key order — `new` writes it and the edit helpers insert a -# missing key at its slot so a hand-edited file keeps reading the same. -TASK_KEYS = ( - "Project", "Summary", "State", "Gates", "Reset", "Witness", "Budget", - "Runs", "Ruling", "Review-minutes", "Epic", "Filed", "Migrated-from", -) -#: `Summary:` is the board's line: the QUESTION the task answers, in at most -#: ten words. It is required on every task file and `check` enforces both. -SUMMARY_MAX_WORDS = 10 -#: Header keys that were retired, and what to say instead of the generic -#: `unknown header key`. `Phase:` died on 2026-09-07: science is unordered -#: ideas, the slug is the identity, the number carried nothing. -DEAD_KEYS = { - "Phase": "Phase: is a retired header — a task is identified by its slug, " - "not a number (delete the line; `Task:` on a ruling names the path)", -} -#: `Batch:` is optional-historical — the 2026-08/09 rulings cite the batch -#: record they were filed from; nothing writes new ones (the slot apparatus -#: was retired 2026-09-03). -RULING_KEYS = ( - "Project", "Task", "Runs", "Ruling", "Supersedes", "Batch", "Reviewed-at", - "Review-minutes-actual", "Follow-ups", "Migrated-from", -) -TASK_SECTIONS = ("Question", "Witness", "Where to look", "Runs", "Ruling") -RULING_SECTIONS = ("Ruling", "Evidence") +LEDGER_DIR = "projects" +SECTIONS = ("Now", "Runs", "Log") +HEADER_KEYS = ("Project", "Issue") +#: what an entry is: a run set off or finished (recorded by this script), a +#: result the human read off, a lesson they want kept, or anything else. +KINDS = ("run", "result", "lesson", "note") +#: a run on the cluster is `open` (submitted, not seen running) or `running`; +#: there is no third state — a finished run is a log entry, not a run line. +RUN_STATES = ("open", "running") +#: how many words a project's one-line summary (the title tail) may hold. +SUMMARY_MAX_WORDS = 15 +#: the board's window on the log. +LOG_WINDOW = 5 PROJECT_FIELDS = ("remote", "local_path", "ral_root", "mirror", "sync_cli", "sync_verbs", "ledger", "assistant", "witness_file", "partition", "status") -#: the one optional field — free text about the row (why a remote is `none`, -#: what a verb does). A row may omit it; an empty `note:` is still drift, and -#: any field outside these two tuples is still an error. PROJECT_OPTIONAL_FIELDS = ("note",) PARTITIONS = ("gpu", "ral", "both") PROJECT_STATUSES = ("active", "dormant", "planned", "retired") # --------------------------------------------------------------------------- # -# grammars (REFERENCE.md) +# grammars # --------------------------------------------------------------------------- # -# Copied verbatim from PyAutoMind/scripts/lifecycle.py: the lookbehind is what -# rejects `owner/Repo#N` — another owner is spelled as a URL. -GATE_REF_RE = re.compile( - r"https://github\.com/([\w.-]+)/([\w.-]+)/(?:issues|pull)/(\d+)" - r"|(?<![\w/])([A-Za-z_][\w.]*)#(\d+)\b" -) -DEFAULT_GATE_OWNER = "PyAutoLabs" - -RUN_LINE_RE = re.compile( - r"^- (?P<stem>\d+)" - r"(?:_(?P<task>\d+)|_\[(?P<tasks>\d+(?:-\d+)?(?:,\d+(?:-\d+)?)*)\])?" - r": (?P<state>submitted|running|done|failed|timeout|void|legacy|legacy_wrong)" - r" — (?P<partition>[a-z][a-z0-9_-]*)" - r" — submitted (?P<date>\d{4}-\d{2}-\d{2})" - r" — wall (?P<wall>\d+:\d{2})" - r"(?: — (?P<note>.+))?$" -) -RUN_CONT_RE = re.compile( - r"^ (?P<key>pulled_to|after|resumes|where|ruled): (?P<value>\S.*)$" -) -RUN_IDENT_RE = re.compile(r"^\d+(?:_\d+|_\[\d+(?:-\d+)?(?:,\d+(?:-\d+)?)*\])?$") -PARTITION_RE = re.compile(r"^[a-z][a-z0-9_-]*$") - -RULING_ID_RE = re.compile(r"^R-(\d{4})(\d{2})(\d{2})-(\d{2})$") -RULING_FILE_RE = re.compile(r"^rulings/(\d{4})/(\d{2})/(R-\d{8}-\d{2})\.md$") -TASK_FILE_RE = re.compile(r"^tasks/([a-z][a-z0-9_]*)/([^/]+)\.md$") PROJECT_KEY_RE = re.compile(r"^[a-z][a-z0-9_]*$") -SLUG_RE = re.compile(r"^[a-z0-9][a-z0-9_-]*$") +LEDGER_FILE_RE = re.compile(r"^projects/([a-z][a-z0-9_]*)\.md$") DATE_RE = re.compile(r"^\d{4}-\d{2}-\d{2}$") WALL_RE = re.compile(r"^\d+:\d{2}$") -INT_RE = re.compile(r"^\d+$") - -#: a light-header line: `Key: value` or a bare `Key:` (empty, or a list follows). +RUN_IDENT_RE = re.compile(r"^\d+(?:_\d+|_\[\d+(?:-\d+)?(?:,\d+(?:-\d+)?)*\])?$") +PARTITION_RE = re.compile(r"^[a-z][a-z0-9_-]*$") +#: `Repo#N` (owner PyAutoLabs), an issue URL, or `none` — the same grammar the +#: Mind uses for a GitHub ref. +ISSUE_RE = re.compile( + r"^(?:none|(?<![\w/])[A-Za-z0-9_.-]+#\d+|" + r"https://github\.com/[\w.-]+/[\w.-]+/issues/\d+)$") +DEFAULT_ISSUE_OWNER = "PyAutoLabs" +RUN_LINE_RE = re.compile( + r"^- (?P<ident>\d+(?:_\d+|_\[[\d,\-]+\])?) — (?P<state>open|running) — " + r"(?P<partition>[a-z][a-z0-9_-]*) — (?P<date>\d{4}-\d{2}-\d{2}) — (?P<what>\S.*)$") +LOG_LINE_RE = re.compile( + r"^- (?P<date>\d{4}-\d{2}-\d{2}) — (?P<kind>run|result|lesson|note) — (?P<text>\S.*)$") +CONT_RE = re.compile(r"^ (?P<text>\S.*)$") HEADER_KEY_RE = re.compile(r"^([A-Z][A-Za-z0-9-]*):(?:[ \t]+(.*?))?[ \t]*$") -HEADER_LINES = 30 +HEADER_LINES = 12 + +TEMPLATE = """# {key} — {summary} + +Project: {key} +Issue: {issue} + +## Now + +{now} + +## Runs + +## Log + +- {today} — note — ledger opened +""" class CortexError(Exception): @@ -166,11 +126,6 @@ def _dash(line: str) -> str: # the light header # --------------------------------------------------------------------------- # def header_span(lines: "list[str]") -> "tuple[int, int]": - """(start, end) line indexes of the header block, end exclusive. - - The block starts at the first `Key:` line after the title (within the - first 30 lines) and ends at the next blank line; (1, 1) when there is - none.""" start = None for i in range(1, min(len(lines), HEADER_LINES)): if HEADER_KEY_RE.match(lines[i]): @@ -185,67 +140,17 @@ def header_span(lines: "list[str]") -> "tuple[int, int]": def parse_header(text: str) -> "tuple[str | None, dict[str, str]]": - """(title, fields) — first occurrence of a key wins; a bare `Key:` followed - by `- item` lines is a list, joined with commas.""" lines = text.split("\n") title = lines[0][2:].strip() if lines and lines[0].startswith("# ") else None fields: "dict[str, str]" = {} start, end = header_span(lines) - last_key = None for line in lines[start:end]: m = HEADER_KEY_RE.match(line) - if m: - key, value = m.group(1), (m.group(2) or "").strip() - if key not in fields: - fields[key] = value - last_key = key - else: - last_key = None - continue - if line.startswith("- ") and last_key is not None: - item = line[2:].strip() - fields[last_key] = f"{fields[last_key]}, {item}" if fields[last_key] else item + if m and m.group(1) not in fields: + fields[m.group(1)] = (m.group(2) or "").strip() return title, fields -def edit_header(text: str, updates: "dict[str, str | None]", order=TASK_KEYS) -> str: - """Return `text` with header keys set (value) or removed (None), in place. - - Every other byte is preserved: a present key has only its value replaced; - a missing key is inserted at its canonical slot; the block, the body and - the line endings are untouched.""" - lines = text.split("\n") - start, end = header_span(lines) - if start == end and any(v is not None for v in updates.values()): - # No header block yet: open one after the title line. - insert_at = 1 - if len(lines) > 1 and lines[1].strip(): - lines.insert(1, "") - lines.insert(insert_at + 1, "") - start = end = insert_at + 1 - for key, value in updates.items(): - idx = next((i for i in range(start, end) if HEADER_KEY_RE.match(lines[i]) - and HEADER_KEY_RE.match(lines[i]).group(1) == key), None) - if value is None: - if idx is not None: - del lines[idx] - end -= 1 - continue - line = f"{key}: {value}" if value else f"{key}:" - if idx is not None: - lines[idx] = line - continue - rank = {k: i for i, k in enumerate(order)} - after = start - 1 - for i in range(start, end): - m = HEADER_KEY_RE.match(lines[i]) - if m and rank.get(m.group(1), -1) < rank.get(key, len(order)): - after = i - lines.insert(after + 1, line) - end += 1 - return "\n".join(lines) - - def sections(text: str) -> "dict[str, tuple[int, int]]": """{name: (first body line index, end index exclusive)} for `## name`.""" lines = text.split("\n") @@ -257,43 +162,29 @@ def sections(text: str) -> "dict[str, tuple[int, int]]": return out -def append_to_section(text: str, name: str, new_lines: "list[str]", - replace_placeholder: "str | None" = None) -> str: - """Append lines at the end of `## name` (before the next heading), - keeping one blank line between the content and the next heading.""" +def _section_lines(text: str, name: str) -> "list[str]": + span = sections(text).get(name) + if span is None: + return [] + return text.split("\n")[span[0]:span[1]] + + +def _replace_section(text: str, name: str, body: "list[str]") -> str: + """Return `text` with the body of `## name` replaced by `body` (one blank + line either side; every other byte preserved).""" lines = text.split("\n") span = sections(text).get(name) if span is None: - raise CortexError(f"no `## {name}` section to write into") - body_start, stop = span - # last content line of the section - last = stop - 1 - while last >= body_start and not lines[last].strip(): - last -= 1 - if replace_placeholder is not None and last >= body_start \ - and lines[last].strip() == replace_placeholder \ - and all(not ln.strip() for ln in lines[body_start:last]): - lines[last:last + 1] = new_lines - return "\n".join(lines) - if last < body_start: - # An empty section: replace whatever blank lines it held with - # `blank, content, blank` so the next heading keeps its gap. - lines[body_start:stop] = [""] + new_lines + [""] - return "\n".join(lines) - lines[last + 1:last + 1] = new_lines - return "\n".join(lines) + raise CortexError(f"no `## {name}` section") + start, stop = span + new = [""] + body + [""] + return "\n".join(lines[:start] + new + lines[stop:]) # --------------------------------------------------------------------------- # -# projects.yaml — PyYAML plus the field validation +# projects.yaml # --------------------------------------------------------------------------- # def parse_projects(text: str) -> "tuple[dict[str, dict], list[str]]": - """Parse projects.yaml into {key: {field: value}} and validate the fields. - - The document itself is PyYAML's job (`yaml.safe_load`); this validates the - shape the Cortex depends on — a mapping of project key → field mapping, - the required and optional field names, and each field's value. An empty or - comment-only file parses to an empty map.""" problems: "list[str]" = [] try: doc = yaml.safe_load(text) @@ -319,7 +210,6 @@ def parse_projects(text: str) -> "tuple[dict[str, dict], list[str]]": def _project_row(key: str, row: dict, problems: "list[str]") -> dict: - """One validated row: unknown/missing fields and bad values are problems.""" out: "dict[str, object]" = {} for field, value in row.items(): if field not in PROJECT_FIELDS + PROJECT_OPTIONAL_FIELDS: @@ -382,398 +272,214 @@ def load_projects(root: Path) -> "tuple[dict[str, dict], list[str]]": # --------------------------------------------------------------------------- # -# run lines +# the ledger file # --------------------------------------------------------------------------- # class Run: - __slots__ = ("ident", "stem", "task", "tasks", "state", "partition", "date", - "wall", "note", "cont", "lineno") + __slots__ = ("ident", "state", "partition", "date", "what", "cont", "lineno") def __init__(self, m: "re.Match", lineno: int): - self.stem = m.group("stem") - self.task = m.group("task") - self.tasks = m.group("tasks") - self.ident = self.stem + (f"_{self.task}" if self.task else - f"_[{self.tasks}]" if self.tasks else "") + self.ident = m.group("ident") self.state = m.group("state") self.partition = m.group("partition") self.date = m.group("date") - self.wall = m.group("wall") - self.note = m.group("note") or "" - self.cont: "dict[str, str]" = {} + self.what = m.group("what").strip() + self.cont: "list[str]" = [] self.lineno = lineno - def task_set(self) -> "set[int] | None": - """The tasks this line covers; None = the whole job (a bare stem).""" - if self.task: - return {int(self.task)} - if self.tasks is None: - return None - out: "set[int]" = set() - for part in self.tasks.split(","): - if "-" in part: - a, b = (int(x) for x in part.split("-")) - out.update(range(a, b + 1)) - else: - out.add(int(part)) - return out - - def ascending(self) -> bool: - if self.tasks is None: - return True - last = -1 - for part in self.tasks.split(","): - a, b = (int(x) for x in part.split("-")) if "-" in part else (int(part),) * 2 - if a <= last or b < a: - return False - last = b - return True - - -def parse_runs(text: str) -> "tuple[list[Run], list[str]]": - """Run lines of `## Runs`; a missing section = no runs (reported by the - section check, not here).""" - lines = text.split("\n") - span = sections(text).get("Runs") - runs: "list[Run]" = [] - problems: "list[str]" = [] - if span is None: - return runs, problems - for i in range(*span): - raw = lines[i] - if not raw.strip(): - continue - line = _dash(raw) - m = RUN_LINE_RE.match(line) - if m: - run = Run(m, i + 1) - if not run.ascending(): - problems.append(f"line {i + 1}: task set not ascending: {run.ident}") - runs.append(run) - continue - m = RUN_CONT_RE.match(line) - if m: - if not runs: - problems.append(f"line {i + 1}: continuation line without a run line") - else: - runs[-1].cont[m.group("key")] = m.group("value").strip() - continue - problems.append(f"line {i + 1}: run line does not parse: {raw.strip()}") - return runs, problems - + def line(self) -> str: + return f"- {self.ident} — {self.state} — {self.partition} — {self.date} — {self.what}" -def _overlaps(a: "set[int] | None", b: "set[int] | None") -> bool: - return a is None or b is None or bool(a & b) + def text(self) -> str: + return " ".join([self.what] + self.cont) -def run_problems(runs: "list[Run]", header_runs: str, state: str, - ruling_ids: "set[str]") -> "list[str]": - problems: "list[str]" = [] - idents = {r.ident for r in runs} - for i, a in enumerate(runs): - for b in runs[i + 1:]: - if a.stem == b.stem and _overlaps(a.task_set(), b.task_set()): - problems.append(f"run {a.ident} overlaps {b.ident} on stem {a.stem}") - body_stems = {r.stem for r in runs} - listed = [s.strip() for s in header_runs.split(",") if s.strip()] - for s in listed: - if not INT_RE.match(s): - problems.append(f"Runs: '{s}' is not a job stem") - if set(listed) != body_stems: - problems.append(f"Runs: header {{{', '.join(sorted(listed))}}} != body stems " - f"{{{', '.join(sorted(body_stems))}}}") - if state == "pulled" and not any(r.state in ("done", "legacy") and "pulled_to" in r.cont - for r in runs): - problems.append("State: pulled needs at least one done | legacy run with pulled_to:") - for r in runs: - if r.state in LEGACY_RUN_STATES and "where" not in r.cont: - problems.append(f"run {r.ident} is {r.state} without where:") - for key in ("after", "resumes"): - target = r.cont.get(key) - if target is not None and (target not in idents or target == r.ident): - problems.append(f"run {r.ident} {key}: {target} names no other run of this task") - ruled = r.cont.get("ruled") - if ruled is not None and ruled not in ruling_ids: - problems.append(f"run {r.ident} ruled: {ruled} does not resolve to a ruling file") - return problems +class Entry: + __slots__ = ("date", "kind", "text_head", "cont", "lineno") + def __init__(self, m: "re.Match", lineno: int): + self.date = m.group("date") + self.kind = m.group("kind") + self.text_head = m.group("text").strip() + self.cont: "list[str]" = [] + self.lineno = lineno -def gate_refs(value: str) -> "tuple[list[str], list[str]]": - """(refs, bad) from a comma-separated `Gates:` / `Follow-ups:` value.""" - refs, bad = [], [] - for token in (t.strip() for t in value.split(",")): - if not token: - continue - if GATE_REF_RE.fullmatch(token): - refs.append(token) - else: - bad.append(token) - return refs, bad + def text(self) -> str: + return " ".join([self.text_head] + self.cont) + def lines(self) -> "list[str]": + return [f"- {self.date} — {self.kind} — {self.text_head}"] + [f" {c}" for c in self.cont] -def gate_url(ref: str) -> str: - """Canonical issues URL for either GATE_REF_RE form (a PR *is* an issue).""" - m = GATE_REF_RE.fullmatch(ref) - owner, repo, num, short_repo, short_num = m.groups() - if owner: - return f"https://github.com/{owner}/{repo}/issues/{num}" - return f"https://github.com/{DEFAULT_GATE_OWNER}/{short_repo}/issues/{short_num}" +class Ledger: + __slots__ = ("key", "path", "rel", "title", "summary", "header", "now", + "runs", "log", "problems") -# --------------------------------------------------------------------------- # -# the tree -# --------------------------------------------------------------------------- # -class Task: - def __init__(self, root: Path, path: Path): + def __init__(self, key: str, path: Path, rel: str): + self.key = key self.path = path - self.rel = path.relative_to(root).as_posix() - self.text = path.read_text(encoding="utf-8") - self.title, self.fields = parse_header(self.text) - self.project_dir = path.parent.name - self.slug = path.stem - self.runs, self.run_parse_problems = parse_runs(self.text) - - def get(self, key: str) -> str: - return self.fields.get(key, "") + self.rel = rel + self.title: "str | None" = None + self.summary = "" + self.header: "dict[str, str]" = {} + self.now = "" + self.runs: "list[Run]" = [] + self.log: "list[Entry]" = [] + self.problems: "list[str]" = [] @property - def state(self) -> str: - return self.get("State") - + def issue(self) -> str: + return self.header.get("Issue", "none") or "none" -class Ruling: - def __init__(self, root: Path, path: Path): - self.path = path - self.rel = path.relative_to(root).as_posix() - self.text = path.read_text(encoding="utf-8") - self.title, self.fields = parse_header(self.text) - self.id = path.stem - - def get(self, key: str) -> str: - return self.fields.get(key, "") + def live(self) -> "list[Run]": + return list(self.runs) + def recent(self, n: int = LOG_WINDOW) -> "list[Entry]": + return self.log[:n] -def _md_files(d: Path): - return sorted(p for p in d.rglob("*.md")) if d.is_dir() else [] + def updated(self) -> str: + """The newest date on the file: the head log entry, else nothing.""" + return self.log[0].date if self.log else "" -def load_tasks(root: Path) -> "tuple[list[Task], list[str]]": - tasks, problems = [], [] - for p in _md_files(root / "tasks"): - rel = p.relative_to(root).as_posix() - if not TASK_FILE_RE.match(rel): - problems.append(f"{rel}: not a task path (tasks/<project>/<slug>.md)") +def _parse_items(lines: "list[str]", offset: int, line_re, factory, problems: "list[str]", + what: str) -> list: + out = [] + for i, raw in enumerate(lines): + if not raw.strip(): continue - tasks.append(Task(root, p)) - return tasks, problems - - -def load_rulings(root: Path) -> "tuple[list[Ruling], list[str]]": - rulings, problems = [], [] - for p in _md_files(root / "rulings"): - rel = p.relative_to(root).as_posix() - if rel == "rulings/AGENTS.md": + line = _dash(raw) + m = line_re.match(line) + if m: + out.append(factory(m, offset + i + 1)) continue - m = RULING_FILE_RE.match(rel) - if not m: - problems.append(f"{rel}: not a ruling path (rulings/<YYYY>/<MM>/R-<YYYYMMDD>-<nn>.md)") + m = CONT_RE.match(raw) + if m: + if not out: + problems.append(f"line {offset + i + 1}: continuation line without a {what} line") + else: + out[-1].cont.append(m.group("text").strip()) continue - yyyy, mm, rid = m.groups() - r = Ruling(root, p) - if rid[2:6] != yyyy or rid[6:8] != mm: - problems.append(f"{rel}: filed under {yyyy}/{mm} but the id is dated {rid[2:6]}-{rid[6:8]}") - rulings.append(r) - return rulings, problems - - -def ruling_files(root: Path) -> "dict[str, Path]": - """{id: path} of every ruling on disk (no validation).""" - out = {} - for p in _md_files(root / "rulings"): - if RULING_ID_RE.match(p.stem): - out.setdefault(p.stem, p) + problems.append(f"line {offset + i + 1}: {what} line does not parse: {raw.strip()}") return out -def batch_records(root: Path) -> "list[Path]": - d = root / "batches" - return sorted(p for p in d.glob("*.md") if p.name != "AGENTS.md") if d.is_dir() else [] +def parse_ledger(text: str, key: str, path: Path, rel: str) -> Ledger: + led = Ledger(key, path, rel) + lines = text.split("\n") + led.title, led.header = parse_header(text) + secs = sections(text) + names = [ln[3:].strip() for ln in lines if ln.startswith("## ")] + if names != list(SECTIONS): + led.problems.append(f"sections must be exactly ## {' · ## '.join(SECTIONS)} in that " + f"order (found: {', '.join(names) or 'none'})") + if "Now" in secs: + a, b = secs["Now"] + led.now = "\n".join(lines[a:b]).strip() + if "Runs" in secs: + a, b = secs["Runs"] + led.runs = _parse_items(lines[a:b], a, RUN_LINE_RE, Run, led.problems, "run") + if "Log" in secs: + a, b = secs["Log"] + led.log = _parse_items(lines[a:b], a, LOG_LINE_RE, Entry, led.problems, "log") + # title: `# <key> — <summary>` + if led.title is None: + led.problems.append("first line must be `# <key> — <summary>`") + else: + m = re.match(r"^([a-z][a-z0-9_]*) — (\S.*)$", _dash(led.title)) + if not m or m.group(1) != key: + led.problems.append(f"title must be `# {key} — <summary>`, not `# {led.title}`") + else: + led.summary = m.group(2).strip() + if len(led.summary.split()) > SUMMARY_MAX_WORDS: + led.problems.append(f"summary is over {SUMMARY_MAX_WORDS} words: {led.summary}") + return led -def batch_reviews(root: Path) -> "list[Path]": - d = root / "batches" / "reviews" - return sorted(p for p in d.glob("*.md") if p.name != "AGENTS.md") if d.is_dir() else [] +def _md_files(d: Path): + return sorted(p for p in d.glob("*.md") if p.name not in ("AGENTS.md", "TEMPLATE.md")) -# --------------------------------------------------------------------------- # -# check -# --------------------------------------------------------------------------- # -def task_problems(root: Path, tasks: "list[Task]", projects: "dict[str, dict]", - rulings: "dict[str, Ruling]", successors: "dict[str, list[str]]") -> "list[str]": +def load_ledgers(root: Path) -> "tuple[list[Ledger], list[str]]": + d = root / LEDGER_DIR + out: "list[Ledger]" = [] problems: "list[str]" = [] - for tk in tasks: - f = tk.fields - p = [f"{tk.rel}: {msg}" for msg in tk.run_parse_problems] - if tk.title is None: - p.append(f"{tk.rel}: line 1 is not `# <title>`") - for key in ("Project", "Summary", "State"): - if not f.get(key): - p.append(f"{tk.rel}: missing header key {key}:") - for key in f: - if key in DEAD_KEYS: - p.append(f"{tk.rel}: {DEAD_KEYS[key]}") - elif key not in TASK_KEYS: - p.append(f"{tk.rel}: unknown header key {key}:") - summary = f.get("Summary", "") - if summary: - words = summary.split() - if len(words) > SUMMARY_MAX_WORDS: - p.append(f"{tk.rel}: Summary: {len(words)} words — at most " - f"{SUMMARY_MAX_WORDS} (the question the task answers)") - for name in TASK_SECTIONS: - if name not in sections(tk.text): - p.append(f"{tk.rel}: missing section ## {name}") - project = f.get("Project", "") - if project and project != tk.project_dir: - p.append(f"{tk.rel}: Project: {project} is not the directory name {tk.project_dir}") - if project and project not in projects: - p.append(f"{tk.rel}: Project: {project} is not a projects.yaml key") - state = f.get("State", "") - if state and state not in TASK_STATES: - p.append(f"{tk.rel}: State: '{state}' is not a task state") - refs, bad = gate_refs(f.get("Gates", "")) - for token in bad: - p.append(f"{tk.rel}: Gates: unrecognised ref '{token}' — Repo#N or an " - f"issue/PR URL; no owner/Repo#N form") - if f.get("Filed") and not DATE_RE.match(f["Filed"]): - p.append(f"{tk.rel}: Filed: '{f['Filed']}' is not YYYY-MM-DD") - if f.get("Budget") and not WALL_RE.match(f["Budget"]): - p.append(f"{tk.rel}: Budget: '{f['Budget']}' is not H+:MM") - if f.get("Review-minutes") and not INT_RE.match(f["Review-minutes"]): - p.append(f"{tk.rel}: Review-minutes: '{f['Review-minutes']}' is not an integer") - # the offline invariants on top of the table - if state == "gated" and not refs: - p.append(f"{tk.rel}: State: gated with an empty Gates:") - if state in WITNESS_STATES and not f.get("Witness"): - p.append(f"{tk.rel}: State: {state} needs a Witness:") - ruling_id = f.get("Ruling", "") - if state in RULED_STATES and not ruling_id: - p.append(f"{tk.rel}: State: {state} needs a Ruling: (reachable only through rule)") - head = None - if ruling_id: - if not RULING_ID_RE.match(ruling_id): - p.append(f"{tk.rel}: Ruling: '{ruling_id}' is not a ruling id") - elif ruling_id not in rulings: - p.append(f"{tk.rel}: Ruling: {ruling_id} does not resolve to a ruling file") - else: - head = rulings[ruling_id] - if successors.get(ruling_id): - p.append(f"{tk.rel}: Ruling: {ruling_id} is not a chain head " - f"(superseded by {', '.join(successors[ruling_id])})") - if head.get("Task") != tk.rel: - p.append(f"{tk.rel}: Ruling: {ruling_id} names {head.get('Task') or '(nothing)'}, not this task") - verb = head.get("Ruling") - if state and verb in VERB_STATES and state not in VERB_STATES[verb]: - p.append(f"{tk.rel}: Ruling: {ruling_id} verb '{verb}' does not fit State: {state}") - # `## Where to look` is rendered, not just parsed: the by-project view - # prints these bullets verbatim as the folders to open, and `collect` - # resolves them to the artefacts it scores. A task that has left - # `planned` with only the template placeholder names nowhere. - if state and state not in WHERE_EXEMPT_STATES: - if not [b for b in _where_to_look(tk) if WHERE_PLACEHOLDER not in b]: - p.append(f"{tk.rel}: ## Where to look names nowhere — a task " - f"past planned needs at least one bullet that is not " - f"'{WHERE_PLACEHOLDER}'") - if state == "pulled" and any(r.state in LIVE_RUN_STATES for r in tk.runs): - if head is None or head.get("Ruling") != "leave-to-finish": - p.append(f"{tk.rel}: State: pulled with a live run needs a leave-to-finish ruling head (--partial)") - p.extend(f"{tk.rel}: {msg}" for msg in - run_problems(tk.runs, f.get("Runs", ""), state, set(rulings))) - problems.extend(p) - return problems + if not d.is_dir(): + return out, problems + for p in _md_files(d): + rel = p.relative_to(root).as_posix() + m = LEDGER_FILE_RE.match(rel) + if not m: + problems.append(f"{rel}: file name must be projects/<key>.md with a bare key") + continue + out.append(parse_ledger(p.read_text(encoding="utf-8"), m.group(1), p, rel)) + return out, problems -def ruling_problems(root: Path, rulings: "list[Ruling]", tasks_by_rel: "dict[str, Task]", - by_id: "dict[str, Ruling]", successors: "dict[str, list[str]]") -> "list[str]": +def ledger_problems(root: Path, ledgers: "list[Ledger]", projects: "dict[str, dict]") -> "list[str]": problems: "list[str]" = [] - seen: "dict[str, str]" = {} - for r in rulings: - p: "list[str]" = [] - prior = seen.setdefault(r.id, r.rel) - if prior != r.rel: - p.append(f"{r.rel}: duplicate ruling id {r.id} (also {prior})") - if r.title is None or not (r.title == r.id or r.title.startswith(f"{r.id} — ")): - p.append(f"{r.rel}: title must be `# {r.id}` or `# {r.id} — <summary>`") - for key in ("Project", "Task", "Ruling"): - if not r.get(key): - p.append(f"{r.rel}: missing header key {key}:") - for key in r.fields: - if key in DEAD_KEYS: - p.append(f"{r.rel}: {DEAD_KEYS[key]}") - elif key not in RULING_KEYS: - p.append(f"{r.rel}: unknown header key {key}:") - for name in RULING_SECTIONS: - if name not in sections(r.text): - p.append(f"{r.rel}: missing section ## {name}") - verb = r.get("Ruling") - if verb and verb not in RULING_VERBS: - p.append(f"{r.rel}: Ruling: '{verb}' is not a ruling verb") - task_rel = r.get("Task") - tk = tasks_by_rel.get(task_rel) - if task_rel and tk is None: - p.append(f"{r.rel}: Task: {task_rel} does not resolve to a task file") - if tk is not None and r.get("Project") and tk.get("Project") != r.get("Project"): - p.append(f"{r.rel}: Task: {task_rel} is not a task of Project: {r.get('Project')}") - if tk is not None: - stems = {s.strip() for s in r.get("Runs").split(",") if s.strip()} - task_stems = {x.stem for x in tk.runs} - if not stems <= task_stems: - p.append(f"{r.rel}: Runs: {{{', '.join(sorted(stems - task_stems))}}} not in the task's runs") - sup = r.get("Supersedes") - if sup: - if sup == r.id: - p.append(f"{r.rel}: Supersedes: itself") - elif sup not in by_id: - p.append(f"{r.rel}: Supersedes: {sup} does not resolve to a ruling file") - else: - old = by_id[sup] - if not sup < r.id: - p.append(f"{r.rel}: Supersedes: {sup} is not earlier than {r.id}") - if (old.get("Project"), old.get("Task")) != (r.get("Project"), r.get("Task")): - p.append(f"{r.rel}: Supersedes: {sup} names a different project/task") - if len(successors.get(r.id, [])) > 1: - p.append(f"{r.rel}: has {len(successors[r.id])} successors " - f"({', '.join(successors[r.id])}) — a chain, not a tree") - batch = r.get("Batch") - if batch and not (root / "batches" / f"{batch}.md").is_file(): - p.append(f"{r.rel}: Batch: {batch} names no batches/{batch}.md") - for key in ("Reviewed-at",): - pass - if r.get("Review-minutes-actual") and not INT_RE.match(r.get("Review-minutes-actual")): - p.append(f"{r.rel}: Review-minutes-actual: not an integer") - _, bad = gate_refs(r.get("Follow-ups")) - for token in bad: - p.append(f"{r.rel}: Follow-ups: unrecognised ref '{token}'") - problems.extend(p) + seen: "set[str]" = set() + for led in ledgers: + seen.add(led.key) + rel = led.rel + problems += [f"{rel}: {p}" for p in led.problems] + if led.key not in projects: + problems.append(f"{rel}: {led.key} is not a projects.yaml key") + row = projects.get(led.key, {}) + for k in HEADER_KEYS: + if k not in led.header: + problems.append(f"{rel}: header is missing `{k}:`") + extra = [k for k in led.header if k not in HEADER_KEYS] + if extra: + problems.append(f"{rel}: unknown header key(s) {', '.join(extra)} — the header " + f"holds only {', '.join(HEADER_KEYS)}") + if led.header.get("Project") not in (None, led.key): + problems.append(f"{rel}: `Project:` must be {led.key}") + issue = led.header.get("Issue") + if issue is not None and not ISSUE_RE.match(issue): + problems.append(f"{rel}: `Issue:` must be Repo#N, an issue URL or none, not {issue!r}") + if row.get("status") == "active" and not led.now: + problems.append(f"{rel}: `## Now` is empty on an active project") + idents: "set[str]" = set() + for r in led.runs: + if r.ident in idents: + problems.append(f"{rel}: run {r.ident} listed twice") + idents.add(r.ident) + if not RUN_IDENT_RE.match(r.ident): + problems.append(f"{rel}: run ident does not parse: {r.ident}") + problems += _date_problems(rel, r.date, f"run {r.ident}") + if row.get("partition") not in (None, "both", r.partition): + problems.append(f"{rel}: run {r.ident} is on `{r.partition}` but the project " + f"runs on `{row['partition']}`") + prev = None + for e in led.log: + problems += _date_problems(rel, e.date, f"log line {e.lineno}") + if prev is not None and e.date > prev: + problems.append(f"{rel}: log is not newest-first at line {e.lineno} " + f"({e.date} after {prev})") + prev = e.date + if row.get("status") == "retired" and led.runs: + problems.append(f"{rel}: a retired project still lists runs") + for key, row in projects.items(): + if row.get("status") == "active" and key not in seen: + problems.append(f"projects.yaml: {key} is active but has no projects/{key}.md") return problems +def _date_problems(rel: str, value: str, what: str) -> "list[str]": + if not DATE_RE.match(value): + return [f"{rel}: {what} date is not YYYY-MM-DD: {value}"] + try: + date.fromisoformat(value) + except ValueError: + return [f"{rel}: {what} date is not a real date: {value}"] + return [] + + def check_problems(root: Path) -> "list[str]": - """Every `check` rule, over the tree at `root`. Hermetic.""" projects, problems = load_projects(root) - tasks, stray = load_tasks(root) - problems.extend(stray) - rulings, stray = load_rulings(root) - problems.extend(stray) - by_id: "dict[str, Ruling]" = {} - for r in rulings: - by_id.setdefault(r.id, r) - successors: "dict[str, list[str]]" = {} - for r in rulings: - if r.get("Supersedes"): - successors.setdefault(r.get("Supersedes"), []).append(r.id) - tasks_by_rel = {tk.rel: tk for tk in tasks} - problems.extend(task_problems(root, tasks, projects, by_id, successors)) - problems.extend(ruling_problems(root, rulings, tasks_by_rel, by_id, successors)) + ledgers, more = load_ledgers(root) + problems += more + problems += ledger_problems(root, ledgers, projects) return problems @@ -789,38 +495,7 @@ def cmd_check(args) -> int: # --------------------------------------------------------------------------- # -# gates -# --------------------------------------------------------------------------- # -def gates_report(root: Path) -> "tuple[list[str], int]": - """(report lines, exit code) — every `gated` task, its refs and their URLs. - - Read-only and offline: nothing asks GitHub whether a ref has cleared. Gate - grading was retired 2026-09-03 — 2 gated refs and 0 flips in its whole - life, while schema decision 54 routes sequencing through prose - `Ready when:` lines. A gated task is moved on by a human reading this - listing and typing `move <task> ready`.""" - tasks, _ = load_tasks(root) - wanted = [tk for tk in tasks - if tk.state == "gated" and gate_refs(tk.get("Gates"))[0]] - if not wanted: - return ["gates: no gated task"], 0 - lines = [f"gates: {len(wanted)} task(s)"] - for tk in wanted: - refs = gate_refs(tk.get("Gates"))[0] - lines.append(f" {tk.rel}: {tk.state} — {', '.join(refs)}") - for ref in refs: - lines.append(f" {ref} → {gate_url(ref)}") - return lines, 0 - - -def cmd_gates(args) -> int: - lines, rc = gates_report(args.root) - print("\n".join(lines)) - return rc - - -# --------------------------------------------------------------------------- # -# shared helpers for the writing verbs +# the verbs that write a ledger # --------------------------------------------------------------------------- # def _today(args) -> date: if getattr(args, "today", None): @@ -831,200 +506,213 @@ def _today(args) -> date: return date.today() -def _task_at(root: Path, ref: str) -> Task: - """The task file `ref` names — repo-relative, or a path under root.""" - p = Path(ref) - path = p if p.is_absolute() else root / p +def _ledger_at(root: Path, key: str) -> Ledger: + if not PROJECT_KEY_RE.match(key or ""): + raise CortexError(f"{key!r} is not a project key") + path = root / LEDGER_DIR / f"{key}.md" if not path.is_file(): - raise CortexError(f"no task file at {ref}") - try: - path = path.resolve() - path.relative_to(root.resolve()) - except ValueError: - raise CortexError(f"{ref} is outside {root}") - rel_path = root.resolve() / path.relative_to(root.resolve()) - if not TASK_FILE_RE.match(path.relative_to(root.resolve()).as_posix()): - raise CortexError(f"{ref} is not tasks/<project>/<slug>.md") - return Task(root.resolve(), rel_path) - - -def _partition_for(root: Path, project: str, given: "str | None") -> str: + raise CortexError(f"no ledger for {key} — `new {key} --summary ...` opens one") + led = parse_ledger(path.read_text(encoding="utf-8"), key, path, + path.relative_to(root).as_posix()) + if led.problems: + raise CortexError(f"{led.rel} does not parse clean — run `check` first: " + + "; ".join(led.problems)) + return led + + +def _write(led: Ledger, text: str) -> None: + led.path.write_text(text, encoding="utf-8") + after = parse_ledger(text, led.key, led.path, led.rel) + if after.problems: + raise CortexError(f"the edit would not read back: " + "; ".join(after.problems)) + + +def _one_line(text: str, what: str) -> str: + text = _dash((text or "").strip()) + if not text: + raise CortexError(f"{what} must say something") + if "\n" in text: + raise CortexError(f"{what} is one line (continuation lines are for the file)") + return text + + +def _prepend_entry(text: str, entry_line: str) -> str: + body = _section_lines(text, "Log") + body = [ln for ln in body if ln.strip() or True] + stripped = [ln for ln in body] + # drop leading/trailing blanks, then put the new entry first + while stripped and not stripped[0].strip(): + stripped.pop(0) + while stripped and not stripped[-1].strip(): + stripped.pop() + return _replace_section(text, "Log", [entry_line] + stripped) + + +def add_entry(root: Path, key: str, kind: str, text: str, today: date) -> str: + if kind not in KINDS: + raise CortexError(f"--kind must be one of {', '.join(KINDS)}") + text = _one_line(text, "the entry") + led = _ledger_at(root, key) + raw = led.path.read_text(encoding="utf-8") + if led.log and today.isoformat() < led.log[0].date: + raise CortexError(f"the log's head is {led.log[0].date}; an entry dated " + f"{today.isoformat()} would break newest-first") + _write(led, _prepend_entry(raw, f"- {today.isoformat()} — {kind} — {text}")) + return f"{led.rel}: {kind} — {text}" + + +def set_now(root: Path, key: str, text: str) -> str: + text = _dash((text or "").strip()) + if not text: + raise CortexError("`now` must say something") + led = _ledger_at(root, key) + raw = led.path.read_text(encoding="utf-8") + _write(led, _replace_section(raw, "Now", text.split("\n"))) + return f"{led.rel}: now — {text.splitlines()[0]}" + + +def _partition_for(row: dict, given: "str | None") -> str: if given: if not PARTITION_RE.match(given): - raise CortexError(f"--partition '{given}' is not a bare partition name") + raise CortexError(f"--partition must be a bare word, not {given!r}") return given - projects, problems = load_projects(root) - row = projects.get(project) - if row is None: - raise CortexError(f"project {project} is not a projects.yaml key") - part = row.get("partition") + part = row.get("partition", "") if part in ("gpu", "ral"): return part - raise CortexError(f"project {project} may use either partition (`both`) — pass --partition") - - -def _run_line(ident: str, state: str, partition: str, day: date, wall: str = "0:00", - note: str = "") -> str: - line = f"- {ident}: {state} — {partition} — submitted {day.isoformat()} — wall {wall}" - return f"{line} — {note}" if note else line - - -def _append_run(tk: Task, ident: str, state: str, partition: str, day: date, *, - note: str = "", cont: "dict[str, str] | None" = None) -> str: - """Task text with one run line (and its continuations) appended to - `## Runs` and the `Runs:` header re-derived from the body.""" - if not RUN_IDENT_RE.match(ident): - raise CortexError(f"--run '{ident}' is not <stem>[_<task>|_[<set>]]") - probe = Run(RUN_LINE_RE.match(_run_line(ident, state, partition, day)), 0) - for r in tk.runs: - if r.stem == probe.stem and _overlaps(r.task_set(), probe.task_set()): - raise CortexError(f"run {ident} overlaps {r.ident} already on this task") - idents = {r.ident for r in tk.runs} - for key, value in (cont or {}).items(): - if key in ("after", "resumes") and value not in idents: - raise CortexError(f"{key}: {value} names no run of this task") - new_lines = [_run_line(ident, state, partition, day, note=note)] - new_lines += [f" {k}: {v}" for k, v in (cont or {}).items()] - text = append_to_section(tk.text, "Runs", new_lines) - stems = [] - for r in tk.runs + [probe]: - if r.stem not in stems: - stems.append(r.stem) - return edit_header(text, {"Runs": ", ".join(stems)}) - - -def _add_pulled_to(tk: Task, text: str, pulled_to: "str | None", states: "set[str]") -> str: - """Give every run in `states` a `pulled_to:` it lacks — `pulled_to` if - given, else the run's own `where:` (a legacy run's quarantine path is - where its results already are). Refuse when no run would carry one, so - `move pulled` never writes a state `check` rejects.""" - targets = [r for r in tk.runs if r.state in states] - if not targets: - raise CortexError(f"no {' | '.join(sorted(states))} run to pull — nothing to review " - "(rule drop, or fix the run lines)") - lacking = [r for r in targets if "pulled_to" not in r.cont] - if not lacking or (not pulled_to and all("where" not in r.cont for r in lacking)): - if any("pulled_to" in r.cont for r in targets): - return text - raise CortexError("no run carries pulled_to: — pass --pulled-to <laptop path>") - lines = text.split("\n") - for r in sorted(lacking, key=lambda x: x.lineno, reverse=True): - path = pulled_to or r.cont.get("where") - if not path: - continue - i = r.lineno # 1-based run line → index of the line after it - while i < len(lines) and RUN_CONT_RE.match(_dash(lines[i])): - i += 1 - lines.insert(i, f" pulled_to: {path}") - return "\n".join(lines) + raise CortexError("the project runs on both partitions — say which with --partition") + + +def _runs_body(runs: "list[Run]") -> "list[str]": + out: "list[str]" = [] + for r in runs: + out.append(r.line()) + out += [f" {c}" for c in r.cont] + return out + + +def add_run(root: Path, key: str, ident: str, what: str, today: date, *, + partition: "str | None" = None, projects: "dict | None" = None) -> str: + if not RUN_IDENT_RE.match(ident or ""): + raise CortexError(f"{ident!r} is not a SLURM job id (`342301`, `342301_3`, " + "`342301_[0-9]`)") + what = _one_line(what, "what the run is") + led = _ledger_at(root, key) + if any(r.ident == ident for r in led.runs): + raise CortexError(f"{ident} is already listed under ## Runs") + rows = projects if projects is not None else load_projects(root)[0] + part = _partition_for(rows.get(key, {}), partition) + raw = led.path.read_text(encoding="utf-8") + line = f"- {ident} — open — {part} — {today.isoformat()} — {what}" + text = _replace_section(raw, "Runs", _runs_body(led.runs) + [line]) + text = _prepend_entry(text, f"- {today.isoformat()} — run — {ident} submitted: {what}") + _write(led, text) + return f"{led.rel}: run {ident} open" + + +def _run_at(led: Ledger, ident: str) -> Run: + for r in led.runs: + if r.ident == ident: + return r + raise CortexError(f"{ident} is not under ## Runs of {led.rel} " + f"({', '.join(r.ident for r in led.runs) or 'no runs'})") + + +def set_running(root: Path, key: str, ident: str) -> str: + led = _ledger_at(root, key) + run = _run_at(led, ident) + if run.state == "running": + return f"{led.rel}: run {ident} already running" + run.state = "running" + raw = led.path.read_text(encoding="utf-8") + _write(led, _replace_section(raw, "Runs", _runs_body(led.runs))) + return f"{led.rel}: run {ident} running" + + +def finish_run(root: Path, key: str, ident: str, today: date, *, failed: bool = False, + wall: "str | None" = None, note: str = "") -> str: + if wall and not WALL_RE.match(wall): + raise CortexError(f"--wall must be H:MM, not {wall!r}") + led = _ledger_at(root, key) + run = _run_at(led, ident) + verb = "failed" if failed else "finished" + bits = [f"{ident} {verb}"] + if wall: + bits.append(f"wall {wall}") + text = " — ".join(bits) + f": {run.what}" + if note: + text += f" — {_one_line(note, '--note')}" + remaining = [r for r in led.runs if r.ident != ident] + raw = led.path.read_text(encoding="utf-8") + out = _replace_section(raw, "Runs", _runs_body(remaining)) + out = _prepend_entry(out, f"- {today.isoformat()} — run — {text}") + _write(led, out) + return f"{led.rel}: run {ident} {verb}" + + +def new_ledger(root: Path, key: str, summary: str, today: date, *, issue: str = "none", + now: str = "") -> str: + projects, problems = load_projects(root) + if problems: + raise CortexError("projects.yaml does not parse clean — run `check` first: " + + "; ".join(problems)) + if key not in projects: + raise CortexError(f"{key} is not a projects.yaml key — add its row first " + "(projects.yaml is code: a human's turn)") + summary = _one_line(summary, "--summary") + if len(summary.split()) > SUMMARY_MAX_WORDS: + raise CortexError(f"--summary is over {SUMMARY_MAX_WORDS} words") + issue = (issue or "none").strip() + if not ISSUE_RE.match(issue): + raise CortexError(f"--issue must be Repo#N, an issue URL or none, not {issue!r}") + path = root / LEDGER_DIR / f"{key}.md" + if path.exists(): + raise CortexError(f"{path.relative_to(root).as_posix()} already exists") + path.parent.mkdir(parents=True, exist_ok=True) + now = _dash((now or "").strip()) or "Just opened — nothing submitted yet." + path.write_text(TEMPLATE.format(key=key, summary=summary, issue=issue, now=now, + today=today.isoformat()), encoding="utf-8") + return f"opened {path.relative_to(root).as_posix()}" # --------------------------------------------------------------------------- # -# move +# the issue-top block # --------------------------------------------------------------------------- # -def move_task(root: Path, ref: str, to: str, *, run: "str | None" = None, - reason: "str | None" = None, - partial: bool = False, partition: "str | None" = None, - after: "str | None" = None, resumes: "str | None" = None, - note: str = "", pulled_to: "str | None" = None, - today: "date | None" = None) -> str: - """Apply one edge of the transition table; return a one-line summary.""" - today = today or date.today() - tk = _task_at(root, ref) - cur = tk.state - if cur not in TASK_STATES: - raise CortexError(f"{tk.rel} has State: '{cur}', which is not a task state") - if to not in TASK_STATES: - raise CortexError(f"'{to}' is not a task state ({' | '.join(TASK_STATES)})") - if cur == "dropped": - raise CortexError(f"{tk.rel} is dropped — terminal; revival is a new slug") - if to == "dropped": - raise CortexError("dropped is a ruling edge: `cortex.py rule <task> drop`") - if to in ("accepted", "rerun"): - raise CortexError(f"{to} is a ruling edge: `cortex.py rule <task> {VERB_FOR_STATE[to]}`" - + (" --supersedes <Ruling:>" if cur == "accepted" else "")) - refs, bad = gate_refs(tk.get("Gates")) - if bad: - raise CortexError(f"{tk.rel}: Gates: unrecognised ref '{bad[0]}' — fix it first") - updates: "dict[str, str | None]" = {} - text = tk.text - cont = {} - if after: - cont["after"] = after - if resumes: - cont["resumes"] = resumes - - if cur == to: - if cur in ("submitted", "running") and run: - part = _partition_for(root, tk.project_dir, partition) - text = _append_run(tk, run, "submitted", part, today, note=note, cont=cont) - tk.path.write_text(text, encoding="utf-8") - return f"{tk.rel}: {cur} — appended run {run}" - raise CortexError(f"{tk.rel} is already {cur}" - + ("" if cur in ("submitted", "running") else - "; --run is for submitted | running tasks")) - edge = (cur, to) - if run and edge not in (("ready", "submitted"), ("submitted", "running")): - raise CortexError("--run applies to ready → submitted and to a submitted | running task") - if edge == ("planned", "gated"): - if not refs: - raise CortexError(f"{tk.rel}: Gates: is empty — planned → ready") - elif edge == ("planned", "ready"): - if refs: - raise CortexError(f"{tk.rel}: Gates: is non-empty — planned → gated") - elif edge == ("gated", "ready"): - pass # the human read `gates` and judged the refs cleared - elif edge == ("ready", "gated"): - raise CortexError("ready → gated is a hand edit of the header — re-gating a " - "ready task is a judgement, not an edge") - elif edge == ("ready", "submitted"): - if not tk.get("Witness"): - raise CortexError(f"{tk.rel}: Witness: is empty — register the witness before submitting") - if not run: - raise CortexError("ready → submitted needs --run <id>") - part = _partition_for(root, tk.project_dir, partition) - text = _append_run(tk, run, "submitted", part, today, note=note, cont=cont) - elif edge == ("ready", "pulled"): - if not tk.runs or not all(r.state in LEGACY_RUN_STATES for r in tk.runs): - raise CortexError("ready → pulled is for a legacy-born task: every run line " - "legacy | legacy_wrong") - if not tk.get("Witness"): - raise CortexError(f"{tk.rel}: Witness: is empty — still mandatory for a legacy-born task") - text = _add_pulled_to(tk, text, pulled_to, {"legacy"}) - elif edge == ("submitted", "running"): - if run: - part = _partition_for(root, tk.project_dir, partition) - text = _append_run(tk, run, "submitted", part, today, note=note, cont=cont) - elif edge in (("submitted", "ready"), ("running", "ready")): - if any(r.state in LIVE_RUN_STATES for r in tk.runs): - raise CortexError(f"{tk.rel}: a run is still submitted | running") - if not any(r.state in RESET_RUN_STATES for r in tk.runs): - raise CortexError(f"{tk.rel}: no failed | timeout | void run to reset from") - if not reason: - raise CortexError(f"{cur} → ready needs --reason") - updates["Reset"] = reason - elif edge == ("running", "pulled"): - if any(r.state in LIVE_RUN_STATES for r in tk.runs) and not partial: - raise CortexError(f"{tk.rel}: a run is still submitted | running — " - "--partial for a partial array (then rule leave-to-finish)") - text = _add_pulled_to(tk, text, pulled_to, {"done"}) - elif edge in (("pulled", "awaiting-ruling"), ("rerun", "ready")): - pass +ISSUE_BEGIN = "<!-- cortex:ledger begin — regenerated from projects/{key}.md; edit there -->" +ISSUE_END = "<!-- cortex:ledger end -->" + + +def issue_url(ref: str) -> str: + if ref.startswith("https://"): + return ref + repo, _, n = ref.partition("#") + return f"https://github.com/{DEFAULT_ISSUE_OWNER}/{repo}/issues/{n}" + + +def issue_block(led: Ledger, row: dict, n: int = LOG_WINDOW, home: str = "") -> str: + """The concise, human-readable ledger that sits at the top of a project's + issue: Now, the runs on the cluster, the last `n` entries. Markdown, fenced + by the two markers so a re-sync replaces exactly this block.""" + link = f"{home}/blob/main/{led.rel}" if home else led.rel + out = [ISSUE_BEGIN.format(key=led.key), + f"**{led.key}** — {led.summary} ", + f"_{row.get('status', '?')} · ledger: [{led.rel}]({link})" + + (f" · updated {led.updated()}" if led.updated() else "") + "_", + "", "**Now**", "", led.now or "_(nothing yet)_", "", "**Runs**", ""] + if led.runs: + out += [f"- `{r.ident}` — {r.state} — {r.partition} — {r.date} — {r.text()}" + for r in led.runs] else: - raise CortexError(f"no edge {cur} → {to} in the transition table") - updates["State"] = to - tk.path.write_text(edit_header(text, updates), encoding="utf-8") - return f"{tk.rel}: {cur} → {to}" + out.append("_nothing on the cluster_") + out += ["", f"**Last {n}**", ""] + out += [f"- {e.date} — *{e.kind}* — {e.text()}" for e in led.recent(n)] or ["_empty_"] + out.append(ISSUE_END) + return "\n".join(out) + "\n" -VERB_FOR_STATE = {"accepted": "accept", "rerun": "rerun", "dropped": "drop"} - - -def cmd_move(args) -> int: - print(move_task(args.root, args.task, args.state, run=args.run, reason=args.reason, - partial=args.partial, partition=args.partition, - after=args.after, resumes=args.resumes, note=args.note or "", - pulled_to=args.pulled_to, today=_today(args))) +def cmd_issue(args) -> int: + led = _ledger_at(args.root, args.project) + projects, _ = load_projects(args.root) + print(issue_block(led, projects.get(args.project, {}), args.n), end="") return 0 @@ -1033,22 +721,10 @@ def cmd_move(args) -> int: # --------------------------------------------------------------------------- # def retire_project(root: Path, key: str, why: str, today: date) -> str: """Flip one `projects.yaml` row to `status: retired` and stamp the reason - on its `note:`. The only verb that writes `projects.yaml`. - - Three things it deliberately does not do. It does not **delete the row**: - that row is the only record of where the project's data lives, and a - retired project still has to be findable. It does not touch a **task** or - a **ruling**: `rulings/` is append-only and history is not rewritten by a - change of status. And it does not retire over **live work** — every state - outside `RULED_STATES` and `planned` is an unfinished question, so the - refusal names each one and the human rules or drops it first. `planned` - stays: an unasked question costs nothing to leave behind. - - The edit is two lines. Everything else in the file — comments, blank - lines, the order of the rows, the other rows' bytes — is preserved, and - the result is re-parsed before it is kept: a `projects.yaml` this verb - could not read back is restored to the bytes it had. - """ + on its `note:`. The only verb that writes `projects.yaml`. It does not + delete the row (it is the one record of where the data lives), it does not + touch the ledger file (history is not rewritten by a change of status), and + it refuses while the ledger still lists a run on the cluster.""" why = (why or "").strip() if not why: raise CortexError("--why must say why, in one line") @@ -1063,21 +739,20 @@ def retire_project(root: Path, key: str, why: str, today: date) -> str: raise CortexError(f"{key} is not a projects.yaml key") if projects[key].get("status") == "retired": raise CortexError(f"{key} is already retired") - tasks, _ = load_tasks(root) - live = [f"{tk.rel} — {tk.state}" for tk in tasks - if tk.project_dir == key - and tk.state not in RULED_STATES | {"planned"}] - if live: - raise CortexError(f"{key} still has live work — rule or drop it " - f"first: {', '.join(live)}") + path_led = root / LEDGER_DIR / f"{key}.md" + if path_led.is_file(): + led = parse_ledger(path_led.read_text(encoding="utf-8"), key, path_led, + path_led.relative_to(root).as_posix()) + if led.runs: + raise CortexError(f"{key} still lists runs on the cluster — `done` them " + f"first: {', '.join(r.ident for r in led.runs)}") path = root / "projects.yaml" original = path.read_text(encoding="utf-8") lines = original.split("\n") starts = [i for i, ln in enumerate(lines) if ln == f"{key}:"] if len(starts) != 1: - raise CortexError(f"projects.yaml: expected one `{key}:` line, " - f"found {len(starts)}") + raise CortexError(f"projects.yaml: expected one `{key}:` line, found {len(starts)}") start = starts[0] stop = start + 1 while stop < len(lines) and lines[stop].startswith(" "): @@ -1103,257 +778,52 @@ def retire_project(root: Path, key: str, why: str, today: date) -> str: raise CortexError("the edit would not read back — projects.yaml is " "unchanged: " + ("; ".join(problems) or f"{key} did not come back retired")) + if path_led.is_file(): + add_entry(root, key, "note", f"retired: {why}", today) return f"retired {key}" -def cmd_retire(args) -> int: - print(retire_project(args.root, args.project, args.why, _today(args))) +# --------------------------------------------------------------------------- # +# CLI +# --------------------------------------------------------------------------- # +def cmd_new(args) -> int: + print(new_ledger(args.root, args.project, args.summary, _today(args), + issue=args.issue, now=args.now or "")) return 0 -# --------------------------------------------------------------------------- # -# rule -# --------------------------------------------------------------------------- # -def next_ruling_id(root: Path, day: date, taken: "set[str] | None" = None) -> str: - stamp = day.strftime("%Y%m%d") - existing = set(ruling_files(root)) | (taken or set()) - for n in range(1, 100): - rid = f"R-{stamp}-{n:02d}" - if rid not in existing: - return rid - raise CortexError(f"99 rulings already filed on {day.isoformat()}") - - -def _ruling_target(tk: Task, verb: str, supersedes: "str | None", by_id: "dict[str, Ruling]", - successors: "dict[str, list[str]]") -> "str | None": - """The task state `verb` writes, or raise if the table forbids it.""" - state = tk.state - if verb not in RULING_VERBS: - raise CortexError(f"'{verb}' is not a ruling verb ({' | '.join(RULING_VERBS)})") - if state == "dropped": - raise CortexError(f"{tk.rel} is dropped — terminal") - if supersedes: - if supersedes not in by_id: - raise CortexError(f"--supersedes {supersedes} does not resolve to a ruling file") - old = by_id[supersedes] - if old.get("Task") != tk.rel: - raise CortexError(f"--supersedes {supersedes} rules on {old.get('Task')}, not {tk.rel}") - if successors.get(supersedes): - raise CortexError(f"{supersedes} already has a successor " - f"({successors[supersedes][0]}) — supersede the head") - if state == "accepted": - if verb not in ("rerun", "drop"): - raise CortexError(f"accepted takes only rerun | drop, with --supersedes {tk.get('Ruling')}") - if supersedes != tk.get("Ruling"): - raise CortexError(f"accepted → {VERB_TARGET[verb]} needs --supersedes {tk.get('Ruling')} " - "(the REWIND case)") - return VERB_TARGET[verb] - if verb == "leave-to-finish": - if state not in ("running", "pulled", "awaiting-ruling"): - raise CortexError(f"leave-to-finish applies to running | pulled | awaiting-ruling, " - f"not {state}") - return None - if state == "awaiting-ruling": - return VERB_TARGET[verb] - if verb == "drop": - return "dropped" - raise CortexError(f"'{verb}' is not allowed from State: {state} " - f"(the table: awaiting-ruling → {VERB_TARGET[verb]})") - - -def rule_task(root: Path, ref: str, verb: str, body: str, *, supersedes: "str | None" = None, - batch: "str | None" = None, minutes: "int | None" = None, - follow_ups: "tuple[str, ...]" = (), - today: "date | None" = None, now: "datetime | None" = None) -> "list[str]": - """File the ruling for one task and update its `Ruling:` and `State:`; - return the ruling paths written (a list of one). - Everything is validated before anything is written.""" - today = today or date.today() - now = now or datetime.now(timezone.utc) - root = root.resolve() - rulings, _ = load_rulings(root) - by_id = {r.id: r for r in rulings} - successors: "dict[str, list[str]]" = {} - for r in rulings: - if r.get("Supersedes"): - successors.setdefault(r.get("Supersedes"), []).append(r.id) - if batch and not (root / "batches" / f"{batch}.md").is_file(): - raise CortexError(f"--batch {batch} names no batches/{batch}.md") - _, bad = gate_refs(", ".join(follow_ups)) - if bad: - raise CortexError(f"--follow-up '{bad[0]}' is not Repo#N or an issue/PR URL " - "(create the issue first)") - body = body.strip("\n") - if not body.strip(): - raise CortexError("--body is empty — the human's words, verbatim") - - tk = _task_at(root, ref) - plan = [(tk, supersedes, _ruling_target(tk, verb, supersedes, by_id, successors))] - - taken: "set[str]" = set() - written = [] - for tk, sup, new_state in plan: - rid = next_ruling_id(root, today, taken) - taken.add(rid) - if sup and not sup < rid: - raise CortexError(f"{sup} is not earlier than the new id {rid} (--today?)") - path = root / "rulings" / rid[2:6] / rid[6:8] / f"{rid}.md" - if path.exists(): - raise CortexError(f"refusing to touch an existing ruling: {path.relative_to(root)}") - plan_item = (tk, sup, new_state, rid, path) - written.append(plan_item) - - stamp = f"{today.isoformat()}T{now.strftime('%H:%M')}Z" - out = [] - for tk, sup, new_state, rid, path in written: - header = [f"Project: {tk.get('Project')}", f"Task: {tk.rel}", - f"Runs: {tk.get('Runs')}" if tk.get("Runs") else "Runs:", - f"Ruling: {verb}"] - if sup: - header.append(f"Supersedes: {sup}") - if batch: - header.append(f"Batch: {batch}") - header.append(f"Reviewed-at: {stamp}") - if minutes is not None: - header.append(f"Review-minutes-actual: {minutes}") - if follow_ups: - header.append(f"Follow-ups: {', '.join(follow_ups)}") - evidence = _where_to_look(tk) or ["- (none given)"] - text = "\n".join([f"# {rid} — {verb} {tk.get('Project')} {tk.slug}", ""] - + header + ["", "## Ruling", "", body, "", "## Evidence", ""] - + evidence) + "\n" - path.parent.mkdir(parents=True, exist_ok=True) - path.write_text(text, encoding="utf-8") - updates: "dict[str, str | None]" = {"Ruling": rid} - if new_state: - updates["State"] = new_state - line = f"{rid} — {verb}" + (f" (supersedes {sup})" if sup else "") - ptext = append_to_section(tk.text, "Ruling", [line], replace_placeholder="(none)") - tk.path.write_text(edit_header(ptext, updates), encoding="utf-8") - out.append(path.relative_to(root).as_posix()) - return out +def cmd_run(args) -> int: + print(add_run(args.root, args.project, args.jobid, args.what, _today(args), + partition=args.partition)) + return 0 -def _where_to_look(tk: Task) -> "list[str]": - span = sections(tk.text).get("Where to look") - if span is None: - return [] - lines = tk.text.split("\n")[span[0]:span[1]] - return [ln for ln in lines if ln.startswith("- ") and ln.strip() != "-"] - - -def cmd_rule(args) -> int: - body_path = Path(args.body) - if not body_path.is_file(): - raise CortexError(f"--body {args.body} is not a file") - paths = rule_task(args.root, args.task, args.verb, body_path.read_text(encoding="utf-8"), - supersedes=args.supersedes, batch=args.batch, minutes=args.minutes, - follow_ups=tuple(args.follow_up or ()), today=_today(args)) - for p in paths: - print(f"wrote {p}") +def cmd_running(args) -> int: + print(set_running(args.root, args.project, args.jobid)) return 0 -# --------------------------------------------------------------------------- # -# new -# --------------------------------------------------------------------------- # -def new_task(root: Path, project: str, slug: str, summary: str, *, gates: str = "", - epic: str = "", legacy_runs: "tuple[str, ...]" = (), - legacy_wrong: "tuple[str, ...]" = (), where: "str | None" = None, - partition: "str | None" = None, witness: str = "", budget: str = "", - minutes: "int | None" = None, title: "str | None" = None, - today: "date | None" = None) -> str: - """Write tasks/<project>/<slug>.md from the template; return its rel path.""" - today = today or date.today() - projects, problems = load_projects(root) - if project not in projects: - raise CortexError(f"project {project} is not a projects.yaml key" - + (f" ({problems[0]})" if problems else "")) - if not SLUG_RE.match(slug): - raise CortexError(f"slug '{slug}' must match {SLUG_RE.pattern}") - summary = " ".join((summary or "").split()) - if not summary: - raise CortexError("--summary must say what the task asks, in one line") - if len(summary.split()) > SUMMARY_MAX_WORDS: - raise CortexError(f"--summary is {len(summary.split())} words — at most " - f"{SUMMARY_MAX_WORDS} (the question the task answers)") - # The slug IS the identity: the path is unique per project, so there is - # nothing else to collide on. - path = root / "tasks" / project / f"{slug}.md" - if path.exists(): - raise CortexError(f"{path.relative_to(root).as_posix()} already exists") - _, bad = gate_refs(gates) - if bad: - raise CortexError(f"--gates '{bad[0]}' is not Repo#N or an issue/PR URL (no owner/Repo#N form)") - legacy = [(r, "legacy") for r in legacy_runs] + [(r, "legacy_wrong") for r in legacy_wrong] - if legacy: - if gates.strip(): - raise CortexError("a legacy-born task cannot be gated — its runs already happened") - if not where: - raise CortexError("--legacy-run needs --where <quarantine path> (check requires where:)") - for ident, _ in legacy: - if not RUN_IDENT_RE.match(ident): - raise CortexError(f"legacy run '{ident}' is not <stem>[_<task>|_[<set>]]") - part = _partition_for(root, project, partition) - if budget and not WALL_RE.match(budget): - raise CortexError(f"--budget '{budget}' is not H+:MM") - state = "ready" if legacy else "planned" - stems = [] - for ident, _ in legacy: - stem = ident.split("_")[0] - if stem not in stems: - stems.append(stem) - words = slug.replace("_", " ").replace("-", " ") - title = title or words - header = [ - f"Project: {project}", f"Summary: {summary}", f"State: {state}", - f"Gates: {gates.strip()}" if gates.strip() else "Gates:", - f"Witness: {witness}" if witness else "Witness:", - f"Budget: {budget}" if budget else "Budget:", - f"Runs: {', '.join(stems)}" if stems else "Runs:", - "Ruling:", - f"Review-minutes: {minutes}" if minutes is not None else "Review-minutes:", - f"Epic: {epic}" if epic else "Epic:", f"Filed: {today.isoformat()}", - ] - run_lines = [] - for ident, rstate in legacy: - run_lines.append(_run_line(ident, rstate, part, today, note="pre-Cortex run, migrated")) - run_lines.append(f" where: {where}") - body = [ - f"# {project.capitalize()} — {title}", "", - *header, "", - "## Question", "", "(the question this task answers)", "", - "## Witness", "", witness or "(not yet registered — a planned task may leave this empty)", "", - # A legacy-born task already knows where to look — `--where` is the - # quarantine path its runs landed in, and it is required for one — so - # the section names it rather than the placeholder `check` refuses on - # anything past `planned`. - "## Where to look", "", - f"- `{where}`" if where else f"- {WHERE_PLACEHOLDER}", "", - "## Runs", "", - ] - if run_lines: - body += run_lines + [""] - body += ["## Ruling", "", "(none)", ""] - path.parent.mkdir(parents=True, exist_ok=True) - path.write_text("\n".join(body), encoding="utf-8") - return path.relative_to(root).as_posix() +def cmd_done(args) -> int: + print(finish_run(args.root, args.project, args.jobid, _today(args), + failed=args.failed, wall=args.wall, note=args.note or "")) + return 0 -def cmd_new(args) -> int: - rel = new_task(args.root, args.project, args.slug, args.summary, gates=args.gates or "", - epic=args.epic or "", legacy_runs=tuple(args.legacy_run or ()), - legacy_wrong=tuple(args.legacy_wrong or ()), where=args.where, - partition=args.partition, witness=args.witness or "", - budget=args.budget or "", minutes=args.minutes, title=args.title, - today=_today(args)) - print(f"wrote {rel}") +def cmd_log(args) -> int: + print(add_entry(args.root, args.project, args.kind, args.text, _today(args))) + return 0 + + +def cmd_now(args) -> int: + print(set_now(args.root, args.project, args.text)) + return 0 + + +def cmd_retire(args) -> int: + print(retire_project(args.root, args.project, args.why, _today(args))) return 0 -# --------------------------------------------------------------------------- # -# main -# --------------------------------------------------------------------------- # def _common(p: argparse.ArgumentParser, dated: bool = False) -> None: p.add_argument("--root", type=Path, default=ROOT, help="the Cortex tree to operate on (default: this checkout)") @@ -1362,72 +832,68 @@ def _common(p: argparse.ArgumentParser, dated: bool = False) -> None: def build_parser() -> argparse.ArgumentParser: - parser = argparse.ArgumentParser( - prog="cortex.py", description=__doc__.split("\n\n")[0], - formatter_class=argparse.RawDescriptionHelpFormatter) + parser = argparse.ArgumentParser(prog="cortex", description=__doc__.split("\n")[0]) sub = parser.add_subparsers(dest="command", required=True) - c = sub.add_parser("check", help="every rule in REFERENCE.md; exit 1 on drift") - _common(c) - c.set_defaults(func=cmd_check) - - g = sub.add_parser("gates", help="list every gated task, its refs and their URLs") - _common(g) - g.set_defaults(func=cmd_gates) - - r = sub.add_parser("rule", help="file a ruling and move the task per the table") - _common(r, dated=True) - r.add_argument("task", help="tasks/<project>/<slug>.md") - r.add_argument("verb", choices=RULING_VERBS) - r.add_argument("--body", required=True, help="file holding the human's words, verbatim") - r.add_argument("--supersedes", help="the ruling id this one replaces (the chain head)") - r.add_argument("--batch", help="the <YYYY-MM-DD>-<slot> the ruling was filed from") - r.add_argument("--minutes", type=int, help="Review-minutes-actual") - r.add_argument("--follow-up", action="append", metavar="REF", - help="Repo#N or an issue/PR URL; repeatable; the issue exists already") - r.set_defaults(func=cmd_rule) - - m = sub.add_parser("move", help="one edge of the transition table") - _common(m, dated=True) - m.add_argument("task", help="tasks/<project>/<slug>.md") - m.add_argument("state", help="the state to move to") - m.add_argument("--run", metavar="ID", help="SLURM job id (<stem>[_<task>|_[<set>]]) to append") - m.add_argument("--reason", help="why a submitted | running task goes back to ready (Reset:)") - m.add_argument("--partial", action="store_true", - help="running → pulled with a run still live (needs a leave-to-finish ruling)") - m.add_argument("--partition", help="the run line's partition (default: the project's row)") - m.add_argument("--after", metavar="RUN", help="afterok dependency of the appended run") - m.add_argument("--resumes", metavar="RUN", help="the run the appended run resumes from") - m.add_argument("--note", help="free text after the appended run line's fourth dash") - m.add_argument("--pulled-to", metavar="PATH", - help="→ pulled: the laptop path written as pulled_to: on each done | legacy " - "run lacking one (default for a legacy run: its where:)") - m.set_defaults(func=cmd_move) - - n = sub.add_parser("new", help="write tasks/<project>/<slug>.md from the template") - _common(n, dated=True) - n.add_argument("project") - n.add_argument("slug") - n.add_argument("--summary", required=True, - help=f"the question the task answers, at most {SUMMARY_MAX_WORDS} words") - n.add_argument("--gates", help="comma-separated Repo#N or issue/PR URLs") - n.add_argument("--epic", help="the epic slug shared with the Mind") - n.add_argument("--legacy-run", action="append", metavar="ID", help="a pre-Cortex run, reusable") - n.add_argument("--legacy-wrong", action="append", metavar="ID", help="a pre-Cortex run, not reusable") - n.add_argument("--where", help="quarantine path written to each legacy run's where:") - n.add_argument("--partition", help="the legacy runs' partition (default: the project's row)") - n.add_argument("--witness", help="the pre-registered checkable claim") - n.add_argument("--budget", help="wall budget per run, H+:MM") - n.add_argument("--minutes", type=int, help="Review-minutes seed") - n.add_argument("--title", help="the title after `# <Project> — ` (default: the slug's words)") - n.set_defaults(func=cmd_new) - - t = sub.add_parser("retire", help="flip a project's row to status: retired") - _common(t, dated=True) - t.add_argument("project", help="the projects.yaml key") - t.add_argument("--why", required=True, - help="one line: why the project is being retired") - t.set_defaults(func=cmd_retire) + p = sub.add_parser("check", help="every structural rule; OK or DRIFT") + _common(p) + p.set_defaults(func=cmd_check) + + p = sub.add_parser("new", help="open a project's ledger") + p.add_argument("project") + p.add_argument("--summary", required=True, help=f"one line, at most {SUMMARY_MAX_WORDS} words") + p.add_argument("--issue", default="none", help="Repo#N or an issue URL (default none)") + p.add_argument("--now", help="the opening `## Now` text") + _common(p, dated=True) + p.set_defaults(func=cmd_new) + + p = sub.add_parser("run", help="record a submission: the run goes under ## Runs as open") + p.add_argument("project") + p.add_argument("jobid") + p.add_argument("what", help="what the run is, one line") + p.add_argument("--partition", help="gpu | ral (needed when the project runs on both)") + _common(p, dated=True) + p.set_defaults(func=cmd_run) + + p = sub.add_parser("running", help="a run has started on the cluster") + p.add_argument("project") + p.add_argument("jobid") + _common(p) + p.set_defaults(func=cmd_running) + + p = sub.add_parser("done", help="a run has finished: it leaves ## Runs and enters the log") + p.add_argument("project") + p.add_argument("jobid") + p.add_argument("--failed", action="store_true") + p.add_argument("--wall", help="H:MM") + p.add_argument("--note", help="one line to carry into the entry") + _common(p, dated=True) + p.set_defaults(func=cmd_done) + + p = sub.add_parser("log", help="write a dated entry in the human's words") + p.add_argument("project") + p.add_argument("text") + p.add_argument("--kind", default="note", choices=KINDS) + _common(p, dated=True) + p.set_defaults(func=cmd_log) + + p = sub.add_parser("now", help="rewrite ## Now") + p.add_argument("project") + p.add_argument("text") + _common(p) + p.set_defaults(func=cmd_now) + + p = sub.add_parser("issue", help="print the block that sits at the top of the project's issue") + p.add_argument("project") + p.add_argument("-n", type=int, default=LOG_WINDOW) + _common(p) + p.set_defaults(func=cmd_issue) + + p = sub.add_parser("retire", help="retire a project's row (the only verb that writes projects.yaml)") + p.add_argument("project") + p.add_argument("--why", required=True) + _common(p, dated=True) + p.set_defaults(func=cmd_retire) return parser diff --git a/scripts/ledger_merge.py b/scripts/ledger_merge.py index 49722bb..d63a878 100755 --- a/scripts/ledger_merge.py +++ b/scripts/ledger_merge.py @@ -7,9 +7,9 @@ it: no workflow looks at a `claude/**` push. The branch sits there until a human writes an explicit "merge this" prompt. -Almost all of what strands is *ledger*: a task file under `tasks/`, a ruling -under `rulings/`, a batch record or review under `batches/`, a check-in stamp. -It is the organism's own bookkeeping, it is template-shaped, its drift check +Almost all of what strands is *ledger*: a project's ledger under `projects/` +(a run recorded, an entry logged, `## Now` rewritten), a check-in stamp, the +two generated pages. It is the organism's own bookkeeping, its drift check (`cortex.py check`) is already automated, and a human reviewing it adds nothing. The minority that is *code* — `scripts/`, `tests/`, `.github/`, `policy/`, `docs/`, `projects.yaml`, the prose pages — is exactly what review @@ -23,12 +23,10 @@ one — a new root file, a new top-level folder — is code. Getting that backwards would auto-merge the next thing nobody thought about. -APPEND-ONLY. `rulings/` is the ledger of record and a ruling, once committed, -is never modified or deleted (rulings/AGENTS.md). When the diff comes from git -(`--base`), the classifier reads `git diff --name-status` and any entry under -`rulings/**` whose status is not `A` (added) — `M`, `D`, `R*`, or anything -else — is code, which leaves the branch for a human. Explicit-path and stdin -inputs carry no status and stay path-only, as in the Mind. +FROZEN. `archive/` holds the retired task, ruling and batch ledgers of +2026-08/09 (the science repos cite their ids). Nothing writes there any more, +so *any* change under it — an addition included — is code and waits for a +human. Usage: python3 scripts/ledger_merge.py classify --base origin/main # diff HEAD vs base @@ -48,11 +46,9 @@ import sys from pathlib import Path -# Directories holding nothing but the run-and-ruling ledger: task files, -# rulings of record, and the batch records and reviews kept as history. Their -# whole contents are ledger (subject to the EXCLUDED_NAMES guard below, and to -# the append-only leg for rulings/ and batches/ in classify_entries). -LEDGER_DIRS = ("tasks/", "rulings/", "batches/") +# The one directory holding nothing but ledger: one file per project. Its +# whole contents are ledger, subject to the EXCLUDED_NAMES guard below. +LEDGER_DIRS = ("projects/",) # Root files that are ledger state. Deliberately NOT here: README.md, # AGENTS.md, CLAUDE.md, REFERENCE.md — prose a human reads, changed rarely @@ -76,12 +72,14 @@ # beside a batch record included. # # INSTRUCTIONAL BY CONTENT: `AGENTS.md` and `TEMPLATE.md` under a ledger dir -# (`rulings/AGENTS.md`, `batches/AGENTS.md`, `batches/reviews/AGENTS.md`) are -# not entries in the ledger — they are the doctrine that says what an entry may be, and the +# (`projects/AGENTS.md`) are not entries in the ledger — they are the doctrine that says what an entry may be, and the # template every future entry is stamped from. They are read by agents as # instructions, so a change to one is a change to behaviour: it needs a human, # exactly as `scripts/` does. Auto-merging a rewrite of `rulings/AGENTS.md` # would let a branch edit the rule that governs its own merge. +# +# `archive/` is not in LEDGER_DIRS at all: it is frozen history (the retired +# task / ruling / batch ledgers), so it falls to the default deny. EXCLUDED_NAMES = ("conftest.py", "test_*.py", "*_test.py", "AGENTS.md", "TEMPLATE.md") @@ -117,29 +115,23 @@ def classify(paths): return ledger, blocked -# The dirs where a change's KIND matters, not only its path: the ledger of -# record is append-only, so only an added file under one is ledger. -# `batches/` joined `rulings/` on 2026-09-03, when the review-slot apparatus -# was retired: the batch records and the human's verbatim reviews stay as -# history — never modified, only added — and 13 rulings cite them. -APPEND_ONLY_DIRS = ("rulings/", "batches/") +# Kept as the retired append-only dirs' name, now empty: `archive/` is frozen +# outright (default deny), so no per-status rule is needed any more. A status +# is still read from git so a rename's old path counts as a deletion. +APPEND_ONLY_DIRS = () def is_append_only_violation(status: str, path: str) -> bool: """True if a `git diff --name-status` entry edits, deletes, renames (or - does anything but add) a file under an append-only dir.""" + does anything but add) a file under an append-only dir (none today).""" parts = [p for p in path.replace("\\", "/").split("/") if p not in ("", ".")] normalised = "/".join(parts) return status[:1] != "A" and any(normalised.startswith(d) for d in APPEND_ONLY_DIRS) def classify_entries(entries): - """Split `(status, path)` entries into (ledger, blocked) paths. - - A path is blocked if it is not ledger material, or if it is a non-`A` - entry under an append-only dir (`rulings/`, `batches/` — an edited, - deleted or renamed ruling, batch record or review). A rename entry - carries both paths; the old one counts as a deletion.""" + """Split `(status, path)` entries into (ledger, blocked) paths. A rename + entry carries both paths; the old one counts as a deletion.""" ledger, blocked, seen = [], [], set() for status, path in entries: path = path.strip() diff --git a/tests/fixtures/empty/projects.yaml b/tests/fixtures/empty/projects.yaml index 516b255..9a55b06 100644 --- a/tests/fixtures/empty/projects.yaml +++ b/tests/fixtures/empty/projects.yaml @@ -1 +1 @@ -# Test fixture — an empty science body map. No rows: parses to an empty map. +# an empty body map — no projects, no ledgers diff --git a/tests/fixtures/empty/rulings/.gitkeep b/tests/fixtures/empty/projects/.keep similarity index 100% rename from tests/fixtures/empty/rulings/.gitkeep rename to tests/fixtures/empty/projects/.keep diff --git a/tests/fixtures/empty/tasks/.gitkeep b/tests/fixtures/empty/tasks/.gitkeep deleted file mode 100644 index e69de29..0000000 diff --git a/tests/fixtures/skeleton/batches/2026-09-01-pm.md b/tests/fixtures/skeleton/batches/2026-09-01-pm.md deleted file mode 100644 index c79a606..0000000 --- a/tests/fixtures/skeleton/batches/2026-09-01-pm.md +++ /dev/null @@ -1,10 +0,0 @@ -# Batch 2026-09-01 pm - -Closed history, kept in the fixture because a ruling cites it: the `Batch:` -field of `rulings/2026/09/R-20260901-03.md` must resolve to a file here. - -- reviewed-at: 2026-09-01T17:10Z -- notes: | - 05_running_array's array task 9 OOMed before its first step and was resubmitted alone - (342102); the board showed the other nine. 10_dropped counted as not - delivered — legacy_wrong, nothing to review but the drop. diff --git a/tests/fixtures/skeleton/checkin.yaml b/tests/fixtures/skeleton/checkin.yaml new file mode 100644 index 0000000..471df3b --- /dev/null +++ b/tests/fixtures/skeleton/checkin.yaml @@ -0,0 +1 @@ +refreshed: 2026-09-02T09:00Z diff --git a/tests/fixtures/skeleton/projects.yaml b/tests/fixtures/skeleton/projects.yaml index eb2e69b..191eedf 100644 --- a/tests/fixtures/skeleton/projects.yaml +++ b/tests/fixtures/skeleton/projects.yaml @@ -1,15 +1,57 @@ -# Test fixture — the science body map with one fictional project. -# Same restricted subset as the live projects.yaml (REFERENCE.md). - +# The skeleton fixture: one active project with a ledger, one active project +# on one partition, one retired project whose ledger is closed history, and +# one dormant row with no ledger at all. Every path is a placeholder. example: - remote: none - local_path: /mnt/c/Users/Jammy/Science/example - ral_root: /mnt/ral/jnightin/example - mirror: /mnt/c/Users/Jammy/Science/example/mirror # the pull root + remote: PyAutoLabs/example + local_path: /tmp/example + ral_root: /mnt/ral/example + mirror: none sync_cli: hpc/sync - sync_verbs: [pull, submit, jobs, tail] + sync_verbs: [push, pull, submit, jobs, tail] ledger: wiki/project/state.md + assistant: example_assistant + witness_file: results/**/*.json + partition: both + status: active + note: "the fixture project" + +single: + remote: none + local_path: /tmp/single + ral_root: /mnt/ral/single + mirror: none + sync_cli: hpc/sync + sync_verbs: [pull] + ledger: NOTES.md assistant: none - witness_file: output/**/checkpoint.hdf5 - partition: gpu + witness_file: results/**/*.json + partition: ral status: active + +wound_down: + remote: none + local_path: /tmp/wound_down + ral_root: /mnt/ral/wound_down + mirror: none + sync_cli: hpc/sync + sync_verbs: [pull] + ledger: README.md + assistant: none + witness_file: results/**/*.json + partition: both + status: retired + note: "retired 2026-09-01: done" + +sleeping: + remote: none + local_path: /tmp/sleeping + ral_root: /mnt/ral/sleeping + mirror: none + sync_cli: hpc/sync + sync_verbs: [pull] + ledger: README.md + assistant: none + witness_file: results/**/*.json + partition: both + status: dormant + note: "no ledger yet" diff --git a/tests/fixtures/skeleton/projects/example.md b/tests/fixtures/skeleton/projects/example.md new file mode 100644 index 0000000..022f7f4 --- /dev/null +++ b/tests/fixtures/skeleton/projects/example.md @@ -0,0 +1,26 @@ +# example — Does the example pipeline recover the truth on the fixture lens + +Project: example +Issue: example#7 + +## Now + +Wave 2 is on the cluster: the array 3002_[0-3] is running and the joint fit 3003 is queued. +Next: pull wave 2 when 3002 lands and compare the parent sigma against wave 1. + +## Runs + +- 3002_[0-3] — running — ral — 2026-09-01 — wave 2, four seeds on the corrected settings +- 3003 — open — gpu — 2026-09-02 — joint fit on the same four seeds + chained after 3002 with afterok + +## Log + +- 2026-09-02 — run — 3003 submitted: joint fit on the same four seeds +- 2026-09-01 — run — 3002_[0-3] submitted: wave 2, four seeds on the corrected settings +- 2026-09-01 — lesson — the adapt image must be capped at S/N 3 before the mesh reads it; wave 1 was not +- 2026-08-31 — result — wave 1 recovered the parent mean but its sigma is 4x too tight + the widths are the finding, not the means +- 2026-08-30 — run — 3001_[0-3] finished — wall 12:10: wave 1, four seeds +- 2026-08-29 — run — 3001_[0-3] submitted: wave 1, four seeds +- 2026-08-28 — note — ledger opened diff --git a/tests/fixtures/skeleton/projects/single.md b/tests/fixtures/skeleton/projects/single.md new file mode 100644 index 0000000..5a7b0dc --- /dev/null +++ b/tests/fixtures/skeleton/projects/single.md @@ -0,0 +1,14 @@ +# single — A project that runs on one partition only + +Project: single +Issue: none + +## Now + +Nothing submitted yet; the first run goes out once the dataset is on RAL. + +## Runs + +## Log + +- 2026-09-01 — note — ledger opened diff --git a/tests/fixtures/skeleton/projects/wound_down.md b/tests/fixtures/skeleton/projects/wound_down.md new file mode 100644 index 0000000..a0f953c --- /dev/null +++ b/tests/fixtures/skeleton/projects/wound_down.md @@ -0,0 +1,16 @@ +# wound_down — A project that finished and was retired + +Project: wound_down +Issue: none + +## Now + +Retired 2026-09-01. + +## Runs + +## Log + +- 2026-09-01 — note — retired: done +- 2026-08-20 — result — the final run answered the question; nothing more to run +- 2026-08-10 — note — ledger opened diff --git a/tests/fixtures/skeleton/rulings/2026/09/R-20260901-01.md b/tests/fixtures/skeleton/rulings/2026/09/R-20260901-01.md deleted file mode 100644 index 949a81c..0000000 --- a/tests/fixtures/skeleton/rulings/2026/09/R-20260901-01.md +++ /dev/null @@ -1,18 +0,0 @@ -# R-20260901-01 — accept example 08_accepted - -Project: example -Task: tasks/example/08_accepted.md -Runs: 342050 -Ruling: accept -Reviewed-at: 2026-09-01T10:30Z -Review-minutes-actual: 5 - -## Ruling - -Accept. theta_E is 1.204 against the published 1.20 ± 0.01; the new sampler -reproduces the anchor. - -## Evidence - -- `/mnt/c/Users/Jammy/Science/example/output/task_08/anchor/samples_summary.json` -- `/mnt/c/Users/Jammy/Science/example/wiki/project/state.md` (task 8) diff --git a/tests/fixtures/skeleton/rulings/2026/09/R-20260901-02.md b/tests/fixtures/skeleton/rulings/2026/09/R-20260901-02.md deleted file mode 100644 index 2884954..0000000 --- a/tests/fixtures/skeleton/rulings/2026/09/R-20260901-02.md +++ /dev/null @@ -1,22 +0,0 @@ -# R-20260901-02 — re-accept example 08_accepted with the corrected evidence pointer - -Project: example -Task: tasks/example/08_accepted.md -Runs: 342050 -Ruling: accept -Supersedes: R-20260901-01 -Batch: 2026-09-01-pm -Reviewed-at: 2026-09-01T17:20Z -Review-minutes-actual: 6 -Follow-ups: PyAutoLens#901 - -## Ruling - -Accept, as before. R-20260901-01 cited the wrong summary file (the task 3 -anchor, not the task 8 one); the number and the verdict stand. File the -sampler-default follow-up. - -## Evidence - -- `/mnt/c/Users/Jammy/Science/example/output/task_08/anchor/samples_summary.json` -- `/mnt/c/Users/Jammy/Science/example/wiki/project/state.md` (task 8) diff --git a/tests/fixtures/skeleton/rulings/2026/09/R-20260901-03.md b/tests/fixtures/skeleton/rulings/2026/09/R-20260901-03.md deleted file mode 100644 index 3025454..0000000 --- a/tests/fixtures/skeleton/rulings/2026/09/R-20260901-03.md +++ /dev/null @@ -1,18 +0,0 @@ -# R-20260901-03 — leave example 05_running_array to finish - -Project: example -Task: tasks/example/05_running_array.md -Runs: 342091, 342102 -Ruling: leave-to-finish -Batch: 2026-09-01-pm -Reviewed-at: 2026-09-01T17:25Z -Review-minutes-actual: 2 - -## Ruling - -Leave it to finish. Nine of ten are in and look right; task 9 is resubmitted -alone and will be on the next board. - -## Evidence - -- `/mnt/c/Users/Jammy/Science/example/output/task_05/` (nine checkpoints) diff --git a/tests/fixtures/skeleton/rulings/2026/09/R-20260901-04.md b/tests/fixtures/skeleton/rulings/2026/09/R-20260901-04.md deleted file mode 100644 index ae4c9b3..0000000 --- a/tests/fixtures/skeleton/rulings/2026/09/R-20260901-04.md +++ /dev/null @@ -1,19 +0,0 @@ -# R-20260901-04 — rerun example 09_rerun on the released stack - -Project: example -Task: tasks/example/09_rerun.md -Runs: 341900 -Ruling: rerun -Batch: 2026-09-01-pm -Reviewed-at: 2026-09-01T17:30Z -Review-minutes-actual: 4 -Follow-ups: https://github.com/PyAutoLabs/PyAutoArray/issues/510 - -## Ruling - -Rerun. The legacy mask edge is one pixel out and the run predates the mask -fix; redo it on the released stack once PyAutoArray#510 lands. - -## Evidence - -- `/mnt/c/Users/Jammy/Science/example/output/legacy/task_09/mask.fits` diff --git a/tests/fixtures/skeleton/rulings/2026/09/R-20260901-05.md b/tests/fixtures/skeleton/rulings/2026/09/R-20260901-05.md deleted file mode 100644 index fd8b501..0000000 --- a/tests/fixtures/skeleton/rulings/2026/09/R-20260901-05.md +++ /dev/null @@ -1,19 +0,0 @@ -# R-20260901-05 — drop the mesh-pix sweep - -Project: example -Task: tasks/example/10_dropped.md -Runs: 341950, 341960 -Ruling: drop -Batch: 2026-09-01-pm -Reviewed-at: 2026-09-01T17:35Z -Review-minutes-actual: 3 - -## Ruling - -Drop. The sweep ran on the wrong regularisation and the question is answered -by task 8; nothing here is reusable. Revive as a new task if mesh-pix comes -back. - -## Evidence - -- `/mnt/c/Users/Jammy/Science/example/output/legacy_wrong/task_10/` diff --git a/tests/fixtures/skeleton/tasks/example/01_scope.md b/tests/fixtures/skeleton/tasks/example/01_scope.md deleted file mode 100644 index e3e1899..0000000 --- a/tests/fixtures/skeleton/tasks/example/01_scope.md +++ /dev/null @@ -1,29 +0,0 @@ -# Example — scope the programme - -Project: example -Summary: Which lenses anchor the programme and at what pixel scale -State: planned -Gates: -Witness: -Budget: 4:00 -Review-minutes: 3 -Epic: example-programme -Filed: 2026-08-25 - -## Question - -Which three lenses anchor the programme, and at what pixel scale? - -## Witness - -(not yet registered — a planned task may leave this empty) - -## Where to look - -- `/mnt/c/Users/Jammy/Science/example/wiki/project/state.md` - -## Runs - -## Ruling - -(none) diff --git a/tests/fixtures/skeleton/tasks/example/02_gated_on_dev.md b/tests/fixtures/skeleton/tasks/example/02_gated_on_dev.md deleted file mode 100644 index 3f466e3..0000000 --- a/tests/fixtures/skeleton/tasks/example/02_gated_on_dev.md +++ /dev/null @@ -1,30 +0,0 @@ -# Example — the pixel-scale sweep, gated on two dev fixes - -Project: example -Summary: Does the pixel-scale sweep change the preferred model -State: gated -Gates: PyAutoArray#431, https://github.com/PyAutoLabs/PyAutoFit/pull/1436 -Witness: three lenses reach log-evidence within 2.0 of the anchor at 0.05 arcsec -Budget: 6:00 -Review-minutes: 6 -Epic: example-programme -Filed: 2026-08-26 - -## Question - -Does the pixel-scale sweep change the preferred model once the mask fix and the -prior-loading fix land? - -## Witness - -`output/task_02/*/samples_summary.json` — log-evidence within 2.0 of the anchor. - -## Where to look - -- `/mnt/c/Users/Jammy/Science/example/output/task_02/` - -## Runs - -## Ruling - -(none) diff --git a/tests/fixtures/skeleton/tasks/example/03_ready_cleared.md b/tests/fixtures/skeleton/tasks/example/03_ready_cleared.md deleted file mode 100644 index bd3dccf..0000000 --- a/tests/fixtures/skeleton/tasks/example/03_ready_cleared.md +++ /dev/null @@ -1,29 +0,0 @@ -# Example — the anchor refit, gate cleared - -Project: example -Summary: Does the refit reproduce the published anchor -State: ready -Gates: PyAutoGalaxy#486 -Witness: anchor lens theta_E within 0.01 arcsec of the published value -Budget: 5:00 -Review-minutes: 4 -Epic: example-programme -Filed: 2026-08-27 - -## Question - -Does the refit on the released stack reproduce the published anchor? - -## Witness - -`output/task_03/anchor/samples_summary.json` — `theta_E` within 0.01 arcsec. - -## Where to look - -- `/mnt/c/Users/Jammy/Science/example/output/task_03/` - -## Runs - -## Ruling - -(none) diff --git a/tests/fixtures/skeleton/tasks/example/04_submitted_override.md b/tests/fixtures/skeleton/tasks/example/04_submitted_override.md deleted file mode 100644 index 6872427..0000000 --- a/tests/fixtures/skeleton/tasks/example/04_submitted_override.md +++ /dev/null @@ -1,33 +0,0 @@ -# Example — the two-band fit, submitted under a gate override - -Project: example -Summary: Do the two bands agree on the source centre -State: submitted -Gates: PyAutoLens#900 -Witness: both bands converge to the same source centre within 0.02 arcsec -Budget: 6:00 -Runs: 342001, 342010 -Review-minutes: 6 -Epic: example-programme -Filed: 2026-08-28 - -## Question - -Do the two bands agree on the source centre once fitted jointly? - -## Witness - -`output/task_04/joint/samples_summary.json` — source centres within 0.02 arcsec. - -## Where to look - -- `/mnt/c/Users/Jammy/Science/example/output/task_04/` - -## Runs - -- 342001: failed — gpu — submitted 2026-08-31 — wall 0:00 — node failure before the first step -- 342010: submitted — gpu — submitted 2026-09-01 — wall 0:00 - -## Ruling - -(none) diff --git a/tests/fixtures/skeleton/tasks/example/05_running_array.md b/tests/fixtures/skeleton/tasks/example/05_running_array.md deleted file mode 100644 index 9845d64..0000000 --- a/tests/fixtures/skeleton/tasks/example/05_running_array.md +++ /dev/null @@ -1,37 +0,0 @@ -# Example — the nine-lens array - -Project: example -Summary: Does Delaunay reach the same theta_E basin on ten lenses -State: running -Gates: PyAutoArray#431 -Witness: nine of ten array tasks write a sane checkpoint.hdf5 within 8:00 wall -Budget: 8:00 -Runs: 342091, 342102 -Ruling: R-20260901-03 -Review-minutes: 8 -Epic: example-programme -Filed: 2026-08-29 - -## Question - -Does the Delaunay pipeline reach the same theta_E basin on all ten lenses? - -## Witness - -`output/task_05/*/checkpoint.hdf5` present for nine lenses, `.err` clean. - -## Where to look - -- `/mnt/c/Users/Jammy/Science/example/output/task_05/` - -## Runs - -- 342091_[0-8,10]: done — gpu — submitted 2026-08-30 — wall 6:12 - pulled_to: /mnt/c/Users/Jammy/Science/example/output/task_05 -- 342091_9: failed — gpu — submitted 2026-08-30 — wall 0:00 — OOM before the first step -- 342102: running — gpu — submitted 2026-09-01 — wall 0:00 — task 9 resubmitted alone - after: 342091_9 - -## Ruling - -R-20260901-03 — leave-to-finish diff --git a/tests/fixtures/skeleton/tasks/example/06_pulled.md b/tests/fixtures/skeleton/tasks/example/06_pulled.md deleted file mode 100644 index cfa9e43..0000000 --- a/tests/fixtures/skeleton/tasks/example/06_pulled.md +++ /dev/null @@ -1,32 +0,0 @@ -# Example — the first half of the shared array, pulled - -Project: example -Summary: Do the five bright lenses converge under the fast preset -State: pulled -Witness: five lenses' checkpoint.hdf5 sane and .err clean under 4:00 wall -Budget: 4:00 -Runs: 342120 -Review-minutes: 5 -Epic: example-programme -Filed: 2026-08-30 - -## Question - -Do the five bright lenses converge under the fast preset? - -## Witness - -`output/task_06/*/checkpoint.hdf5` sane for all five. - -## Where to look - -- `/mnt/c/Users/Jammy/Science/example/output/task_06/` - -## Runs - -- 342120_[0-4]: done — gpu — submitted 2026-08-31 — wall 3:40 — array shared with 07_awaiting_ruling - pulled_to: /mnt/c/Users/Jammy/Science/example/output/task_06 - -## Ruling - -(none) diff --git a/tests/fixtures/skeleton/tasks/example/07_awaiting_ruling.md b/tests/fixtures/skeleton/tasks/example/07_awaiting_ruling.md deleted file mode 100644 index 3bb82c3..0000000 --- a/tests/fixtures/skeleton/tasks/example/07_awaiting_ruling.md +++ /dev/null @@ -1,34 +0,0 @@ -# Example — the second half of the shared array, awaiting a ruling - -Project: example -Summary: Do the five faint lenses converge once resumed -State: awaiting-ruling -Witness: five faint lenses' checkpoint.hdf5 sane after the checkpoint resubmit -Budget: 8:00 -Runs: 342110, 342120 -Review-minutes: 5 -Epic: example-programme -Filed: 2026-08-30 - -## Question - -Do the five faint lenses converge once resumed from the timed-out checkpoint? - -## Witness - -`output/task_07/*/checkpoint.hdf5` sane for all five. - -## Where to look - -- `/mnt/c/Users/Jammy/Science/example/output/task_07/` - -## Runs - -- 342110: timeout — gpu — submitted 2026-08-30 — wall 8:00 — hit the budget; resubmitted from checkpoint -- 342120_[5-9]: done — gpu — submitted 2026-08-31 — wall 3:55 — array shared with 06_pulled - resumes: 342110 - pulled_to: /mnt/c/Users/Jammy/Science/example/output/task_07 - -## Ruling - -(none) diff --git a/tests/fixtures/skeleton/tasks/example/08_accepted.md b/tests/fixtures/skeleton/tasks/example/08_accepted.md deleted file mode 100644 index 84897bf..0000000 --- a/tests/fixtures/skeleton/tasks/example/08_accepted.md +++ /dev/null @@ -1,35 +0,0 @@ -# Example — the anchor comparison, accepted - -Project: example -Summary: Does the new sampler reproduce the anchor -State: accepted -Gates: PyAutoFit#1400 -Witness: anchor theta_E within 0.01 arcsec of the published value on the new sampler -Budget: 6:00 -Runs: 342050 -Ruling: R-20260901-02 -Review-minutes: 6 -Epic: example-programme -Filed: 2026-08-24 - -## Question - -Does the new sampler reproduce the anchor within the published error? - -## Witness - -`output/task_08/anchor/samples_summary.json` — `theta_E` within 0.01 arcsec. - -## Where to look - -- `/mnt/c/Users/Jammy/Science/example/output/task_08/` - -## Runs - -- 342050: done — gpu — submitted 2026-08-28 — wall 5:10 - pulled_to: /mnt/c/Users/Jammy/Science/example/output/task_08 - ruled: R-20260901-02 - -## Ruling - -R-20260901-02 — accept (supersedes R-20260901-01) diff --git a/tests/fixtures/skeleton/tasks/example/09_rerun.md b/tests/fixtures/skeleton/tasks/example/09_rerun.md deleted file mode 100644 index c260297..0000000 --- a/tests/fixtures/skeleton/tasks/example/09_rerun.md +++ /dev/null @@ -1,34 +0,0 @@ -# Example — the legacy-born mask test, ruled rerun - -Project: example -Summary: Is the pre-Cortex mask-edge run reusable -State: rerun -Witness: the legacy run's mask edge matches the released mask within one pixel -Budget: 8:00 -Runs: 341900 -Ruling: R-20260901-04 -Review-minutes: 5 -Epic: example-programme -Filed: 2026-08-31 - -## Question - -Is the pre-Cortex mask-edge run reusable, or must it be redone on the released stack? - -## Witness - -`output/legacy/task_09/mask.fits` edge vs the released mask. - -## Where to look - -- `/mnt/c/Users/Jammy/Science/example/output/legacy/task_09/` - -## Runs - -- 341900: legacy — gpu — submitted 2026-08-20 — wall 7:50 — pre-Cortex run, quarantined as reusable - where: /mnt/c/Users/Jammy/Science/example/output/legacy/task_09 - ruled: R-20260901-04 - -## Ruling - -R-20260901-04 — rerun diff --git a/tests/fixtures/skeleton/tasks/example/10_dropped.md b/tests/fixtures/skeleton/tasks/example/10_dropped.md deleted file mode 100644 index b4bad40..0000000 --- a/tests/fixtures/skeleton/tasks/example/10_dropped.md +++ /dev/null @@ -1,34 +0,0 @@ -# Example — the mesh-pix sweep, dropped - -Project: example -Summary: Does the mesh-pix sweep beat the Delaunay anchor -State: dropped -Witness: mesh-pix log-evidence beats the Delaunay anchor by 5.0 -Budget: 6:00 -Runs: 341950, 341960 -Ruling: R-20260901-05 -Review-minutes: 4 -Epic: example-programme -Filed: 2026-08-31 - -## Question - -Does the mesh-pix sweep beat the Delaunay anchor? - -## Witness - -`output/legacy_wrong/task_10/*/samples_summary.json` log-evidence vs the anchor. - -## Where to look - -- `/mnt/c/Users/Jammy/Science/example/output/legacy_wrong/task_10/` - -## Runs - -- 341950_[0-3]: legacy_wrong — gpu — submitted 2026-08-22 — wall 6:00 — ran on the wrong regularisation - where: /mnt/c/Users/Jammy/Science/example/output/legacy_wrong/task_10 -- 341960: void — gpu — submitted 2026-08-23 — wall 0:00 — cancelled before the first step - -## Ruling - -R-20260901-05 — drop diff --git a/tests/test_cortex.py b/tests/test_cortex.py index 4c6d94d..3e8b776 100644 --- a/tests/test_cortex.py +++ b/tests/test_cortex.py @@ -1,14 +1,13 @@ -"""Contract tests for scripts/cortex.py — the Cortex's check, gates, rule, move, new. +"""Contract tests for scripts/cortex.py — check, new, run, running, done, log, now, issue, retire. -Two things these tests deliberately do, matching the Mind's `test_lifecycle_check.py`: +Two things these tests deliberately do: -1. **The fixture is the witness.** `tests/fixtures/skeleton/` holds one project, - one task per state, five rulings (one chain of two), one batch record and - one review; `tests/fixtures/empty/` is an empty map. Both must pass `check` +1. **The fixture is the witness.** `tests/fixtures/skeleton/` holds four rows + (two active with ledgers, one retired with a closed ledger, one dormant + with none) and `tests/fixtures/empty/` an empty map. Both pass `check` unchanged, and every writing verb runs on a `copytree` of the skeleton. 2. **Prove each leg FAILS.** Every `check` rule in REFERENCE.md is driven with - a `tmp_path` mutation that must trip it, asserting the ` - …` line; every - transition the table allows is taken and every one it refuses is refused. + a `tmp_path` mutation that must trip it, asserting the ` - …` line. """ import shutil @@ -27,1104 +26,385 @@ SCRIPT = REPO / "scripts" / "cortex.py" SKELETON = REPO / "tests" / "fixtures" / "skeleton" EMPTY = REPO / "tests" / "fixtures" / "empty" -TODAY = "2026-09-02" +TODAY = "2026-09-03" +EX = "projects/example.md" -P = {n: f"tasks/example/{n}.md" for n in ( - "01_scope", "02_gated_on_dev", "03_ready_cleared", "04_submitted_override", - "05_running_array", "06_pulled", "07_awaiting_ruling", "08_accepted", - "09_rerun", "10_dropped")} - -# --------------------------------------------------------------------------- # -# helpers -# --------------------------------------------------------------------------- # def _copy(tmp_path: Path, src: Path = SKELETON) -> Path: root = tmp_path / "tree" shutil.copytree(src, root) return root -def _run(*args, root=None, stdin=""): +def _run(*args, root=None): cmd = [sys.executable, str(SCRIPT), *args] if root is not None: cmd += ["--root", str(root)] - return subprocess.run(cmd, input=stdin, capture_output=True, text=True) + return subprocess.run(cmd, capture_output=True, text=True) def _problems(root: Path): return cortex.check_problems(root) -def _edit(root: Path, rel: str, old: str, new: str, count: int = 1): +def _edit(root: Path, rel: str, old: str, new: str): p = root / rel text = p.read_text() assert old in text, f"{old!r} not in {rel}" - p.write_text(text.replace(old, new, count)) - - -def _fields(root: Path, rel: str): - return cortex.parse_header((root / rel).read_text())[1] - - -def _assert_drift(root: Path, *needles: str): - problems = _problems(root) - joined = "\n".join(problems) - for needle in needles: - assert needle in joined, f"expected {needle!r} in:\n{joined}" - return problems - + p.write_text(text.replace(old, new, 1)) -def _move(root, rel, state, *extra, today=TODAY): - return _run("move", rel, state, "--today", today, *extra, root=root) - -def _body(tmp_path: Path, text="Accept. The human's words, verbatim.\n") -> Path: - p = tmp_path / "body.txt" - p.write_text(text) - return p +def _ledger(root: Path, key: str = "example") -> cortex.Ledger: + leds, _ = cortex.load_ledgers(root) + return next(l for l in leds if l.key == key) # --------------------------------------------------------------------------- # -# the witness — both fixtures pass check unchanged +# check # --------------------------------------------------------------------------- # -def test_skeleton_fixture_passes_check(): - result = _run("check", root=SKELETON) - assert result.returncode == 0, result.stdout + result.stderr - assert result.stdout.strip() == "cortex check: OK" - - -def test_empty_fixture_passes_check(): - result = _run("check", root=EMPTY) - assert result.returncode == 0, result.stdout + result.stderr - assert result.stdout.strip() == "cortex check: OK" - - -def test_check_problems_is_quiet_on_the_witness(): +def test_the_fixtures_check_clean(): assert _problems(SKELETON) == [] assert _problems(EMPTY) == [] + assert _run("check", root=SKELETON).returncode == 0 + assert _run("check", root=EMPTY).returncode == 0 -def test_check_reports_drift_in_lifecycle_shape(tmp_path): - root = _copy(tmp_path) - _edit(root, P["03_ready_cleared"], "State: ready\n", "State: flying\n") - result = _run("check", root=root) - assert result.returncode == 1 - lines = result.stdout.splitlines() - assert lines[0] == "cortex check: DRIFT" - assert lines[1].startswith(" - tasks/example/03_ready_cleared.md: ") - - -def test_no_import_time_side_effects_and_root_on_every_verb(): - help_text = _run("--help").stdout - for verb in ("check", "gates", "rule", "move", "new"): - assert verb in help_text - assert "--root" in _run(verb, "--help").stdout - - -# --------------------------------------------------------------------------- # -# the light header and the grammars -# --------------------------------------------------------------------------- # -def test_header_first_occurrence_wins_block_ends_at_blank_line_list_keys(): - text = ("# Title\n\nProject: one\nProject: two\nGates:\n- A#1\n- B#2\nState: ready\n\n" - "Task: 99\n") - title, fields = cortex.parse_header(text) - assert title == "Title" - assert fields["Project"] == "one" - assert fields["Gates"] == "A#1, B#2" - assert "Task" not in fields # after the blank line — body, not header - - -def test_header_beyond_30_lines_is_body(): - text = "# T\n\n" + "\n".join(f"K{i}: v" for i in range(40)) + "\n" - _, fields = cortex.parse_header(text) - assert "K27" in fields and "K28" not in fields - - -def test_edit_header_preserves_every_other_byte(tmp_path): - root = _copy(tmp_path) - before = (root / P["03_ready_cleared"]).read_text() - after = cortex.edit_header(before, {"State": "submitted"}) - assert after.replace("State: submitted", "State: ready") == before - # a removed key drops exactly one line; an inserted key lands at its slot - removed = cortex.edit_header(before, {"Budget": None}) - assert removed == before.replace("Budget: 5:00\n", "") - inserted = cortex.edit_header(removed, {"Reset": "why"}) - lines = inserted.splitlines() - assert lines[lines.index("Gates: PyAutoGalaxy#486") + 1] == "Reset: why" - - -def test_run_line_regex_matches_reference_examples(): - for line in ( - "- 342091_[0-8,10]: done — gpu — submitted 2026-08-30 — wall 6:12", - "- 342091_9: failed — gpu — submitted 2026-08-30 — wall 0:00 — OOM before the first step", - "- 342102: running — gpu — submitted 2026-09-01 — wall 0:00 — task 9 resubmitted alone", - ): - assert cortex.RUN_LINE_RE.match(line), line - assert cortex.RUN_CONT_RE.match(" after: 342091_9") - assert not cortex.RUN_CONT_RE.match(" after: 342091_9") - assert not cortex.RUN_LINE_RE.match("- 342102: running — GPU — submitted 2026-09-01 — wall 0:00") - - -def test_double_dash_is_read_as_the_em_dash(tmp_path): - root = _copy(tmp_path) - _edit(root, P["06_pulled"], " — gpu — submitted 2026-08-31 — wall 3:40 — ", - " -- gpu -- submitted 2026-08-31 -- wall 3:40 -- ") - assert _problems(root) == [] - - -def test_gate_ref_regex_rejects_owner_repo_form(): - assert cortex.GATE_REF_RE.fullmatch("PyAutoArray#431") - assert cortex.GATE_REF_RE.fullmatch("https://github.com/PyAutoLabs/PyAutoFit/pull/1436") - assert not cortex.GATE_REF_RE.fullmatch("PyAutoLabs/PyAutoArray#431") - assert cortex.gate_url("PyAutoArray#431") == "https://github.com/PyAutoLabs/PyAutoArray/issues/431" - assert cortex.gate_url("https://github.com/o/r/pull/7") == "https://github.com/o/r/issues/7" +def test_the_live_tree_checks_clean(): + r = _run("check", root=REPO) + assert r.returncode == 0, r.stdout -# --------------------------------------------------------------------------- # -# projects.yaml — PyYAML plus the field validation -# --------------------------------------------------------------------------- # -def test_comment_only_projects_yaml_parses_to_an_empty_map(): - rows, problems = cortex.parse_projects((EMPTY / "projects.yaml").read_text()) - assert rows == {} and problems == [] - - -def test_the_live_body_map_parses_clean(): - """The seeded map (task 3) is the real witness for the field validation: - it is the only file that exercises `note` and a 17-verb list. It stopped - exercising `planned` on 2026-09-07, when euclid_dr1_prelim's task 4 - launched and its row flipped to `active` — the last `planned` row in the - live map. `planned` as a legal status is covered on the fixture by - `test_planned_and_retired_are_legal_statuses`.""" - text = (REPO / "projects.yaml").read_text() - rows, problems = cortex.parse_projects(text) - assert problems == [] - assert {"inference_programme", "subhalo_validation", "euclid", - "euclid_dr1_prelim"} <= set(rows) - assert rows["euclid_dr1_prelim"]["status"] == "active" - assert rows["euclid"]["remote"] == "none" - assert "PyAutoLabs remote" in rows["euclid"]["note"] - assert rows["subhalo_validation"]["assistant"] == "autolens_assistant" - assert rows["inference_programme"]["assistant"] == "none" - - -def test_a_block_list_and_a_quoted_scalar_are_ordinary_yaml(tmp_path): - """What the retired hand parser refused, PyYAML reads: a block list, and a - quoted scalar holding a colon or a hash.""" +def test_check_prints_one_line_per_finding_and_exits_1(tmp_path): root = _copy(tmp_path) - _edit(root, "projects.yaml", " sync_verbs: [pull, submit, jobs, tail]\n", - " sync_verbs:\n - pull\n - submit\n") - _edit(root, "projects.yaml", " status: active\n", - ' status: active\n note: "a fact: with a colon # and a hash"\n') - assert _problems(root) == [] - rows, problems = cortex.parse_projects((root / "projects.yaml").read_text()) - assert problems == [] - assert rows["example"]["sync_verbs"] == ["pull", "submit"] - assert rows["example"]["note"] == "a fact: with a colon # and a hash" - + _edit(root, EX, "Issue: example#7", "Issue: nonsense") + r = _run("check", root=root) + assert r.returncode == 1 + assert "cortex check: DRIFT" in r.stdout + assert " - projects/example.md: `Issue:` must be" in r.stdout -def test_a_document_pyyaml_cannot_read_is_one_problem(): - rows, problems = cortex.parse_projects("example:\n remote: [unclosed\n") - assert rows == {} and len(problems) == 1 - assert "not valid YAML" in problems[0] - -def test_a_document_that_is_not_a_mapping_of_rows(): - rows, problems = cortex.parse_projects("- example\n") - assert rows == {} and problems == ["projects.yaml: the document is not a mapping of project keys"] - rows, problems = cortex.parse_projects("example: a string\n") - assert rows == {} and problems == ["projects.yaml: example is not a mapping of fields"] - - -def test_sync_verbs_must_hold_bare_words(tmp_path): +def test_an_active_project_needs_a_ledger(tmp_path): root = _copy(tmp_path) - _edit(root, "projects.yaml", " sync_verbs: [pull, submit, jobs, tail]\n", - " sync_verbs: [pull, 'Not A Verb']\n") - _assert_drift(root, "example.sync_verbs holds a non-bare word") - - -def test_unknown_projects_field_is_an_error(tmp_path): - root = _copy(tmp_path) - _edit(root, "projects.yaml", " status: active\n", " status: active\n colour: blue\n") - _assert_drift(root, "projects.yaml: unknown field `colour` on example") - - -def test_missing_projects_field_and_bad_enums(tmp_path): - root = _copy(tmp_path) - _edit(root, "projects.yaml", " partition: gpu\n", " partition: cpu\n") - _edit(root, "projects.yaml", " ledger: wiki/project/state.md\n", "") - _assert_drift(root, "example is missing ledger", "partition must be gpu | ral | both") - - -def test_planned_and_retired_are_legal_statuses(tmp_path): - root = _copy(tmp_path) - for status in ("planned", "retired"): - _edit(root, "projects.yaml", " status: active\n", f" status: {status}\n") - assert _problems(root) == [], status - _edit(root, "projects.yaml", f" status: {status}\n", " status: active\n") - _edit(root, "projects.yaml", " status: active\n", " status: someday\n") - _assert_drift(root, "status must be active | dormant | planned | retired") - - -def test_note_is_optional_but_never_empty_and_the_field_set_still_closes(tmp_path): - """`note` is the one field a row may omit. An empty one is still drift, - and everything outside the field set is still an unknown field.""" - root = _copy(tmp_path) - assert _problems(root) == [], "the fixture carries no `note:`" - _edit(root, "projects.yaml", " status: active\n", - ' status: active\n note: "a fact: with a colon # and a hash"\n') - assert _problems(root) == [] - rows, problems = cortex.parse_projects((root / "projects.yaml").read_text()) - assert problems == [] and rows["example"]["note"] == \ - "a fact: with a colon # and a hash" - _edit(root, "projects.yaml", ' note: "a fact: with a colon # and a hash"\n', - " note:\n") - _assert_drift(root, "example.note is empty") - - -def test_assistant_is_required_and_names_a_bare_workspace_directory(tmp_path): - """`assistant` is required like every field but `note`. Its value is - exactly `none` or a bare workspace-relative directory name — the Cortex - names the assistant, it never reads it, so no path and no stat.""" - root = _copy(tmp_path) - assert _problems(root) == [], "the fixture carries `assistant: none`" - _edit(root, "projects.yaml", " assistant: none\n", - " assistant: autolens_assistant\n") - assert _problems(root) == [] - rows, problems = cortex.parse_projects((root / "projects.yaml").read_text()) - assert problems == [] and rows["example"]["assistant"] == "autolens_assistant" - _edit(root, "projects.yaml", " assistant: autolens_assistant\n", - " assistant:\n") - _assert_drift(root, "example.assistant is empty") - _edit(root, "projects.yaml", " assistant:\n", "") - _assert_drift(root, "example is missing assistant") - for bad in ("../autolens_assistant", "/home/jammy/Code/PyAutoLabs/autolens_assistant"): - root = _copy(tmp_path / bad.replace("/", "_")) - _edit(root, "projects.yaml", " assistant: none\n", f" assistant: {bad}\n") - _assert_drift(root, "example.assistant must be 'none' or a bare " - "workspace-relative directory name") - - -# --------------------------------------------------------------------------- # -# retire -# --------------------------------------------------------------------------- # -#: The states `retire` allows a project to still hold: the three ruled ones -#: plus `planned`, which is an unasked question. -_QUIET = {"01_scope", "08_accepted", "09_rerun", "10_dropped"} - - -def _quiesce(root: Path, project: str = "example") -> "list[str]": - """Leave `project` holding only ruled and planned tasks — dropping the - rulings that named the tasks removed, so `check` stays clean.""" - gone = [] - for p in sorted((root / "tasks" / project).glob("*.md")): - if p.stem not in _QUIET: - gone.append(p.relative_to(root).as_posix()) - p.unlink() - for r in sorted((root / "rulings").rglob("*.md")): - if any(f"Task: {g}" in r.read_text() for g in gone): - r.unlink() - assert _problems(root) == [], _problems(root) - return gone - - -def test_retire_flips_the_status_and_stamps_the_note(tmp_path): - """Two lines change and nothing else — the row, its tasks and its - rulings all stay, and `check` is clean afterwards.""" - root = _copy(tmp_path) - _quiesce(root) - before = (root / "projects.yaml").read_text().split("\n") - tasks_before = sorted(p.name for p in (root / "tasks" / "example").glob("*.md")) - assert cortex.retire_project(root, "example", "the question was answered", - date(2026, 9, 4)) == "retired example" - after = (root / "projects.yaml").read_text().split("\n") - assert " status: retired" in after - assert ' note: "retired 2026-09-04: the question was answered"' in after - # every byte outside the two edited lines is identical: the fixture row - # carries no `note:`, so the note is an appended line. - assert [ln for ln in after if not ln.startswith((" status:", " note:"))] == \ - [ln for ln in before if not ln.startswith(" status:")] - assert sorted(p.name for p in (root / "tasks" / "example").glob("*.md")) == \ - tasks_before - assert _problems(root) == [] - rows, problems = cortex.parse_projects((root / "projects.yaml").read_text()) - assert problems == [] and rows["example"]["status"] == "retired" + (root / "projects" / "single.md").unlink() + assert any("single is active but has no projects/single.md" in p for p in _problems(root)) -def test_retire_rewrites_an_existing_note_in_place(tmp_path): +def test_a_ledger_needs_a_row(tmp_path): root = _copy(tmp_path) - _quiesce(root) - _edit(root, "projects.yaml", " status: active\n", - ' status: active\n note: "a fact: recorded"\n') - before = (root / "projects.yaml").read_text() - cortex.retire_project(root, "example", "no science since 2026-05", - date(2026, 9, 4)) - after = (root / "projects.yaml").read_text() - assert ' note: "a fact: recorded"' not in after - assert ' note: "retired 2026-09-04: no science since 2026-05"' in after - assert after.count(" note:") == before.count(" note:") == 1 - assert _problems(root) == [] + shutil.copy(root / EX, root / "projects" / "stranger.md") + _edit(root, "projects/stranger.md", "# example —", "# stranger —") + _edit(root, "projects/stranger.md", "Project: example", "Project: stranger") + assert any("stranger is not a projects.yaml key" in p for p in _problems(root)) -def test_retire_refuses_while_a_task_is_still_live(tmp_path): - """Every state outside `accepted | rerun | dropped | planned` is an - unfinished question, and the refusal names each one.""" - root = _copy(tmp_path) - before = (root / "projects.yaml").read_text() - with pytest.raises(cortex.CortexError) as e: - cortex.retire_project(root, "example", "why", date(2026, 9, 4)) - message = str(e.value) - assert "still has live work" in message - for name in ("02_gated_on_dev", "03_ready_cleared", "04_submitted_override", - "05_running_array", "06_pulled", "07_awaiting_ruling"): - assert f"tasks/example/{name}.md — " in message - # planned and the ruled states are not named — they are allowed to stay - for name in ("01_scope", "08_accepted", "09_rerun", "10_dropped"): - assert f"tasks/example/{name}.md" not in message - assert (root / "projects.yaml").read_text() == before - - -def test_retire_refuses_an_unknown_key_a_repeat_and_a_bad_why(tmp_path): - root = _copy(tmp_path) - _quiesce(root) - before = (root / "projects.yaml").read_text() - for why, needle in ((" ", "--why must say why"), - ('a "quoted" reason', "double quote")): - with pytest.raises(cortex.CortexError) as e: - cortex.retire_project(root, "example", why, date(2026, 9, 4)) - assert needle in str(e.value) - with pytest.raises(cortex.CortexError) as e: - cortex.retire_project(root, "nobody", "why", date(2026, 9, 4)) - assert "not a projects.yaml key" in str(e.value) - assert (root / "projects.yaml").read_text() == before - cortex.retire_project(root, "example", "done", date(2026, 9, 4)) - with pytest.raises(cortex.CortexError) as e: - cortex.retire_project(root, "example", "done again", date(2026, 9, 4)) - assert "already retired" in str(e.value) - - -def test_retire_round_trips_through_the_cli(tmp_path): +def test_the_file_name_is_the_key(tmp_path): root = _copy(tmp_path) - r = _run("retire", "example", "--why", "the question was answered", - "--today", TODAY, root=root) - assert r.returncode == 1 and "still has live work" in r.stderr - _quiesce(root) - r = _run("retire", "example", "--why", "the question was answered", - "--today", TODAY, root=root) - assert r.returncode == 0, r.stderr - assert r.stdout.strip() == "retired example" - text = (root / "projects.yaml").read_text() - assert " status: retired" in text - assert f' note: "retired {TODAY}: the question was answered"' in text - assert _run("check", root=root).returncode == 0 + (root / "projects" / "Example-Two.md").write_text("# x — y\n") + assert any("file name must be projects/<key>.md" in p for p in _problems(root)) -# --------------------------------------------------------------------------- # -# check — tasks -# --------------------------------------------------------------------------- # -def test_summary_is_required(tmp_path): - """`Summary:` is the board's line: a task without one is drift, and so is - a task whose `Summary:` is present but empty.""" +def test_the_title_names_the_key_and_carries_a_short_summary(tmp_path): root = _copy(tmp_path) - _edit(root, P["02_gated_on_dev"], - "Summary: Does the pixel-scale sweep change the preferred model\n", "") - _assert_drift(root, "02_gated_on_dev.md: missing header key Summary:") + _edit(root, EX, "# example — Does", "# other — Does") + assert any("title must be `# example — <summary>`" in p for p in _problems(root)) root = _copy(tmp_path / "b") - _edit(root, P["02_gated_on_dev"], - "Summary: Does the pixel-scale sweep change the preferred model\n", "Summary:\n") - _assert_drift(root, "02_gated_on_dev.md: missing header key Summary:") - - -def test_summary_is_capped_at_ten_words(tmp_path): - """Ten words is the cap the board is readable at; eleven is drift.""" - root = _copy(tmp_path) - ten = " ".join(f"w{i}" for i in range(10)) - _edit(root, P["02_gated_on_dev"], - "Summary: Does the pixel-scale sweep change the preferred model\n", - f"Summary: {ten}\n") - assert _problems(root) == [] - _edit(root, P["02_gated_on_dev"], f"Summary: {ten}\n", f"Summary: {ten} w10\n") - _assert_drift(root, "02_gated_on_dev.md: Summary: 11 words — at most 10") - - -def test_phase_header_is_dead(tmp_path): - """The number went on 2026-09-07: `Phase:` is named as retired rather than - passing as a generic unknown key, on a task file and on a ruling alike.""" - root = _copy(tmp_path) - _edit(root, P["01_scope"], "State: planned\n", "Phase: 1\nState: planned\n") - _edit(root, R + "R-20260901-05.md", "Task: tasks/example/10_dropped.md\n", - "Phase: tasks/example/10_dropped.md\nTask: tasks/example/10_dropped.md\n") - _assert_drift(root, "01_scope.md: Phase: is a retired header", - "R-20260901-05.md: Phase: is a retired header") - - -def test_unknown_project(tmp_path): - root = _copy(tmp_path) - shutil.move(root / "tasks" / "example", root / "tasks" / "other") - for p in (root / "tasks" / "other").glob("*.md"): - p.write_text(p.read_text().replace("Project: example", "Project: other")) - _assert_drift(root, "tasks/other/01_scope.md: Project: other is not a projects.yaml key") - - -def test_directory_must_equal_project(tmp_path): - root = _copy(tmp_path) - _edit(root, P["01_scope"], "Project: example\n", "Project: sample\n") - _assert_drift(root, "01_scope.md: Project: sample is not the directory name example", - "01_scope.md: Project: sample is not a projects.yaml key") - - -def test_where_to_look_must_name_somewhere_past_planned(tmp_path): - """The section is rendered, not just parsed — the dashboard's - `## By project` view and `checkin` print these bullets verbatim as the - folders to open. A task that has left `planned` carrying only the - template placeholder names nowhere.""" - root = _copy(tmp_path) - holder = f"- {cortex.WHERE_PLACEHOLDER}\n" - # `planned` is the one state the placeholder is honest on - _edit(root, P["01_scope"], "- `/mnt/c/Users/Jammy/Science/example/wiki/project/state.md`\n", - holder) - assert _problems(root) == [] - _edit(root, P["07_awaiting_ruling"], - "- `/mnt/c/Users/Jammy/Science/example/output/task_07/`\n", holder) - _assert_drift(root, "07_awaiting_ruling.md: ## Where to look names nowhere") + _edit(root, EX, "# example — Does the example pipeline recover the truth on the fixture lens", + "# example — " + " ".join(["word"] * 16)) + assert any("summary is over 15 words" in p for p in _problems(root)) -def test_a_new_task_ships_the_placeholder_and_no_other_bullet(tmp_path): - """`new` writes `planned`, so the task it writes passes `check` — and - the placeholder it writes is the exact string the rule exempts.""" +def test_the_header_holds_project_and_issue_only(tmp_path): root = _copy(tmp_path) - _run("new", "example", "11_fresh", "--summary", "Does the fresh idea hold", - "--today", TODAY, root=root) - text = (root / "tasks/example/11_fresh.md").read_text() - assert f"- {cortex.WHERE_PLACEHOLDER}" in text - assert _problems(root) == [] - - -def test_illegal_state_unknown_key_and_missing_section(tmp_path): - root = _copy(tmp_path) - _edit(root, P["01_scope"], "State: planned\n", "State: queued\nGate-cleared: x\n") - _edit(root, P["01_scope"], "## Where to look", "## Where") - _assert_drift(root, "State: 'queued' is not a task state", "unknown header key Gate-cleared:", - "missing section ## Where to look") - - -def test_migrated_from_is_legal_on_a_task_and_a_ruling(tmp_path): - """Decision 53: the provenance key joins both key tables; everything else - outside them is still drift.""" - root = _copy(tmp_path) - _edit(root, P["01_scope"], "Filed: 2026-08-25\n", - "Filed: 2026-08-25\nMigrated-from: PyAutoMind/draft/research/example/scope.md\n") - _edit(root, R + "R-20260901-05.md", "Ruling: drop\n", - "Ruling: drop\nMigrated-from: PyAutoMind/batches/reviews/2026-09-01-pm.md\n") - assert _problems(root) == [] - _edit(root, P["01_scope"], "Migrated-from: PyAutoMind", "Migrated-to: PyAutoMind") - _edit(root, R + "R-20260901-05.md", "Migrated-from: PyAutoMind", "Migrated-to: PyAutoMind") - _assert_drift(root, "01_scope.md: unknown header key Migrated-to:", - "R-20260901-05.md: unknown header key Migrated-to:") - - -def test_owner_repo_gate_form_is_refused(tmp_path): - root = _copy(tmp_path) - _edit(root, P["02_gated_on_dev"], "Gates: PyAutoArray#431,", "Gates: PyAutoLabs/PyAutoArray#431,") - _assert_drift(root, "Gates: unrecognised ref 'PyAutoLabs/PyAutoArray#431'") - - -def test_gated_with_empty_gates(tmp_path): - root = _copy(tmp_path) - _edit(root, P["02_gated_on_dev"], "Gates: PyAutoArray#431, https://github.com/PyAutoLabs/PyAutoFit/pull/1436\n", - "Gates:\n") - _assert_drift(root, "02_gated_on_dev.md: State: gated with an empty Gates:") - - -@pytest.mark.parametrize("rel", [P["04_submitted_override"], P["05_running_array"], P["06_pulled"], - P["07_awaiting_ruling"], P["08_accepted"], P["09_rerun"]]) -def test_witness_invariant(tmp_path, rel): - root = _copy(tmp_path) - witness = _fields(root, rel)["Witness"] - _edit(root, rel, f"Witness: {witness}\n", "Witness:\n") - _assert_drift(root, f"{rel}: State: {_fields(root, rel)['State']} needs a Witness:") - - -def test_planned_task_may_leave_the_witness_empty(): - assert _fields(SKELETON, P["01_scope"])["Witness"] == "" - assert _problems(SKELETON) == [] - - -@pytest.mark.parametrize("rel,rid", [(P["08_accepted"], "R-20260901-02"), - (P["09_rerun"], "R-20260901-04"), - (P["10_dropped"], "R-20260901-05")]) -def test_ruled_states_need_a_ruling(tmp_path, rel, rid): - root = _copy(tmp_path) - _edit(root, rel, f"Ruling: {rid}\n", "Ruling:\n") - _assert_drift(root, f"{rel}: State: {_fields(root, rel)['State']} needs a Ruling:") - - -def test_header_body_run_mismatch(tmp_path): - root = _copy(tmp_path) - _edit(root, P["04_submitted_override"], "Runs: 342001, 342010\n", "Runs: 342001\n") - _assert_drift(root, "04_submitted_override.md: Runs: header {342001} != body stems {342001, 342010}") + _edit(root, EX, "Issue: example#7", "Issue: example#7\nState: running") + assert any("unknown header key(s) State" in p for p in _problems(root)) + root = _copy(tmp_path / "b") + _edit(root, EX, "Issue: example#7\n", "") + assert any("header is missing `Issue:`" in p for p in _problems(root)) + root = _copy(tmp_path / "c") + _edit(root, EX, "Project: example", "Project: other") + assert any("`Project:` must be example" in p for p in _problems(root)) -def test_overlapping_task_sets_on_one_stem(tmp_path): - root = _copy(tmp_path) - _edit(root, P["05_running_array"], "- 342091_9: failed", "- 342091_[7-9]: failed") - _assert_drift(root, "05_running_array.md: run 342091_[0-8,10] overlaps 342091_[7-9] on stem 342091") +def test_issue_grammar(): + for ok in ("none", "example#7", "PyAutoCortex#12", + "https://github.com/Jammy2211/ic50_workspace/issues/3"): + assert cortex.ISSUE_RE.match(ok), ok + for bad in ("PyAutoLabs/example#7", "example", "#7", ""): + assert not cortex.ISSUE_RE.match(bad), bad + assert cortex.issue_url("example#7") == "https://github.com/PyAutoLabs/example/issues/7" -def test_bare_stem_overlaps_every_task_of_that_stem(tmp_path): +def test_the_three_sections_in_order_and_nothing_else(tmp_path): root = _copy(tmp_path) - _edit(root, P["05_running_array"], "- 342091_9: failed", "- 342091: failed") - _assert_drift(root, "run 342091_[0-8,10] overlaps 342091 on stem 342091") - - -def test_one_array_may_feed_two_tasks(): - """Job ids are unique per task, not globally: 342120 sits in tasks 6 and 7.""" - assert _fields(SKELETON, P["06_pulled"])["Runs"] == "342120" - assert "342120" in _fields(SKELETON, P["07_awaiting_ruling"])["Runs"] - assert _problems(SKELETON) == [] + _edit(root, EX, "## Runs", "## Jobs") + assert any("sections must be exactly ## Now · ## Runs · ## Log" in p for p in _problems(root)) + root = _copy(tmp_path / "b") + _edit(root, EX, "## Log", "## Notes\n\n## Log") + assert any("sections must be exactly" in p for p in _problems(root)) -def test_run_line_that_does_not_parse_and_stray_continuation(tmp_path): +def test_an_active_project_must_say_where_it_is(tmp_path): root = _copy(tmp_path) - _edit(root, P["06_pulled"], "- 342120_[0-4]: done — gpu — submitted 2026-08-31 — wall 3:40", - "- 342120_[0-4]: done — gpu — 2026-08-31 — wall 3:40") - _edit(root, P["04_submitted_override"], "## Runs\n\n", "## Runs\n\n after: 1\n") - _assert_drift(root, "06_pulled.md: line 27: run line does not parse", - "04_submitted_override.md: line 28: continuation line without a run line") + p = root / EX + text = p.read_text() + a, b = text.index("## Now"), text.index("## Runs") + p.write_text(text[:a] + "## Now\n\n" + text[b:]) + assert any("`## Now` is empty on an active project" in p for p in _problems(root)) -def test_pulled_needs_a_pulled_to(tmp_path): +def test_run_lines_parse_to_open_or_running_with_a_partition_and_a_date(tmp_path): + led = _ledger(SKELETON) + assert [(r.ident, r.state, r.partition, r.date) for r in led.runs] == [ + ("3002_[0-3]", "running", "ral", "2026-09-01"), + ("3003", "open", "gpu", "2026-09-02")] + assert led.runs[1].text() == "joint fit on the same four seeds chained after 3002 with afterok" root = _copy(tmp_path) - _edit(root, P["06_pulled"], " pulled_to: /mnt/c/Users/Jammy/Science/example/output/task_06\n", "") - _assert_drift(root, "06_pulled.md: State: pulled needs at least one done | legacy run with pulled_to:") + _edit(root, EX, "- 3003 — open — gpu", "- 3003 — submitted — gpu") + assert any("run line does not parse" in p for p in _problems(root)) + root = _copy(tmp_path / "b") + _edit(root, EX, "- 3003 — open — gpu", "- 3003 — open — cpu") + assert not any("3003" in p and "partition" in p for p in _problems(root)) # `both` allows any + _edit(root, "projects/single.md", "## Runs\n", "## Runs\n\n- 9 — open — gpu — 2026-09-01 — x\n") + assert any("run 9 is on `gpu` but the project runs on `ral`" in p for p in _problems(root)) -def test_legacy_needs_where(tmp_path): +def test_a_run_is_listed_once_and_a_retired_project_lists_none(tmp_path): root = _copy(tmp_path) - _edit(root, P["10_dropped"], " where: /mnt/c/Users/Jammy/Science/example/output/legacy_wrong/task_10\n", "") - _assert_drift(root, "10_dropped.md: run 341950_[0-3] is legacy_wrong without where:") + _edit(root, EX, "- 3003 — open — gpu — 2026-09-02 — joint fit", + "- 3003 — open — gpu — 2026-09-02 — joint fit\n- 3003 — open — gpu — 2026-09-02 — again") + assert any("run 3003 listed twice" in p for p in _problems(root)) + root = _copy(tmp_path / "b") + _edit(root, "projects/wound_down.md", "## Runs\n", "## Runs\n\n- 1 — open — ral — 2026-09-01 — x\n") + assert any("a retired project still lists runs" in p for p in _problems(root)) -def test_after_and_resumes_name_a_run_of_the_same_task(tmp_path): - root = _copy(tmp_path) - _edit(root, P["05_running_array"], " after: 342091_9", " after: 342091") - _edit(root, P["07_awaiting_ruling"], " resumes: 342110", " resumes: 342120_[5-9]") - _assert_drift(root, "05_running_array.md: run 342102 after: 342091 names no other run of this task", - "07_awaiting_ruling.md: run 342120_[5-9] resumes: 342120_[5-9] names no other run") +def test_the_dash_may_be_typed_as_two_hyphens(): + led = cortex.parse_ledger( + "# k -- summary\n\nProject: k\nIssue: none\n\n## Now\n\nx\n\n## Runs\n\n" + "- 1 -- open -- ral -- 2026-09-01 -- a run\n\n## Log\n\n- 2026-09-01 -- note -- hi\n", + "k", Path("k.md"), "projects/k.md") + assert led.problems == [] + assert led.summary == "summary" and led.runs[0].state == "open" and led.log[0].kind == "note" -def test_ruled_continuation_must_resolve(tmp_path): +def test_log_lines_carry_a_date_a_kind_and_text_newest_first(tmp_path): + led = _ledger(SKELETON) + assert [e.kind for e in led.log] == ["run", "run", "lesson", "result", "run", "run", "note"] + assert led.log[3].text() == ("wave 1 recovered the parent mean but its sigma is 4x too " + "tight the widths are the finding, not the means") + assert led.updated() == "2026-09-02" + assert [e.date for e in led.recent(2)] == ["2026-09-02", "2026-09-01"] root = _copy(tmp_path) - _edit(root, P["08_accepted"], " ruled: R-20260901-02", " ruled: R-20260901-09") - _assert_drift(root, "08_accepted.md: run 342050 ruled: R-20260901-09 does not resolve to a ruling file") + _edit(root, EX, "- 2026-09-01 — lesson —", "- 2026-09-01 — verdict —") + assert any("log line does not parse" in p for p in _problems(root)) + root = _copy(tmp_path / "b") + _edit(root, EX, "- 2026-08-28 — note — ledger opened", "- 2026-09-09 — note — out of order") + assert any("log is not newest-first" in p for p in _problems(root)) + root = _copy(tmp_path / "c") + _edit(root, EX, "- 2026-08-28 — note", "- 2026-02-30 — note") + assert any("not a real date" in p for p in _problems(root)) -def test_pulled_with_a_live_run_needs_leave_to_finish(tmp_path): +def test_a_continuation_line_needs_something_to_continue(tmp_path): root = _copy(tmp_path) - _edit(root, P["05_running_array"], "State: running\n", "State: pulled\n") - assert _problems(root) == [] # R-20260901-03 is a leave-to-finish head - _edit(root, P["05_running_array"], "Ruling: R-20260901-03\n", "Ruling:\n") - _assert_drift(root, "05_running_array.md: State: pulled with a live run needs a leave-to-finish ruling head") + _edit(root, EX, "## Log\n\n", "## Log\n\n orphan\n") + assert any("continuation line without a log line" in p for p in _problems(root)) -def test_stray_file_under_tasks(tmp_path): +def test_projects_yaml_is_validated(tmp_path): root = _copy(tmp_path) - (root / "tasks" / "notes.md").write_text("# stray\n") - _assert_drift(root, "tasks/notes.md: not a task path") + _edit(root, "projects.yaml", " partition: both\n status: active\n note: \"the fixture project\"", + " partition: moon\n status: active\n note: \"the fixture project\"") + assert any("example.partition must be gpu | ral | both" in p for p in _problems(root)) + root = _copy(tmp_path / "b") + _edit(root, "projects.yaml", " status: active\n note: \"the fixture project\"", + " status: active\n colour: red\n note: \"the fixture project\"") + assert any("unknown field `colour` on example" in p for p in _problems(root)) # --------------------------------------------------------------------------- # -# check — rulings and chains +# the writing verbs # --------------------------------------------------------------------------- # -R = "rulings/2026/09/" - - -def test_dangling_supersedes(tmp_path): - root = _copy(tmp_path) - _edit(root, R + "R-20260901-02.md", "Supersedes: R-20260901-01\n", "Supersedes: R-20260831-01\n") - _assert_drift(root, "R-20260901-02.md: Supersedes: R-20260831-01 does not resolve to a ruling file") - - -def test_self_supersedes(tmp_path): - root = _copy(tmp_path) - _edit(root, R + "R-20260901-02.md", "Supersedes: R-20260901-01\n", "Supersedes: R-20260901-02\n") - _assert_drift(root, "R-20260901-02.md: Supersedes: itself") - - -def test_supersedes_must_be_earlier_and_same_task(tmp_path): - root = _copy(tmp_path) - _edit(root, R + "R-20260901-01.md", "Ruling: accept\n", "Ruling: accept\nSupersedes: R-20260901-03\n") - _assert_drift(root, "R-20260901-01.md: Supersedes: R-20260901-03 is not earlier than R-20260901-01", - "R-20260901-01.md: Supersedes: R-20260901-03 names a different project/task") - - -def test_duplicate_successor_is_a_tree_not_a_chain(tmp_path): - root = _copy(tmp_path) - _edit(root, R + "R-20260901-03.md", "Ruling: leave-to-finish\n", - "Ruling: leave-to-finish\nSupersedes: R-20260901-01\n") - _assert_drift(root, "R-20260901-01.md: has 2 successors (R-20260901-02, R-20260901-03) — a chain, not a tree") - - -def test_superseded_ruling_as_task_head(tmp_path): - root = _copy(tmp_path) - _edit(root, P["08_accepted"], "Ruling: R-20260901-02\n", "Ruling: R-20260901-01\n") - _assert_drift(root, "08_accepted.md: Ruling: R-20260901-01 is not a chain head (superseded by R-20260901-02)") - - -def test_task_ruling_must_name_that_task(tmp_path): - root = _copy(tmp_path) - _edit(root, P["09_rerun"], "Ruling: R-20260901-04\n", "Ruling: R-20260901-05\n") - _assert_drift(root, "09_rerun.md: Ruling: R-20260901-05 names tasks/example/10_dropped.md, not this task", - "09_rerun.md: Ruling: R-20260901-05 verb 'drop' does not fit State: rerun") - - -def test_verb_state_mismatch(tmp_path): +def test_new_opens_a_ledger_for_a_row_and_refuses_the_rest(tmp_path): root = _copy(tmp_path) - _edit(root, P["08_accepted"], "State: accepted\n", "State: awaiting-ruling\n") - _assert_drift(root, "08_accepted.md: Ruling: R-20260901-02 verb 'accept' does not fit State: awaiting-ruling") - - -def test_ruling_id_filename_title_and_directory(tmp_path): - root = _copy(tmp_path) - _edit(root, R + "R-20260901-05.md", "# R-20260901-05 — drop", "# R-20260901-06 — drop") - (root / "rulings" / "2026" / "08").mkdir() - shutil.copy(root / R / "R-20260901-04.md", root / "rulings" / "2026" / "08" / "R-20260901-04.md") - (root / "rulings" / "2026" / "stray.md").write_text("# stray\n") - _assert_drift(root, "R-20260901-05.md: title must be `# R-20260901-05`", - "rulings/2026/08/R-20260901-04.md: filed under 2026/08 but the id is dated 2026-09", - "duplicate ruling id R-20260901-04", - "rulings/2026/stray.md: not a ruling path") - - -def test_ruling_runs_subset_task_path_verb_and_batch(tmp_path): - root = _copy(tmp_path) - _edit(root, R + "R-20260901-03.md", "Runs: 342091, 342102\n", "Runs: 342091, 342999\n") - _edit(root, R + "R-20260901-03.md", "Batch: 2026-09-01-pm\n", "Batch: 2026-09-01-am\n") - _edit(root, R + "R-20260901-04.md", "Ruling: rerun\n", "Ruling: redo\n") - _edit(root, R + "R-20260901-05.md", "Task: tasks/example/10_dropped.md\n", "Task: tasks/example/11.md\n") - _assert_drift(root, "R-20260901-03.md: Runs: {342999} not in the task's runs", - "R-20260901-03.md: Batch: 2026-09-01-am names no batches/2026-09-01-am.md", - "R-20260901-04.md: Ruling: 'redo' is not a ruling verb", - "R-20260901-05.md: Task: tasks/example/11.md does not resolve to a task file") - - -def test_ruling_follow_up_uses_the_gate_grammar(tmp_path): - root = _copy(tmp_path) - _edit(root, R + "R-20260901-02.md", "Follow-ups: PyAutoLens#901\n", "Follow-ups: PyAutoLabs/PyAutoLens#901\n") - _assert_drift(root, "R-20260901-02.md: Follow-ups: unrecognised ref 'PyAutoLabs/PyAutoLens#901'") - - -def test_two_chains_of_one_on_a_task_are_fine(tmp_path): - """leave-to-finish then accept need not chain; Supersedes: replaces, never sequences.""" - root = _copy(tmp_path) - _move(root, P["05_running_array"], "pulled", "--partial") - _move(root, P["05_running_array"], "awaiting-ruling") - r = _run("rule", P["05_running_array"], "accept", "--body", str(_body(tmp_path)), + (root / "projects" / "single.md").unlink() + r = _run("new", "single", "--summary", "A fresh question", "--issue", "single#1", "--today", TODAY, root=root) assert r.returncode == 0, r.stderr + led = _ledger(root, "single") + assert led.summary == "A fresh question" and led.issue == "single#1" + assert led.now.startswith("Just opened") + assert led.log[0].text() == "ledger opened" and led.log[0].date == TODAY assert _problems(root) == [] - assert _fields(root, P["05_running_array"])["Ruling"] == "R-20260902-01" - assert "Supersedes" not in _fields(root, "rulings/2026/09/R-20260902-01.md") - - -# --------------------------------------------------------------------------- # -# move — every legal transition, every refused one -# --------------------------------------------------------------------------- # -def test_planned_to_gated_or_ready_by_gates(tmp_path): - root = _copy(tmp_path) - r = _move(root, P["01_scope"], "gated") - assert r.returncode == 1 and "Gates: is empty — planned → ready" in r.stderr - r = _move(root, P["01_scope"], "ready") - assert r.returncode == 0 and r.stdout.strip() == "tasks/example/01_scope.md: planned → ready" - assert _fields(root, P["01_scope"])["State"] == "ready" - _edit(root, P["01_scope"], "State: ready\nGates:\n", "State: planned\nGates: PyAutoLens#1\n") - r = _move(root, P["01_scope"], "ready") - assert r.returncode == 1 and "Gates: is non-empty — planned → gated" in r.stderr - assert _move(root, P["01_scope"], "gated").returncode == 0 - assert _problems(root) == [] + assert _run("new", "single", "--summary", "again", root=root).returncode == 1 + r = _run("new", "nobody", "--summary", "x", root=root) + assert r.returncode == 1 and "not a projects.yaml key" in r.stderr + r = _run("new", "sleeping", "--summary", " ".join(["w"] * 16), root=root) + assert r.returncode == 1 and "over 15 words" in r.stderr + r = _run("new", "sleeping", "--summary", "x", "--issue", "PyAutoLabs/x#1", root=root) + assert r.returncode == 1 and "--issue must be" in r.stderr -def test_gated_to_ready_is_a_plain_move(tmp_path): - """Gate grading is retired: the human reads `gates`, judges the refs and - moves the task. No flag, and nothing written but `State:`.""" +def test_run_records_a_submission_as_open_and_logs_it(tmp_path): root = _copy(tmp_path) - before = (root / P["02_gated_on_dev"]).read_text() - r = _move(root, P["02_gated_on_dev"], "ready") + r = _run("run", "example", "3004_[0-9]", "wave 3, ten seeds", "--partition", "ral", + "--today", TODAY, root=root) assert r.returncode == 0, r.stderr - assert _fields(root, P["02_gated_on_dev"])["State"] == "ready" - assert (root / P["02_gated_on_dev"]).read_text() == \ - before.replace("State: gated", "State: ready") + led = _ledger(root) + assert [x.ident for x in led.runs] == ["3002_[0-3]", "3003", "3004_[0-9]"] + assert led.runs[-1].state == "open" and led.runs[-1].partition == "ral" + assert led.log[0].kind == "run" and led.log[0].text() == "3004_[0-9] submitted: wave 3, ten seeds" assert _problems(root) == [] + # again → refused; a project on both partitions needs --partition; a bad id is refused + assert _run("run", "example", "3004_[0-9]", "x", "--partition", "ral", root=root).returncode == 1 + r = _run("run", "example", "3005", "x", root=root) + assert r.returncode == 1 and "--partition" in r.stderr + r = _run("run", "example", "job-five", "x", "--partition", "ral", root=root) + assert r.returncode == 1 and "not a SLURM job id" in r.stderr + # a one-partition project needs no flag + assert _run("run", "single", "77", "first run", "--today", TODAY, root=root).returncode == 0 + assert _ledger(root, "single").runs[0].partition == "ral" -def test_ready_to_gated_is_a_hand_edit(tmp_path): - root = _copy(tmp_path) - r = _move(root, P["03_ready_cleared"], "gated") - assert r.returncode == 1 and "hand edit of the header" in r.stderr - - -def test_ready_to_submitted_needs_witness_and_run(tmp_path): +def test_running_flips_the_state_and_done_moves_the_run_into_the_log(tmp_path): root = _copy(tmp_path) - r = _move(root, P["03_ready_cleared"], "submitted") - assert r.returncode == 1 and "--run" in r.stderr - _edit(root, P["03_ready_cleared"], "Witness: anchor lens theta_E within 0.01 arcsec of the published value\n", - "Witness:\n") - r = _move(root, P["03_ready_cleared"], "submitted", "--run", "350000") - assert r.returncode == 1 and "Witness: is empty" in r.stderr - _edit(root, P["03_ready_cleared"], "Witness:\n", "Witness: theta_E within 0.01\n") - before = (root / P["03_ready_cleared"]).read_text() - r = _move(root, P["03_ready_cleared"], "submitted", "--run", "350000", "--note", "first wave") + assert _run("running", "example", "3003", root=root).returncode == 0 + assert _ledger(root).runs[1].state == "running" + assert "already running" in _run("running", "example", "3003", root=root).stdout + r = _run("done", "example", "3002_[0-3]", "--wall", "20:05", "--note", "all four landed", + "--today", TODAY, root=root) assert r.returncode == 0, r.stderr - after = (root / P["03_ready_cleared"]).read_text() - assert "- 350000: submitted — gpu — submitted 2026-09-02 — wall 0:00 — first wave\n" in after - f = _fields(root, P["03_ready_cleared"]) - assert f["State"] == "submitted" and f["Runs"] == "350000" - # the body outside `## Runs` and the rest of the header are untouched - assert after.split("## Runs")[0].replace("State: submitted", "State: ready") \ - .replace("Runs: 350000\n", "") == before.split("## Runs")[0] - assert after.split("## Ruling")[1] == before.split("## Ruling")[1] - assert _problems(root) == [] - - -def test_run_self_loop_appends_a_wave_and_keeps_state(tmp_path): - root = _copy(tmp_path) - r = _move(root, P["04_submitted_override"], "submitted", "--run", "342020", "--after", "342010") - assert r.returncode == 0 and "appended run 342020" in r.stdout - text = (root / P["04_submitted_override"]).read_text() - assert "- 342020: submitted — gpu — submitted 2026-09-02 — wall 0:00\n after: 342010\n" in text - f = _fields(root, P["04_submitted_override"]) - assert f["State"] == "submitted" and f["Runs"] == "342001, 342010, 342020" - r = _move(root, P["05_running_array"], "running", "--run", "342103", "--resumes", "342091_9") + led = _ledger(root) + assert [x.ident for x in led.runs] == ["3003"] + assert led.log[0].text() == ("3002_[0-3] finished — wall 20:05: wave 2, four seeds on the " + "corrected settings — all four landed") + r = _run("done", "example", "3003", "--failed", "--today", TODAY, root=root) assert r.returncode == 0 - assert _fields(root, P["05_running_array"])["State"] == "running" + led = _ledger(root) + assert led.runs == [] and led.log[0].text().startswith("3003 failed: joint fit") assert _problems(root) == [] + r = _run("done", "example", "3003", root=root) + assert r.returncode == 1 and "is not under ## Runs" in r.stderr + r = _run("done", "single", "1", "--wall", "20", root=root) + assert r.returncode == 1 and "--wall must be H:MM" in r.stderr -def test_run_self_loop_refuses_overlap_and_dangling_after(tmp_path): +def test_log_writes_the_humans_words_newest_first(tmp_path): root = _copy(tmp_path) - r = _move(root, P["05_running_array"], "running", "--run", "342091_[9-10]") - assert r.returncode == 1 and "overlaps" in r.stderr - r = _move(root, P["05_running_array"], "running", "--run", "342200", "--after", "999") - assert r.returncode == 1 and "names no run of this task" in r.stderr - r = _move(root, P["03_ready_cleared"], "ready", "--run", "1") - assert r.returncode == 1 and "already ready" in r.stderr - - -def test_submitted_to_running_and_back_to_ready(tmp_path): - root = _copy(tmp_path) - assert _move(root, P["04_submitted_override"], "running").returncode == 0 - r = _move(root, P["04_submitted_override"], "ready", "--reason", "node failure") - assert r.returncode == 1 and "still submitted | running" in r.stderr - _edit(root, P["04_submitted_override"], "- 342010: submitted", "- 342010: failed") - r = _move(root, P["04_submitted_override"], "ready") - assert r.returncode == 1 and "--reason" in r.stderr - r = _move(root, P["04_submitted_override"], "ready", "--reason", "both waves died on node failure") + r = _run("log", "example", "the sigma is the finding, not the mean", "--kind", "lesson", + "--today", TODAY, root=root) assert r.returncode == 0, r.stderr - f = _fields(root, P["04_submitted_override"]) - assert f["State"] == "ready" and f["Reset"] == "both waves died on node failure" - assert _problems(root) == [] - # a task whose runs all succeeded has nothing to reset from - _edit(root, P["03_ready_cleared"], "State: ready\n", "State: running\n") - _edit(root, P["03_ready_cleared"], "## Runs\n\n", "## Runs\n\n- 1: done — gpu — submitted 2026-09-01 — wall 1:00\n") - _edit(root, P["03_ready_cleared"], "Runs:\n", "Runs: 1\n") if "Runs:" in (root / P["03_ready_cleared"]).read_text() \ - else _edit(root, P["03_ready_cleared"], "Budget: 5:00\n", "Budget: 5:00\nRuns: 1\n") - r = _move(root, P["03_ready_cleared"], "ready", "--reason", "x") - assert r.returncode == 1 and "no failed | timeout | void run" in r.stderr - - -def test_running_to_pulled_and_partial(tmp_path): - root = _copy(tmp_path) - r = _move(root, P["05_running_array"], "pulled") - assert r.returncode == 1 and "--partial" in r.stderr - r = _move(root, P["05_running_array"], "pulled", "--partial") - assert r.returncode == 0 - assert _problems(root) == [] # the leave-to-finish head closes the partial pull - _edit(root, P["05_running_array"], "- 342102: running", "- 342102: done") - _edit(root, P["05_running_array"], "State: pulled\n", "State: running\n") - assert _move(root, P["05_running_array"], "pulled").returncode == 0 + led = _ledger(root) + assert led.log[0].kind == "lesson" and led.log[0].date == TODAY + assert len(led.log) == 8 and _problems(root) == [] + assert _run("log", "example", "plain", "--today", TODAY, root=root).returncode == 0 + assert _ledger(root).log[0].kind == "note" + r = _run("log", "example", "backdated", "--today", "2026-08-01", root=root) + assert r.returncode == 1 and "newest-first" in r.stderr + r = _run("log", "example", " ", root=root) + assert r.returncode == 1 + r = _run("log", "example", "x", "--kind", "verdict", root=root) + assert r.returncode == 2 -def test_running_to_pulled_writes_pulled_to(tmp_path): +def test_now_rewrites_the_head_pointer_and_nothing_else(tmp_path): root = _copy(tmp_path) - _edit(root, P["04_submitted_override"], "State: submitted\n", "State: running\n") - _edit(root, P["04_submitted_override"], "- 342010: submitted", "- 342010: done") - r = _move(root, P["04_submitted_override"], "pulled") - assert r.returncode == 1 and "--pulled-to" in r.stderr - r = _move(root, P["04_submitted_override"], "pulled", "--pulled-to", "/mnt/c/out/task_04") + before = _ledger(root) + r = _run("now", "example", "Pulled wave 2.\nNext: compare against wave 1.", root=root) assert r.returncode == 0, r.stderr - text = (root / P["04_submitted_override"]).read_text() - assert "- 342010: done — gpu — submitted 2026-09-01 — wall 0:00\n pulled_to: /mnt/c/out/task_04\n" in text - assert "- 342001: failed — gpu — submitted 2026-08-31 — wall 0:00 — node failure before the first step\n- 342010" in text + led = _ledger(root) + assert led.now == "Pulled wave 2.\nNext: compare against wave 1." + assert [x.ident for x in led.runs] == [x.ident for x in before.runs] + assert len(led.log) == len(before.log) assert _problems(root) == [] + assert _run("now", "example", "", root=root).returncode == 1 -def test_pulled_to_awaiting_ruling_and_rerun_to_ready(tmp_path): - root = _copy(tmp_path) - assert _move(root, P["06_pulled"], "awaiting-ruling").returncode == 0 - assert _move(root, P["09_rerun"], "ready").returncode == 0 - assert _problems(root) == [] - - -def test_legacy_born_ready_to_pulled(tmp_path): - root = _copy(tmp_path) - _edit(root, P["09_rerun"], "State: rerun\nWitness: the legacy run's mask edge matches the released mask within one pixel\n", - "State: ready\nWitness:\n") - r = _move(root, P["09_rerun"], "pulled") - assert r.returncode == 1 and "Witness: is empty" in r.stderr - _edit(root, P["09_rerun"], "Witness:\n", "Witness: mask edge within one pixel\n") - assert _move(root, P["09_rerun"], "pulled").returncode == 0 - _edit(root, P["09_rerun"], "State: pulled\n", "State: ready\n") - _edit(root, P["09_rerun"], "- 341900: legacy", "- 341900: done") - r = _move(root, P["09_rerun"], "pulled") - assert r.returncode == 1 and "legacy-born" in r.stderr - - -@pytest.mark.parametrize("rel,to,needle", [ - (P["07_awaiting_ruling"], "accepted", "rule <task> accept"), - (P["07_awaiting_ruling"], "rerun", "rule <task> rerun"), - (P["07_awaiting_ruling"], "dropped", "rule <task> drop"), - (P["01_scope"], "dropped", "rule <task> drop"), - (P["08_accepted"], "rerun", "--supersedes"), - (P["10_dropped"], "ready", "terminal"), - (P["04_submitted_override"], "pulled", "no edge submitted → pulled"), - (P["03_ready_cleared"], "running", "no edge ready → running"), - (P["06_pulled"], "ready", "no edge pulled → ready"), - (P["01_scope"], "flying", "not a task state"), -]) -def test_refused_edges(tmp_path, rel, to, needle): +def test_a_verb_refuses_a_ledger_that_does_not_parse(tmp_path): root = _copy(tmp_path) - before = (root / rel).read_text() - r = _move(root, rel, to) - assert r.returncode == 1, r.stdout - assert needle in r.stderr, r.stderr - assert (root / rel).read_text() == before + _edit(root, EX, "- 3003 — open — gpu", "- 3003 — weird — gpu") + r = _run("log", "example", "x", root=root) + assert r.returncode == 1 and "does not parse clean" in r.stderr + r = _run("log", "nobody", "x", root=root) + assert r.returncode == 1 and "no ledger for nobody" in r.stderr # --------------------------------------------------------------------------- # -# rule +# the issue block # --------------------------------------------------------------------------- # -def _rule(root, rel, verb, body, *extra, today=TODAY): - return _run("rule", rel, verb, "--body", str(body), "--today", today, *extra, root=root) - - -def test_rule_assigns_ids_in_sequence_and_moves_the_task(tmp_path): - root = _copy(tmp_path) - body = _body(tmp_path) - r = _rule(root, P["07_awaiting_ruling"], "accept", body, "--batch", "2026-09-01-pm", - "--minutes", "4", "--follow-up", "PyAutoLens#902") +def test_issue_prints_the_concise_ledger_between_markers(): + r = _run("issue", "example", "-n", "3", root=SKELETON) assert r.returncode == 0, r.stderr - assert r.stdout.strip() == "wrote rulings/2026/09/R-20260902-01.md" - rf = _fields(root, "rulings/2026/09/R-20260902-01.md") - assert rf["Task"] == P["07_awaiting_ruling"] and rf["Ruling"] == "accept" - assert rf["Runs"] == "342110, 342120" and rf["Batch"] == "2026-09-01-pm" - assert rf["Review-minutes-actual"] == "4" and rf["Follow-ups"] == "PyAutoLens#902" - assert rf["Reviewed-at"].startswith("2026-09-02T") - text = (root / "rulings/2026/09/R-20260902-01.md").read_text() - assert text.startswith("# R-20260902-01 — accept example 07_awaiting_ruling\n") - assert "## Ruling\n\nAccept. The human's words, verbatim.\n\n## Evidence\n\n- " in text - pf = _fields(root, P["07_awaiting_ruling"]) - assert pf["State"] == "accepted" and pf["Ruling"] == "R-20260902-01" - assert "R-20260902-01 — accept" in (root / P["07_awaiting_ruling"]).read_text().split("## Ruling")[1] - assert _problems(root) == [] - # the second ruling of the day is -02 - _move(root, P["06_pulled"], "awaiting-ruling") - r = _rule(root, P["06_pulled"], "rerun", body) - assert r.stdout.strip() == "wrote rulings/2026/09/R-20260902-02.md" - assert _fields(root, P["06_pulled"])["State"] == "rerun" - assert _problems(root) == [] + out = r.stdout + assert out.startswith(cortex.ISSUE_BEGIN.format(key="example")) + assert out.rstrip().endswith(cortex.ISSUE_END) + assert "**Now**" in out and "Wave 2 is on the cluster" in out + assert "`3002_[0-3]` — running — ral" in out + assert "**Last 3**" in out + assert out.count("\n- `") == 2 # two runs + assert out.count("\n- 2026-") == 3 # three entries + assert "2026-08-31 — *result*" not in out # the fourth entry is outside the window -def test_rule_supersedes_an_accepted_ruling(tmp_path): - root = _copy(tmp_path) - body = _body(tmp_path, "REWIND: the gates were wrong.\n") - r = _rule(root, P["08_accepted"], "rerun", body) - assert r.returncode == 1 and "--supersedes R-20260901-02" in r.stderr - r = _rule(root, P["08_accepted"], "rerun", body, "--supersedes", "R-20260901-01") - assert r.returncode == 1 and "already has a successor" in r.stderr - r = _rule(root, P["08_accepted"], "accept", body, "--supersedes", "R-20260901-02") - assert r.returncode == 1 and "only rerun | drop" in r.stderr - r = _rule(root, P["08_accepted"], "rerun", body, "--supersedes", "R-20260901-02") - assert r.returncode == 0, r.stderr - rf = _fields(root, "rulings/2026/09/R-20260902-01.md") - assert rf["Supersedes"] == "R-20260901-02" - pf = _fields(root, P["08_accepted"]) - assert pf["State"] == "rerun" and pf["Ruling"] == "R-20260902-01" - assert _problems(root) == [] +def test_issue_block_names_the_home_when_given(): + led = _ledger(SKELETON) + block = cortex.issue_block(led, {"status": "active"}, 5, home="https://github.com/x/y") + assert "(https://github.com/x/y/blob/main/projects/example.md)" in block -def test_rule_refuses_a_verb_the_table_forbids(tmp_path): - root = _copy(tmp_path) - body = _body(tmp_path) - for rel, verb, needle in ( - (P["05_running_array"], "accept", "not allowed from State: running"), - (P["06_pulled"], "rerun", "not allowed from State: pulled"), - (P["10_dropped"], "drop", "terminal"), - (P["01_scope"], "leave-to-finish", "running | pulled | awaiting-ruling"), - (P["09_rerun"], "accept", "not allowed from State: rerun"), - ): - before = (root / rel).read_text() - r = _rule(root, rel, verb, body) - assert r.returncode == 1, (rel, verb, r.stdout) - assert needle in r.stderr, r.stderr - assert (root / rel).read_text() == before - assert not list((root / "rulings" / "2026" / "09").glob("R-20260902-*")) - - -def test_rule_leave_to_finish_keeps_state(tmp_path): +# --------------------------------------------------------------------------- # +# retire +# --------------------------------------------------------------------------- # +def test_retire_flips_the_row_logs_it_and_refuses_over_live_runs(tmp_path): root = _copy(tmp_path) - r = _rule(root, P["06_pulled"], "leave-to-finish", _body(tmp_path, "Wait.\n")) + r = _run("retire", "example", "--why", "done", "--today", TODAY, root=root) + assert r.returncode == 1 and "still lists runs" in r.stderr + r = _run("retire", "single", "--why", "never started", "--today", TODAY, root=root) assert r.returncode == 0, r.stderr - f = _fields(root, P["06_pulled"]) - assert f["State"] == "pulled" and f["Ruling"] == "R-20260902-01" + rows, _ = cortex.load_projects(root) + assert rows["single"]["status"] == "retired" + assert rows["single"]["note"] == f"retired {TODAY}: never started" + assert _ledger(root, "single").log[0].text() == "retired: never started" assert _problems(root) == [] + assert _run("retire", "single", "--why", "again", root=root).returncode == 1 + r = _run("retire", "sleeping", "--why", "no ledger, still fine", "--today", TODAY, root=root) + assert r.returncode == 0 + assert (root / "projects.yaml").read_text().count("status: retired") == 3 -def test_rule_refuses_to_touch_an_existing_ruling(tmp_path): - root = _copy(tmp_path) - r = _rule(root, P["07_awaiting_ruling"], "accept", _body(tmp_path), today="2026-09-01") - assert r.returncode == 0 # -06 is free on 2026-09-01 - assert r.stdout.strip() == "wrote rulings/2026/09/R-20260901-06.md" - existing = root / "rulings/2026/09/R-20260901-06.md" - before = existing.read_text() - # an id that exists somewhere unexpected is still taken - (root / "rulings" / "2026" / "10").mkdir() - for n in range(7, 100): - (root / "rulings" / "2026" / "09" / f"R-20260901-{n:02d}.md").write_text("# x\n") - _move(root, P["06_pulled"], "awaiting-ruling") - r = _rule(root, P["06_pulled"], "accept", _body(tmp_path), today="2026-09-01") - assert r.returncode == 1 and "99 rulings already filed" in r.stderr - assert existing.read_text() == before - - -def test_rule_validates_batch_follow_up_and_body(tmp_path): +def test_retire_preserves_every_other_byte_of_projects_yaml(tmp_path): root = _copy(tmp_path) - body = _body(tmp_path) - r = _rule(root, P["07_awaiting_ruling"], "accept", body, "--batch", "2026-09-09-x") - assert r.returncode == 1 and "names no batches/2026-09-09-x.md" in r.stderr - r = _rule(root, P["07_awaiting_ruling"], "accept", body, "--follow-up", "PyAutoLabs/PyAutoLens#1") - assert r.returncode == 1 and "--follow-up" in r.stderr - r = _rule(root, P["07_awaiting_ruling"], "accept", _body(tmp_path, "\n\n")) - assert r.returncode == 1 and "--body is empty" in r.stderr - assert _fields(root, P["07_awaiting_ruling"])["State"] == "awaiting-ruling" - - -# --------------------------------------------------------------------------- # -# gates -# --------------------------------------------------------------------------- # -def test_gates_offline_lists_gated_tasks(): - r = _run("gates", root=SKELETON) - assert r.returncode == 0 - assert r.stdout.splitlines() == [ - "gates: 1 task(s)", - " tasks/example/02_gated_on_dev.md: gated — PyAutoArray#431, " - "https://github.com/PyAutoLabs/PyAutoFit/pull/1436", - " PyAutoArray#431 → https://github.com/PyAutoLabs/PyAutoArray/issues/431", - " https://github.com/PyAutoLabs/PyAutoFit/pull/1436 → " - "https://github.com/PyAutoLabs/PyAutoFit/issues/1436", - ] - r = _run("gates", root=EMPTY) - assert r.returncode == 0 and r.stdout.strip() == "gates: no gated task" + before = (root / "projects.yaml").read_text() + _run("retire", "sleeping", "--why", "x", "--today", TODAY, root=root) + after = (root / "projects.yaml").read_text() + diff = [(a, b) for a, b in zip(before.split("\n"), after.split("\n")) if a != b] + assert diff == [(" status: dormant", " status: retired"), + (' note: "no ledger yet"', f' note: "retired {TODAY}: x"')] # --------------------------------------------------------------------------- # -# new +# the CLI surface # --------------------------------------------------------------------------- # -def test_new_writes_a_parseable_task(tmp_path): - root = _copy(tmp_path) - r = _run("new", "example", "11_next", "--summary", "Does the next idea hold up", - "--gates", "PyAutoLens#1, https://github.com/o/r/pull/2", - "--epic", "example-programme", "--budget", "6:00", "--minutes", "5", "--witness", "a claim", - "--today", TODAY, root=root) - assert r.returncode == 0, r.stderr - assert r.stdout.strip() == "wrote tasks/example/11_next.md" - text = (root / "tasks/example/11_next.md").read_text() - assert text.startswith("# Example — 11 next\n\nProject: example\n" - "Summary: Does the next idea hold up\nState: planned\n") - f = _fields(root, "tasks/example/11_next.md") - assert f["Summary"] == "Does the next idea hold up" and "Phase" not in f - assert f["Gates"] == "PyAutoLens#1, https://github.com/o/r/pull/2" and "Lane" not in f - assert f["Filed"] == TODAY and f["Budget"] == "6:00" and f["Review-minutes"] == "5" - for name in cortex.TASK_SECTIONS: - assert f"\n## {name}\n" in text - assert _problems(root) == [] - assert _move(root, "tasks/example/11_next.md", "gated").returncode == 0 - - -def test_new_refuses_duplicates_and_unknown_projects(tmp_path): - """The slug is the identity: the only collision left is the path itself. - A second task alongside `03_ready_cleared` is fine — it just needs its own - slug — while re-using that slug is refused.""" - root = _copy(tmp_path) - s = ("--summary", "Does the anchor refit reproduce the published value") - r = _run("new", "example", "03_again", *s, "--today", TODAY, root=root) - assert r.returncode == 0, r.stderr - assert _problems(root) == [] - r = _run("new", "example", "03_ready_cleared", *s, root=root) - assert r.returncode == 1 and "already exists" in r.stderr - r = _run("new", "nobody", "x", *s, root=root) - assert r.returncode == 1 and "not a projects.yaml key" in r.stderr - r = _run("new", "example", "x", *s, "--gates", "PyAutoLabs/PyAutoLens#1", root=root) - assert r.returncode == 1 and "no owner/Repo#N form" in r.stderr - assert not (root / "tasks/example/x.md").exists() +def test_every_verb_is_registered(): + parser = cortex.build_parser() + verbs = set(parser._subparsers._group_actions[0].choices) + assert verbs == {"check", "new", "run", "running", "done", "log", "now", "issue", "retire"} -def test_new_requires_a_summary_within_the_cap(tmp_path): - """`new` refuses what `check` would reject, before writing anything.""" - root = _copy(tmp_path) - r = _run("new", "example", "11_next", root=root) - assert r.returncode == 2 and "--summary" in r.stderr - r = _run("new", "example", "11_next", "--summary", " ", root=root) - assert r.returncode == 1 and "--summary must say what the task asks" in r.stderr - eleven = " ".join(f"w{i}" for i in range(11)) - r = _run("new", "example", "11_next", "--summary", eleven, root=root) - assert r.returncode == 1 and "--summary is 11 words — at most 10" in r.stderr - assert not (root / "tasks/example/11_next.md").exists() +def test_a_bad_root_is_a_usage_error(tmp_path): + assert _run("check", root=tmp_path / "nowhere").returncode == 2 -def test_new_legacy_born_task_is_ready_and_pullable(tmp_path): - root = _copy(tmp_path) - r = _run("new", "example", "12_legacy", "--summary", "Is the quarantined run reusable", - "--legacy-run", "300000", - "--legacy-wrong", "300001_[0-3]", "--today", TODAY, root=root) - assert r.returncode == 1 and "--where" in r.stderr - r = _run("new", "example", "12_legacy", "--summary", "Is the quarantined run reusable", - "--legacy-run", "300000", - "--legacy-wrong", "300001_[0-3]", "--where", "/mnt/q", "--witness", "w", "--today", TODAY, root=root) - assert r.returncode == 0, r.stderr - f = _fields(root, "tasks/example/12_legacy.md") - assert f["State"] == "ready" and f["Runs"] == "300000, 300001" - text = (root / "tasks/example/12_legacy.md").read_text() - assert "- 300000: legacy — gpu — submitted 2026-09-02 — wall 0:00 — pre-Cortex run, migrated\n where: /mnt/q\n" in text - assert "- 300001_[0-3]: legacy_wrong — gpu" in text - assert _problems(root) == [] - assert _move(root, "tasks/example/12_legacy.md", "pulled").returncode == 0 - text = (root / "tasks/example/12_legacy.md").read_text() - # the reusable run gets pulled_to: (its where:), the wrong one does not - assert " where: /mnt/q\n pulled_to: /mnt/q\n- 300001_[0-3]: legacy_wrong" in text - assert text.count("pulled_to:") == 1 - assert _problems(root) == [] - # legacy_wrong-only: nothing to pull — rule drop is the door - r = _run("new", "example", "14", "--summary", "Is the wrong-stack run reusable", - "--legacy-wrong", "2", "--where", "/q", "--witness", "w", root=root) - assert r.returncode == 0, r.stderr - r = _move(root, "tasks/example/14.md", "pulled") - assert r.returncode == 1 and "no legacy run to pull" in r.stderr - r = _run("new", "example", "13", "--summary", "Is the gated legacy run reusable", - "--legacy-run", "1", "--where", "/q", "--gates", "PyAutoLens#1", root=root) - assert r.returncode == 1 and "cannot be gated" in r.stderr +def test_no_verb_of_the_old_schema_survives(): + text = SCRIPT.read_text() + for gone in ("def move_task", "def rule_task", "def gates_report", "awaiting-ruling", + "def load_tasks", "def load_rulings"): + assert gone not in text, gone -def test_partition_comes_from_the_project_row_or_the_flag(tmp_path): +def test_today_is_injectable_and_validated(tmp_path): root = _copy(tmp_path) - _edit(root, "projects.yaml", " partition: gpu\n", " partition: both\n") - r = _move(root, P["03_ready_cleared"], "submitted", "--run", "1") - assert r.returncode == 1 and "--partition" in r.stderr - r = _move(root, P["03_ready_cleared"], "submitted", "--run", "1", "--partition", "ral") - assert r.returncode == 0 - assert "- 1: submitted — ral — " in (root / P["03_ready_cleared"]).read_text() - - -def test_bad_root_and_bad_today(): - r = _run("check", root=Path("/nonexistent/tree")) - assert r.returncode == 2 - r = _run("move", P["01_scope"], "ready", "--today", "yesterday", root=SKELETON) + r = _run("log", "example", "x", "--today", "yesterday", root=root) assert r.returncode == 1 and "--today must be YYYY-MM-DD" in r.stderr + assert _run("log", "example", "x", root=root).returncode == 0 + assert _ledger(root).log[0].date == date.today().isoformat() diff --git a/tests/test_ledger_merge.py b/tests/test_ledger_merge.py index 56c55d1..3ab8ed1 100644 --- a/tests/test_ledger_merge.py +++ b/tests/test_ledger_merge.py @@ -2,13 +2,12 @@ This gate decides what lands on `main` with no human in the loop, so the properties that matter are the refusals: default deny for anything -unclassified, no traversal or dotfile route past the ledger prefixes, no -pytest-collectable file smuggled in beside a packet, every code home the repo +unclassified, no traversal or dotfile route past the ledger prefix, no +pytest-collectable file smuggled in beside a ledger, every code home the repo has (scripts/, tests/, .github/, policy/, docs/, projects.yaml, the prose pages) staying on the human side of the line, no doctrine file (AGENTS.md, -TEMPLATE.md) riding along because it happens to sit under a ledger dir — and, -the Cortex's own addition, a MODIFIED, DELETED or RENAMED ruling never -auto-merging (rulings/ is append-only). +TEMPLATE.md) riding along because it sits under `projects/` — and the frozen +`archive/` (the retired task, ruling and batch ledgers) never auto-merging. The one thing this gate must NOT refuse is the generated board: `dashboard.md` and `dashboard.html` are rendered from the ledger and self-healed on main, so @@ -27,13 +26,10 @@ REPO = SCRIPT.resolve().parents[1] -def test_ledger_dirs_and_registry_files_are_ledger(): +def test_ledger_dir_and_registry_files_are_ledger(): for path in ( - "tasks/example/01_scope.md", - "tasks/newproject/01_first.md", - "rulings/2026/09/R-20260901-01.md", - "batches/2026-09-01-pm.md", - "batches/reviews/2026-09-01-pm.md", + "projects/example.md", + "projects/newproject.md", "checkin.yaml", # generated from the ledger, self-healed on main by dashboard_refresh.yml "dashboard.md", @@ -47,14 +43,13 @@ def test_every_code_home_needs_a_human(): "scripts/cortex.py", "scripts/ledger_merge.py", "tests/test_cortex.py", - "tests/fixtures/skeleton/tasks/example/01_scope.md", + "tests/fixtures/skeleton/projects/example.md", ".github/workflows/ledger_merge.yml", ".github/workflows/cortex_check.yml", ".github/workflows/dashboard_refresh.yml", ".github/workflows/pages_dashboard.yml", ".gitignore", ".claude/settings.json", - ".claude/hooks/session-start.sh", "policy/never_rewrite_history.md", "docs/schema_decisions.md", # CODE: sync_cli and local_path are paths the conductor executes under. @@ -68,97 +63,84 @@ def test_every_code_home_needs_a_human(): assert not ledger_merge.is_ledger_path(path), path +def test_the_archive_is_frozen_even_for_additions(): + """The retired ledgers are history the science repos cite by id; nothing + writes there, so an added file is as much a human's call as an edit.""" + for path in ( + "archive/tasks/example/01_scope.md", + "archive/rulings/2026/09/R-20260901-01.md", + "archive/rulings/2026/09/R-20260999-01.md", + "archive/batches/2026-09-01-pm.md", + "archive/batches/reviews/2026-09-01-pm.md", + "archive/anything_new.md", + ): + assert not ledger_merge.is_ledger_path(path), path + + def test_unclassified_paths_default_to_deny(): """A root file or top-level folder nobody has thought about is code.""" - for path in ("brand_new_root_file.md", "newfolder/thing.md", "notes.txt", "skills/x/SKILL.md"): + for path in ("brand_new_root_file.md", "newfolder/thing.md", "notes.txt", + "skills/x/SKILL.md", "tasks/example/x.md", "rulings/2026/09/R-1.md"): assert not ledger_merge.is_ledger_path(path), path def test_traversal_cannot_smuggle_code_behind_a_ledger_prefix(): - for path in ("tasks/../scripts/evil.py", "rulings/../../etc/passwd", ".."): + for path in ("projects/../scripts/evil.py", "projects/../../etc/passwd", ".."): assert not ledger_merge.is_ledger_path(path), path def test_dot_paths_are_never_ledger_wherever_they_sit(): - for path in ("tasks/.github/workflows/x.yml", "batches/.hidden", ".tasks/x.md"): + for path in ("projects/.github/workflows/x.yml", "projects/.hidden", ".projects/x.md"): assert not ledger_merge.is_ledger_path(path), path -def test_inert_assets_ride_along_but_collectable_tests_do_not(): - assert ledger_merge.is_ledger_path("batches/reviews/2026-09-01-pm.md") - for path in ( - "batches/conftest.py", - "tasks/example/test_thing.py", - "rulings/2026/09/thing_test.py", - ): +def test_collectable_tests_do_not_ride_along(): + for path in ("projects/conftest.py", "projects/test_thing.py", "projects/thing_test.py"): assert not ledger_merge.is_ledger_path(path), path -def test_doctrine_under_a_ledger_dir_needs_a_human(): - """A ledger dir holds entries; `AGENTS.md` and `TEMPLATE.md` inside one are - not entries, they are the doctrine that says what an entry may be and the - template every future entry is stamped from. A branch that could auto-merge - `rulings/AGENTS.md` could edit the rule governing its own merge.""" - for path in ( - "rulings/AGENTS.md", - "batches/AGENTS.md", - "batches/reviews/AGENTS.md", - # by name, not by that fixed list — a doctrine file in a dir nobody has - # created yet is code on the day it appears - "tasks/example/AGENTS.md", - ): +def test_doctrine_under_the_ledger_dir_needs_a_human(): + """`projects/` holds entries; `AGENTS.md` and `TEMPLATE.md` inside it are + the doctrine that says what an entry may be. A branch that could + auto-merge them could edit the rule governing its own merge.""" + for path in ("projects/AGENTS.md", "projects/TEMPLATE.md"): assert not ledger_merge.is_ledger_path(path), path - # ordinary entries beside them still merge - assert ledger_merge.is_ledger_path("batches/2026-09-01-pm.md") - assert ledger_merge.is_ledger_path("batches/reviews/2026-09-01-pm.md") + assert ledger_merge.is_ledger_path("projects/example.md") -def test_every_tracked_file_under_a_ledger_dir_gets_the_right_verdict(): - """The witness on the live tree: every tracked path under tasks/, rulings/ - and batches/ is ledger unless it is doctrine.""" - out = subprocess.run(["git", "ls-files", "tasks", "rulings", "batches"], cwd=REPO, +def test_every_tracked_file_under_the_ledger_dir_gets_the_right_verdict(): + out = subprocess.run(["git", "ls-files", "projects"], cwd=REPO, capture_output=True, text=True, check=True).stdout.split() - assert out, "no ledger-dir paths tracked" + assert out, "no ledger paths tracked" for path in out: doctrine = Path(path).name in ("AGENTS.md", "TEMPLATE.md") assert ledger_merge.is_ledger_path(path) is not doctrine, path +def test_every_tracked_archive_file_is_code(): + out = subprocess.run(["git", "ls-files", "archive"], cwd=REPO, + capture_output=True, text=True, check=True).stdout.split() + assert out, "no archive paths tracked" + for path in out: + assert not ledger_merge.is_ledger_path(path), path + + def test_classify_splits_and_dedupes_preserving_order(): ledger, blocked = ledger_merge.classify( - ["checkin.yaml", "scripts/x.py", "checkin.yaml", "tasks/a/b.md", "", " "] + ["checkin.yaml", "scripts/x.py", "checkin.yaml", "projects/a.md", "", " "] ) - assert ledger == ["checkin.yaml", "tasks/a/b.md"] + assert ledger == ["checkin.yaml", "projects/a.md"] assert blocked == ["scripts/x.py"] -# --------------------------------------------------------------------------- # -# append-only rulings/ and batches/ -# --------------------------------------------------------------------------- # -def test_added_ledger_of_record_is_ledger_but_any_other_status_is_code(): - """`rulings/` is append-only because a ruling is superseded, never edited; - `batches/` joined it on 2026-09-03 when the slot apparatus was retired and - the records became history — never modified, only added.""" - for rel in ("rulings/2026/09/R-20260901-01.md", "batches/2026-09-01-pm.md", - "batches/reviews/2026-09-01-pm.md"): - assert not ledger_merge.is_append_only_violation("A", rel), rel - for status in ("M", "D", "R100", "R", "T", "C75", "U"): - assert ledger_merge.is_append_only_violation(status, rel), (rel, status) - # a modified task file is still ledger — tasks move, that is their job - assert not ledger_merge.is_append_only_violation("M", "tasks/example/01_scope.md") - - -def test_classify_entries_blocks_a_modified_ruling_or_record(): - ledger, blocked = ledger_merge.classify_entries([ - ("A", "rulings/2026/09/R-20260901-06.md"), - ("M", "rulings/2026/09/R-20260901-01.md"), - ("M", "tasks/example/08_accepted.md"), - ("D", "batches/2026-09-01-pm.md"), - ("A", "scripts/new.py"), - ]) - assert ledger == ["rulings/2026/09/R-20260901-06.md", "tasks/example/08_accepted.md"] - assert blocked == ["rulings/2026/09/R-20260901-01.md", "batches/2026-09-01-pm.md", - "scripts/new.py"] +def test_classify_entries_blocks_the_archive_whatever_the_status(): + entries = [("A", "projects/a.md"), ("M", "projects/b.md"), ("D", "projects/c.md"), + ("A", "archive/rulings/2026/09/R-20260901-09.md"), + ("M", "archive/tasks/example/01_scope.md")] + ledger, blocked = ledger_merge.classify_entries(entries) + assert ledger == ["projects/a.md", "projects/b.md", "projects/c.md"] + assert blocked == ["archive/rulings/2026/09/R-20260901-09.md", + "archive/tasks/example/01_scope.md"] def _git(cwd, *args): @@ -171,135 +153,107 @@ def _repo(tmp_path): _git(repo, "init", "-q", "-b", "main") _git(repo, "config", "user.email", "t@example.com") _git(repo, "config", "user.name", "t") - (repo / "rulings" / "2026" / "09").mkdir(parents=True) - (repo / "tasks" / "p").mkdir(parents=True) - (repo / "rulings/2026/09/R-20260901-01.md").write_text("# R-20260901-01\n\nRuling: accept\n") - (repo / "tasks/p/01_a.md").write_text("# a\n\nState: ready\n") - (repo / "checkin.yaml").write_text("refreshed: 2026-09-04T12:00Z\n") - _git(repo, "add", "-A") - _git(repo, "commit", "-q", "-m", "base") - _git(repo, "checkout", "-q", "-b", "claude/work") + (repo / "projects").mkdir() + (repo / "projects" / "a.md").write_text("# a — x\n") + (repo / "archive").mkdir() + (repo / "archive" / "old.md").write_text("old\n") + _git(repo, "add", ".") + _git(repo, "commit", "-qm", "base") + _git(repo, "checkout", "-qb", "claude/x") return repo def test_changed_entries_reads_name_status_from_git(tmp_path): repo = _repo(tmp_path) - (repo / "rulings/2026/09/R-20260901-02.md").write_text("# R-20260901-02\n") - (repo / "tasks/p/01_a.md").write_text("# a\n\nState: submitted\n") - (repo / "rulings/2026/09/R-20260901-01.md").write_text("# R-20260901-01\n\nRuling: drop\n") - _git(repo, "add", "-A") - _git(repo, "commit", "-q", "-m", "work") + (repo / "projects" / "a.md").write_text("# a — y\n") + (repo / "projects" / "b.md").write_text("# b — z\n") + _git(repo, "add", ".") + _git(repo, "commit", "-qm", "ledger") entries = ledger_merge.changed_entries("main", cwd=repo) - assert sorted(entries) == [("A", "rulings/2026/09/R-20260901-02.md"), - ("M", "rulings/2026/09/R-20260901-01.md"), - ("M", "tasks/p/01_a.md")] + assert sorted(entries) == [("A", "projects/b.md"), ("M", "projects/a.md")] ledger, blocked = ledger_merge.classify_entries(entries) - assert blocked == ["rulings/2026/09/R-20260901-01.md"] - assert set(ledger) == {"rulings/2026/09/R-20260901-02.md", "tasks/p/01_a.md"} + assert not blocked -def test_a_renamed_or_deleted_ruling_is_code_but_added_ones_merge(tmp_path): +def test_a_change_under_the_archive_is_code_on_a_real_diff(tmp_path): repo = _repo(tmp_path) - _git(repo, "mv", "rulings/2026/09/R-20260901-01.md", "rulings/2026/09/R-20260901-03.md") - _git(repo, "commit", "-q", "-m", "rename") - entries = ledger_merge.changed_entries("main", cwd=repo) - _, blocked = ledger_merge.classify_entries(entries) - assert "rulings/2026/09/R-20260901-01.md" in blocked - assert "rulings/2026/09/R-20260901-03.md" in blocked - # a branch that only ADDS rulings and moves tasks is ledger-only - _git(repo, "checkout", "-q", "main") - _git(repo, "checkout", "-q", "-b", "claude/clean") - (repo / "rulings/2026/09/R-20260901-04.md").write_text("# R-20260901-04\n") - (repo / "tasks/p/01_a.md").write_text("# a\n\nState: accepted\n") - (repo / "checkin.yaml").write_text("refreshed: 2026-09-04T13:00Z\n") - _git(repo, "add", "-A") - _git(repo, "commit", "-q", "-m", "clean") + (repo / "archive" / "old.md").write_text("edited\n") + _git(repo, "add", ".") + _git(repo, "commit", "-qm", "touch the archive") ledger, blocked = ledger_merge.classify_entries(ledger_merge.changed_entries("main", cwd=repo)) - assert blocked == [] and len(ledger) == 3 + assert blocked == ["archive/old.md"] -def _run(*args, stdin=""): - return subprocess.run( - [sys.executable, str(SCRIPT), "classify", *args], - input=stdin, - capture_output=True, - text=True, - ) +def test_a_rename_out_of_the_ledger_dir_is_code(tmp_path): + repo = _repo(tmp_path) + _git(repo, "mv", "projects/a.md", "renamed.md") + _git(repo, "commit", "-qm", "rename") + entries = ledger_merge.changed_entries("main", cwd=repo) + ledger, blocked = ledger_merge.classify_entries(entries) + assert "renamed.md" in blocked + + +def _cli(*args, stdin=""): + return subprocess.run([sys.executable, str(SCRIPT), "classify", *args], + input=stdin, capture_output=True, text=True) def test_cli_exit_codes_separate_ledger_from_code(): - assert _run("checkin.yaml", "tasks/a/b.md", "rulings/2026/09/R-20260901-01.md").returncode == 0 - result = _run("checkin.yaml", "scripts/cortex.py") - assert result.returncode == 1 - assert "scripts/cortex.py" in result.stdout + assert _cli("projects/a.md", "checkin.yaml").returncode == 0 + assert _cli("projects/a.md", "scripts/cortex.py").returncode == 1 + assert _cli("archive/rulings/2026/09/R-20260901-01.md").returncode == 1 -def test_explicit_paths_carry_no_status_and_stay_path_only(): - """Without a git diff there is no status to read, so a ruling path given - by hand is judged on its path alone — as in the Mind.""" - assert _run("rulings/2026/09/R-20260901-01.md").returncode == 0 - assert _run(stdin="rulings/2026/09/R-20260901-01.md\n").returncode == 0 +def test_stdin_paths_are_read(): + assert _cli(stdin="projects/a.md\n").returncode == 0 + assert _cli(stdin="projects/a.md\nREADME.md\n").returncode == 1 def test_an_empty_diff_is_not_permission_to_merge(): - """Exit 0 means 'go'. Nothing to merge must never read as go.""" - result = _run(stdin="\n") - assert result.returncode == 1 - assert "nothing to merge" in result.stdout + r = _cli(stdin="") + assert r.returncode == 1 def test_every_fixture_path_is_classified(): - """The witness: every path under tests/fixtures/ gets a verdict, and it is - code — `tests/` is a code home, whatever shape the files inside take.""" - out = subprocess.run(["git", "ls-files", "tests/fixtures"], cwd=REPO, - capture_output=True, text=True, check=True).stdout.split() - assert out, "no fixture paths tracked" - for path in out: - assert not ledger_merge.is_ledger_path(path), path + for p in (REPO / "tests" / "fixtures").rglob("*"): + if p.is_file(): + assert not ledger_merge.is_ledger_path(p.relative_to(REPO).as_posix()) def test_this_repos_own_workflow_cannot_auto_merge_itself(): - """Self-consistency: the gate is on the code side of its own line.""" assert not ledger_merge.is_ledger_path(".github/workflows/ledger_merge.yml") - assert not ledger_merge.is_ledger_path("scripts/ledger_merge.py") def test_workflow_invokes_the_script_as_implemented(): - workflow = (REPO / ".github" / "workflows" / "ledger_merge.yml").read_text() - assert "python3 scripts/ledger_merge.py classify --base origin/main" in workflow - assert "python3 scripts/cortex.py check" in workflow + text = (REPO / ".github" / "workflows" / "ledger_merge.yml").read_text() + assert "scripts/ledger_merge.py classify --base origin/main" in text + assert "scripts/cortex.py check" in text def test_every_workflow_this_repo_has_is_code(): - """Enumeration, not a sample: no workflow file may ever auto-merge, and the - two added with the dashboard (one of which pushes to main) are named here so - the list cannot silently fall behind the directory.""" - tracked = set(subprocess.run(["git", "ls-files", ".github/workflows"], cwd=REPO, - capture_output=True, text=True, check=True).stdout.split()) - for name in ("cortex_check.yml", "ledger_merge.yml", "dashboard_refresh.yml", - "pages_dashboard.yml"): - assert f".github/workflows/{name}" in tracked, name - for path in tracked: - assert not ledger_merge.is_ledger_path(path), path + for p in (REPO / ".github" / "workflows").glob("*.yml"): + assert not ledger_merge.is_ledger_path(p.relative_to(REPO).as_posix()) + + +def test_the_workflows_watch_the_ledger_dir_not_the_retired_ones(): + for name in ("cortex_check.yml", "dashboard_refresh.yml"): + text = (REPO / ".github" / "workflows" / name).read_text() + assert '- "projects/**"' in text, name + assert '- "tasks/**"' not in text and '- "rulings/**"' not in text, name def test_the_page_workflow_spells_the_conductor_as_implemented(): - """`--cortex` is a flag of the SUBCOMMAND, not a global: `dashboard --cortex - . --check` runs, `--cortex . dashboard --check` exits 2 (argparse) and the - check() wrapper would report a renderer failure. Pin the spelling.""" - refresh = (REPO / ".github" / "workflows" / "dashboard_refresh.yml").read_text() - assert 'python3 "$BRAIN" dashboard --cortex . --check' in refresh - assert 'python3 "$BRAIN" dashboard --cortex . --apply' in refresh - assert "git add dashboard.md dashboard.html" in refresh + text = (REPO / ".github" / "workflows" / "dashboard_refresh.yml").read_text() + assert "agents/conductors/cortex/_cortex.py" in text + assert "dashboard --cortex . --check" in text + assert "dashboard --cortex . --apply" in text def test_no_scheduled_job_mutates_the_ledger(): - """Gate grading was the one cron that wrote task headers; it was retired - on 2026-09-03. Nothing schedules a ledger write any more.""" - workflows = (REPO / ".github" / "workflows") - for path in sorted(workflows.glob("*.yml")): - text = path.read_text() - if "schedule:" not in text: - continue - assert "git add tasks" not in text, path.name - assert "git add rulings" not in text, path.name + """The only scheduled writer renders the two generated pages.""" + for p in (REPO / ".github" / "workflows").glob("*.yml"): + text = p.read_text() + if "schedule:" in text: + assert p.name == "dashboard_refresh.yml", p.name + assert "git add dashboard.md dashboard.html" in text + assert "projects/" not in text.split("git add", 1)[1].split("\n")[0] From 3ea45c7168b14434ec48ba2e38d918eb59d2aa1d Mon Sep 17 00:00:00 2001 From: Claude <noreply@anthropic.com> Date: Sat, 12 Sep 2026 05:33:49 +0000 Subject: [PATCH 2/4] cortex: render the board over the project ledgers Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UbmGoiKHHsNmMWp9GH4se --- dashboard.html | 318 ++++++++++++++++++++------------------ dashboard.md | 405 ++++++++++++++++++++----------------------------- 2 files changed, 330 insertions(+), 393 deletions(-) diff --git a/dashboard.html b/dashboard.html index 3be4411..6d0688b 100644 --- a/dashboard.html +++ b/dashboard.html @@ -4,7 +4,7 @@ <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1"> <title>PyAutoCortex Dashboard - + +.fresh{padding:.1rem .9rem;margin:1.2rem 0;background:var(--tint)}.fresh p{margin:.5rem 0}table.map{width:100%;border-collapse:collapse;font-size:.95em}table.map th{text-align:left;color:var(--muted);font-weight:600}table.map td,table.map th{border-bottom:1px solid var(--line);padding:.4rem .5rem .4rem 0;vertical-align:top}.checkin{display:flex;gap:.9rem;align-items:center;padding:.9rem 1.2rem;border:2px solid var(--line);border-radius:.6rem;margin:.2rem 0 1.4rem}.checkin .mark{font-size:2rem;line-height:1}.checkin .label{display:block;font-size:.75rem;letter-spacing:.06em;text-transform:uppercase;color:var(--muted)}.checkin time{font-size:1.6rem;font-weight:700;font-variant-numeric:tabular-nums;display:block}.checkin .note{font-size:.85rem;color:var(--muted)}.checkin.fresh-ok{border-color:var(--ok);background:color-mix(in srgb,var(--ok) 12%,transparent)}.checkin.fresh-ok .mark{color:var(--ok)}.checkin.fresh-bad{border-color:var(--bad);background:color-mix(in srgb,var(--bad) 12%,transparent)}.checkin.fresh-bad .mark{color:var(--bad)}.project h3{font-size:1.35rem;font-weight:700;padding-bottom:.2rem;border-bottom:2px solid var(--accent);letter-spacing:.01em;margin-top:1.6rem}.pathchip{display:inline-block;background:var(--accent);color:#fff;font-family:ui-monospace,SFMono-Regular,Menlo,monospace;font-size:.88em;padding:.12em .5em;border-radius:.3rem;overflow-wrap:anywhere}@media(prefers-color-scheme:dark){.pathchip{background:color-mix(in srgb,var(--accent) 55%,#000)}}.stale{color:var(--bad);font-weight:600} -

PyAutoCortexDashboard

Question. Run. Rule.

Every science task the Cortex is holding, one line each under its project. Tap a task's 📋 and its next command is on your clipboard. The verdict is always yours — this page hands you the command, never the ruling.

-
  • 1Awaiting ruling
  • 37Recent rulings
-

Last updated 2026-09-11.

+

PyAutoCortexDashboard

Question. Run. Rule.

One ledger per science project: what is on the cluster right now, what the human said they would do next, and the last few things they wrote down. Nothing here is inferred from a result — the board hands you the check-in and the resume prompt; what a run meant is yours to log.

+
  • 1Running
  • 17Open
  • 7Projects
+

Last updated 2026-09-12.

markdown version · GitHub Page

-

check in since last time

+

check in since last time

Last check-in

Summary

-

One row per active project — the next thing, what it is holding, and when it last ruled.

- - - - - - - - + + + + + + + +
ProjectNext taskAwaitingRunningLast ruling
analytic_gaussianIs criterion 2's mu threshold or the minimal-EP control wrong002026-09-10
autolens_inferenceDo six backend legs agree on the HST SLaM posterior?00-
ep_toy_gaussianDo three EP fits agree with NUTS at N=5001-
euclid_dr1_prelimDo ordered Nautilus reruns reproduce the euclid catalogue values?102026-09-10
ic50_workspaceDoes EP match the graphical joint fit end to end01-
slope_hierarchy_scaleDoes hierarchical slope recovery hold at N=25 to 5001-
subhalo_validationDoes rectangular_adapt false-detect a subhalo on the clumpy source052026-09-07
ProjectRunningOpenLast update
subhalo_validation052026-09-07
euclid_dr1_prelim122026-09-11
analytic_gaussian002026-09-10
ep_toy_gaussian022026-09-10
slope_hierarchy_scale042026-09-09
ic50_workspace042026-09-09
autolens_inference002026-09-11
-

Nothing open

- - - - - - - - -
ProjectStatusTasksRetire
concrdormantnone
cowls_dianadormantnone
eucliddormantplanned 3
pj011646dormantnone
profilingdormantnone
subhalo_simulationsdormantnone
-
2 retired
    -
  • inference_programme — retired 2026-09-07: restarting from scratch: the project setup was unsatisfactory and it informed how the Cortex manages projects; RAL outputs stashed at /mnt/ral/jnightin/inference_programme_retired_2026-09-07
    • -
  • slope_hierarchy — retired 2026-09-08: wrapped up 2026-07-22 at N=5; succeeded by slope_hierarchy_scale; tree vaulted under z_projects_complete/
    • -
-

Projects markdown version

+

Projects

-

analytic_gaussian

-

Local /mnt/c/Users/Jammy/Science/analytic_gaussian

-

RAL /mnt/ral/jnightin/analytic_gaussian

-

active · ral partition · tasks: accepted 1 · planned 1

-

Is criterion 2's mu threshold or the minimal-EP control wrongplanned

-

tasks/analytic_gaussian/

+

subhalo_validation — Do the SLaM subhalo pipelines false-detect on lenses with no subhalo

+

active · ral · no issue yet · projects/subhalo_validation.md · local /mnt/c/Users/Jammy/Science/subhalo_validation · RAL /mnt/ral/jnightin/subhalo_validation

+

Now

+

Five chains on RAL under the corrected adapt settings: rectangular_adapt on pl_eff_0 / pl_eff_1_outer (job A done, job B 342311 submitted) and the delaunay_adapt_split reruns plus rectangular_adapt on pl_sersic_0. pl_sersic_0 on delaunay_adapt_split is accepted (no false detection, +2.557).
Next: pull when the numba chains land and read the evidence_increase of each; no more 1000-pixel runs.

+

Runs

+
    +
  • 342299_1 — open — ral — 2026-09-07 — delaunay_adapt_split_pl_eff_0: job B, numba chain at the standard 1250-pixel Delaunay mesh (8c / 96gb / 48 h); reloads job A 342231_1; the 09-07 ruling: run al…
    • +
  • 342299_2 — open — ral — 2026-09-07 — delaunay_adapt_split_pl_eff_1_outer: job B, numba chain at the standard 1250-pixel Delaunay mesh (8c / 96gb / 48 h); reloads job A 342231_2; the 09-07 ruling:…
    • +
  • 342311_1 — open — ral — 2026-09-07 — rectangular_adapt_pl_eff_0: job B, numba chain (8c / 96gb / 48 h) at the standard 1250-pixel mesh under PIPELINE=rectangular_adapt; reloads job A 342237_1; the…
    • +
  • 342311_2 — open — ral — 2026-09-07 — rectangular_adapt_pl_eff_1_outer: job B, numba chain (8c / 96gb / 48 h) at the standard 1250-pixel mesh under PIPELINE=rectangular_adapt; reloads job A 342237_…
    • +
  • 342311_0 — open — ral — 2026-09-07 — rectangular_adapt_pl_sersic_0: job B, numba chain (8c / 96gb / 48 h) at the standard 1250-pixel mesh under PIPELINE=rectangular_adapt; reloads job A 342237_0;…
    • +
+

Last 5

+
    +
  • 2026-09-07 — result — accepted delaunay_adapt_split_pl_sersic_0 (R-20260907-01): Accepted. The numba delaunay_adapt_split chain on pl_sersic_0 (job 342273_0, 22 h 08 m, 1250 Hilbert / 1310 Delaunay pixels) lands evidence_increase +2.557 (< 5): no false detectio…
    • +
  • 2026-09-07 — run — 342311_0 submitted: rectangular_adapt_pl_sersic_0 — job B, numba chain (8c / 96gb / 48 h) at the standard 1250-pixel mesh under PIPELINE=rectangular_adapt; reloads job A 342237_0; the 09-07 ask: submit the rectangular runs too
    • +
  • 2026-09-07 — run — 342311_2 submitted: rectangular_adapt_pl_eff_1_outer — job B, numba chain (8c / 96gb / 48 h) at the standard 1250-pixel mesh under PIPELINE=rectangular_adapt; reloads job A 342237_2; the 09-07 ask: submit the rectangular runs too
    • +
  • 2026-09-07 — run — 342311_1 submitted: rectangular_adapt_pl_eff_0 — job B, numba chain (8c / 96gb / 48 h) at the standard 1250-pixel mesh under PIPELINE=rectangular_adapt; reloads job A 342237_1; the 09-07 ask: submit the rectangular runs too
    • +
  • 2026-09-07 — run — 342299_2 submitted: delaunay_adapt_split_pl_eff_1_outer — job B, numba chain at the standard 1250-pixel Delaunay mesh (8c / 96gb / 48 h); reloads job A 342231_2; the 09-07 ruling: run all datasets at 1250, no more pix1000
    • +
+

full log

+

resume subhalo_validation

-

autolens_inference

-

Local /home/jammy/Code/PyAutoLabs/autolens_inference

-

RAL /mnt/ral/jnightin/autolens_inference

-

active · both partition · tasks: ready 1

-

Do six backend legs agree on the HST SLaM posterior?ready

-

tasks/autolens_inference/ · issues

+

euclid_dr1_prelim — Refit the ten DR1-prelim tiles and rebuild the catalogue on RAL

+

active · both · no issue yet · projects/euclid_dr1_prelim.md · local /mnt/c/Users/Jammy/Science/euclid_dr1_prelim · RAL /mnt/ral/jnightin/euclid_dr1_prelim

+

Now

+

342629 reran the ten tiles on refreshed libraries; eight finished and confirmed all four 09-10 defects fixed, tiles 1 and 3 are still in vis_lp. The SED chain ran on GPU for the eight (342648) and the CPU verification on tile 9 (342668_9) passed in 9:24, faster than the GPU — the CPU route is the one the main experiment uses.
Next: pull, bundle with output_sed, compare magnitudes (µJy fluxes) against the 20260623 catalogue; then tiles 1/3 SED on the CPU script. The parity clause needs restating: rerun scatter, not ordering, is what fails it.

+

Runs

+
    +
  • 342629_[0-9] — running — ral — 2026-09-10 — ordered_rerun_catalogue_parity: two-stage vis_lp(n_live=750)+vis_pix(n_live=300), array 0-9, ten dr1_prelim tiles; libraries refreshed BEFORE submit (PyAutoNer…
    • +
  • 342648_[0,2,4-9] — open — gpu — 2026-09-11 — ordered_rerun_catalogue_parity: SED chain `hpc/batch_gpu/submit_sersic_waveband_8tiles` (a copy of `submit_sersic_waveband` differing only in `--array=0,2,4-9`…
    • +
  • 342668_9 — open — ral — 2026-09-11 — ordered_rerun_catalogue_parity: CPU verification of the SED chain via hpc/batch_cpu/submit_sersic_waveband (pipeline PR #70, sbatch --array=9): tile 102008848,…
    • +
+

Last 5

+
    +
  • 2026-09-11 — note — **CPU verification PASSED and was faster than the GPU.** `342668_9` (ral, 8 CPU, JAX on the CPU backend) COMPLETED in **9:24** wall, exit 0, all eight `sersic_lens_model/{vis,nir_y,nir_j,nir_h,decam_g,decam_r,decam_i,decam_z}` searches wri…
    • +
  • 2026-09-11 — note — **342629 scored on the eight tiles that finished; all four 09-10 defects confirmed fixed; parity fails on rerun scatter, not ordering.** `sacct`: tasks 0,2,4,5,6,7,8,9 COMPLETED (walls 5:45–9:42, MaxRSS 7.6–10.9 GB); tasks 1 (`Tile10200729…
    • +
  • 2026-09-11 — run — 342668_9 submitted: ordered_rerun_catalogue_parity — CPU verification of the SED chain via hpc/batch_cpu/submit_sersic_waveband (pipeline PR #70, sbatch --array=9): tile 102008848, JAX on the CPU backend, 8 CPU; same seeded vis_lp zip as t…
    • +
  • 2026-09-11 — run — 342648_[0,2,4-9] submitted: ordered_rerun_catalogue_parity — SED chain `hpc/batch_gpu/submit_sersic_waveband_8tiles` (a copy of `submit_sersic_waveband` differing only in `--array=0,2,4-9`, because `hpc/sync submit` passes no sbatch argume…
    • +
  • 2026-09-10 — result — accepted dr1_prelim_10_lens_science_run (R-20260910-03): For euclid_dr1_prelim, I am content these 3 are success so wrap them up and then do the 4th task on the catalogue stuff
    • +
+

full log

+

resume euclid_dr1_prelim

-

ep_toy_gaussian

-

Local /mnt/c/Users/Jammy/Science/ep_toy_gaussian

-

RAL /mnt/ral/jnightin/ep_toy_gaussian

-

active · ral partition · tasks: submitted 1

-

Do three EP fits agree with NUTS at N=50submitted — runs 342639, 342640 · wall 0:00 of 24:00 (0%)

-

tasks/ep_toy_gaussian/ · issues

+

analytic_gaussian — Graphical and EP against a closed-form Gaussian posterior, at ensemble scale

+

active · ral · no issue yet · projects/analytic_gaussian.md · local /mnt/c/Users/Jammy/Science/analytic_gaussian · RAL /mnt/ral/jnightin/analytic_gaussian

+

Now

+

Wave 1 (342413, 200 seeds at N=5) is accepted as the baseline: autofit EP is exact on the Gaussian leg, leg B sigma misses as pre-registered (78/200), collapse rate 0/200. Follow-ups filed through /intake.
Next: settle whether criterion 2's mu threshold or the minimal-EP control is wrong before any rerun; the N=25 rung is written but not submitted.

+

Runs

+

nothing on the cluster

+

Last 5

+
    +
  • 2026-09-10 — result — accepted ensemble_parity (R-20260910-04): ok accept and intake the things to address. we can do another run down the line so also achieve the results in the analytic_gaussian project for future comparison
    • +
  • 2026-09-10 — note — question: Is criterion 2's mu threshold or the minimal-EP control wrong — planned, never run [archive/tasks/analytic_gaussian/minimal_ep_legb_mu_threshold.md]
    • +
  • 2026-09-09 — run — 342413_[0-199] submitted (done, wall 0:05): ensemble_parity — the N=5 seed ensemble, 200 seeds, 50 concurrent, `hpc/batch_cpu/submit_ensemble_n5`, sample `ens_n5`. RAL PyAuto mirror verified at PyAutoFit `66f9f8d5d` before submission, whic…
    • +
  • 2026-09-09 — note — question: Do graphical and EP recover closed-form means and errors [archive/tasks/analytic_gaussian/ensemble_parity.md]
    • +
+

full log

+

resume analytic_gaussian

-

euclid_dr1_prelim

-

Local /mnt/c/Users/Jammy/Science/euclid_dr1_prelim

-

RAL /mnt/ral/jnightin/euclid_dr1_prelim

-

active · both partition · tasks: accepted 3 · awaiting-ruling 1

-

Do ordered Nautilus reruns reproduce the euclid catalogue values?awaiting-ruling

-

↳ the results are good — accept and open the next task

-

↳ run it again

-

tasks/euclid_dr1_prelim/

+

ep_toy_gaussian — NUTS versus three EP fits on the #1405 collapse toy at N=50

+

active · ral · no issue yet · projects/ep_toy_gaussian.md · local /mnt/c/Users/Jammy/Science/ep_toy_gaussian · RAL /mnt/ral/jnightin/ep_toy_gaussian

+

Now

+

n50_seed42 is on RAL: NUTS + one-by-one as 342639, EP x3 on identical data as 342640_[0-2]. The one-by-one leg finished (50/50 in 319 s); the EP repeats are turning per-factor Dynesty fits.
Next: pull and run scripts/compare.py --sample n50_seed42; settle first whether a prior factor with only NO_CHANGE rows counts as zero SUCCESS updates.

+

Runs

+
    +
  • 342639 — open — ral — 2026-09-10 — nuts_vs_ep_x3_n50: one_by_one + BlackJAX NUTS, partition ral
    • +
  • 342640 — open — ral — 2026-09-10 — nuts_vs_ep_x3_n50: EP x3 on identical data, array 0-2
    • +
+

Last 5

+
    +
  • 2026-09-10 — run — 342640 submitted: nuts_vs_ep_x3_n50 — EP x3 on identical data, array 0-2
    • +
  • 2026-09-10 — run — 342639 submitted: nuts_vs_ep_x3_n50 — one_by_one + BlackJAX NUTS, partition ral
    • +
  • 2026-09-10 — note — question: Do three EP fits agree with NUTS at N=50 [archive/tasks/ep_toy_gaussian/nuts_vs_ep_x3_n50.md]
    • +
+

full log

+

resume ep_toy_gaussian

-

ic50_workspace

-

Local /mnt/c/Users/Jammy/Science/ic50_workspace

-

RAL /mnt/ral/jnightin/ic50_workspace

-

active · ral partition · tasks: submitted 1

-

Does EP match the graphical joint fit end to endsubmitted — runs 342408, 342409, 342411, 342412 · wall 0:00 of 48:00 (0%)

-

tasks/ic50_workspace/ · issues

+

slope_hierarchy_scale — Hierarchical slope recovery with graphical and EP at N=25 to 50

+

active · both · no issue yet · projects/slope_hierarchy_scale.md · local /mnt/c/Users/Jammy/Science/slope_hierarchy_scale · RAL /mnt/ral/jnightin/slope_hierarchy_scale

+

Now

+

The N=25 scale-up is running: graphical array 342348_[0-24], joint fits 342350_0 / 342351_0, and the EP rerun 342410 on the mirror that carries #1580.
Next: pull wave 1 and compare EP against the NUTS parent sigma; then decide whether to go to N=50.

+

Runs

+
    +
  • 342348_[0-24] — open — gpu — 2026-09-08 — n25_scale_up: one lens per array task, hpc/batch_gpu/submit_one_by_one, sample_n25_seed42
    • +
  • 342350_0 — open — gpu — 2026-09-08 — n25_scale_up: joint hierarchical NUTS fit, hpc/batch_gpu/submit_graphical
    • +
  • 342351_0 — open — ral — 2026-09-08 — n25_scale_up: EP arm on the CPU partition, hpc/batch_cpu/submit_ep, max_steps 12; RAL PyAutoFit mirror 68ff9bd57 predates PyAutoFit#1580, so this arm runs with…
    • +
  • 342410 — open — ral — 2026-09-09 — n25_scale_up: EP arm rerun on the JAX-on-CPU path (no multiprocessing pool); replaces 342351_0
    • +
+

Last 5

+
    +
  • 2026-09-09 — run — 342410 submitted: n25_scale_up — EP arm rerun on the JAX-on-CPU path (no multiprocessing pool); replaces 342351_0
    • +
  • 2026-09-08 — run — 342351_0 submitted: n25_scale_up — EP arm on the CPU partition, hpc/batch_cpu/submit_ep, max_steps 12; RAL PyAutoFit mirror 68ff9bd57 predates PyAutoFit#1580, so this arm runs without the stale-mask fixed-point fix
    • +
  • 2026-09-08 — run — 342350_0 submitted: n25_scale_up — joint hierarchical NUTS fit, hpc/batch_gpu/submit_graphical
    • +
  • 2026-09-08 — run — 342348_[0-24] submitted: n25_scale_up — one lens per array task, hpc/batch_gpu/submit_one_by_one, sample_n25_seed42
    • +
  • 2026-07-22 — note — question: Does hierarchical slope recovery hold at N=25 to 50 [archive/tasks/slope_hierarchy_scale/n25_scale_up.md]
    • +
+

full log

+

resume slope_hierarchy_scale

-

slope_hierarchy_scale

-

Local /mnt/c/Users/Jammy/Science/slope_hierarchy_scale

-

RAL /mnt/ral/jnightin/slope_hierarchy_scale

-

active · both partition · tasks: running 1

-

Does hierarchical slope recovery hold at N=25 to 50running — runs 342348_[0-24], 342350_0, 342351_0, 342410 · wall 0:00 of 48:00 (0%)

-

tasks/slope_hierarchy_scale/ · issues

+

ic50_workspace — EP against the graphical joint fit on IC50 dose-response data, scaling up

+

active · ral · no issue yet · projects/ic50_workspace.md · local /mnt/c/Users/Jammy/Science/ic50_workspace · RAL /mnt/ral/jnightin/ic50_workspace

+

Now

+

Parity at N=5 achieved on RAL (33/33 within 3σ for both methods, runs 342408 / 342409); the scale ladder runs 342411 and 342412 are submitted.
Next: pull the ladder and see where EP's cost and hill_coef width go as N grows; ep_lbfgs_jax is the scale lever.

+

Runs

+
    +
  • 342408 — open — ral — 2026-09-09 — ep_scale_up: EP sim, nlive 150, max_steps 12
    • +
  • 342409 — open — ral — 2026-09-09 — ep_scale_up: graphical sim joint Dynesty, nlive 150
    • +
  • 342411 — open — ral — 2026-09-09 — ep_scale_up: EP scale ladder, sim rungs 5/10/25/50, nlive 150 max_steps 12
    • +
  • 342412 — open — ral — 2026-09-09 — ep_scale_up: graphical scale ladder, sim rungs 5/10/25/50, nlive 150
    • +
+

Last 5

+
    +
  • 2026-09-09 — run — 342412 submitted: ep_scale_up — graphical scale ladder, sim rungs 5/10/25/50, nlive 150
    • +
  • 2026-09-09 — run — 342411 submitted: ep_scale_up — EP scale ladder, sim rungs 5/10/25/50, nlive 150 max_steps 12
    • +
  • 2026-09-09 — run — 342409 submitted: ep_scale_up — graphical sim joint Dynesty, nlive 150
    • +
  • 2026-09-09 — run — 342408 submitted: ep_scale_up — EP sim, nlive 150, max_steps 12
    • +
  • 2026-08-19 — note — question: Does EP match the graphical joint fit end to end [archive/tasks/ic50_workspace/ep_scale_up.md]
    • +
+

full log

+

resume ic50_workspace

-

subhalo_validation

-

Local /mnt/c/Users/Jammy/Science/subhalo_validation

-

RAL /mnt/ral/jnightin/subhalo_validation

-

active · ral partition · tasks: accepted 1 · running 2 · submitted 3

-

Does rectangular_adapt false-detect a subhalo on the clumpy sourcerunning — runs 342237_1, 342240_1, 342311_1 · wall 1:52 of 48:00 (4%) · ⚠️ failed runs: 342240_1

-

Does rectangular_adapt false-detect a subhalo on the outer-component lensrunning — runs 342237_2, 342240_2, 342311_2 · wall 1:52 of 48:00 (4%) · ⚠️ failed runs: 342240_2

-

Does delaunay_adapt_split false-detect a subhalo on the clumpy sourcesubmitted — runs 342027_1, 342231_1, 342234_1, 342299_1 · wall 45:10 of 48:00 (94%) · ⚠️ failed runs: 342234_1

-

Does delaunay_adapt_split false-detect on the outer-component lens and AdaptSplit fixsubmitted — runs 342027_2, 342093_0, 342231_2, 342234_2, 342299_2 · wall 47:05 of 48:00 (98%) · ⚠️ failed runs: 342234_2

-

Does rectangular_adapt behave like delaunay_adapt_split on the clean nullsubmitted — runs 342094_0, 342095_0, 342237_0, 342240_0, 342311_0 · wall 18:01 of 48:00 (38%) · ⚠️ failed runs: 342240_0

-

tasks/subhalo_validation/ · issues

+

autolens_inference — Do the PyAutoLens inference backends agree on the HST SLaM posterior

+

active · both · no issue yet · projects/autolens_inference.md · local /home/jammy/Code/PyAutoLabs/autolens_inference · RAL /mnt/ral/jnightin/autolens_inference

+

Now

+

Born 2026-09-10 as the from-scratch restart of inference_programme; slam_hst_base is the first question and is ready to submit — nothing has run yet.
Next: hpc/sync submit the six backend legs on the HST SLaM dataset and record the job ids here.

+

Runs

+

nothing on the cluster

+

Last 5

+
    +
  • 2026-09-11 — note — question: Do six backend legs agree on the HST SLaM posterior? — ready, never run [archive/tasks/autolens_inference/slam_hst_base.md]
    • +
+

full log

+

resume autolens_inference

-

Awaiting ruling markdown version

Results are in and nothing is running — the human's verdict is the only thing outstanding. Ordered failures first, then the tasks a ruling is required for, then the clean ones.

-

Do ordered Nautilus reruns reproduce the euclid catalogue values?awaiting-ruling

-

↳ the results are good — accept and open the next task

-

↳ run it again

-

Recent rulings markdown version

The ledger of record, newest first. A verdict recorded only outside the Cortex does not exist.

+
+

euclid — The original Euclid DR1 grade-AB catalogue project, superseded by euclid_dr1_prelim

+

dormant · both · no issue yet · projects/euclid.md · local /mnt/c/Users/Jammy/Science/euclid · RAL /mnt/ral/jnightin/euclid_strong_lens_modeling_pipeline

+

Now

+

Dormant since 2026-09-07: superseded by euclid_dr1_prelim. Three questions were planned and never run (see the archive).

+

Runs

+

nothing on the cluster

+

Last 5

+
    +
  • 2026-08-28 — note — question: Do we recover Sersic indices or is prior pile-up artefact — planned, never run [archive/tasks/euclid/sersic_index_recovery.md]
    • +
  • 2026-08-28 — note — question: Can a fitted Euclid lens be resimulated and recovered — planned, never run [archive/tasks/euclid/resimulate_fitted_lens_simulator.md]
    • +
  • 2026-08-28 — note — question: How robust are magnification estimates under model match and mismatch — planned, never run [archive/tasks/euclid/magnification_robustness.md]
    • +
+

full log

+
+
2 retired
    +
  • inference_programme — The first inference programme, retired and restarted as autolens_inference — retired 2026-09-07: restarting from scratch: the project setup was unsatisfactory and it informed how the Cortex manages projects; RAL outputs stashed at /mnt/ral/jnightin/inference_programme_retired_2026-09-07
    • +
  • slope_hierarchy — Hierarchical slope recovery at N=5, wrapped up and succeeded by slope_hierarchy_scale — retired 2026-09-08: wrapped up 2026-07-22 at N=5; succeeded by slope_hierarchy_scale; tree vaulted under z_projects_complete/
    • +
+

No ledger

- - - - - - - - - - - - - + + + + + +
RulingVerbTaskBatch
R-20260910-04accepttasks/analytic_gaussian/ensemble_parity.md-
R-20260910-03accepttasks/euclid_dr1_prelim/dr1_prelim_10_lens_science_run.md-
R-20260910-02accepttasks/euclid_dr1_prelim/ordered_mge_control_unordered_102005065.md-
R-20260910-01accepttasks/euclid_dr1_prelim/ordered_mge_witness_102005065.md-
R-20260907-06droptasks/inference_programme/legacy_point_output_sweep.md-
R-20260907-05droptasks/inference_programme/cluster_extended_source_inference.md-
R-20260907-04droptasks/inference_programme/likelihood_term_levers.md-
R-20260907-03droptasks/inference_programme/prodigy_mass_pix.md-
R-20260907-02droptasks/inference_programme/nautilus_mass_pix_baseline.md-
R-20260907-01accepttasks/subhalo_validation/delaunay_adapt_split_pl_sersic_0.md-
R-20260904-05droptasks/inference_programme/multiband_compile_census_completion.md-
R-20260904-04droptasks/inference_programme/mass_pix_gradient_cost_probe.md-
ProjectStatusNote
profilingdormant2026-05 checkpointed sweep harness, PyAutoNSS venv; name collides with autolens_profiling; ral_root planned, sync_verbs unverified
cowls_dianadormantlocal git, no remote; ral_root planned, sync_verbs unverified
subhalo_simulationsdormantgit init, zero commits, 21 GB; ral_root planned, sync_verbs unverified
pj011646dormantpersonal remote, recorded as a fact — not a PyAutoLabs repo; ral_root planned, sync_verbs unverified
concrdormantpersonal remote Jammy2211/cosmology_and_cancer, recorded as a fact; ral_root planned, sync_verbs unverified