Repository navigation
maintenance: EP cortex check-in clean-up 2026-09-30 (stale runs, projects.yaml de-dup, wave-2 prep) #50
Description
Overview
The 2026-09-30 Cortex check-in of the four active EP projects (ep_toy_gaussian, analytic_gaussian, ic50_workspace, slope_hierarchy_scale) found small bookkeeping and diagnosis issues standing between us and a new wave of runs: stale open ledger runs, two out-of-date ## Now sections, a tripled projects.yaml, a stranded ledger edit blocking check-in push, and four per-project science questions (wave-2 memory sizing, criterion-2 opinion, an EP N=50 non-finite sufficient-statistics failure, the moment-matching gate). This task clears them in order and stops at a go/no-go table for new submissions.
Plan
- Commit the stranded euclid_dr1 ledger edit as ledger; close every stale
openEP run withcortex.py doneafter confirming exit codes from logs + dated sacct; rewrite the two stale## Nowsections in the human's words. - De-duplicate
projects.yaml(commit 25e0168 tripled it; copies are byte-identical), add a duplicate-key guard toload_projectswith a test, re-render the dashboard → Cortex PR. - ep_toy_gaussian: size NUTS memory from the N=50 model, raise
--memabove 8 GB, re-verify the RAL mirror contains PyAutoFit#1632/#1634 and dependency floors, pick a new sample name (n50_seed42_w2); no submit. - analytic_gaussian: get astra's (Codex) independent opinion on criterion 2 vs the minimal-EP control; record verbatim as a
note; human rules. - ic50_workspace: diagnose the N=50
assert np.isfinite(suff_stats).all()failure; file/intakebug if a library defect; report EP cost and hill_coef width vs N from the completed rungs. - slope_hierarchy_scale: log the never-recorded 342348 / 342350_0 outcomes; surface the moment-matching prompt's size/status for the human go/no-go.
- Stop at a per-project go/no-go table (concrete next submission or named blocker).
Detailed implementation plan
Affected Repositories
- PyAutoCortex (primary) — ledgers on main via
cortex checkin --apply;projects.yaml+ guard + test via PR - PyAutoFit — read-only diagnosis; any fix routes through
/intake(bug, autofit)
Branch Survey
| Repository | Current Branch | Dirty? |
|---|---|---|
| organs/PyAutoCortex | main | projects/euclid_dr1.md modified (ledger, 2026-09-30 09:22, to be committed) |
| fit/PyAutoFit | main | clean |
Suggested branch: feature/ep-checkin-cleanup (worktree ~/Code/PyAutoLabs-wt/ep-checkin-cleanup/)
Facts established at planning
projects.yaml: commit25e0168(2026-09-17) took the file 262 → 610 lines. Copy 1 = lines 118–291, copy 2 = 292–451, copy 3 = 452–610; copies 2 and 3 are byte-identical to copy 1 minus theeuclid_sersicsrow. Keep lines 1–291, delete 292–610.load_projects(scripts/cortex.py:268) usesyaml.safe_load, which keeps the last duplicate silently.- Dirty
projects/euclid_dr1.md: +15 ledger lines (run 374152 submitted today; 09-28 v2-grade vis_lp batches 366938/367039/367141/367243/367244; two notes) written by a sibling auto-mode session. Commit, do not discard. - Local science checkouts (
/mnt/c/Users/Jammy/Science/<project>) hold the SLURM logs.ic50_workspace/hpc/batch_cpu/error/error.342411.errends inautofit/messages/abstract.py:316 assert np.isfinite(suff_stats).all()viaresult.projected_model→prior.project→message.projectinsideoptimiser.optimise(factor_approx)(RAL mirror/mnt/ral/jnightin/PyAuto/PyAutoFit). - ep_toy_gaussian
hpc/batch_cpu/submit_nuts:10andsubmit_ep:11both#SBATCH --mem=8gb.
Implementation Steps
- Phase 1 — bookkeeping (canonical Cortex main). Pull ff-only; commit
projects/euclid_dr1.mdascortex(euclid_dr1): ledger — …. Confirm each stale run from log tails +ssh euclid_jump sacct -S 2026-09-08 -E 2026-09-11 -j <ids> -X; thencortex.py done ic50_workspace 342408|342409|342412and342411 --failed;cortex.py done slope_hierarchy_scale 342348_[0-24]|342350_0|342351_0. Draft## Nowfor ic50_workspace and slope_hierarchy_scale, confirm wording with the human, write withcortex.py now.cortex.py check→ OK;pyauto-brain cortex checkin --apply. - Phase 1c — projects.yaml (worktree, PR). Delete lines 292–610; verify
yaml.safe_loadstill yields 11 keys. Inscripts/cortex.pyload_projects, pre-scan column-0 keys (^([a-z][a-z0-9_]*):\s*(#.*)?$) beforeyaml.safe_loadand emitprojects.yaml: duplicate project key 'X' (lines a, b)as a problem. Addtests/test_cortex.py::test_projects_yaml_rejects_duplicate_project_keys. One-line rule in REFERENCE.md "restricted YAML subset". Re-render dashboard;pytest tests -q. - Phase 2 — ep_toy_gaussian. Read the 342639 OOM evidence, size NUTS memory for N=50 window adaptation (propose 32 GB unless evidence says otherwise), edit
hpc/batch_cpu/submit_nutslocally. Verify the RAL mirror HEAD is an ancestor-descendant of #1632 and #1634 and diffpip freezevs dependency floors. Record viacortex.py log … note; draftnow. - Phase 3 — analytic_gaussian. Brief = pre-registered criteria (
archive/tasks/analytic_gaussian/minimal_ep_legb_mu_threshold.md), wave-1 aggregate (analytic_gaussianb44390d), minimal-EP control code.codex exec --skip-git-repo-check …; record verbatim asnote; bring to the human. - Phase 4 — ic50_workspace. From
error.342411.err/output.342411.outidentify the factor, message class andsuff_statsat the failingproject, the EP step and rung; establish the mirror commit at run time and whether latermessages/abstract.pychanges touch the path. Library bug →/intakebug prompt with the traceback as Witness. Report EP cost + hill_coef width vs N from rungs 5/10/25; record next-ladder options. - Phase 5 — slope_hierarchy_scale. Extract 342348 per-lens and 342350_0 joint-NUTS outcomes from the output logs;
cortex.py log … note. Surfacedraft/feature/autofit/ep_hierarchical_scatter_moment_matching.md(medium, supervised, ready, epic graphical-ep) for the go/no-go. - Phase 6 — go/no-go + ship. Table per project;
ship_libraryfor the Cortex PR;cortex checkin --applyoncenowwordings are confirmed.
Key Files
organs/PyAutoCortex/projects.yaml— de-dup (keep lines 1–291)organs/PyAutoCortex/scripts/cortex.py—load_projectsduplicate-key guardorgans/PyAutoCortex/tests/test_cortex.py— new guard testorgans/PyAutoCortex/projects/{ep_toy_gaussian,analytic_gaussian,ic50_workspace,slope_hierarchy_scale,euclid_dr1}.md— ledgers (cortex.py verbs only)/mnt/c/Users/Jammy/Science/ep_toy_gaussian/hpc/batch_cpu/submit_nuts— memory raisefit/PyAutoFit/autofit/messages/abstract.py— read-only diagnosis
Verification
cortex.py checkOK; census shows zeroopenruns for ic50_workspace and slope_hierarchy_scale.projects.yamlloads with 11 keys / 291 lines; new test red on the 610-line file, green after;pytest tests -qgreen;ledger_merge.pyclassifies the change as code.- analytic_gaussian ledger carries astra's
note; each of the four## Nownames a submission or blocker.
Original Prompt
Click to expand starting prompt
EP cortex check-in clean-up (2026-09-30) — clear the small issues, then relaunch runs
Type: maintenance
Target: PyAutoCortex
Repos:
- PyAutoCortex
- PyAutoFit
Themes: - ep
- hpc
Difficulty: large
Autonomy: supervised
Priority: high
Status: formalised
Consequence: glance
Witness:cortex.pycensus shows zeroopenruns for ic50_workspace and slope_hierarchy_scale;projects.yamlloads with a duplicate-key check passing (one row per key); the analytic_gaussian ledger carries astra's criterion-2 opinion as anote; and each of the four ledgers'## Nownames a concrete next submission or a named blocker.
Review-minutes: 3
Unattended: needs-slicing
The 2026-09-30 Cortex check-in of the four active EP projects (ep_toy_gaussian,
analytic_gaussian, ic50_workspace, slope_hierarchy_scale) found a set of small
issues standing between us and a new wave of runs. Work through them one by
one, in this order, then stop at a go/no-go for new submissions. Every ledger
write is a cortex.py verb; a result/lesson entry is the human's words only.
1. Cortex bookkeeping
1a. Close stale ledger runs. These are listed open but their SLURM logs
show they ended long ago (the project hpc/sync sacct only covers today, so
exit codes were not confirmed from the cluster — confirm via the logs /
a dated sacct, then record with cortex.py done):
- ic50_workspace: 342408, 342409 (N=5 parity, completed), 342412 (graphical
ladder, all rungs passed, finished 2026-09-09 22:48), 342411 (EP ladder,
finished 2026-09-09 23:19 but rung N=50 FAILED → --failed).
- slope_hierarchy_scale: 342348_[0-24] (one_by_one, ended 2026-09-08
20:37), 342350_0 (joint NUTS, ended 2026-09-09 13:06), 342351_0 (EP arm,
superseded by 342410 / 343299 — output parked as ep_dead_342351).
1b. Refresh ## Now for ic50_workspace (still says the ladder is
"submitted") and slope_hierarchy_scale (still lists the three runs as open)
in the human's words.
1c. De-duplicate PyAutoCortex/projects.yaml. Several project rows are
repeated up to three times (e.g. analytic_gaussian at lines 118/292/452,
ep_toy_gaussian 132/306/466, slope_hierarchy_scale, ic50_workspace,
autolens_inference, profiling, cowls_diana, subhalo_simulations,
slope_hierarchy, concr). YAML keeps the last duplicate silently. Find the
commit that introduced the repetition, keep one canonical row per key
(diff the copies first — keep any field that differs), re-render the
dashboard, and add a duplicate-key check so it cannot recur.
1d. Check-in could not push because the Cortex checkout had an unrelated
uncommitted edit to projects/euclid_dr1.md — find its owner session and
get it committed or discarded (do not discard blind), then run
pyauto-brain cortex checkin --apply.
2. ep_toy_gaussian — prepare wave 2 (collapse question at N=50 unanswered)
- Wave 1 failed on infrastructure only: NUTS 342639 OOM at the 8 GB SBATCH cap
in window adaptation; EP x3 342640_[0-2] EMFILE (fixed by PyAutoFit#1632/#1634,
RAL mirror a73684012 contains both). - Raise the NUTS SBATCH memory above 8 GB (size it from the N=50 model).
- Re-verify the RAL PyAutoFit mirror still contains #1632/#1634 and dependency
floors (see RAL dep-floor lesson: diff pip freeze vs floors). - Choose a NEW sample name (e.g.
n50_seed42_w2) — never rerun into an
existing sample directory. - Ready to submit; submission itself waits for the human go.
3. analytic_gaussian — settle criterion 2 before any rerun
- Open question (planned, never run): is criterion 2's mu threshold wrong, or
is the minimal-EP control wrong? Task archive:
archive/tasks/analytic_gaussian/minimal_ep_legb_mu_threshold.md. - Wave 1 baseline (342413, 200 seeds N=5): EP exact on Gaussian leg, leg B
sigma misses as pre-registered 78/200, collapse 0/200, witness 7 met / 4 miss. - Ask astra (Codex GPT-6-astra) for an independent opinion on the question:
hand it the pre-registered criteria, the wave-1 aggregate (analytic_gaussian
b44390d), and the minimal-EP control code; ask whether the threshold, the
control, or neither is at fault and why. Record its answer verbatim as a
noteand bring it to the human — the human rules, not astra. - Then run the planned check if the human agrees, and only then decide whether
the N=25 rung (written, not submitted) goes out.
4. ic50_workspace — the EP N=50 rung failure
- 342411 rung N=50 failed with
AssertionError: assert np.isfinite(suff_stats).all()
inautofit/messages/abstract.pyproject(see
hpc/batch_cpu/error/error.342411.err). Rungs 5/10/25 completed; graphical
ladder (342412) passed all rungs including N=50. - Diagnose: which factor/message produced non-finite sufficient statistics,
and whether the RAL mirror at the time predated a relevant fix. If it is a
library bug, file it through /intake (bug, autofit) rather than fixing inline. - Then decide the next ladder: rerun N=50 (fixed/current mirror) and/or try the
ep_lbfgs_jaxscale lever; report EP cost and hill_coef width vs N from the
completed rungs.
5. slope_hierarchy_scale — the moment-matching gate
- 343299 showed the hierarchical factor never updates under the Laplace
projection (0/50 SUCCESS: 27 BAD_PROJECTION + 23 FAILURE); parent = prior. - Human decision needed: go/no-go on
draft/feature/autofit/ep_hierarchical_scatter_moment_matching.md. Surface
its current size and status; do not start it inside this task. - Record the per-lens (342348) and joint NUTS (342350_0) outcomes that were
never logged, so the graphical baseline for the EP comparison is on file.
Done when
Every stale run is closed, every ## Now is current, projects.yaml has one row
per key with a guard, and each project has a concrete, human-approvable next
submission (or a named blocker). New runs themselves are the follow-up.
🤖 Generated with Claude Code
Activity
Jammy2211 commentedon Sep 30, 2026
astra (Codex gpt-6-astra) on analytic_gaussian criterion 2 — verbatim, 2026-09-30
Question: Is criterion 2's mu threshold wrong, is the minimal-EP control wrong, or neither — and why? Give the diagnostic you would run to decide.
Command:
codex exec --skip-git-repo-check -s read-only -m gpt-6-astrafrom the analytic_gaussian checkout; brief in the task scratchpad (astra_brief_analytic_gaussian.md).Neither is established as “wrong” by this result alone. I favour inadequate calibration of the threshold, with moderate confidence, but would not yet certify the control. Criterion 2 remains NOT MET.
Doubling the maximum over five seeds does not establish a 95% ensemble bound. Gaussian EP can reproduce each tilted distribution’s marginal moments while missing the full posterior’s moments: with unknown sigma, the shared mu–sigma dependence matters. Consequently, accurate sigma mean/std and excellent x_i rows do not guarantee accurate mu. However, “Gaussian-site bias” alone does not demonstrate that these particular mu errors are intrinsic. Nor does convergence prove correctness; leg A does not exercise the uncertain-scatter integration.
Inspection finds the expected cavity subtraction at line 538, projected-minus-cavity updates at line 561, and total-variance calculation at line 342. The stopping condition also excludes sweeps with skipped updates. I have not identified a demonstrated defect.
The decisive diagnostic would independently reconstruct the same EP factorisation and approximation family, using adaptive quadrature, on a passing seed, a threshold-edge failure, and the worst mu failure. At the reported fixed point, reconstruct every cavity from the other sites and independently integrate each tilted distribution; compare its first two moments with the corresponding q moments. Check undamped residuals, skipped updates, and stability under tighter tolerance, wider integration windows, alternative initialisation, and reversed update order. Merely increasing grid density in the existing implementation cannot expose a shared algebraic error.
One specific audit target is support handling: line 333 clips every hierarchical tilt to prior support, while natural parameters represent ordinary Gaussians. Establish whether that support convention is consistent with the intended EP equations; this is a question to test, not a proven cause.
If independent, consistent EP reproduces the mu discrepancies, the five-seed calibration underestimated approximation error. Any revised criterion then needs explicit prospective justification and fresh validation seeds; wave 1 stays failed. If independent moments or fixed-point residuals disagree, investigate the control before changing tolerances. The minimal column is an empirical comparator, not a mathematical upper bound on EP accuracy.
Opus reviewer's independent view (written before reading astra):
OPUS SUBAGENT'S INDEPENDENT VIEW (written before reading astra's answer)
On the evidence in the wave-1 sidecars, the threshold is wrong and the control is (very probably) not.
Three facts from results/ens_n5/seed_*.json carry it. (1) The mu displacement is not noise: the signed
a_mu = (E_EP[mu] - E_ref[mu])/std_ref is almost perfectly linear in how far the closed-form mu sits from
the hyper-prior mean, a_mu ~= 0.041 * (E_ref[mu] - m0)/std_ref with m0 = 50 (corr 0.98, intercept -3e-4,
residual std 0.004, max |residual| 0.016). That is the signature of a shrinkage-weight error: EP's
mu is ~4% less pulled toward the N(50, 10^2) hyper-prior than the exact posterior. (2) Its cause is
visible in the same column: E_EP[sigma] is below the closed form on 200/200 seeds (signed sigma a
median -0.040), so every hierarchical site hands mu a data precision 1/(sigma^2 + v_i) that is too
large, and mu's std is inflated by ~5% (b_mu median +0.050, positive on 88% of seeds) -- the expected
downstream footprint of a Gaussian q(sigma) on a skewed sigma posterior, not of a cavity or
parameterisation bug (a bug would not scale cleanly with |E_ref[mu] - m0| and vanish at mu = m0; all
200 runs also report converged=True in 13-20 sweeps). (3) The 0.05 tolerance was calibrated by the
"2x the max over seeds 0-4" rule, and seeds 0-4 happen to have |E_ref[mu] - m0| small: their |a_mu| is
0.002-0.020, the p80 of the 200-seed distribution is 0.044, the p95 is 0.077. A 5-seed max doubled
cannot set a 95%-of-seeds bar on a statistic whose size is set by the data draw. So criterion 2's mu
row threshold is under-calibrated; the 166/200 is a sampling-of-the-calibration failure, not a
breach of the ceiling. The one residual caveat is that "EP-intrinsic" still has to be shown to be
the size the Gaussian-site approximation predicts, not merely the right shape. The diagnostic: for
each seed, take the converged minimal-EP q(sigma) (or its sigma-marginal fixed point) and compute,
by direct quadrature with no EP code, E[mu] under the exact joint but with p(sigma | data) replaced
by that Gaussian (truncated to sigma>0) -- that is a_pred. If a_obs - a_pred is ~0 (the task's own
|.|<=0.01 on >=32/34, median <=0.005) the control is faithful and the threshold should be re-derived
from the 200-seed p95 of a_pred (~0.08, so ~0.10); a secondary cheap check is to rerun the 34 seeds at
n_grid=1601, half_width=12, tol=1e-12 and confirm a_mu does not move. A structural flag worth naming
but not a proven defect: theta_grid clips every hierarchical site's tilted to sigma > 0, so the
positivity constraint enters N+1 times rather than once; testing the variant that clips only the
hyper-prior site would show whether that duplication contributes to the low E[sigma].
Jammy2211 commentedon Sep 30, 2026
Progress — 2026-09-30 check-in clean-up (all phases run)
Landed on Cortex
main(ledger class): c1135f3 stranded euclid_dr1 ledger edit committed; d7ce4a8 seven stale runs closed after log + dated-sacct confirmation (ic50 342408/342409/342412 finished, 342411 failed at rung N=50; slope 342348_[0-24] finished, 342350_0 OOM-killed after sampling, 342351_0 hung then cancelled); 47ad968 six check-in notes; d75ae0c## Nowrewritten for all four EP projects.cortex.py checkOK; no EP project has an open run.Code half (this issue's PR, pending Heart):
feature/ep-checkin-cleanup— projects.yaml 610 → 290 lines (tripled by 25e0168; copies byte-identical), duplicate-key guard inparse_projects+ test + REFERENCE note; 65 tests green.Per-project state
submit_nuts --mem8 → 64 GB locally (uncommitted); RAL mirror 404b3e5f7 has #1632/#1634; needs atoy.SAMPLESentry forn50_seed42_w2andSAMPLEexported to the job.factor_step; stochastic trigger). Bug promptPyAutoMind/draft/bug/autofit/ep_project_nonfinite_suff_stats_ic50_n50.md(Mind dcc36aee), human-approved for start_dev. Ladder: EP 26/50/140/~300 s per sweep at N=5/10/25/50 vs graphical 21/42/119/281 s wall; graphical overconfident beyond N=25; EP hill_coef widths not yet comparable (factor-message widths).draft/feature/autofit/ep_hierarchical_scatter_moment_matching.md) human-approved to start; not started anywhere before.Blocked on Heart RED (as of the 10:51 tick):
release validation FAILED (stage integrate)plus transient behind-origin library reasons. Ship of this PR and the two PyAutoFit start_dev launches wait on the human's RED override or a green Heart.Jammy2211 commentedon Sep 30, 2026
Library PR Created
PR: #51 (label
pending-release; branchfeature/ep-checkin-cleanup, rebased onto main 67b6939 — commits be69ddb de-duplicateprojects.yaml, ccc9801 reject duplicate project keys)Heart RED override (development only)
cortex.py checkOK; ledger_merge classify → code (human review).Workspace impact
None (organ repo; no workspace scripts import it) — option (iii); merge stays human via /prm once required checks are green.
Ledger commits already on main
c1135f3, d7ce4a8, 47ad968, d75ae0c, check-in 67b6939.
Merge pull request #51 from PyAutoLabs/feature/ep-checkin-cleanup
Jammy2211 commentedon Sep 30, 2026
Shipped
projects.yamlde-duplicated (610 → 290 lines, 17 keys unchanged).cortex.py checknow rejects a repeated project key (projects.yaml: duplicate project key '<key>' (lines …)).REFERENCE.mddocuments the one-row-per-key rule.mainas ledger commits c1135f3, d7ce4a8, 47ad968 and d75ae0c, plus the check-in 67b6939. The seven stale open runs are closed, the## Nowsections are rewritten, and the stranded euclid_dr1 ledger edit has landed.Follow-ups (not in this PR):
projects.yamlheader comment (lines 12–13) still cites a REFERENCE.md "restricted YAML subset" section that no longer exists.samples.csvsummary.Closed via /prm.