[Jammy2211]

Summary

B5c-2a of the search-extensibility epic. This PR pulls the correlated_nd pilot (RAL 410866), judges it under correlated_nd@1, calibrates it, rules on the OOM cells (amendment A2) and freezes correlated_nd@2. No scored run exists and nothing was submitted.

• Pull. results/searches/ of 410866 came from the PRIVATE copy /mnt/ral/jnightin/inference/autofit_inference_b5c, tarred per dataset on the login node. It holds 930 rows, committed verbatim in 66910c5. The 8 exact references there are sha256-identical to the committed ones. No output/, no SLURM logs, and the shared mirror was never touched.
• Accounting. judge_correlated_nd.py writes results/accounting/correlated_nd_pilot.json; lint runs --check. It reconciles the rows with the committed sacct extract results/accounting/correlated_nd_pilot_sacct.tsv:
  • 1040 expected = 749 rows + 177 failed + 0 missing + 110 deferred + 4 oom
  • 930 admitted, 0 refused
  • sacct COMPLETED → 749 row + 177 failed; OUT_OF_MEMORY → 4 oom
• Verdicts under @1. 307 accepted, 623 rejected; the pilot ranks nothing. The catalogue's correlated_nd@1 section and the README table are grouped by task, then dataset, in menu order, with nothing ranked across families.
• Failures.
  • 80 Emcee/Zeus rows: a harness bug, now fixed.
  • 60 d100 Nautilus rows: n_live_100 needs n_live > D; n_live_200/400 hit the nautilus HDF5 checkpoint limit at D = 100.
  • 10 d100 warm NUTS rows: their provider failed.
  • 27 rows at the 4 h cap: d50 Nautilus n_live_100 and d50 warm NUTS.
• Calibration. calibrate_correlated_nd.py writes results/calibration/correlated_nd_wave1.json; lint runs --check.
  • σ-ratio bands per task: evidence [0.85, 1.1] on 79 rows; posterior kept at [0.5, 2.0] on 1 row. Both contain the §6 exact draws' [0.98, 1.02].
  • The ESS and R-hat bars are kept.
  • 75 cells get timeouts max(600 s, 10 × median pilot attempt wall). 8 cells are probe-first. 21 cells are deferred, each with its reason.
  • Memory requests per cell class.
  • The family's wall-rates table is wall/rates.py CORRELATED_ND_STEP_RATE.
• A2 and freeze. Protocol §11 A2. _protocol.FROZEN_CND2 / THRESHOLDS_CND2 were committed in 44c5e77 (44c5e77f1fb0779f675a8f51247d812183e93b48). The sha is recorded in §7, §11 and state.md by c12b203; this PR's merge makes it reachable from main. protocol_for resolves correlated_nd@1 for pilot rows (wave 1) and correlated_nd@2 for scored rows (wave 2). The test test_protocol_id_resolves_per_wave checks this on one real pilot row and one synthetic scored row, and confirms the verdict strings carry the id. The runner still refuses a wave-2 correlated_nd run: the scored wave is declared and submitted by B5c-2b.

Inputs for B5c-2b:

• Scored menu: 83 cells × 50 seeds × 3 realisations = 12 450 runs, about 1 430 task-hours at pilot medians. The largest timeout is 36 875 s (d100 DynestyDynamic numpy).
• Deferred cells: see scored_deferred in the calibration file.
• Memory: 16 G for d50 Nautilus n_live_400, 17 G for d100 cold NUTS, 8 G elsewhere.
• Pin: the pilot's, df6d218f10cd / 7f3055b7f3d1.

Commits: 66910c5 rows · b3845e8 oom class, reconcile, judge CLI and harness fix · 31ba4a6 oom rows, sacct and accounting · 44c5e77 freeze · a5f2d6e catalogue and README · c12b203 freeze sha recorded · a663671 summary (producer revision only). Siblings: PyAutoCortex and PyAutoInsight PRs (B5c-2a).

Decisions taken

Each one is flagged for the human. Each has a one-line revert.

1. A2 memory headroom is 2×. --mem = max(8 G, ceil(2 × peak)) per cell class (dataset × search × settings, both backends). The peak is the largest sampled sacct MaxRSS, or 8 G when any pilot task was OOM-killed. Sampled RSS undercounts the peak at a kill: task 304 read 3.1 G when it died at 8 G. This raises two cells: d50 Nautilus n_live_400 to 16 G and d100 cold NUTS to 17 G (MaxRSS 8.1 G). Revert: set MEMORY_HEADROOM in calibrate_correlated_nd.py and regenerate.
2. d100 Nautilus n_live_200 is deferred from the scored menu, along with n_live_400 and n_live_100. All 40 n_live_200/400 attempts failed at the pin within 31–74 s, before the first bound was sampled. The cause is nautilus's HDF5 checkpoint: it cannot store a D = 100 bound attribute (h5py "object header message is too large"). Affordability therefore cannot be judged from the pilot walls; for comparison, d50 n_live_200 took a median of 820 s. n_live_100 is deferred by construction: Nautilus needs n_live > D. d100 warm NUTS is deferred with its provider. Revert: drop the entries from _correlated_nd.A2_DEFERRED in a later amendment, after a library or nautilus fix.
3. The OOM cells. d50 Nautilus n_live_100 (both backends) and d50 warm NUTS are deferred by the §7 cap rule, read as "no attempt ended on its own". In each, 9 of 10 attempts hit the 4 h cap and the 10th was OOM-killed before the cap. Revert: calibrate with "every attempt Terminated at the cap" only. Those two cells would then get timeouts from zero completed rows, so they would be probe-first.
4. σ-ratio bands are per task under @2, read literally from "a band moves only on ≥ 20 completed pilot rows of its task". The evidence task moves to [0.85, 1.1] on 79 rows. The posterior task has 1 row, so it keeps [0.5, 2.0]. @1 had one shared band. With a single band, 34 of the 307 accepted pilot rows would have flipped, all of them BlackJAX NUTS (500 samples), judged by Nautilus σ noise. Revert: FROZEN_CND2 bands as plain tuples (one band for both tasks).
5. The evidence calibration set also requires §4(c). This was added after seeing the pilot. 6 collapsed banana Nautilus n_live_200 rows (σ ratios 0.04–0.13) pass the medians against the banana's wide reference σ, and without §4(c) they would drag the lower band edge to 0. Revert: drop the (c) filter in calibrate_correlated_nd.calibrate.
6. New oom attempt class. SLURM's OOM SIGKILL exits 137 under timeout, the same code as the cap's own kill. The pilot submit therefore wrote cap stubs (process wall = the 4 h limit) for the 4 OOM tasks, which would have been counted as censored cap failures. cnd_cap_stub.py --reconcile-oom rewrote them from the committed sacct extract (MaxRSS, real wall). Under @2, an oom run is rerun once at twice its cell's request, and a second oom is a failed attempt. Revert: git revert the reconcile commit.
7. Harness fix in this phase. All 80 Emcee/Zeus pilot rows (d20 and the banana) failed after the search in _posterior.chain_diagnostics, which relabelled by g0.centre. The fix is relabel=False for families with no label symmetry. Pilot verdicts stay @1 (failed). Those 8 cells have no completed row, so they are probe-first in the scored wave. Revert: revert commit b3845e8's _posterior/_runner hunks.

Witness checklist

• Every task of RAL 410866 is accounted for: results/ was pulled from the PRIVATE copy, a stub row exists for every task killed without a row, and the four OUT_OF_MEMORY tasks (263/273/304/383) are carried as oom, distinct from missing/deferred/failed.
• Every row is judged under correlated_nd@1. The catalogue's correlated_nd section is grouped by task and dataset and ranks nothing across families.
• The §7 placeholders (σ-ratio bands, ESS/R-hat bars), the per-cell timeouts (max(600 s, 10 × median pilot wall)) and the censoring rule are calibrated from the pilot by the §7 rules. A band moves only on ≥ 20 completed rows of its task and never past the 200 000 exact draws. A cell whose every attempt hit the cap is deferred with its reason.
• Amendment A2 rules on the OOM cells (memory request from the measured MaxRSS plus headroom, or deferral with a reason) and on d100 Nautilus affordability from the pilot walls.
• correlated_nd@2 is frozen. Protocol §7/§11 and state.md carry the freeze sha, and PROTOCOL_ID resolves @1 for pilot rows and @2 for scored rows, with a test.
• NO scored run exists or was submitted in this phase.
• The Cortex Log entry and the Insight correlated_nd_pilot row are closed with the pilot verdicts (sibling PRs).

Local gates: ruff check, ruff format --check, pytest scripts/misc/test (174 passed), and every lint --check (catalogue, readme, summary, wave-2 plan/probe/accounting, cnd plan, cnd accounting, cnd calibration and rates, WALL-BASIS). calibrate_pilot.py --check is not a lint gate and was already stale on main; it is untouched.

Closes #17

🤖 Generated with Claude Code