[Jammy2211]

Summary

B5b of the search-extensibility epic. This PR pulls the scored wave 2 back from RAL, judges every row under gaussian_x3@2 at the pinned PyAutoFit revision df6d218f1, and publishes the results as a gaussian_x3@2 section of the catalogue. It also writes the timeouts for the probe-first cells and submits their remaining runs on RAL. Closes #10.

Rows.

• All 11,770 pulled rows are admitted by _wave2.admit; none are refused. That is 11,750 from the scored arrays and 20 from the probe.

• judge_wave2.py writes the accounting to results/accounting/gaussian_x3_wave2.json. It adds up to the 13,000 expected runs:

  Part	Expected	Outcome
  Scored arrays	11,750	10,580 rows, 1,170 failed, 0 missing, 0 operational kills
  Probe	20	20 rows
  Probe-first remainder	980	missing: submitted as job 409930
  Deferred (blend warm NUTS)	250	deferred, with reasons

References.

• All 20 references are complete: 2 datasets × 2 backends × 5 realisations. Each one uses only the Nautilus runs. On every reference the ln Z spread between DynestyStatic and Nautilus is 0.49–7.3 nat, which is recorded as a §6 limitation.
• The JAX blend realisation-1 reference was rebuilt on the laptop from three Nautilus seeds. It passes the §6 agreement test: ln Z spread 0.012 nat (limit 0.2) and worst median offset 0.016 σ (limit 0.1).
• The NSS ln 3! offset is validated (measured −1.728).

Probe-first timeouts come from TIMEOUT_RULE_V2 applied to the 20 probe rows. They are written to results/calibration/gaussian_x3_wave2_probe.json:

• NSS 50-step on JAX: 600 s (the floor; median 26.6 s over 10 runs).
• Zeus on JAX: 1168 s (median 116.8 s over 10 runs).

cells() reads these after the wave-1 timeouts. plan_wave2.py --write generated submit_wave2_remainder (20 tasks, 980 runs). The submit files B5a actually ran are byte-identical.

Catalogue. catalogue/search_catalogue.json gains a gaussian_x3@2 section. For each leg (dataset × search × settings × backend, pooled over 5 realisations × 50 seeds) it gives:

• the success rate with its Wilson 95 % interval;
• the expected wall per right answer with its bootstrap 95 % interval;
• §7 censoring.

Legs are ordered only within one task and one dataset. The wave-1 pilot section is byte-identical to before (verified).

Summary. dashboard/summary.json stays inference-summary@1. It now carries a protocols list and the wave-2 accounting. The wave-2 records themselves are withheld because of file size; see decision 1.

Commits, in this order:

1. Rows: 18e439d.
2. Tooling: 5c9e022.
3. Catalogue: f2cf3be.
4. Docs: cd4fc41.
5. Summary: ee51ad5.
6. Review fixes: 0eba8d2 (tooling), 5b26b6a (catalogue), be2d148 (docs), 5d5f835 (summary).
7. Re-review fixes: 49849e8.

output/ is never committed.

Decisions taken (decide-and-flag)

1. Wave-2 records withheld from dashboard/summary.json (question on research: B5b — pull, judge and publish scored wave 2 (gaussian_x3@2) #10).
   • The @1 contract can carry a per-record protocol_id. The full export, with 223 + 11,770 records, per-record protocol_id: gaussian_x3@2 and a top-level protocols list, classifies ok against PyAutoInsight/insight/summary.py.
   • The problem is size: that file is ~84 MB (63 MB as compact JSON). That is above GitHub's 50 MB warning, and the Insight snapshot would copy it again on every refresh.
   • So the committed summary keeps the pilot records and adds the wave-2 accounting, marked records: withheld with the reason.
   • --wave2-records --output <path> writes the full export; it refuses to write the committed path.
   • The question is posted on research: B5b — pull, judge and publish scored wave 2 (gaussian_x3@2) #10. The options are: the fit instance moves to inference-summary@2 (recommended); per-leg @1 records; or commit the 84 MB file or put it in LFS.
2. The ordering in the @2 section is the catalogue's own sort key, not a protocol rule.
   • §8 defines the headline numbers but no ordering. Legs are sorted by expected wall per right answer, then by success rate, and only legs with at least one right answer get a position.
   • A cheap, unreliable search can therefore lead. On the blend evidence task, DynestyStatic (14 % success) is first, ahead of Nautilus (100 %).
   • Entries whose bootstrap intervals overlap are marked tied (=n in the README).
   • A protocol amendment fixing the ordering is left to the human.
3. Seed-1 run used for the JAX blend reference. It is the RAL run of task run nautilus 1 1 ral_jax_cpu_fp64 (output/reference/gaussian_x3_blend/ral_jax_cpu_fp64/nautilus_seed1/a7506fab…, pulled with a targeted rsync).
   • The stray local local_jax_cpu_fp64/nautilus_seed1/a7506fab… is a pilot-era run with no row, so it is not used. It has the same hash because the identity is the same, but the builder reads the row's output_path.
   • The reference mixes PyAutoFit commits: seeds 0 and 2 are at 0dbf258 (pilot) and seed 1 is at the pin.
4. The constant-likelihood file is RAL's own. run_constant_likelihood.py merges results in place, so the RAL file is exactly the committed file plus offsets.nss.jax_cpu, and it was taken as-is.
5. Pilot rows are not re-judged (ruling 1). _protocol.PILOT_NOT_REJUDGED keeps the 78 JAX blend pilot rows not_assessed, with the reason "reference completed after the pilot (B5b)". No other pilot record changes.
6. How the remainder reached RAL. The two generated files (submit_wave2_remainder and wave2/remainder.tsv) were copied with scp into the RAL clone, which stays at f0b3eef; it was not pulled with git. Their sha256 matches the branch. The runner, the protocol values and the pin are unchanged by B5b. The throttle is %20, i.e. min(60, 20 tasks).
7. Probe-sourced legs. They are open and unranked until all 250 of their runs have rows, which happens in B5b-2.
8. Rules applied as written, with the deviation documented.
   • Timeout censoring compares the consumer's total_wall_s, as frozen, even though a warm-start timeout was derived from the median attempt wall.
   • Missing runs are censored per closed cell.
   • The bootstrap resamples attempts (seed × realisation), not seeds alone.
   • None of these changes a number today.
9. Carried limitations, recorded and not fixed.
   • Timeouts are lenient on RAL (10× laptop medians; RAL is 4–7× faster), and no scored attempt hit its timeout. The probe-derived timeouts are RAL-measured and so are not lenient in the same way.
   • LBFGS returning +inf on blend numpy at the pin is reported as an outcome (210 of 250 runs).

Disk decision

The laptop root had 6.1 GB free at the start, down to 4.0–4.5 GB later (other sessions are using it too).

• Remote du would not finish. du -d1 output did not complete in 15 minutes on the RAL NFS, so the size of output/ (11,770 run directories with samples.csv) could not be shown to fit with 2 GB to spare. output/ was not pulled.
• Logs were skipped. sacct reports all 1,629 tasks COMPLETED with exit 0, and the logs are not needed to judge.
• results/ came over as a parallel tar. rsync of results/ ran at about 0.16 files/s, so I ran a 12-way parallel per-directory tar on the login node instead, using only my own files in /tmp, removed afterwards. That gave a 19 MB tgz, 171 MB when extracted.
• One targeted output pull. The only output/ brought back is the JAX blend Nautilus seed-1 reference directory (13 MB). Seeds 0 and 2 are symlinked to the canonical checkout's existing pilot outputs. The NSS constant-likelihood output/ was not needed, because the json carries both ln Z values.

RAL

• 409930 [0-19]%20: submit_wave2_remainder, 980 runs, on --partition=ral with --exclude=euclid-ral-compute-[1-4], 2 CPUs and 6 GB per task, no gres, PyAutoFit pin guards. Running at hand-back. B5b-2 pulls it.

Per-cell headline (gaussian_x3@2)

The State column means:

• closed: every expected run has been judged.
• open: the cell's remainder is still running.
• deferred: not run, with the reason recorded.

Task	Dataset	Search · settings · backend	Accepted / attempted	Success (Wilson 95 %)	Wall per right answer, s (bootstrap 95 %)	State
point/MAP	blend	lbfgs · default · numpy	0/250	0% [0%, 2%]	unbounded (≥ 214)	closed
point/MAP	blend	lbfgs · default · jax_cpu	0/250	0% [0%, 2%]	unbounded (≥ 236)	closed
point/MAP	blend	bfgs · default · numpy	0/250	0% [0%, 2%]	unbounded (≥ 175)	closed
point/MAP	blend	bfgs · default · jax_cpu	0/250	0% [0%, 2%]	unbounded (≥ 171)	closed
point/MAP	blend	multi_start_adam · starts_48_steps_300 · jax_cpu	0/250	0% [0%, 2%]	unbounded (≥ 833)	closed
point/MAP	blend	multi_start_prodigy · starts_48_steps_300 · jax_cpu	152/250	61% [55%, 67%]	18 [17, 20]	closed
point/MAP	blend	multi_start_adabelief · starts_48_steps_300 · jax_cpu	0/250	0% [0%, 2%]	unbounded (≥ 794)	closed
point/MAP	blend	multi_start_lion · starts_48_steps_300 · jax_cpu	0/250	0% [0%, 2%]	unbounded (≥ 535)	closed
point/MAP	separated	lbfgs · default · numpy	0/250	0% [0%, 2%]	unbounded (≥ 462)	closed
point/MAP	separated	lbfgs · default · jax_cpu	0/250	0% [0%, 2%]	unbounded (≥ 352)	closed
point/MAP	separated	bfgs · default · numpy	0/250	0% [0%, 2%]	unbounded (≥ 437)	closed
point/MAP	separated	bfgs · default · jax_cpu	0/250	0% [0%, 2%]	unbounded (≥ 322)	closed
point/MAP	separated	multi_start_adam · starts_48_steps_300 · jax_cpu	0/250	0% [0%, 2%]	unbounded (≥ 697)	closed
point/MAP	separated	multi_start_prodigy · starts_48_steps_300 · jax_cpu	250/250	100% [98%, 100%]	9 [9, 9]	closed
point/MAP	separated	multi_start_adabelief · starts_48_steps_300 · jax_cpu	0/250	0% [0%, 2%]	unbounded (≥ 703)	closed
point/MAP	separated	multi_start_lion · starts_48_steps_300 · jax_cpu	0/250	0% [0%, 2%]	unbounded (≥ 389)	closed
posterior	blend	emcee · walkers_50_steps_2000 · numpy	0/250	0% [0%, 2%]	unbounded (≥ 4063)	closed
posterior	blend	emcee · walkers_50_steps_2000 · jax_cpu	0/250	0% [0%, 2%]	unbounded (≥ 4896)	closed
posterior	blend	zeus · walkers_50_steps_2000 · numpy	0/250	0% [0%, 2%]	unbounded (≥ 11233)	closed
posterior	blend	zeus · walkers_50_steps_2000 · jax_cpu	0/5	0% [0%, 43%]	unbounded (≥ 271)	open
posterior	blend	blackjax_nuts · warmup_200_samples_500 · jax_cpu	0/250	0% [0%, 2%]	unbounded (≥ 418)	closed
posterior	blend	blackjax_nuts_warm · warmup_200_samples_500 · jax_cpu	—	—	—	deferred
posterior	blend	smc · default · jax_cpu	0/250	0% [0%, 2%]	unbounded (≥ 1921)	closed
posterior	separated	emcee · walkers_50_steps_2000 · numpy	0/250	0% [0%, 2%]	unbounded (≥ 4085)	closed
posterior	separated	emcee · walkers_50_steps_2000 · jax_cpu	0/250	0% [0%, 2%]	unbounded (≥ 5137)	closed
posterior	separated	zeus · walkers_50_steps_2000 · numpy	198/250	79% [74%, 84%]	216 [201, 233]	closed
posterior	separated	zeus · walkers_50_steps_2000 · jax_cpu	4/5	80% [38%, 96%]	134 [82, 296]	open
posterior	separated	blackjax_nuts · warmup_200_samples_500 · jax_cpu	0/250	0% [0%, 2%]	unbounded (≥ 1490)	closed
posterior	separated	blackjax_nuts_warm · warmup_200_samples_500 · jax_cpu	250/250	100% [98%, 100%]	237 [233, 241]	closed
posterior	separated	smc · default · jax_cpu	0/250	0% [0%, 2%]	unbounded (≥ 1310)	closed
evidence	blend	nautilus · n_live_100 · numpy	250/250	100% [98%, 100%]	247 [242, 251]	closed
evidence	blend	nautilus · n_live_100 · jax_cpu	250/250	100% [98%, 100%]	253 [248, 259]	closed
evidence	blend	nautilus · n_live_200 · numpy	250/250	100% [98%, 100%]	306 [301, 312]	closed
evidence	blend	nautilus · n_live_200 · jax_cpu	250/250	100% [98%, 100%]	299 [297, 302]	closed
evidence	blend	nautilus · n_live_400 · numpy	250/250	100% [98%, 100%]	522 [514, 531]	closed
evidence	blend	nautilus · n_live_400 · jax_cpu	250/250	100% [98%, 100%]	513 [508, 518]	closed
evidence	blend	dynesty_static · nlive_200 · numpy	36/250	14% [11%, 19%]	170 [128, 237]	closed
evidence	blend	dynesty_static · nlive_200 · jax_cpu	36/250	14% [11%, 19%]	190 [144, 266]	closed
evidence	blend	dynesty_dynamic · default · numpy	59/250	24% [19%, 29%]	346 [281, 444]	closed
evidence	blend	dynesty_dynamic · default · jax_cpu	59/250	24% [19%, 29%]	378 [307, 484]	closed
evidence	blend	nss · n_live_200_mcmc_steps_50 · jax_cpu	3/5	60% [23%, 88%]	40 [24, 121]	open
evidence	separated	nautilus · n_live_100 · numpy	250/250	100% [98%, 100%]	344 [336, 352]	closed
evidence	separated	nautilus · n_live_100 · jax_cpu	250/250	100% [98%, 100%]	339 [332, 347]	closed
evidence	separated	nautilus · n_live_200 · numpy	250/250	100% [98%, 100%]	293 [285, 301]	closed
evidence	separated	nautilus · n_live_200 · jax_cpu	250/250	100% [98%, 100%]	284 [276, 292]	closed
evidence	separated	nautilus · n_live_400 · numpy	250/250	100% [98%, 100%]	393 [387, 399]	closed
evidence	separated	nautilus · n_live_400 · jax_cpu	250/250	100% [98%, 100%]	414 [407, 422]	closed
evidence	separated	dynesty_static · nlive_200 · numpy	103/250	41% [35%, 47%]	62 [54, 72]	closed
evidence	separated	dynesty_static · nlive_200 · jax_cpu	103/250	41% [35%, 47%]	69 [60, 81]	closed
evidence	separated	dynesty_dynamic · default · numpy	145/250	58% [52%, 64%]	143 [130, 160]	closed
evidence	separated	dynesty_dynamic · default · jax_cpu	145/250	58% [52%, 64%]	156 [141, 175]	closed
evidence	separated	nss · n_live_200_mcmc_steps_50 · jax_cpu	5/5	100% [57%, 100%]	29 [29, 29]	open

Heart RED override

This is quoted from #10. The live human answered the AskUserQuestion "Invoke the Heart RED development override for this batch?" with "Yes, override for development (Recommended)" on 2026-10-15. The answer covers tasks search-ext-b5b-score, search-ext-a5-consumers and search-ext-b4b-dev-retire.

These are the exact RED reasons at the door (Heart verdict, 2026-10-15):

• "Heart: RED — Release RED; monitoring RED · 10/100 · 415 unresolved"
• "[red] release validation FAILED (stage integrate)"
• "[red] PyAutoHands/workflows/Python Version Matrix: failure"
• "[red] PyAutoHeart/workflows/Release Integrate: failure"
• "[red] PyAutoMind/workflows/Lifecycle Drift: failure"
• "[red] PyAutoArray/Tests: red"
• "[red] autofit_workspace_test/Smoke Tests: red"
• "[red] autogalaxy_workspace_test/Smoke Tests: red"
• "[red] autolens_workspace_test/Smoke Tests: red"
• "[red] Release validation: NOT release_ready — v2026.10.15.1.dev81701 profile=release (2026-10-15T07:04:34+00:00)"
• "[red] Worktree drift: 5 orphan / 0 missing / 4 dirty"
• "[red] autolens_inference: red"
• "[red] autolens_profiling: red"
• "[red] euclid_strong_lens_modeling_pipeline: red"

Permitted: commit, push, PR-open. Merge stays a separate human act. There is no release or rehearsal, and this branch does not fix Heart.

Gate

tests	smoke	review	heart	witness
pytest 118 passed; ruff check and format clean; all seven lint --check gates green. The reviewer re-ran these in a CI-like clean venv: 111 passed, 5 skipped, about 90 s.	n/a: a benchmark repo with no downstream script surface or curated smoke list; no library code changed	Independent fresh-context review (Opus subagent): FINDINGS, then fixed and re-reviewed, then one remaining should-fix (a contradictory clause in state.md) plus one nit applied in 49849e8; that last change was not re-reviewed again	RED, overridden for development (above)	6 of 7 met; the summary-records part is open (#10). See below.

Review dispositions:

1. Ordering is not a §8 rule: fixed. The key and caveat are stated, intervals and ties are marked, and the protocol amendment is left to the human.
2. Probe legs could never close: fixed, with a test; operational kills are now excluded from the closure count.
3. The timeout is checked against the consumer wall: accepted and documented. That is the frozen rule.
4. Missing runs are censored per closed cell, with no provider cost: accepted and documented.
5. The bootstrap resamples attempts, not seeds: accepted and documented.
6. Prose figures in state.md: fixed.
7. Misplaced comment: fixed.
8. Missing tests: fixed. The duplicated row scanners are accepted.
9. Tie nits (adjacent-only comparison; an unbounded upper bound never ties): accepted.

Witness, item by item:

• ✅ Every pulled wave-2 row is admitted or refused by _wave2.admit, and the per-leg accounting() sums equal the expected counts: 11,750 scored + 20 probe, 250 deferred with reasons. 11,770 are admitted, 0 refused; judge_wave2.py --check is green.
• ✅ The catalogue has a gaussian_x3@2 section whose success rates and walls per right answer come only from @2 rows at df6d218f1. The section records pyautofit_pin and pyautofit_commits_pulled = [df6d218f1…].
• ⚠️ dashboard/summary.json validates against the Insight contract and check --offline passes (Insight PR). However, the wave-2 records are withheld on size (decision 1), and the fit snapshot can only refresh after this merges: --from is a local read and writes no snapshot.
• ✅ The JAX blend realisation-1 reference reaches status: complete from three Nautilus seeds and passes §6.
• ✅ The probe-first cells' @2 timeouts come from the probe medians by the frozen rule, and their remainder is submitted on --partition=ral as job 409930.
• ✅ cortex.py done moved 408244–408367 into the log: cortex: autofit_inference — wave 2 judged under gaussian_x3@2, remainder 409930 (B5b) PyAutoCortex#65.
• ✅ The lint gate is green: export_inference_summary.py, plan_wave2.py, check_submits.py and build_catalogue.py with --check, plus the new write_probe_timeouts.py --check and judge_wave2.py --check.

Test plan

• python -m pytest -q scripts/misc/test: 118 passed.
• ruff check . and ruff format --check ..
• build_catalogue.py --check, build_readme.py --check, export_inference_summary.py --check, plan_wave2.py --check, check_submits.py --check, write_probe_timeouts.py --check, judge_wave2.py --check.
• The wave-1 catalogue section is byte-identical, and the B5a as-run submits are unchanged.
• The full --wave2-records export (84 MB) classifies ok against insight.summary.
• CI lint on this PR.

Merge order (human /prm): this PR, then PyAutoLabs/PyAutoCortex#65, then the PyAutoInsight PR.

🤖 Generated with Claude Code