[Jammy2211]

Summary

This is B5c stage 1 of the search-extensibility epic. It adds a second toy-likelihood family, correlated_nd, so search rankings stop resting on gaussian_x3 alone. Everything below was committed before any correlated_nd row exists, and the 930-run pilot is submitted on RAL. Closes #14.

The family. Four datasets, each with an exact posterior and an exact ln Z by construction, so no reference search is run:

• correlated Gaussians at D = 20, 50 and 100, each a seeded Haar rotation of eigenvalues log-spaced from 0.1 to 100, giving condition number 10³ exactly;
• one D = 20 hybrid Rosenbrock banana (Pagani et al. 2022, n1 = 2, n2 = 19).

Protocol first. wiki/project/protocol_correlated_nd.md (correlated_nd@1) was first committed as 9ec37795f404, ahead of every other commit in this PR.

• It sets the tasks, the verdict rules (§4(b)3 is a whitened-coordinate check under the exact whitening map; it replaces gaussian_x3's ppc and mode criteria), the §6 agreement test against the exact reference, the per-dimension deferral rules with reasons, the pilot and the freeze rule.
• Amendment A1 (§11) was made in place after review round 1, still before any row or submission.

Generator and references.

• simulators/correlated_nd.py is seeded by structure_seed and data_seed. Tests regenerate every committed dataset byte for byte.
• models/correlated_nd_exact.py computes ln Z, the marginals, the mode and the whitening map. Tests check them against brute-force grid integration on small toys, against exact iid draws, and with a quadrature-refinement check.
• reference/exact_correlated_nd.py writes 8 references (4 datasets × numpy/JAX), all complete. Each passed the §6 test:
  • the backend's own Fitness.call equals the analytic ln L to ≤ 6e-14 nat;
  • 200 000 exact draws reproduce every median within 0.006 σ;
  • the truncation is bounded;
  • the control verdicts are accepted.

Harness wiring.

• The runner gains model_type correlated_nd, with 60 literal leaves, the A4 search-level seed on every family row, the whitened statistic, process_elapsed_s, and a refusal of any RAL run that is not on the pin.
• _protocol gains correlated_nd@1 (THRESHOLDS_CND1, PLACEHOLDERs marked). Every gaussian_x3 verdict is unchanged.
• searches/_correlated_nd.py holds the expected-run manifest, the deferrals, the identity (_wave2.manifest_key plus model), and the attempt accounting.
• Family rows are kept out of every gaussian_x3 catalogue, calibration, wave-2 scan and summary record.

Catalogue. A new correlated_nd@1 section groups the family by task and then dataset, with ranking: none and cross_family: none. The README has its own region.

Pilot. It has 1040 expected runs: 930 planned and 110 deferred. The deferred ones are Emcee/Zeus at D ≥ 50 (fewer than 2D walkers) and DynestyStatic/NSS at D = 100 (n_live < H = 267.6 nat).

• submit_correlated_nd_pilot is generated by plan_correlated_nd.py, which lint runs with --check.
• It runs on --partition=ral only, excludes euclid-ral-compute-[1-4], requests no gres, and has explicit pin guards.
• Each run gets a 4 h process cap. A run killed without writing a row gets a censored stub row from cnd_cap_stub.py.
• check_submits.py --check is green.

RAL.

• The login-node dry run passed: pins at their commits and importing, a row written at the pin under correlated_nd@1, sbatch --test-only OK.
• The first submission, 409957, failed at import in every task and was cancelled; it wrote no row. Another session had switched the shared clone to an old local main (stash "pre-main-switch", 19:24 UTC), and it lost _autofit_inference_cli.py and config/ as the array started.
• The pilot was therefore resubmitted from a private copy, /mnt/ral/jnightin/inference/autofit_inference_b5c. That copy holds 191 delivered files from 245a3ef, sha256-verified, plus its own copy of .pyauto_pin/. It runs as RAL 410866_[0-929]%100, PENDING at PR-open, with an estimated start of 2026-10-16 ~19:30 UTC.
• The shared clone was left as found.

CI.

• lint installs scipy for the exact-reference tests and runs plan_correlated_nd.py --check.
• witness adds one LBFGS on corr_gaussian_d20, which must be accepted under correlated_nd@1.

Decision taken (decide-and-flag)

The pilot is 930 runs, not the brief's "about 600–800", because Nautilus is exempt from the live-point floor (protocol §11 A1(i)).

• The √(H/n_live) scatter bound is classic nested sampling's. Nautilus's ln Z comes from importance sampling over neural-network bounds, so deferring it was not "cannot answer by construction" (review round 1). §4(c) now judges it directly.
• The banana is in the pilot (it is one of the "3 dimensions" at D = 20), so all four datasets get calibration rows.
• Rejected alternative: the first version's 820 runs, with Nautilus n_live_100 deferred at d50, d100 and banana, n_live_200 deferred at d100, and their warm NUTS deferred with them.
• Revert: set LIVE_POINT_FLOOR_CLASSES in scripts/misc/searches/_correlated_nd.py to include "Nautilus", then run plan_correlated_nd.py --write and the catalogue/README rebuilds. The cells come back as deferred and the rows are never admitted.

Also disclosed. These are not judgement calls:

• the private RAL copy (forced by the disturbed shared clone);
• the submit is named submit_correlated_nd_pilot rather than ….sh, following the repo's naming;
• one run per array task, because there is no measured wall to block seeds by.

Ship gate

Leg	Result
1 Tests	167 passed with the pinned PyAutoFit on the path; 162 passed and 5 skipped in a CI emulation without PyAuto; ruff check and format clean; all 8 --check gates green (catalogue, readme, export, plan_wave2, probe timeouts, judge_wave2, plan_correlated_nd, check_submits).
2 Smoke	n/a: no workspace smoke script is touched. Local end-to-end runs at the pin passed: Nautilus n_live_200 D=20 numpy accepted (all of (b)1–(b)3 and (c)), BlackJAXNUTS D=100 accepted, MultiStartAdam D=100 rejected (a), LBFGS D=20 accepted.
3 Review	Two independent fresh-context Opus rounds, each given the diff only. Round 1: 0 blockers, 2 major, 4 minor, 4 nits; all fixed (A1, pin refusal, process-clock censoring, scanner and exporter isolation, seeding, quadrature). Round 2: 0 blockers, 1 major (no-row SIGKILL → censored stub) and 4 minor, all fixed; 3 nits noted (one fixed).
4 Heart	RED. Development-only override recorded on #14 (quoted below).
Witness	Met. ✓ Own pre-registered protocol correlated_nd@1 (9ec3779, A1 before any row). ✓ Seeded datasets (byte-for-byte regeneration tested). ✓ References per backend with the §6 agreement test (8/8 complete). ✓ Wave-1 pilot manifest committed before any row; no correlated_nd row exists. ✓ Catalogue groups it by task, separately from gaussian_x3, and ranks nothing across families. The pilot itself is submitted (410866, pending).

Merge order: this PR → PyAutoCortex → PyAutoInsight.

Heart RED — development-only override

Heart at the door: Heart: RED — Release RED; monitoring RED · 10/100 · 415 unresolved; [red] release validation FAILED (stage integrate); [red] PyAutoHands/workflows/Python Version Matrix: failure; [red] PyAutoHeart/workflows/Release Integrate: failure; [red] PyAutoArray/Tests: red; [red] autofit_workspace_test/Smoke Tests: red; [red] autogalaxy_workspace_test/Smoke Tests: red; [red] autolens_workspace_test/Smoke Tests: red; [red] Release validation: NOT release_ready; [red] Worktree drift; [red] autolens_inference: red; [red] autolens_profiling: red; [red] euclid_strong_lens_modeling_pipeline: red. Live human in session 6ffb59e9 answered AskUserQuestion "Both, override for development (Recommended)" on 2026-10-15 (third grant) for this launch. Permitted: commit, push, PR-open. Merge is a separate human act; this never reaches a release.

🤖 Generated with Claude Code