B5c: second model family correlated_nd — protocol correlated_nd@1, exact references, 930-run pilot

.github/workflows/lint.yml

@@ -33,7 +33,8 @@ jobs:
 
       - name: Install lint tools
         run: |
-          pip install ruff numpy pytest pyyaml
+          # scipy: the correlated_nd exact-reference tests (normal CDFs of the banana).
+          pip install ruff numpy scipy pytest pyyaml
           # lychee is a Rust binary; fetch the prebuilt release. The tarball
           # nests everything under lychee-x86_64-unknown-linux-gnu/, so strip
           # that leading directory and extract just the binary.
@@ -76,6 +77,9 @@ jobs:
       - name: wave-2 accounting --check
         run: python scripts/misc/searches/judge_wave2.py --check
 
+      - name: correlated_nd pilot plan --check
+        run: python scripts/misc/tooling/plan_correlated_nd.py --check
+
       - name: WALL-BASIS gate
         run: python scripts/misc/wall/check_submits.py --check
 

.github/workflows/witness.yml

@@ -80,3 +80,24 @@ jobs:
           python scripts/misc/tooling/export_inference_summary.py \
             --rows-root "$RUNNER_TEMP/witness" --output "$RUNNER_TEMP/witness/summary.json" \
             --require-accepted
+
+      # The second family (B5c): one LBFGS on the D = 20 correlated Gaussian, judged by
+      # correlated_nd@1 against the committed exact reference; must be `accepted`.
+      - name: LBFGS, numpy, seed 0, corr_gaussian_d20 (correlated_nd@1)
+        working-directory: autofit_inference
+        env:
+          PYTHONPATH: ${{ github.workspace }}/PyAutoNerves:${{ github.workspace }}/PyAutoFit
+          NUMBA_CACHE_DIR: /tmp/numba_cache
+          MPLCONFIGDIR: /tmp/matplotlib
+          OMP_NUM_THREADS: "1"
+        run: |
+          python scripts/corr_gaussian_d20/searches/lbfgs/correlated_nd.py \
+            --config-name local_numpy_fp64 --seed 0 \
+            --output-dir "$RUNNER_TEMP/output_cnd" --results-root "$RUNNER_TEMP/witness_cnd"
+          python - "$RUNNER_TEMP/witness_cnd" <<'PY'
+          import json, pathlib, sys
+          (path,) = pathlib.Path(sys.argv[1]).rglob("search_seed0.json")
+          verdict = json.loads(path.read_text())["scientific"]
+          print(verdict["protocol_id"], verdict["acceptance"], verdict["acceptance_reason"])
+          assert verdict["protocol_id"] == "correlated_nd@1" and verdict["acceptance"] == "accepted"
+          PY

AGENTS.md

@@ -45,6 +45,14 @@ and it changes only through a new version committed before the first row it judg
 The thresholds live once, in `scripts/misc/searches/_protocol.py`; PLACEHOLDER values are
 calibrated in the wave-1 pilot and then frozen. Wave 1 ranks nothing.
 
+The second family, `correlated_nd` (B5c: correlated Gaussians at D = 20/50/100 and a D = 20
+Rosenbrock banana, exact references), has its own pre-registered protocol
+[`wiki/project/protocol_correlated_nd.md`](wiki/project/protocol_correlated_nd.md)
+(`correlated_nd@1`, `_protocol.THRESHOLDS_CND1`), its own manifest
+(`searches/_correlated_nd.py`) and its own catalogue section. Its rows are never mixed into
+`gaussian_x3` catalogues, calibration or summary records, and nothing is ranked across
+families.
+
 ## Import model and leaves
 
 `ruff.toml` is the root sentinel: every script walks up to it and puts the root and
@@ -79,16 +87,21 @@ _autofit_inference_cli.py  flags, the {local,ral}_{numpy,jax_cpu}_{fp64,fp32} gr
 config/general.yaml        PyAutoConf overrides (outputs kept)
 dataset/<name>/            committed data/noise_map/model/truth JSON (seeded)
 scripts/<dataset>/searches/<sampler>/gaussian_x3.py   the leaves
-scripts/misc/models/       the gaussian_x3 model, priors and Analysis
+scripts/<dataset>/searches/<sampler>/correlated_nd.py the second family's leaves (B5c)
+scripts/misc/models/       the gaussian_x3 and correlated_nd models, priors and Analysis;
+                           correlated_nd_exact.py (every exact posterior quantity)
 scripts/misc/searches/     _runner.py (run_search + registry), _posterior.py, _protocol.py,
                            _metrics.py (MLTracker), _pilot.py (wave-1 manifest),
                            _wave2.py (scored wave-2 manifest, identity, accounting, pin),
-                           judge_wave2.py (@2 admit/accounting CLI -> results/accounting/)
-scripts/misc/reference/    reference-posterior, MAP and constant-likelihood runs + combiner
-scripts/misc/simulators/   the seeded dataset simulator
+                           judge_wave2.py (@2 admit/accounting CLI -> results/accounting/),
+                           _correlated_nd.py (the correlated_nd pilot manifest, deferrals)
+scripts/misc/reference/    reference-posterior, MAP and constant-likelihood runs + combiner;
+                           exact_correlated_nd.py (the exact references + §6 self-test)
+scripts/misc/simulators/   the seeded dataset simulators (gaussian_x3, correlated_nd)
 scripts/misc/tooling/      build_readme.py, export_inference_summary.py, build_catalogue.py
                            (wave-1 pilot + gaussian_x3@2 sections), plan_wave2.py,
-                           write_probe_timeouts.py (probe-first @2 timeouts)
+                           write_probe_timeouts.py (probe-first @2 timeouts),
+                           plan_correlated_nd.py (the correlated_nd pilot submit)
 scripts/misc/wall/         the WALL-BASIS submit gate + rates from the pilot walls
 scripts/misc/test/         unit tests + the exporter fixture
 results/reference/         reference posteriors per dataset and backend

README.md

@@ -18,6 +18,10 @@ of the
   closed (250 blend warm-NUTS runs deferred). They are judged under the frozen `@2`
   thresholds and ranked within one task and one dataset among the legs that clear a
   reliability floor (see "Scored wave 2" below).
+- **A second family, `correlated_nd` (B5c).** High-D correlated Gaussians and a
+  Rosenbrock banana with exact references, under its own pre-registered protocol
+  `correlated_nd@1`; its 930-run pilot runs on RAL and ranks nothing (see "The second
+  family" below).
 - **The wave-1 pilot** (223 rows, below) ranks nothing. It calibrated the protocol and
   ran on PyAutoFit main before phase A2.
 - **Summary.** The repo reports to
@@ -47,6 +51,31 @@ coverage, evidence searches on `ln Z` within one backend under the `ln 3!` conve
 Wave 1 is a pilot that ranks nothing; its PLACEHOLDER thresholds are calibrated and then
 frozen before the scored wave 2.
 
+## The second family: `correlated_nd` (B5c)
+
+High-dimensional, ill-conditioned targets with **exact** posteriors and evidences, so no
+reference search is run: correlated Gaussians at D = 20, 50 and 100 with condition number
+10³ (a seeded Haar-random rotation of log-spaced eigenvalues), and a D = 20 hybrid
+Rosenbrock banana (Pagani et al. 2022). Datasets come from
+`scripts/misc/simulators/correlated_nd.py`; the exact references
+(`scripts/misc/reference/exact_correlated_nd.py`) pass a per-backend agreement test before
+any row is judged. Rows are judged by their own pre-registered protocol,
+[`wiki/project/protocol_correlated_nd.md`](wiki/project/protocol_correlated_nd.md)
+(`correlated_nd@1`), and catalogued by task and dataset **apart from `gaussian_x3`;
+nothing is ranked across families.** The pilot (930 runs: 10 seeds × every non-deferred
+cell of the gaussian_x3 wave-2 menu × the four datasets) runs on RAL `--partition=ral`.
+
+<!-- BEGIN auto-table:correlated_nd -->
+| Dataset | D | Exact ln Z | H (nat) | Exact references | Planned cells | Deferred cells | Pilot rows (failed) |
+|---|---|---|---|---|---|---|---|
+| `corr_gaussian_d20` | 20 | -93.020 | 53.1 | `jax_cpu` complete, `numpy` complete | 26 | 0 | 0 (0) |
+| `corr_gaussian_d50` | 50 | -233.019 | 133.3 | `jax_cpu` complete, `numpy` complete | 22 | 4 | 0 (0) |
+| `corr_gaussian_d100` | 100 | -467.088 | 267.6 | `jax_cpu` complete, `numpy` complete | 19 | 7 | 0 (0) |
+| `rosenbrock_d20` | 20 | -140.946 | 133.3 | `jax_cpu` complete, `numpy` complete | 26 | 0 | 0 (0) |
+
+Pilot: 1040 expected runs — 0 rows, 0 failed, 930 not yet pulled, 110 deferred with reasons (`catalogue/search_catalogue.json`, section `correlated_nd@1`). Wave 1 (pilot): ranks nothing (protocol correlated_nd §9). Cross-family ranking: none: this family is catalogued apart from gaussian_x3 and no leg is ever ranked or compared across families (protocol correlated_nd §9).
+<!-- END auto-table:correlated_nd -->
+
 ## Running
 
 ```bash

_autofit_inference_cli.py

@@ -51,8 +51,16 @@
 #: ``(where, backend, precision)`` — one group per field.
 CONFIG_NAME_RE = re.compile(r"^(local|ral)_(numpy|jax_cpu)_(fp64|fp32)$")
 
-#: The datasets a leaf may name (``dataset/<name>/``).
-DATASETS = ("gaussian_x3_blend", "gaussian_x3_separated")
+#: The datasets a leaf may name (``dataset/<name>/``): the gaussian_x3 pair, then the
+#: second family's (``correlated_nd``, B5c; ``scripts/misc/simulators/correlated_nd.py``).
+GAUSSIAN_X3_DATASETS = ("gaussian_x3_blend", "gaussian_x3_separated")
+CORRELATED_ND_DATASETS = (
+    "corr_gaussian_d20",
+    "corr_gaussian_d50",
+    "corr_gaussian_d100",
+    "rosenbrock_d20",
+)
+DATASETS = GAUSSIAN_X3_DATASETS + CORRELATED_ND_DATASETS
 
 #: ``--compile-cache``: ``cold`` disables JAX's persistent compilation cache so
 #: ``compile_s`` is a full trace + lower + compile; ``warm`` leaves it on. The two are
@@ -157,8 +165,8 @@ def build_parser(default_config_name: str = "local_numpy_fp64") -> argparse.Argu
         type=int,
         choices=(1, 2),
         default=1,
-        help="1 = the wave-1 pilot (gaussian_x3@1); 2 = scored wave 2 (gaussian_x3@2, "
-        "refused unless autofit is the pinned revision)",
+        help="1 = the wave-1 pilot (gaussian_x3@1; correlated_nd@1 for that family); "
+        "2 = scored wave 2 (gaussian_x3@2, refused unless autofit is the pinned revision)",
     )
     return parser
 
@@ -263,19 +271,27 @@ def library_revisions(libraries=("autonerves", "autofit")) -> dict:
 def dataset_path(dataset: str, data_seed: int | None = None, root: Path = ROOT) -> Path:
     """``dataset/<name>/`` (or ``…/data_seed<n>/`` for a non-default realisation)."""
     sys.path.insert(0, str(root / "scripts" / "misc"))
-    from simulators.gaussian_x3 import dataset_dir
+    if dataset in CORRELATED_ND_DATASETS:
+        from simulators.correlated_nd import dataset_dir
+    else:
+        from simulators.gaussian_x3 import dataset_dir
 
     return dataset_dir(dataset, data_seed, root)
 
 
+def simulator_for(dataset: str) -> str:
+    """The simulator script (below ``scripts/misc/simulators/``) that writes ``dataset``."""
+    return "correlated_nd.py" if dataset in CORRELATED_ND_DATASETS else "gaussian_x3.py"
+
+
 def auto_simulate_if_missing(dataset: str, data_seed: int | None = None, root: Path = ROOT) -> Path:
     """Simulate the dataset when its ``data.json`` is absent; return its directory."""
     path = dataset_path(dataset, data_seed, root)
     if (path / "data.json").exists():
         return path
     command = [
         sys.executable,
-        str(root / "scripts" / "misc" / "simulators" / "gaussian_x3.py"),
+        str(root / "scripts" / "misc" / "simulators" / simulator_for(dataset)),
         "--name",
         dataset,
         "--output-root",