inference-summary@3 record shards; fit pinned @3 (migrating_from @1); gaussian_x3 wave 2 closed (B5b-2)

.gitignore

@@ -17,3 +17,4 @@ _mind/
 # Half-written receipt/snapshot files (written then renamed into place)
 receipts/*.tmp
 snapshots/*.tmp
+snapshots/**/*.tmp

REFERENCE.md

@@ -9,11 +9,11 @@ There is no inference conductor and no new scientific execution engine.
 
 `registry.yaml` version `schema: 1` lists instances with `instance`, `repo`,
 `summary_path`, `supported_schema`, `dashboard_url`, `library_refs` and
-`cortex_project`. Repository identity/location resolves from Mind's `repos.yaml`.
+`cortex_project`, plus an optional `migrating_from` (below). Repository identity/location resolves from Mind's `repos.yaml`.
 A source is registered once. A fixture demonstrates generic reading without being
 registered as a real second adopter. Initial real producer: autolens_inference.
 
-The reader supports `schema: inference-summary`, integer `version: 1` or `2`.
+The reader supports `schema: inference-summary`, integer `version: 1`, `2` or `3`.
 Required envelope keys: project, scope, generated_at, evidence_updated_at,
 producer_revision, coverage, records, comparisons, comparison_policy, limitations.
 `project` equals the registry's Mind identity. Times are UTC; unknown evidence time
@@ -191,6 +191,52 @@ lists remain a valid "no measurements" state. The producer and executor must
 validate model/prior/artifact provenance; the dashboard and assistant consume
 the same project-owned catalogue rather than independently reconstructing it.
 
+## Version 3: record shards
+
+V3 is v1's envelope and record contract with the records held in shard files, so
+a producer with many records never commits one large file and a refresh copies
+only what changed. First producer: `fit` (autofit_inference, B5b-2; pinned 2026-10-15,
+first capture pending the producer's merge).
+It does not add v2's setups; v1 and v2 remain valid for the instances pinned to
+them.
+
+The summary file is an **index**: every v1 envelope key except `records`, which
+must be absent, plus `record_shards`, a list of
+`{id, protocol_id, path, sha256, records}`. `id` and `path` are unique; `path` is
+a safe path relative to the index's directory; `sha256` is the lowercase hex
+digest of the shard file's exact bytes; `records` is its record count. Each shard
+is an object with `schema: inference-summary-records`, `version: 3`, the index's
+`project`, `shard` (its index `id`), `protocol_id` and `records`. A record that
+names a protocol (`protocol_id` or `scientific.protocol_id`) must name its
+shard's. The producer chooses the split (autofit_inference: one shard per protocol
+× dataset × search × settings × configuration, one record per line).
+
+The reader verifies every shard's digest, header and count, then assembles the
+records in index order and validates the result exactly as v1, with record ids
+unique across shards and comparisons resolved across them. An altered or
+mismatched shard makes the capture `invalid` and a shard that cannot be fetched
+makes it `unavailable`; either way the whole capture is refused and the last
+good one stays shown. No partial capture is shown. The
+board lists one row per shard (protocol, records, execution, convergence and
+acceptance counts, a link to the shard at the captured commit), never every
+record; each shard file holds the full record provenance.
+
+A capture keeps the index in `snapshots/<instance>.json` (no inline records) and
+each shard's bytes, unchanged, at `snapshots/<instance>/<path>`. A refresh reuses
+every kept shard whose digest is unchanged and fetches only the others, at the same
+resolved commit; it writes shards only after the whole capture validates, then
+the index, then removes kept shards the new index no longer names. An interrupted
+write leaves digests that no longer match (offline: `invalid`), and the next
+refresh re-fetches the mismatched shards. Offline reads re-verify the
+kept shards. A local `--from` read takes shards from beside the index.
+
+**Pin migration.** A row moving to a new version may declare `migrating_from:
+<schema>@<older version>`. Until the first capture under the new pin, the
+committed capture made under that older schema remains valid last-good evidence,
+labelled as awaiting its refresh; offline checks pass and say so. A fresh fetch
+must return the pinned version (an older one is `unsupported`, and the old
+capture is shown cached). Remove the field once the new capture is committed.
+
 ## Ingest and output
 
 `bin/pyauto-insight fetch` resolves each project branch once and fetches the declared

campaigns.yaml

@@ -93,17 +93,16 @@ campaigns:
   title: PyAutoFit search benchmark on gaussian_x3 (autofit_inference)
   status: running
   updated: '2026-10-15'
-  next: B5b-2 — when RAL job 409930 (the 980-run probe-first remainder, Zeus JAX + NSS 50-step) finishes, pull
-    and judge it under gaussian_x3@2 to close wave 2, refresh the fit snapshot from main, and apply the human's
-    choice of form for the wave-2 summary records (autofit_inference#10).
-  recent_progress: Scored wave 2 pulled and judged under gaussian_x3@2 (B5b, 2026-10-15, autofit_inference#10)
-    — 11,770 rows admitted at the pin df6d218f10cd; all 20 per-realisation references complete; catalogue
-    gaussian_x3@2 section orders legs within one task and one dataset; probe-first remainder submitted as 409930.
-  blockers: the wave-2 summary records are withheld on size (~84 MB) pending a human choice of form
-    (autofit_inference#10); the probe-first cells stay open until 409930 is pulled.
-  job_status: unknown — live scheduler not queried; the Cortex ledger records 409930 submitted 2026-10-15 on
-    ral
-  evidence: https://github.com/PyAutoLabs/autofit_inference/issues/10
+  next: After the B5b-2 PRs merge, capture the fit snapshot from main under inference-summary@3, run
+    pyauto-insight check online and drop the registry's migrating_from; then B5c (a second model family).
+  recent_progress: Wave 2 closed (B5b-2, 2026-10-15, autofit_inference#12) — RAL 409930 pulled, 12,750 rows
+    admitted at the pin df6d218f10cd with 0 missing; NSS 50-step leads both evidence rankings; the catalogue
+    ranks only legs with success rate >= 50 % over >= 20 attempts (post-hoc presentation convention); fit
+    records published as inference-summary@3 (101 record shards).
+  blockers: none recorded; the fit snapshot stays the committed @1 capture until the first @3 refresh.
+  job_status: unknown — live scheduler not queried; the Cortex ledger records 409930 COMPLETED (all 20
+    tasks, 2026-10-15)
+  evidence: https://github.com/PyAutoLabs/autofit_inference/issues/12
   project: autofit_inference
 tasks:
 - id: drop_1e3_prior_centring_after_631
@@ -183,6 +182,6 @@ tasks:
   priority: high
   path: tasks/gaussian_x3_search_wave2.md
   updated: '2026-10-15'
-  next: B5b judged the scored arrays under gaussian_x3@2 and submitted the probe-first remainder (409930); B5b-2
-    pulls and judges it, then refreshes the fit snapshot from main.
-  evidence: https://github.com/PyAutoLabs/autofit_inference/issues/10
+  next: Wave 2 is closed and judged (B5b-2); the fit snapshot is captured from main under inference-summary@3
+    after the B5b-2 PRs merge.
+  evidence: https://github.com/PyAutoLabs/autofit_inference/issues/12

dashboard.md

@@ -1,6 +1,6 @@
 # PyAutoInsight — inference control room
 
-<!-- insight-inputs:fc9e31f44edcef644a874bb0d0bfd233c5b8b872d26e9564f456cf8f879e7b41 -->
+<!-- insight-inputs:401cd93fc2d479242b5e4a37d1ca248a2de2844aea87e2c27ff248c29b11ce0f -->
 
 | Where | Count |
 |---|---:|
@@ -10,7 +10,7 @@
 | [Cached](#evidence) | 0 |
 | [Failed](#evidence) | 0 |
 
-<!-- insight-campaigns:9e10c87a19b139911692bf88e95eb66eeb201f2b6c230f01dbb4069808e59b2d -->
+<!-- insight-campaigns:bc4e241a4b624a5cbaceb3a77abc2c84cf5bb57a506760acc1f2fb98acc21210 -->
 ## Check in on all inference work
 
 Copy this into one chat. Add a campaign focus or idea before or after it, or leave it unchanged.
@@ -46,7 +46,7 @@ After taking action, report what changed, what the evidence supports and what re
 | [N=50 NUTS versus three EP repeats](https://github.com/PyAutoLabs/PyAutoCortex/blob/d45f198804615c090e6b6694cf725ddb0be2354e/projects/ep_toy_gaussian.md) | Cortex records failed wave 1 (EMFILE/OOM), wave-2 preparation and no submission. | Present the existing wave-2 proposal with current dependency evidence for decision. |
 | [Hierarchical slope recovery at scale](https://github.com/PyAutoLabs/PyAutoCortex/blob/d45f198804615c090e6b6694cf725ddb0be2354e/projects/slope_hierarchy_scale.md) | Cortex retains graphical baseline, post-processing OOM and stale EP hierarchical factor; Mind records later moments fix. | Verify the moments fix release and project mirror; recover missing summaries from existing samples before a proposed rerun. |
 | [IC50 EP and graphical scale ladder](https://github.com/PyAutoLabs/PyAutoCortex/blob/d45f198804615c090e6b6694cf725ddb0be2354e/projects/ic50_workspace.md) | Cortex records N=5/10/25 results, failed N=50 EP rung and width-interpretation caveat. | Check implementation close-outs, then propose a seeded ladder with appropriate output and mean-field diagnostics. |
-| [PyAutoFit search benchmark on gaussian_x3 (autofit_inference)](https://github.com/PyAutoLabs/autofit_inference/issues/10) | Scored wave 2 pulled and judged under gaussian_x3@2 (B5b, 2026-10-15, autofit_inference#10) — 11,770 rows admitted at the pin df6d218f10cd; all 20 per-realisation references complete; catalogue gaussian_x3@2 section orders legs within one task and one dataset; probe-first remainder submitted as 409930. | B5b-2 — when RAL job 409930 (the 980-run probe-first remainder, Zeus JAX + NSS 50-step) finishes, pull and judge it under gaussian_x3@2 to close wave 2, refresh the fit snapshot from main, and apply the human&#x27;s choice of form for the wave-2 summary records (autofit_inference#10). |
+| [PyAutoFit search benchmark on gaussian_x3 (autofit_inference)](https://github.com/PyAutoLabs/autofit_inference/issues/12) | Wave 2 closed (B5b-2, 2026-10-15, autofit_inference#12) — RAL 409930 pulled, 12,750 rows admitted at the pin df6d218f10cd with 0 missing; NSS 50-step leads both evidence rankings; the catalogue ranks only legs with success rate &gt;= 50 % over &gt;= 20 attempts (post-hoc presentation convention); fit records published as inference-summary@3 (101 record shards). | After the B5b-2 PRs merge, capture the fit snapshot from main under inference-summary@3, run pyauto-insight check online and drop the registry&#x27;s migrating_from; then B5c (a second model family). |
 
 ## Active tasks
 
@@ -58,7 +58,7 @@ After taking action, report what changed, what the evidence supports and what re
 | [Graphical Model Scale-Up — Scoping](https://github.com/PyAutoLabs/PyAutoInsight/blob/main/tasks/graphical_scoping.md) | graphical-ep | needs-slicing | high | Reconcile current Cortex scale results and library dependencies; select one bounded inference phase with a witness. |
 | [NUTS/HMC trial on validated likelihoods](https://github.com/PyAutoLabs/PyAutoInsight/blob/main/tasks/gradient_sampler_validated_likelihoods.md) | point-source-admission | needs-decision | normal | An unscoped idea, not permission to submit: reconcile point-source admission with imaging and weak-lensing candidate targets. |
 | [gaussian_x3 search benchmark — wave 1 (the pilot)](https://github.com/PyAutoLabs/PyAutoInsight/blob/main/tasks/gaussian_x3_search_wave1.md) | gaussian-x3-search-benchmark | blocked | normal | Pilot stopped at 223 of 520 runs and is not restarted; freeze gaussian_x3@2 from its 17-row calibration before any scored run; the deferred cells move to scored wave 2 (after A2 and A4). |
-| [gaussian_x3 search benchmark — wave 2 (scored)](https://github.com/PyAutoLabs/PyAutoInsight/blob/main/tasks/gaussian_x3_search_wave2.md) | gaussian-x3-search-benchmark | running | high | B5b judged the scored arrays under gaussian_x3@2 and submitted the probe-first remainder (409930); B5b-2 pulls and judges it, then refreshes the fit snapshot from main. |
+| [gaussian_x3 search benchmark — wave 2 (scored)](https://github.com/PyAutoLabs/PyAutoInsight/blob/main/tasks/gaussian_x3_search_wave2.md) | gaussian-x3-search-benchmark | running | high | Wave 2 is closed and judged (B5b-2); the fit snapshot is captured from main under inference-summary@3 after the B5b-2 PRs merge. |
 
 [Ledger](https://github.com/PyAutoLabs/PyAutoInsight/blob/main/campaigns.yaml)
 
@@ -12133,6 +12133,7 @@ Archived rows belong to this project's September 2026 runs and are historical, n
 ## autofit_inference
 Integrity: ok. Freshness: freshness policy unspecified · evidence 2026-10-08T18:53:07Z. Qualification: 35/223 accepted by producer; remaining assessments shown below.
 Capture source branch: main; revision: e60dc01f9f7a3f338c60946bcf4f48a1b2c48497; fetched 2026-10-15T16:02:01Z; latest attempt 2026-10-15T16:02:01Z.
+Captured under inference-summary@1; the registry pins inference-summary@3, awaiting the first refresh under it.
 
 <details><summary>Full captured evidence and provenance</summary>
 

insight/board.py

@@ -6,6 +6,7 @@
 import html
 import json
 import re
+from collections import Counter
 from datetime import UTC, datetime
 from pathlib import Path
 
@@ -29,6 +30,24 @@ def comparisons(s):
     return (s.doc or {}).get("comparisons", [])
 
 
+def sharded(s):
+    return (s.doc or {}).get("version") == summary.SHARDED
+
+
+def shard_rows(s):
+    """``(shard entry, its records)`` of a v3 capture: the assembled records are
+    the shards' records concatenated in index order with the declared counts."""
+    recs, at = records(s), 0
+    for entry in (s.doc or {}).get("record_shards", []):
+        yield entry, recs[at : at + entry["records"]]
+        at += entry["records"]
+
+
+def _tally(recs, pick):
+    counts = Counter(pick(r) for r in recs)
+    return ", ".join(f"{k} {n}" for k, n in sorted(counts.items(), key=lambda kv: str(kv[0])))
+
+
 def integrity(s):
     if s.source == "local":
         return f"local checkout {s.commit or 'unknown'} ({'dirty' if s.dirty else 'clean' if s.dirty is False else 'unknown state'})"
@@ -75,7 +94,9 @@ def input_marker(views, data):
             "instance": vars(s.instance),
             "outcome": s.outcome,
             "source": s.source,
-            "doc": s.doc,
+            # A v3 index's sha256 digests pin every shard's records.
+            "doc": summary.index_of(s.doc),
+            "migrating": s.migrating,
             "commit": s.commit,
             "fetched_at": s.fetched_at,
             "errors": s.errors,
@@ -149,6 +170,8 @@ def _detail(s, now=None):
         parts.append(
             '<p class="warn">Local preview, not a published capture. No receipt written.</p>'
         )
+    if s.migrating:
+        parts.append('<p class="warn">' + e(s.migrating) + ".</p>")
     if d.get("version") == 2:
         nav, _ = setup_browser.render(s, now)
         warnings = []
@@ -169,6 +192,8 @@ def _detail(s, now=None):
     ]
     if not records(s):
         parts.append("<p>No measurements. An empty feed does not mean scientific success.</p>")
+    if sharded(s):
+        return "".join(parts + [_shard_html(s)]) + "</details></div>"
     for archived, label in [(False, "Current results"), (True, "Historical project results")]:
         rows = []
         for r in records(s):
@@ -263,6 +288,62 @@ def _detail(s, now=None):
     return "".join(parts) + "</details></div>"
 
 
+def _shard_html(s):
+    """A v3 capture, one row per record shard (a protocol × leg): its counts and a
+    link to the shard at the captured commit. Records are not listed one by one;
+    each shard file holds their full provenance."""
+    inst = s.instance
+    rows = []
+    for entry, recs in shard_rows(s):
+        where = summary.shard_location(inst.summary_path, entry["path"])
+        rows.append(
+            [
+                e(entry["protocol_id"]),
+                e(entry["id"]),
+                e(len(recs)),
+                e(_tally(recs, lambda r: str(r["execution"]["status"]).split(":", 1)[0])),
+                e(_tally(recs, lambda r: r["scientific"]["convergence"])),
+                e(_tally(recs, lambda r: r["scientific"]["acceptance"])),
+                f'<a href="{e(inst.blob_url(s.commit, where))}">records</a>',
+            ]
+        )
+    return (
+        "<h3>Record shards</h3><p>inference-summary@3: the records are held in one file per "
+        "protocol and leg, each verified against the index's sha256 at the captured commit. "
+        "Counts are the producer's assertions, shown without ranking; each shard file "
+        "holds every record's diagnostics, timings and provenance.</p>"
+        + table(
+            [
+                "Protocol",
+                "Shard",
+                "Records",
+                "Execution",
+                "Convergence",
+                "Acceptance",
+                "Evidence",
+            ],
+            rows,
+        )
+        + "<h3>Comparisons and parity groups</h3><p>Targets, seeds and stages are retained. No best-seed selection, cross-target ranking or automatic convergence threshold. Declared groups do not themselves certify parity.</p>"
+        + (
+            "<p>No certified comparison pairs supplied by this producer.</p>"
+            if not comparisons(s)
+            else "".join(
+                "<p>"
+                + e(c["id"])
+                + ": "
+                + e(
+                    "refused: " + "; ".join(reasons)
+                    if (reasons := summary.comparison_refusals(s.doc, c))
+                    else "producer-declared protocol " + c["protocol"]
+                )
+                + "</p>"
+                for c in comparisons(s)
+            )
+        )
+    )
+
+
 def _captured_at(views):
     """Oldest successful observation of displayed inputs; failed or missing inputs stay unknown.
 
@@ -345,8 +426,41 @@ def render_markdown(views, now=None, campaign_data=None):
         ]
         if s.cached:
             out.append("Cached: original evidence and capture times retained.")
+        if s.migrating:
+            out.append(s.migrating[0].upper() + s.migrating[1:] + ".")
         if not s.doc:
             continue
+        if sharded(s):
+            out += [
+                "",
+                "Coverage: " + _dump(s.doc["coverage"]),
+                *s.doc["limitations"],
+                "",
+                "| Protocol | Shard | Records | Execution | Convergence | Acceptance | Evidence |",
+                "|---|---|---:|---|---|---|---|",
+            ]
+            for entry, recs in shard_rows(s):
+                where = summary.shard_location(s.instance.summary_path, entry["path"])
+                vals = [
+                    entry["protocol_id"],
+                    entry["id"],
+                    len(recs),
+                    _tally(recs, lambda r: str(r["execution"]["status"]).split(":", 1)[0]),
+                    _tally(recs, lambda r: r["scientific"]["convergence"]),
+                    _tally(recs, lambda r: r["scientific"]["acceptance"]),
+                ]
+                out.append(
+                    "| "
+                    + " | ".join(md(v) for v in vals)
+                    + f" | [records]({s.instance.blob_url(s.commit, where)}) |"
+                )
+            for c in comparisons(s):
+                out.append(
+                    f"Comparison {md(c['id'])}: "
+                    + md(summary.comparison_refusals(s.doc, c) or c["protocol"])
+                )
+            out.append("")
+            continue
         if s.doc.get("version") == 2:
             for setup in s.doc["setups"]:
                 out.append(