diff --git a/AGENTS.md b/AGENTS.md index d9060a0..971a8dd 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -53,6 +53,14 @@ human confirms once, and the chosen papers are filed in one PR. failed approaches) lives in **PyAutoMind** (the `complete///` records, GitHub issues), not here. Memory = what the science says; Mind = what the organism did. +- **Inference and profiling campaign records** — campaign intent, coverage and + evidence presentation belong to **PyAutoInsight** and **PyAutoPulse** + respectively. Their `*_inference` / `*_profiling` projects own protocols, + detailed results and provenance; Cortex retains scientific run records and + human conclusions. Keep short pointers and reusable methodological lessons + here, not duplicate counts, per-search statuses, timing tables or pending work. + Route by information type, not permanence. Before removing an existing summary, + verify a merged, reachable destination preserves its evidence and provenance. - **Workflow state, health, execution** — Mind / Heart / Build respectively. ## Scope: personal research material, out of scope for user-facing repos diff --git a/wiki/methods/concepts/sampler-benchmarks.md b/wiki/methods/concepts/sampler-benchmarks.md index a747a69..aebfae7 100644 --- a/wiki/methods/concepts/sampler-benchmarks.md +++ b/wiki/methods/concepts/sampler-benchmarks.md @@ -73,42 +73,24 @@ From the `autolens_profiling` HPC runs ("searches first-class" series): Raw outputs, job scripts and result JSON/PNGs live in `autolens_profiling` (per-run directories; see that repo's AGENTS.md). -## The search catalogue — every PyAutoFit search, per task (2026-10) - -The organism's first benchmark that covers **every search in the -manifest** rather than a hand-picked few lives in `autofit_inference`: -`catalogue/search_catalogue.json` (schema -`autofit-inference-search-catalogue`, version 1), read here at commit -`3463aa19014b6bd83563018568f0fb4a2376f8eb`. - -- **Protocol id `gaussian_x3@1`** — a toy likelihood (three blended or - separated 1D Gaussians, datasets `gaussian_x3_blend` / - `gaussian_x3_separated`), pre-registered in that repo's - `wiki/project/protocol_gaussian_x3.md`. Campaign task - `gaussian_x3_search_wave1`. -- **Grouped by requested task, never compared across tasks**: - `point_map` (judged by protocol §4(a): max log posterior within 1 nat - of the reference MAP), `posterior` (§4(b): relabelled medians, - sigma-ratio band, PPC χ², mode coverage), `evidence` (§4(b)+§4(c): - posterior accuracy and ln Z within 1 nat). -- **Wave 1 ranks nothing** (protocol §9): its rows calibrate the - protocol's thresholds. 223 of 520 expected runs landed before the - B3 wrap-up ruling (2026-10-08) stopped the pilot; the rest are - deferred to wave 2 on RAL. The pilot ran on PyAutoFit `main` before - phase A2 (gradients under JAX), so the BFGS / L-BFGS non-finite - results it records are not a judgement on those searches. -- Per-search status at that commit: 13 `measured`, NSS `deferred` - (phase A3b), Drawer `unsupported` (a sanity floor that answers no - task). -- The catalogue embeds the search-manifest snapshot it was built - against (`catalogue/search_manifest.json`, `search-manifest@1`), so a - row is keyed by the same registry name the rosters, the capability - matrix and the Brain's samplers faculty use. The faculty lists the - catalogue's cells as its `mature (autofit_inference search - catalogue)` tier. - -Read it as coverage and failure modes per task until a ranking wave -lands; quote no "fastest search" from wave 1. +## The search catalogue — campaign and evidence pointers + +[Insight's wave-1 campaign record](https://github.com/PyAutoLabs/PyAutoInsight/blob/main/tasks/gaussian_x3_search_wave1.md) +is the entry point for the `gaussian_x3` pilot's intent, coverage and interpretation. +Follow [wave 2](https://github.com/PyAutoLabs/PyAutoInsight/blob/main/tasks/gaussian_x3_search_wave2.md) +for its successor campaign. Detailed per-search evidence remains in +[autofit_inference's catalogue](https://github.com/PyAutoLabs/autofit_inference/blob/main/catalogue/search_catalogue.json), +with the [project protocol](https://github.com/PyAutoLabs/autofit_inference/blob/main/wiki/project/protocol_gaussian_x3.md). + +The A5 summary previously recorded here used +[catalogue revision 3463aa1](https://github.com/PyAutoLabs/autofit_inference/blob/3463aa19014b6bd83563018568f0fb4a2376f8eb/catalogue/search_catalogue.json). +That source preserves the historical snapshot; consult the campaign and source +catalogue for counts, statuses and protocol thresholds rather than maintaining +another copy here. + +The reusable lesson is to distinguish point estimation, posterior inference and +evidence estimation before comparing searches. Exploratory calibration alone +does not establish a speed ranking. ## Benchmark hygiene diff --git a/wiki/methods/index.md b/wiki/methods/index.md index e619896..b5366b3 100644 --- a/wiki/methods/index.md +++ b/wiki/methods/index.md @@ -24,9 +24,8 @@ should use this wiki. vmap batching, chunked vmap. - [[initialization-chaining]] — which samplers provide vs require starting points; PyAutoFit chained searches. -- [[sampler-benchmarks]] — the benchmarking-campaign record (MLTracker - contract, minimal + A100 results, raw-output locations) and the - per-task search catalogue (`autofit_inference`, protocol `gaussian_x3@1`). +- [[sampler-benchmarks]] — benchmark methodology, historical minimal + A100 + results, and pointers to Insight's search campaigns and the project-owned catalogue. - [[gradient-optimizer-benchmarks]] — JAX MAP/optimization study on the MGE lens likelihood (CPU + A100): diversity × gradients; multi-start Adam scales (GIGA-Lens), SVGD reaches truth, CMA-ES collapses. diff --git a/wiki/methods/log.md b/wiki/methods/log.md index b558eb2..ba726d3 100644 --- a/wiki/methods/log.md +++ b/wiki/methods/log.md @@ -142,3 +142,13 @@ hand-written section pointing at `autofit_inference`'s `gaussian_x3@1`): per-task grouping, wave 1 ranks nothing, the pilot's coverage and statuses. The index entry names it. No benchmark verdict was added. + +## 2026-10-15 — Route search campaign records through Insight + +Replaced the duplicate A5 campaign summary in [[sampler-benchmarks]] with links +to Insight's existing wave-1 and wave-2 records and the project-owned catalogue +and protocol. Retained the pinned catalogue revision for historical provenance +and the reusable task-comparability lesson. Earlier benchmark sections remain +historical evidence pending a separate destination/provenance audit. The write +contract now routes inference campaigns through Insight and profiling through +Pulse; permanence alone does not make a campaign record Memory content.