Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,14 @@ human confirms once, and the chosen papers are filed in one PR.
failed approaches) lives in **PyAutoMind** (the `complete/<YYYY>/<MM>/`
records, GitHub issues), not here. Memory = what the science says; Mind = what
the organism did.
- **Inference and profiling campaign records** — campaign intent, coverage and
evidence presentation belong to **PyAutoInsight** and **PyAutoPulse**
respectively. Their `*_inference` / `*_profiling` projects own protocols,
detailed results and provenance; Cortex retains scientific run records and
human conclusions. Keep short pointers and reusable methodological lessons
here, not duplicate counts, per-search statuses, timing tables or pending work.
Route by information type, not permanence. Before removing an existing summary,
verify a merged, reachable destination preserves its evidence and provenance.
- **Workflow state, health, execution** — Mind / Heart / Build respectively.

## Scope: personal research material, out of scope for user-facing repos
Expand Down
54 changes: 18 additions & 36 deletions wiki/methods/concepts/sampler-benchmarks.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,42 +73,24 @@ From the `autolens_profiling` HPC runs ("searches first-class" series):
Raw outputs, job scripts and result JSON/PNGs live in
`autolens_profiling` (per-run directories; see that repo's AGENTS.md).

## The search catalogue — every PyAutoFit search, per task (2026-10)

The organism's first benchmark that covers **every search in the
manifest** rather than a hand-picked few lives in `autofit_inference`:
`catalogue/search_catalogue.json` (schema
`autofit-inference-search-catalogue`, version 1), read here at commit
`3463aa19014b6bd83563018568f0fb4a2376f8eb`.

- **Protocol id `gaussian_x3@1`** — a toy likelihood (three blended or
separated 1D Gaussians, datasets `gaussian_x3_blend` /
`gaussian_x3_separated`), pre-registered in that repo's
`wiki/project/protocol_gaussian_x3.md`. Campaign task
`gaussian_x3_search_wave1`.
- **Grouped by requested task, never compared across tasks**:
`point_map` (judged by protocol §4(a): max log posterior within 1 nat
of the reference MAP), `posterior` (§4(b): relabelled medians,
sigma-ratio band, PPC χ², mode coverage), `evidence` (§4(b)+§4(c):
posterior accuracy and ln Z within 1 nat).
- **Wave 1 ranks nothing** (protocol §9): its rows calibrate the
protocol's thresholds. 223 of 520 expected runs landed before the
B3 wrap-up ruling (2026-10-08) stopped the pilot; the rest are
deferred to wave 2 on RAL. The pilot ran on PyAutoFit `main` before
phase A2 (gradients under JAX), so the BFGS / L-BFGS non-finite
results it records are not a judgement on those searches.
- Per-search status at that commit: 13 `measured`, NSS `deferred`
(phase A3b), Drawer `unsupported` (a sanity floor that answers no
task).
- The catalogue embeds the search-manifest snapshot it was built
against (`catalogue/search_manifest.json`, `search-manifest@1`), so a
row is keyed by the same registry name the rosters, the capability
matrix and the Brain's samplers faculty use. The faculty lists the
catalogue's cells as its `mature (autofit_inference search
catalogue)` tier.

Read it as coverage and failure modes per task until a ranking wave
lands; quote no "fastest search" from wave 1.
## The search catalogue — campaign and evidence pointers

[Insight's wave-1 campaign record](https://github.com/PyAutoLabs/PyAutoInsight/blob/main/tasks/gaussian_x3_search_wave1.md)
is the entry point for the `gaussian_x3` pilot's intent, coverage and interpretation.
Follow [wave 2](https://github.com/PyAutoLabs/PyAutoInsight/blob/main/tasks/gaussian_x3_search_wave2.md)
for its successor campaign. Detailed per-search evidence remains in
[autofit_inference's catalogue](https://github.com/PyAutoLabs/autofit_inference/blob/main/catalogue/search_catalogue.json),
with the [project protocol](https://github.com/PyAutoLabs/autofit_inference/blob/main/wiki/project/protocol_gaussian_x3.md).

The A5 summary previously recorded here used
[catalogue revision 3463aa1](https://github.com/PyAutoLabs/autofit_inference/blob/3463aa19014b6bd83563018568f0fb4a2376f8eb/catalogue/search_catalogue.json).
That source preserves the historical snapshot; consult the campaign and source
catalogue for counts, statuses and protocol thresholds rather than maintaining
another copy here.

The reusable lesson is to distinguish point estimation, posterior inference and
evidence estimation before comparing searches. Exploratory calibration alone
does not establish a speed ranking.

## Benchmark hygiene

Expand Down
5 changes: 2 additions & 3 deletions wiki/methods/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,8 @@ should use this wiki.
vmap batching, chunked vmap.
- [[initialization-chaining]] — which samplers provide vs require
starting points; PyAutoFit chained searches.
- [[sampler-benchmarks]] — the benchmarking-campaign record (MLTracker
contract, minimal + A100 results, raw-output locations) and the
per-task search catalogue (`autofit_inference`, protocol `gaussian_x3@1`).
- [[sampler-benchmarks]] — benchmark methodology, historical minimal + A100
results, and pointers to Insight's search campaigns and the project-owned catalogue.
- [[gradient-optimizer-benchmarks]] — JAX MAP/optimization study on the MGE
lens likelihood (CPU + A100): diversity × gradients; multi-start Adam scales
(GIGA-Lens), SVGD reaches truth, CMA-ES collapses.
Expand Down
10 changes: 10 additions & 0 deletions wiki/methods/log.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,3 +142,13 @@ hand-written section pointing at `autofit_inference`'s
`gaussian_x3@1`): per-task grouping, wave 1 ranks nothing, the
pilot's coverage and statuses. The index entry names it. No benchmark
verdict was added.

## 2026-10-15 — Route search campaign records through Insight

Replaced the duplicate A5 campaign summary in [[sampler-benchmarks]] with links
to Insight's existing wave-1 and wave-2 records and the project-owned catalogue
and protocol. Retained the pinned catalogue revision for historical provenance
and the reusable task-comparability lesson. Earlier benchmark sections remain
historical evidence pending a separate destination/provenance audit. The write
contract now routes inference campaigns through Insight and profiling through
Pulse; permanence alone does not make a campaign record Memory content.
Loading