Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion badge.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"schemaVersion": 1,
"label": "insight",
"message": "1 projects \u00b7 56 records \u00b7 0 cached",
"message": "2 projects \u00b7 56 records \u00b7 0 cached",
"color": "yellow"
}
24 changes: 24 additions & 0 deletions campaigns.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,20 @@ campaigns:
job_status: unknown — live scheduler not queried; historical ledger states are not current job evidence
evidence: https://github.com/PyAutoLabs/PyAutoCortex/blob/d45f198804615c090e6b6694cf725ddb0be2354e/projects/ic50_workspace.md
project: ic50_workspace
- id: gaussian-x3-search-benchmark
title: PyAutoFit search benchmark on gaussian_x3 (autofit_inference)
status: blocked
updated: '2026-10-08'
next: Run the 297 deferred wave-1 runs (and the JAX blend reference seed 1) on RAL partition ral on a PyAutoFit
revision after A2, then freeze gaussian_x3@2 before any wave-2 row.
recent_progress: Wave-1 pilot stopped at 223 of 520 expected runs by the B3 wrap-up ruling (2026-10-08);
pilot ranks nothing; catalogue, calibration record and inference-summary@1 published.
blockers: Phase A2 (PyAutoFit#1679, gradients under JAX) must merge first; NSS waits on A3b; no compute
submission is authorized by this ledger.
job_status: unknown — live scheduler not queried; the local pilot was stopped and this ledger records no
RAL submission
evidence: https://github.com/PyAutoLabs/autofit_inference/issues/4
project: autofit_inference
tasks:
- id: drop_1e3_prior_centring_after_631
campaign: slam-hst-base
Expand Down Expand Up @@ -150,3 +164,13 @@ tasks:
next: 'An unscoped idea, not permission to submit: reconcile point-source admission with imaging and weak-lensing
candidate targets.'
evidence: https://github.com/PyAutoLabs/PyAutoMind/blob/8dd1edc7e04c960daae3663513c61395823eddb0/ideas.md
- id: gaussian_x3_search_wave1
campaign: gaussian-x3-search-benchmark
title: gaussian_x3 search benchmark — wave 1 (the pilot)
status: blocked
priority: normal
path: tasks/gaussian_x3_search_wave1.md
updated: '2026-10-08'
next: Pilot partial (223 of 520 runs); run the deferred cells on RAL partition ral after A2, then freeze
gaussian_x3@2.
evidence: https://github.com/PyAutoLabs/autofit_inference/issues/4
9 changes: 5 additions & 4 deletions dashboard.html

Large diffs are not rendered by default.

29 changes: 26 additions & 3 deletions dashboard.md
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
# PyAutoInsight — inference control room

<!-- insight-inputs:f70051ff43253519801b4a7d148faab5708eee5feeb4f55c4f6efca3a0b2c865 -->
<!-- insight-inputs:4d0d489a4e3ad725a1cce864a9cb930eb030864e01ae55fc468b4eb68df27e08 -->

| Where | Count |
|---|---:|
| [Projects](#evidence) | 1 |
| [Projects](#evidence) | 2 |
| [Records](#evidence) | 56 |
| [Comparisons](#evidence) | 0 |
| [Cached](#evidence) | 0 |
| [Failed](#evidence) | 0 |

<!-- insight-campaigns:cebf9dfa86c60986e2ae3381ae314259f7e491233af2ec497ff330f710f3e901 -->
<!-- insight-campaigns:b76456132d7495bee180722f43f437e05e473221388e08c82463a9dd7dbb4481 -->
## Check in on all inference work

Copy this into one chat. Add a campaign focus or idea before or after it, or leave it unchanged.
Expand Down Expand Up @@ -46,6 +46,7 @@ After taking action, report what changed, what the evidence supports and what re
| [N=50 NUTS versus three EP repeats](https://github.com/PyAutoLabs/PyAutoCortex/blob/d45f198804615c090e6b6694cf725ddb0be2354e/projects/ep_toy_gaussian.md) | Cortex records failed wave 1 (EMFILE/OOM), wave-2 preparation and no submission. | Present the existing wave-2 proposal with current dependency evidence for decision. |
| [Hierarchical slope recovery at scale](https://github.com/PyAutoLabs/PyAutoCortex/blob/d45f198804615c090e6b6694cf725ddb0be2354e/projects/slope_hierarchy_scale.md) | Cortex retains graphical baseline, post-processing OOM and stale EP hierarchical factor; Mind records later moments fix. | Verify the moments fix release and project mirror; recover missing summaries from existing samples before a proposed rerun. |
| [IC50 EP and graphical scale ladder](https://github.com/PyAutoLabs/PyAutoCortex/blob/d45f198804615c090e6b6694cf725ddb0be2354e/projects/ic50_workspace.md) | Cortex records N=5/10/25 results, failed N=50 EP rung and width-interpretation caveat. | Check implementation close-outs, then propose a seeded ladder with appropriate output and mean-field diagnostics. |
| [PyAutoFit search benchmark on gaussian_x3 (autofit_inference)](https://github.com/PyAutoLabs/autofit_inference/issues/4) | Wave-1 pilot stopped at 223 of 520 expected runs by the B3 wrap-up ruling (2026-10-08); pilot ranks nothing; catalogue, calibration record and inference-summary@1 published. | Run the 297 deferred wave-1 runs (and the JAX blend reference seed 1) on RAL partition ral on a PyAutoFit revision after A2, then freeze gaussian_x3@2 before any wave-2 row. |

## Active tasks

Expand All @@ -56,6 +57,7 @@ After taking action, report what changed, what the evidence supports and what re
| [Expectation Propagation Scale-Up — Scoping](https://github.com/PyAutoLabs/PyAutoInsight/blob/main/tasks/ep_scoping.md) | graphical-ep | needs-slicing | high | Reconcile completed wrapper work and Pulse profiling ownership before selecting a bounded remaining inference phase. |
| [Graphical Model Scale-Up — Scoping](https://github.com/PyAutoLabs/PyAutoInsight/blob/main/tasks/graphical_scoping.md) | graphical-ep | needs-slicing | high | Reconcile current Cortex scale results and library dependencies; select one bounded inference phase with a witness. |
| [NUTS/HMC trial on validated likelihoods](https://github.com/PyAutoLabs/PyAutoInsight/blob/main/tasks/gradient_sampler_validated_likelihoods.md) | point-source-admission | needs-decision | normal | An unscoped idea, not permission to submit: reconcile point-source admission with imaging and weak-lensing candidate targets. |
| [gaussian_x3 search benchmark — wave 1 (the pilot)](https://github.com/PyAutoLabs/PyAutoInsight/blob/main/tasks/gaussian_x3_search_wave1.md) | gaussian-x3-search-benchmark | blocked | normal | Pilot partial (223 of 520 runs); run the deferred cells on RAL partition ral after A2, then freeze gaussian_x3@2. |

[Ledger](https://github.com/PyAutoLabs/PyAutoInsight/blob/main/campaigns.yaml)

Expand Down Expand Up @@ -12126,6 +12128,27 @@ Archived rows belong to this project's September 2026 runs and are historical, n
</details>


<!-- insight:instance name=fit receipt=ec35be493712d693b8cab4d071d97d9a242137b6 outcome=ok shown=ec35be493712d693b8cab4d071d97d9a242137b6 -->
## autofit_inference
Integrity: ok. Freshness: freshness policy unspecified · evidence time unknown. Qualification: no measurements.
Capture source branch: main; revision: ec35be493712d693b8cab4d071d97d9a242137b6; fetched 2026-10-08T19:29:28Z; latest attempt 2026-10-08T19:29:28Z.

<details><summary>Full captured evidence and provenance</summary>


Coverage: {"excluded": [], "expected": {"reason": "No expected-run manifest yet; wave 1 (B3) defines one", "runs": null}, "observed": {"records": 0, "runs": 0}}
Verdicts are producer-asserted under protocol gaussian_x3@1, whose PLACEHOLDER thresholds are calibrated in the wave-1 pilot and then frozen
Rows are judged only against a reference with the same dataset, backend, data seed and assertion mechanism; log evidences are compared only within one backend and one assertion mechanism
Nested-search convergence needs the search's own termination condition: Nautilus is read from its sampler; other nested searches are not_assessed until PyAutoFit exposes it (phase A3)
Completion is execution, not convergence; convergence is reported separately from acceptance
Sample archives remain in project storage; current availability unknown

| Target | Stage | Seed | Execution | Convergence | Acceptance | Sampling s | Total s | Archive | Evidence |
|---|---|---|---|---|---|---:|---:|---|---|

</details>


<!-- decision-history:4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 -->
## Decision History

Expand Down
17 changes: 17 additions & 0 deletions receipts/fit.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{
"instance": "fit",
"repo": "autofit_inference",
"github": "PyAutoLabs/autofit_inference",
"branch": "main",
"resolved_commit": "ec35be493712d693b8cab4d071d97d9a242137b6",
"summary_path": "dashboard/summary.json",
"source": "remote",
"fetched_at": "2026-10-08T19:29:28Z",
"outcome": "ok",
"schema": "inference-summary",
"version": 1,
"record_count": 0,
"errors": [],
"cached_from": null,
"refreshed_at": "2026-10-08T19:30:00Z"
}
2 changes: 1 addition & 1 deletion receipts/lens.json
Original file line number Diff line number Diff line change
Expand Up @@ -13,5 +13,5 @@
"record_count": 56,
"errors": [],
"cached_from": null,
"refreshed_at": "2026-10-08T11:16:58Z"
"refreshed_at": "2026-10-08T19:29:59Z"
}
7 changes: 7 additions & 0 deletions registry.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,10 @@ instances:
dashboard_url: https://github.com/PyAutoLabs/autolens_inference#readme
library_refs: [PyAutoLens, PyAutoGalaxy, PyAutoArray, PyAutoFit]
cortex_project: autolens_inference
- instance: fit
repo: autofit_inference
summary_path: dashboard/summary.json
supported_schema: inference-summary@1
dashboard_url: https://github.com/PyAutoLabs/autofit_inference#readme
library_refs: [PyAutoFit]
cortex_project: autofit_inference
58 changes: 58 additions & 0 deletions snapshots/fit.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
{
"instance": "fit",
"repo": "autofit_inference",
"github": "PyAutoLabs/autofit_inference",
"summary_path": "dashboard/summary.json",
"commit": "ec35be493712d693b8cab4d071d97d9a242137b6",
"fetched_at": "2026-10-08T19:29:28Z",
"summary": {
"comparison_policy": {
"id": "gaussian_x3-task-groups-v1",
"note": "Grouped by task (point/MAP, posterior, evidence); wave 1 is a pilot that ranks nothing; never best-seed",
"ranking": "none"
},
"comparisons": [],
"coverage": {
"excluded": [],
"expected": {
"reason": "No expected-run manifest yet; wave 1 (B3) defines one",
"runs": null
},
"observed": {
"records": 0,
"runs": 0
}
},
"evidence_updated_at": null,
"evidence_updated_at_reason": "No result rows yet (wave 1 is phase B3)",
"generated_at": "2026-10-08T12:57:54Z",
"generation_basis": "Source commit timestamp; not a measurement or check-in time",
"limitations": [
"Verdicts are producer-asserted under protocol gaussian_x3@1, whose PLACEHOLDER thresholds are calibrated in the wave-1 pilot and then frozen",
"Rows are judged only against a reference with the same dataset, backend, data seed and assertion mechanism; log evidences are compared only within one backend and one assertion mechanism",
"Nested-search convergence needs the search's own termination condition: Nautilus is read from its sampler; other nested searches are not_assessed until PyAutoFit exposes it (phase A3)",
"Completion is execution, not convergence; convergence is reported separately from acceptance",
"Sample archives remain in project storage; current availability unknown"
],
"producer_revision": "91bfcead296c7b3c558db72923b6ab3cf0288931",
"project": "autofit_inference",
"protocol_id": "gaussian_x3@1",
"records": [],
"references": {
"gaussian_x3_blend/jax_cpu": {
"assertion_mechanism": null,
"data_seed": null,
"status": "pending"
},
"gaussian_x3_blend/numpy": {
"assertion_mechanism": "raise_resample",
"data_seed": 1,
"status": "complete"
}
},
"schema": "inference-summary",
"scope": "inference",
"valid_until": null,
"version": 1
}
}
11 changes: 9 additions & 2 deletions state.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@
"organ": "insight",
"repo": "PyAutoInsight",
"status": "yellow",
"headline": "1 projects \u00b7 56 records \u00b7 0 cached",
"updated": "2026-10-04T08:09:23Z",
"headline": "2 projects \u00b7 56 records \u00b7 0 cached",
"updated": "2026-10-08T19:29:29Z",
"pages_url": "https://pyautolabs.github.io/PyAutoInsight/",
"items": [
{
Expand All @@ -13,6 +13,13 @@
"text": "lens: ok \u00b7 56 records \u00b7 0/56 accepted by producer; remaining assessments shown below \u00b7 freshness policy unspecified \u00b7 evidence time unknown",
"url": "https://pyautolabs.github.io/PyAutoInsight/#lens",
"prompt": null
},
{
"id": "insight:fit",
"severity": "yellow",
"text": "fit: ok \u00b7 0 records \u00b7 no measurements \u00b7 freshness policy unspecified \u00b7 evidence time unknown",
"url": "https://pyautolabs.github.io/PyAutoInsight/#fit",
"prompt": null
}
]
}
73 changes: 73 additions & 0 deletions tasks/gaussian_x3_search_wave1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
# gaussian_x3 search benchmark — wave 1 (the pilot)

Type: research
Target: autofit_inference
Campaign: gaussian-x3-search-benchmark
Epic: search-extensibility (phase B3)
Issue: https://github.com/PyAutoLabs/autofit_inference/issues/4
Filed: 2026-10-08

## Intent

Wave 1 is the **exploratory pilot** of the `gaussian_x3` search benchmark
(autofit_inference, protocol `gaussian_x3@1`,
[`wiki/project/protocol_gaussian_x3.md`](https://github.com/PyAutoLabs/autofit_inference/blob/main/wiki/project/protocol_gaussian_x3.md)).
It **ranks nothing**. It exists to:

1. run every registered PyAutoFit search once per seed on the toy problem, so the
catalogue (`catalogue/search_catalogue.json`) can give every search a status —
measured, unsupported, deferred or failed — grouped by the task it answers
(point/MAP, posterior, evidence), never compared across tasks;
2. calibrate the protocol's PLACEHOLDER thresholds (σ-ratio band, ppc tolerance, mode
radius, ESS and R-hat bars), the per-search timeouts and the censoring rule on the
reference runs' seed scatter and the pilot (decision D16 iii), and freeze them as
`gaussian_x3@2` before any wave-2 row.

## Menu (protocol §1, §9)

10 search seeds × both datasets (`gaussian_x3_blend`, the `gaussian_x3_separated`
disjoint-prior control) × `local_numpy_fp64`, and `local_jax_cpu_fp64` where the search
is JAX-native, on PyAutoFit `main` (post-A1), every row flagged `pilot` and stamped with
its PyAutoFit commit:

- point/MAP: LBFGS, BFGS, MultiStartAdam, MultiStartProdigy, MultiStartADABelief,
MultiStartLion;
- posterior: Emcee, Zeus (numpy and JAX legs), BlackJAXNUTS cold and warm from a short
Nautilus, SMC;
- evidence: Nautilus (`n_live` 100/200/400), DynestyStatic, DynestyDynamic; NSS is
`deferred` until phase A3b (NSS onto Fitness);
- Drawer is a sanity floor and is not benchmarked (`unsupported` in the catalogue).

The expected-run manifest is `scripts/misc/searches/_pilot.py`; every expected run with
no row is published as deferred with its reason (`coverage.expected` of the summary), so
a missing run is never silently absent.

## Where things live

- **Execution and rows**: autofit_inference (`results/searches/`, the exporter's
`dashboard/summary.json`, instance `fit` in this registry).
- **Science record**: PyAutoCortex row `autofit_inference` (ledger
`wiki/project/state.md` in the project repo).
- **Verdict commentary**: autofit_inference `wiki/project/state.md` (journal entry
2026-10-08). The planned autofit_assistant campaign page is not made in B3:
autofit_assistant gitignores dated `wiki/project/` entries (they are per-clone memory,
never shipped with the template), so its home needs a human decision.
- **Bounded implementation work**: PyAutoMind (the search-extensibility epic).

## Boundaries

Local CPU only. Wave 2 (scored: 50 seeds × 5 data realisations) runs on RAL
`--partition=ral` only — never `gpu`, `ral,gpu` or `gpu,ral` — on a PyAutoFit revision
frozen after phases A2 and A4 (phase B5). This task authorizes no compute submission.

## Status at the B3 wrap-up (2026-10-08)

The pilot was stopped by the human wrap-up ruling with **223 of 520** expected runs on
disk (autofit_inference#4). It ranks nothing. Every run without a row is published as
`deferred` with its reason: wave 2 on RAL `--partition=ral`, NSS to phase A3b. The
pilot ran on PyAutoFit main `0dbf258c4f5e`, before phase A2 (PyAutoFit#1679, gradients
under JAX); BFGS/LBFGS non-finite results are expected to change once A2 merges and are
not a judgement on those searches. The JAX blend reference is pending (2 of 3 Nautilus
runs), so JAX blend rows are `not_assessed`. The calibration record keeps `@1` (17
calibration rows < 20); the `gaussian_x3@2` freeze is deferred and still precedes any
wave-2 row.
12 changes: 10 additions & 2 deletions tests/test_registry.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,17 @@ def _doc(*rows):

def test_the_committed_registry_is_valid_against_a_body_map(body_map):
data = yaml.safe_load((ORGAN_ROOT / "registry.yaml").read_text())
body_map.update({"PyAutoArray": {"github": "PyAutoLabs/PyAutoArray"}})
body_map.update(
{
"PyAutoArray": {"github": "PyAutoLabs/PyAutoArray"},
"autofit_inference": {
"path": "fit/autofit_inference",
"github": "PyAutoLabs/autofit_inference",
},
}
)
assert registry.validate(data, body_map) == []
assert [r["instance"] for r in data["instances"]] == ["lens"]
assert [r["instance"] for r in data["instances"]] == ["lens", "fit"]


def test_load_resolves_path_and_github_from_the_body_map(registry_file, fake_mind):
Expand Down
Loading