Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 10 additions & 9 deletions campaigns.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -93,10 +93,11 @@ campaigns:
title: PyAutoFit search benchmark on gaussian_x3 (autofit_inference)
status: running
updated: '2026-10-16'
next: 'B5c-2: RAL 410866 (the correlated_nd pilot, 930 runs, from the private copy inference/autofit_inference_b5c)
finished 2026-10-16 with 926/930 completed and 4 OUT_OF_MEMORY at 8 G; pull results/ only, settle the 4 OOM
cells (more memory or excluded), judge under correlated_nd@1, freeze correlated_nd@2 from the pilot and submit its scored wave; the family is
catalogued apart from gaussian_x3 and ranks nothing across families.'
next: 'B5c-2b (draft/research/autofit_inference/search_ext_b5c2b_scored_wave_submit.md): realisations 2-3 and
their exact references, the correlated_nd@2 scored-wave manifest (83 cells x 50 seeds x 3 realisations,
12 450 runs) and its probe for 8 probe-first cells, then the submit on RAL --partition=ral from a private
pinned copy with the A2 memory requests; correlated_nd@2 is frozen (autofit_inference 44c5e77f1fb0); the
family is catalogued apart from gaussian_x3 and ranks nothing across families.'
recent_progress: Wave 2 closed (B5b-2, 2026-10-15, autofit_inference#12) — RAL 409930 pulled, 12,750 rows
admitted at the pin df6d218f10cd with 0 missing; NSS 50-step leads both evidence rankings; the catalogue
ranks only legs with success rate >= 50 % over >= 20 attempts (post-hoc presentation convention); fit
Expand Down Expand Up @@ -180,14 +181,14 @@ tasks:
- id: correlated_nd_pilot
campaign: gaussian-x3-search-benchmark
title: correlated_nd search benchmark — the pilot (wave 1, second model family)
status: running
status: complete
priority: normal
path: tasks/correlated_nd_pilot.md
updated: '2026-10-16'
next: RAL 410866, the 930-run correlated_nd@1 pilot (protocol first committed 9ec37795f404, A1 before any
row), finished 2026-10-16 (926/930 completed, 4 OUT_OF_MEMORY at 8 G, results unpulled); B5c-2 pulls it,
settles the 4 OOM cells, judges it and freezes correlated_nd@2.
evidence: https://github.com/PyAutoLabs/autofit_inference/issues/14
next: Done (B5c-2a, autofit_inference#17) — pilot judged under correlated_nd@1 (1040 expected = 749 rows +
177 failed + 4 oom + 0 missing + 110 deferred; 307 accepted); correlated_nd@2 frozen in 44c5e77f1fb0 from
results/calibration/correlated_nd_wave1.json; the scored wave is B5c-2b.
evidence: https://github.com/PyAutoLabs/autofit_inference/issues/17
- id: gaussian_x3_search_wave2
campaign: gaussian-x3-search-benchmark
title: gaussian_x3 search benchmark — wave 2 (scored)
Expand Down
2 changes: 1 addition & 1 deletion dashboard.html

Large diffs are not rendered by default.

7 changes: 3 additions & 4 deletions dashboard.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# PyAutoInsight — inference control room

<!-- insight-inputs:004fbffeb895e547f2c938b514aa10c7f79e8b20dc7b32f18ab17eb3e64d8c88 -->
<!-- insight-inputs:25cabeb3e20fa27f0eceeaf73870e49f1bc78c35b6ed5f15573959f0075a6bec -->

| Where | Count |
|---|---:|
Expand All @@ -10,7 +10,7 @@
| [Cached](#evidence) | 0 |
| [Failed](#evidence) | 0 |

<!-- insight-campaigns:c09fff24924c3c8bc5e2bfb0e9e37109d0202153cc34c91f3b9ee6281893322a -->
<!-- insight-campaigns:be44ab685722080abcb342d8094de6ec3457b9c959757f7ba2962ef1304dc2ee -->
## Check in on all inference work

Copy this into one chat. Add a campaign focus or idea before or after it, or leave it unchanged.
Expand Down Expand Up @@ -46,7 +46,7 @@ After taking action, report what changed, what the evidence supports and what re
| [N=50 NUTS versus three EP repeats](https://github.com/PyAutoLabs/PyAutoCortex/blob/d45f198804615c090e6b6694cf725ddb0be2354e/projects/ep_toy_gaussian.md) | Cortex records failed wave 1 (EMFILE/OOM), wave-2 preparation and no submission. | Present the existing wave-2 proposal with current dependency evidence for decision. |
| [Hierarchical slope recovery at scale](https://github.com/PyAutoLabs/PyAutoCortex/blob/d45f198804615c090e6b6694cf725ddb0be2354e/projects/slope_hierarchy_scale.md) | Cortex retains graphical baseline, post-processing OOM and stale EP hierarchical factor; Mind records later moments fix. | Verify the moments fix release and project mirror; recover missing summaries from existing samples before a proposed rerun. |
| [IC50 EP and graphical scale ladder](https://github.com/PyAutoLabs/PyAutoCortex/blob/d45f198804615c090e6b6694cf725ddb0be2354e/projects/ic50_workspace.md) | Cortex records N=5/10/25 results, failed N=50 EP rung and width-interpretation caveat. | Check implementation close-outs, then propose a seeded ladder with appropriate output and mean-field diagnostics. |
| [PyAutoFit search benchmark on gaussian_x3 (autofit_inference)](https://github.com/PyAutoLabs/autofit_inference/issues/12) | Wave 2 closed (B5b-2, 2026-10-15, autofit_inference#12) — RAL 409930 pulled, 12,750 rows admitted at the pin df6d218f10cd with 0 missing; NSS 50-step leads both evidence rankings; the catalogue ranks only legs with success rate &gt;= 50 % over &gt;= 20 attempts (post-hoc presentation convention); fit records published as inference-summary@3 (101 record shards). | B5c-2: RAL 410866 (the correlated_nd pilot, 930 runs, from the private copy inference/autofit_inference_b5c) finished 2026-10-16 with 926/930 completed and 4 OUT_OF_MEMORY at 8 G; pull results/ only, settle the 4 OOM cells (more memory or excluded), judge under correlated_nd@1, freeze correlated_nd@2 from the pilot and submit its scored wave; the family is catalogued apart from gaussian_x3 and ranks nothing across families. |
| [PyAutoFit search benchmark on gaussian_x3 (autofit_inference)](https://github.com/PyAutoLabs/autofit_inference/issues/12) | Wave 2 closed (B5b-2, 2026-10-15, autofit_inference#12) — RAL 409930 pulled, 12,750 rows admitted at the pin df6d218f10cd with 0 missing; NSS 50-step leads both evidence rankings; the catalogue ranks only legs with success rate &gt;= 50 % over &gt;= 20 attempts (post-hoc presentation convention); fit records published as inference-summary@3 (101 record shards). | B5c-2b (draft/research/autofit_inference/search_ext_b5c2b_scored_wave_submit.md): realisations 2-3 and their exact references, the correlated_nd@2 scored-wave manifest (83 cells x 50 seeds x 3 realisations, 12 450 runs) and its probe for 8 probe-first cells, then the submit on RAL --partition=ral from a private pinned copy with the A2 memory requests; correlated_nd@2 is frozen (autofit_inference 44c5e77f1fb0); the family is catalogued apart from gaussian_x3 and ranks nothing across families. |

## Active tasks

Expand All @@ -58,7 +58,6 @@ After taking action, report what changed, what the evidence supports and what re
| [Graphical Model Scale-Up — Scoping](https://github.com/PyAutoLabs/PyAutoInsight/blob/main/tasks/graphical_scoping.md) | graphical-ep | needs-slicing | high | Reconcile current Cortex scale results and library dependencies; select one bounded inference phase with a witness. |
| [NUTS/HMC trial on validated likelihoods](https://github.com/PyAutoLabs/PyAutoInsight/blob/main/tasks/gradient_sampler_validated_likelihoods.md) | point-source-admission | needs-decision | normal | An unscoped idea, not permission to submit: reconcile point-source admission with imaging and weak-lensing candidate targets. |
| [gaussian_x3 search benchmark — wave 1 (the pilot)](https://github.com/PyAutoLabs/PyAutoInsight/blob/main/tasks/gaussian_x3_search_wave1.md) | gaussian-x3-search-benchmark | blocked | normal | Pilot stopped at 223 of 520 runs and is not restarted; freeze gaussian_x3@2 from its 17-row calibration before any scored run; the deferred cells move to scored wave 2 (after A2 and A4). |
| [correlated_nd search benchmark — the pilot (wave 1, second model family)](https://github.com/PyAutoLabs/PyAutoInsight/blob/main/tasks/correlated_nd_pilot.md) | gaussian-x3-search-benchmark | running | normal | RAL 410866, the 930-run correlated_nd@1 pilot (protocol first committed 9ec37795f404, A1 before any row), finished 2026-10-16 (926/930 completed, 4 OUT_OF_MEMORY at 8 G, results unpulled); B5c-2 pulls it, settles the 4 OOM cells, judges it and freezes correlated_nd@2. |
| [gaussian_x3 search benchmark — wave 2 (scored)](https://github.com/PyAutoLabs/PyAutoInsight/blob/main/tasks/gaussian_x3_search_wave2.md) | gaussian-x3-search-benchmark | running | high | Wave 2 is closed and judged (B5b-2); the fit snapshot is captured from main under inference-summary@3 after the B5b-2 PRs merge. |

[Ledger](https://github.com/PyAutoLabs/PyAutoInsight/blob/main/campaigns.yaml)
Expand Down
49 changes: 39 additions & 10 deletions tasks/correlated_nd_pilot.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,13 +66,42 @@ Every dataset × backend has an exact reference, complete in `results/reference/
- The first submission, 409957, failed at import in every task: the shared RAL clone had
been switched by another session. It was cancelled and wrote no row.

This task authorises nothing further. Pulling, judging and the `correlated_nd@2` freeze
are B5c-2.

## Next (B5c-2)

1. Pull the pilot's rows (`results/` only) and judge them under `correlated_nd@1`; settle
the 4 OOM cells (more memory or excluded).
2. Calibrate the PLACEHOLDERs and timeouts, then freeze `correlated_nd@2` before any
scored row.
3. Submit the scored wave: 50 seeds × realisations 1–3 × every non-deferred cell.
## Judged and closed (2026-10-16, B5c-2a, autofit_inference#17)

**Status: complete.** `results/` of 410866 was pulled from the private copy (930 rows; the
8 exact references there are byte-identical to the committed ones) and judged under
`correlated_nd@1` (`results/accounting/correlated_nd_pilot.json`, reconciled with the
committed `sacct` extract):

- **1040 expected = 749 completed rows + 177 failed + 4 `oom` + 0 missing + 110 deferred**;
930 admitted, 0 refused.
- Under `@1`: **307 accepted, 623 rejected**. The pilot ranks nothing.
- `oom` is a new attempt class. The submit had written cap stubs for the 4 OUT_OF_MEMORY
tasks, because SLURM's OOM SIGKILL exits 137 under `timeout`. They were rewritten from
sacct, with MaxRSS 7.6 / 7.7 / 3.1 / 8.4 G at the 8 G request.
- The failures:
- 80 Emcee/Zeus rows (d20, banana) crashed in the harness's chain diagnostics after the
search. That bug is fixed.
- 60 d100 Nautilus rows, which fail at the pin.
- 10 d100 warm NUTS rows.
- 27 rows stopped at the 4 h cap (d50 Nautilus `n_live_100`, d50 warm NUTS).

**Calibration and freeze.** The calibration is
[`results/calibration/correlated_nd_wave1.json`](https://github.com/PyAutoLabs/autofit_inference/blob/main/results/calibration/correlated_nd_wave1.json).
`correlated_nd@2` (protocol §11 A2) was frozen in autofit_inference `44c5e77f1fb0` before any
scored row:
- σ-ratio bands are per task. The evidence band is [0.85, 1.1] (79 rows). The posterior
band is kept at [0.5, 2.0] (1 row, < 20).
- The ESS and R-hat bars are kept.
- 75 timed cells and 8 probe-first cells.
- 21 cells are deferred, each with its reason. They include the OOM cells d50 Nautilus
`n_live_100` and d50 warm NUTS (by the cap rule) and every d100 Nautilus cell.
- Memory is `max(8 G, ceil(2 × the pilot peak))`: 16 G for d50 Nautilus `n_live_400`,
17 G for d100 cold NUTS.

## Next

B5c-2b (`draft/research/autofit_inference/search_ext_b5c2b_scored_wave_submit.md`):
realisations 2–3 and their exact references, then the scored-wave manifest (83 cells × 50
seeds × 3 realisations = 12 450 runs) and its probe, then the submit. Nothing is submitted
by this task.
Loading