Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
89 changes: 70 additions & 19 deletions campaigns.yaml
Original file line number Diff line number Diff line change
@@ -1,29 +1,57 @@
version: 1
last_checkin: null
last_checkin: '2026-10-17T11:18:53Z'
last_checkin_report: checkins/2026-10-17T111853Z.md
campaigns:
- id: slam-hst-base
title: HST SLaM base backend parity
status: active
updated: '2026-10-04'
next: Reconcile all seeds and stages with current project rows and Cortex before proposing another run.
recent_progress: Project and Cortex ledgers retain the 24 September rerun submissions; published rows require
coverage reconciliation.
blockers: Live RAL status and output/archive availability unverified.
job_status: unknown — live scheduler not queried; historical ledger states are not current job evidence
evidence: https://github.com/PyAutoLabs/PyAutoCortex/blob/d45f198804615c090e6b6694cf725ddb0be2354e/projects/autolens_inference.md
status: superseded
updated: '2026-10-17'
next: 'Superseded 2026-10-17 by hst-imaging-mass-samplers (human: clean slate). The 2026-09-24 re-run rows
stay as history in autolens_inference PR #23; no further run under these campaigns.'
recent_progress: 'The 2026-09-24 slam_base re-runs landed as 6 rows (A100 jax_gpu dense and sparse, numba_cpu
sparse; seeds 0-1) in autolens_inference PR #23, pending merge. A100 walls are clean single-attempt; the numba_cpu
sparse rows carry a resumed source_pix[1] stage.'
blockers: The three CPU legs jax_cpu dense, jax_cpu sparse and numba_cpu dense have not run; numba_cpu sparse
chain walls are multi-attempt.
job_status: 'All 2026-09-24 re-runs COMPLETED 0:0 per sacct queried 2026-10-17T11:10Z: 350674 and 350675 (gpu),
350682 (ral).'
evidence: https://github.com/PyAutoLabs/autolens_inference/pull/23
project: autolens_inference
checkin_report: checkins/2026-10-17T111853Z.md
- id: slam-hst-delaunay
title: HST SLaM Delaunay variant
status: active
updated: '2026-10-04'
next: Retrieve or account for every rerun, failed seed and missing stage; preserve the archived comparison
context.
recent_progress: Delaunay is a separate target/recipe; pre-speedup runs are archived, reruns recorded on
24 September.
blockers: Do not merge the Delaunay target with rectangular comparisons; current job status unknown.
job_status: unknown — live scheduler not queried; historical ledger states are not current job evidence
evidence: https://github.com/PyAutoLabs/autolens_inference/blob/146a84346eaa51eda158634cae520d677f3e6213/wiki/project/state.md
status: superseded
updated: '2026-10-17'
next: 'Superseded 2026-10-17 by hst-imaging-mass-samplers (human: clean slate). The 2026-09-24 re-run rows
stay as history in autolens_inference PR #23; no further run under these campaigns.'
recent_progress: 'The 2026-09-24 delaunay_1250 re-runs landed as 6 rows (A100 jax_gpu dense and sparse, numba_cpu
sparse; seeds 0-1) in autolens_inference PR #23, pending merge. A100 walls are clean single-attempt; the numba_cpu
sparse rows carry a resumed mass_total[1] stage.'
blockers: The seed-0 mass_total[1] offset (5.98, 5.12 and 6.62 sigma on dense, sparse and numba_cpu) reproduces
on three backends; human reading pending in Cortex. Do not merge the Delaunay target with rectangular comparisons.
job_status: 'All 2026-09-24 re-runs COMPLETED 0:0 per sacct queried 2026-10-17T11:10Z: 350678 and 350679 (gpu),
350684 (ral).'
evidence: https://github.com/PyAutoLabs/autolens_inference/pull/23
project: autolens_inference
checkin_report: checkins/2026-10-17T111853Z.md
- id: hst-imaging-mass-samplers
title: 'HST imaging, clean slate: Nautilus base on A100 and jax_cpu at a pinned release, mass stage with linear
lens light and with it fixed, then other samplers on the mass stage'
status: needs-slicing
updated: '2026-10-17'
next: 'Phase 1 via Mind start_dev: lens-light mode as a run variant (--lens-light linear|fixed|both, fixed = ordinary
profiles with intensities from the light[1] fit), both rows from one chain, --export-prepared for both modes,
library revisions and SLURM job id recorded in every row, submit scripts for A100 dense/sparse and ral jax_cpu
dense/sparse against a private pinned checkout; then phase 2 compute by separate authorisation.'
recent_progress: 'Opened 2026-10-17 as a fresh start by human decision: the base is re-measured from scratch on
PyAuto release 2026.10.17.1 (PyAutoLens tag v2026.10.17.1, released 2026-10-17) in fresh output trees; the
2026-09-24 rows and both earlier campaigns are superseded history.'
blockers: Phase 1 runner change not started (Mind start_dev pending); no pinned private checkout exists on RAL
yet; compute not authorised until phase 1 ships.
job_status: none submitted
evidence: https://github.com/PyAutoLabs/autolens_inference/pull/23
project: autolens_inference
checkin_report: checkins/2026-10-17T111853Z.md
- id: point-source-admission
title: Point-source sampler admission
status: needs-decision
Expand Down Expand Up @@ -116,7 +144,7 @@ tasks:
path: tasks/drop_1e3_prior_centring_after_631.md
updated: '2026-10-04'
next: Verify the release containing Galaxy#634 and Lens#754 before changing priors; retain zero-centre gradient
witness.
witness. Carried into hst-imaging-mass-samplers phase 1 as a prior-identity question for the frozen model.
evidence: https://github.com/PyAutoLabs/PyAutoMind/blob/8dd1edc7e04c960daae3663513c61395823eddb0/draft/feature/autolens_inference/drop_1e3_prior_centring_after_631.md
- id: reconcile_cortex_ledger_references
campaign: slam-hst-base
Expand Down Expand Up @@ -199,3 +227,26 @@ tasks:
next: Wave 2 is closed and judged (B5b-2); the fit snapshot is captured from main under inference-summary@3
after the B5b-2 PRs merge.
evidence: https://github.com/PyAutoLabs/autofit_inference/issues/12
- id: hst_mass_base_light_modes
campaign: hst-imaging-mass-samplers
title: 'HST imaging mass stage, clean slate: Nautilus base on A100 and jax_cpu at release 2026.10.17.1, linear
and fixed lens light'
status: needs-slicing
priority: high
path: tasks/hst_mass_base_light_modes.md
updated: '2026-10-17'
next: 'Phase 1 via Mind start_dev: --lens-light linear|fixed|both, both rows from one chain, --export-prepared
for both modes, library revisions and SLURM job id in every row, A100 and ral jax_cpu submit scripts against
a private pinned checkout; no compute until phase 1 ships and phase 2 is authorised.'
evidence: https://github.com/PyAutoLabs/autolens_inference/pull/23
- id: hst_mass_samplers_wave1
campaign: hst-imaging-mass-samplers
title: 'HST imaging mass stage: sampler comparison, wave 1'
status: needs-slicing
priority: normal
path: tasks/hst_mass_samplers_wave1.md
updated: '2026-10-17'
next: Depends on the frozen mass_total[1] problems (both light modes, A100 and jax_cpu) from the pinned base
in hst_mass_base_light_modes; candidate samplers from the experiments runner (nautilus, emcee, nuts, smc) and
the promoted PyAutoFit catalogue (NSS, DynestyStatic/Dynamic, Zeus) are not yet sliced.
evidence: https://github.com/PyAutoLabs/autolens_inference/pull/23
94 changes: 94 additions & 0 deletions checkins/2026-10-17T111853Z.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
# Inference check-in — 2026-10-17T11:18:53Z

Previous: none (`last_checkin` was null; the initial migration was not a check-in)
Scope: all open campaigns; optional user focus "i want to begin our first proper Insight campaign, using PyAutoLens imaging likelihood function, starting with nautilus base for cpu and gpu (a100 on RAL) with the mass phase doing both with linear light profiles and with them removed. Get the base up and then we try other samplers on mass"
User decisions: the direction quoted above, given in the Insight chat on 2026-10-17. It opens a campaign; it authorises no compute. Later in the same chat on 2026-10-17 the human decided: "Let's do this as a fresh start we did kind of start a campaign but weve done lots of improvements since warranting a clean slate." Then, answering four explicit questions: (1) clean slate: "Yes, supersede and re-run on the pin"; (2) CPU backend: "jax_cpu, dense + sparse"; (3) light mode: "Fix lens light to ordinary profiles"; (4) test fix: "Yes, relax the two assertions". These decisions authorise no compute.

## Sources inspected

| Source | Commit / query time | Evidence | Availability |
|---|---|---|---|
| autolens_inference main | `df5fd2c` | https://github.com/PyAutoLabs/autolens_inference/blob/df5fd2c03cce/wiki/project/state.md | verified |
| autolens_inference row landing PR | head `fc561a9` | https://github.com/PyAutoLabs/autolens_inference/pull/23 | verified, open, not merged |
| RAL scheduler (`sacct -X`) for 350674, 350675, 350678, 350679, 350682, 350684 | 2026-10-17T11:10Z | queried by the session architect over SSH | verified |
| RAL result rows | 2026-10-17 | `/mnt/ral/jnightin/inference/autolens_inference/results/slam/imaging/hst/` (12 JSON + 12 PNG) | verified; 8 pulled by `hpc/sync` earlier, 4 `numba_cpu` copied 2026-10-17 |
| Pulled SLURM logs | 2026-10-17 | autolens_inference checkout `logs/{gpu,cpu}/output/output.<job>_<seed>.out`, each ending `Finished.` with the row path | verified (local, untracked) |
| PyAutoCortex `projects/autolens_inference.md` | origin/main `9e8b3d4` before this sweep; `ab738e4` (merged as `a61e18a`) after | https://github.com/PyAutoLabs/PyAutoCortex/blob/a61e18a/projects/autolens_inference.md | verified; six runs closed with `cortex.py done`, Now rewritten |
| PyAutoInsight | `4e9a34d` | this repository | verified |
| PyAutoMind | origin/main `11ec4317` | no `draft/*/autolens_inference` prompt and no autolens_inference claim found | verified |
| autofit_inference (gaussian-x3-search-benchmark) | not re-inspected this sweep | ledger state retained from 2026-10-16 | not re-inspected |
| graphical_ep (graphical-ep) | not re-inspected this sweep | ledger state retained from 2026-10-04 | not re-inspected |
| analytic_gaussian (analytic-gaussian) | not re-inspected this sweep | ledger state retained from 2026-10-04 | not re-inspected |
| ep_toy_gaussian (ep-toy-gaussian) | not re-inspected this sweep | ledger state retained from 2026-10-04 | not re-inspected |
| slope_hierarchy_scale (slope-hierarchy-scale) | not re-inspected this sweep | ledger state retained from 2026-10-04 | not re-inspected |
| ic50_workspace (ic50-scale) | not re-inspected this sweep | ledger state retained from 2026-10-04 | not re-inspected |

## Campaign sweep

| Campaign | Change since previous | Coordination state | Job evidence | Blocker | Next bounded step |
|---|---|---|---|---|---|
| slam-hst-base | All four `slam_base` re-run arrays completed; 6 rows landed in PR #23 | active | 350674, 350675 (gpu) and 350682 (ral) COMPLETED 0:0, sacct 2026-10-17T11:10Z | Three CPU legs (`jax_cpu` dense and sparse, `numba_cpu` dense) have not run; `numba_cpu` sparse walls are multi-attempt | Human reads the parity view, then decides whether the remaining CPU legs are still wanted |
| slam-hst-delaunay | Both `delaunay_1250` A100 arrays and the `numba_cpu` sparse array completed; 6 rows landed in PR #23 | active | 350678, 350679 (gpu) and 350684 (ral) COMPLETED 0:0, sacct 2026-10-17T11:10Z | Seed-0 `mass_total[1]` offset reproduces on three backends; human reading pending in Cortex | Human reads the seed-0 offset |
| hst-imaging-mass-samplers | New, opened from the user direction above | needs-decision | none submitted | Three human choices (below) and a bounded runner change | Decide the choices, then `start_dev` the runner change |
| point-source-admission | No change; project state.md 2026-10-02 entry is still the latest record | needs-decision | none new | Gradient cost share and sampler admission unmeasured | Unchanged |
| graphical-ep | Not re-inspected | needs-slicing | not queried | Unchanged | Unchanged |
| analytic-gaussian | Not re-inspected | blocked | not queried | Unchanged | Unchanged |
| ep-toy-gaussian | Not re-inspected | needs-decision | not queried | Unchanged | Unchanged |
| slope-hierarchy-scale | Not re-inspected | blocked | not queried | Unchanged | Unchanged |
| ic50-scale | Not re-inspected | blocked | not queried | Unchanged | Unchanged |
| gaussian-x3-search-benchmark | Not re-inspected | running | not queried; ledger records 409930 COMPLETED | Unchanged | Unchanged (B5c-2b) |

Row summary for the two SLaM campaigns (search wall in seconds, chain = sum of the five stage walls; `mass_total[1]` log Z and largest absolute `truth_delta_sigma`), seed 0 / seed 1:

| variant / config | chain | mass_total[1] wall | mass log Z | max abs Δσ | resumed stage |
|---|---|---|---|---|---|
| slam_base / A100 dense | 2,468 / 2,365 | 562 / 521 | 31,589.22 / 31,590.06 | 1.52 / 0.55 | none |
| slam_base / A100 sparse | 3,930 / 3,915 | 837 / 825 | 31,597.52 / 31,594.05 | 1.98 / 1.20 | none |
| slam_base / numba_cpu sparse | 168,824 / 177,740 | 44,221 / 50,086 | 31,592.05 / 31,595.97 | 1.30 / 3.18 | source_pix[1] |
| delaunay_1250 / A100 dense | 4,390 / 5,210 | 1,217 / 1,538 | 31,499.14 / 31,528.44 | 5.98 / 2.18 | none |
| delaunay_1250 / A100 sparse | 7,454 / 8,013 | 1,492 / 1,699 | 31,531.15 / 31,538.40 | 5.12 / 3.57 | none |
| delaunay_1250 / numba_cpu sparse | 772,484 / 757,842 | 345,641 / 360,538 | 31,515.84 / 31,544.76 | 6.62 / 3.92 | mass_total[1] |

## Coverage and limits

- Both `numba_cpu` arrays were requeued after their first 2026-09-24 start (the 09-24 project journal saw them RUNNING at +1 min; sacct now reports Start 2026-09-25T09:45 for 350682 and 2026-10-03T09:41 for 350684, Reason BeginTime). Each row carries one `resumed` stage, so the CPU chain walls and the resumed stage walls span more than one attempt and are not single-attempt measurements. The clean CPU stage walls are the non-resumed ones, notably `slam_base` `mass_total[1]` at 44,221 / 50,086 s.
- The A100 chain walls are single-attempt.
- Result schema v2 records no job id or library revisions. The job ids come from the pulled logs and the Cortex ledger; the library revisions (Array `3de624b5`, Lens `86054bbc1`, Fit `dd9fbe0aa`, Galaxy `70a61e26`, Nerves `1fa613a`) come from the 2026-09-24 project journal.
- One simulated target, two seeds per config. All rows report `status: complete`; that is execution completion, not convergence assessment or scientific acceptance.
- The published `inference-summary` snapshot for lens was not refreshed by this sweep; it refreshes after PR #23 merges.
- Six campaigns outside autolens_inference were not re-inspected; their ledger rows are unchanged.

Scientific observations and any human conclusion belong in the Cortex ledger for autolens_inference; none was written this sweep.

## Decisions and handoff

Proposed interpretations, for the human (not rulings):

- (a) The inference-level speedup from the likelihood work is modest on the A100: 0.98 to 1.22x on the chain across the eight legs, compared with the archived pre-speedup rows.
- (b) The `delaunay_1250` seed-0 `mass_total[1]` offset (5.98σ dense, 5.12σ sparse, 6.62σ CPU, against 2.18σ / 3.57σ / 3.92σ at seed 1) reproduces on three backends, which would answer the 2026-09-24 "confirm or retire" question as confirmed, as a property of that target and seed rather than of a backend.
- (c) `slam_base` `mass_total[1]` log Z spans about 8 nats across the six legs (31,589.2 to 31,597.5) against 1 to 4 nats seed to seed.

Proposed for `hst-imaging-mass-samplers` (decision-gated; see `tasks/hst_mass_base_light_modes.md` and `tasks/hst_mass_samplers_wave1.md`):

- CPU backend for the base: `jax_cpu` is recommended because the NUTS/SMC experiment path needs a JAX-frozen analysis; `numba_cpu` would instead complete the parity row.
- "Linear light removed": fixed ordinary light profiles with intensities from `light[1]` (the plan recorded in Cortex), or lens-light subtraction from the image.
- Seed count.

Human decisions recorded after the sweep, in the Insight chat on 2026-10-17:

- "Let's do this as a fresh start we did kind of start a campaign but weve done lots of improvements since warranting a clean slate."
- (1) Clean slate: "Yes, supersede and re-run on the pin".
- (2) CPU backend: "jax_cpu, dense + sparse".
- (3) Light mode: "Fix lens light to ordinary profiles".
- (4) Test fix: "Yes, relax the two assertions".

Consequences recorded in the ledger:

- `slam-hst-base` and `slam-hst-delaunay` are `superseded` by `hst-imaging-mass-samplers`. Their 2026-09-24 re-run rows stay as history in autolens_inference PR #23; no further run is planned under those campaigns. The proposed interpretations (a) to (c) above stay proposals and are not carried into the new campaign.
- `hst-imaging-mass-samplers` is reframed as a clean slate at status `needs-slicing`: the base is re-measured from scratch on PyAuto release 2026.10.17.1 (PyAutoLens tag v2026.10.17.1) in fresh output trees, on A100 `jax_gpu` dense and sparse and ral `jax_cpu` dense and sparse, seeds 0 and 1, with `mass_total[1]` run in two light modes (linear lens light, and lens light fixed to ordinary profiles with intensities from the `light[1]` fit). `numba_cpu` is not in this campaign. Phase 1 is a bounded runner change through Mind `start_dev`, not yet started; phase 2 compute needs separate authorisation.
- The task `drop_1e3_prior_centring_after_631` keeps its campaign and status and is carried into the new campaign's phase 1 as a prior-identity question for the frozen model.
- `tests/test_campaigns.py::test_current_ledger_and_migration_validate` had two migration-era assertions (`last_checkin` null, every `job_status` starting with "unknown") that this first real check-in breaks; per decision (4) they are relaxed to accept a check-in stamp with an existing report and any non-empty `job_status`.

Authorised actions this sweep: landing the 12 rows (autolens_inference PR #23, open, not merged), closing the six runs in the Cortex ledger (`ab738e4`, merged by ledger_merge as `a61e18a`), and this report with its campaigns.yaml update. No Mind implementation issue was opened.

Compute submitted: none. No decision above authorises compute.
2 changes: 1 addition & 1 deletion dashboard.html

Large diffs are not rendered by default.

Loading
Loading