diff --git a/campaigns.yaml b/campaigns.yaml
index ec0606c..a5a1d61 100644
--- a/campaigns.yaml
+++ b/campaigns.yaml
@@ -1,6 +1,6 @@
version: 1
-last_checkin: '2026-10-17T11:18:53Z'
-last_checkin_report: checkins/2026-10-17T111853Z.md
+last_checkin: '2026-10-18T17:00:00Z'
+last_checkin_report: checkins/2026-10-18T170000Z.md
campaigns:
- id: slam-hst-base
title: HST SLaM base backend parity
@@ -35,23 +35,17 @@ campaigns:
project: autolens_inference
checkin_report: checkins/2026-10-17T111853Z.md
- id: hst-imaging-mass-samplers
- title: 'HST imaging, clean slate: Nautilus base on A100 and jax_cpu at a pinned release, mass stage with linear
+ title: 'HST imaging, clean slate: Nautilus base on A100, jax_cpu and numba_cpu at a pinned release, mass stage with linear
lens light and with it fixed, then other samplers on the mass stage'
- status: needs-slicing
- updated: '2026-10-17'
- next: 'Phase 1 via Mind start_dev: lens-light mode as a run variant (--lens-light linear|fixed|both, fixed = ordinary
- profiles with intensities from the light[1] fit), both rows from one chain, --export-prepared for both modes,
- library revisions and SLURM job id recorded in every row, submit scripts for A100 dense/sparse and ral jax_cpu
- dense/sparse against a private pinned checkout; then phase 2 compute by separate authorisation.'
- recent_progress: 'Opened 2026-10-17 as a fresh start by human decision: the base is re-measured from scratch on
- PyAuto release 2026.10.17.1 (PyAutoLens tag v2026.10.17.1, released 2026-10-17) in fresh output trees; the
- 2026-09-24 rows and both earlier campaigns are superseded history.'
- blockers: Phase 1 runner change not started (Mind start_dev pending); no pinned private checkout exists on RAL
- yet; compute not authorised until phase 1 ships.
- job_status: none submitted
- evidence: https://github.com/PyAutoLabs/autolens_inference/pull/23
+ status: running
+ updated: '2026-10-18'
+ next: '(1) Human decision on filing a Mind bug task for the jax_gpu sparse fixed-light mass_total[1] divergence (both seeds) and reproducing it on the laptop from the exported prepared problem, before any CPU or sampler wave uses the sparse fixed-light problem. (2) Human authorisation for GPU wave 1 attempt 2 (nautilus+emcee, --array=0-11%4) after git pull of autolens_inference on the RAL login node to ca505be. (3) numba_cpu dense DROPPED by human decision 2026-10-18 (decisions/hst-cleanslate-drop-numba-dense.md): no rerun. (4) Wait for 412539, 412541 and 412565 to finish (jax_cpu sparse source_pix[1] near its end), then pull and read. (5) Then start_dev the filed Mind prompt draft/refactor/autolens_inference/flatten_cleanslate_output_layout.md (Mind 734e1bf8; human 2026-10-18: flat output/autolens/imaging/... layout, no dated folder; blocked until those arrays finish).'
+ recent_progress: 'Read 2026-10-18 (checkins/2026-10-18T170000Z.md). A100 base complete: 412535 (dense) and 412537 (sparse) rows for both light modes and seeds read. Dense fixed-light mass_total[1] is the best-recovered row (worst abs delta-sigma 0.24 / 0.34). Sparse fixed-light mass_total[1] diverged on both seeds (log Z -43567 / -43202, max logL ~-43500 vs ~+31649 linear; slope pinned at 2.96); observation recorded in Cortex, human reading pending. jax_cpu dense seed 1 linear row written (chain ~21 h vs ~40 min on A100; worst 1.29 sigma). numba_cpu dense 412563 FAILED at the first source_pix[1] batch (two Nautilus pool workers SIGSEGV, MaxRSS 356 MB, not OOM). Wave-1 fix merged as autolens_inference#34 (ca505be) 2026-10-17T19:25Z; attempt 2 not submitted.'
+ blockers: 'Sparse fixed-light mass-stage divergence unexplained (needs reproduction); wave 1 attempt 2 needs RAL checkout pull (b91ce9c -> ca505be) and human authorisation; numba_cpu dense dropped 2026-10-18 (human decision), no longer a blocker.'
+ job_status: 'Read 2026-10-18 ~16:55Z (sacct/squeue by the architect), pin 2026.10.17.1. 412535 (jax_gpu dense) and 412537 (jax_gpu sparse) COMPLETED 0:0 both seeds. 412539 (jax_cpu dense) RUNNING 23 h: seed 1 linear row written, seed 1 on fixed mass_total[1], seed 0 in linear mass_total[1]. 412541 (jax_cpu sparse) RUNNING 23 h, both seeds in source_pix[1], no row. 412563 (numba_cpu dense) FAILED 1:0 both seeds after 00:00:57 (pool-worker SIGSEGV; failed rows). 412565 (numba_cpu sparse) RUNNING 21 h, both seeds in source_pix[1], no row. 412543 wave 1 attempt 1: 18 arms FAILED, 18-23 cancelled; attempt 2 not submitted.'
+ evidence: https://github.com/PyAutoLabs/autolens_inference/pull/34
project: autolens_inference
- checkin_report: checkins/2026-10-17T111853Z.md
+ checkin_report: checkins/2026-10-18T170000Z.md
- id: point-source-admission
title: Point-source sampler admission
status: needs-decision
@@ -231,22 +225,20 @@ tasks:
campaign: hst-imaging-mass-samplers
title: 'HST imaging mass stage, clean slate: Nautilus base on A100 and jax_cpu at release 2026.10.17.1, linear
and fixed lens light'
- status: needs-slicing
+ status: running
priority: high
path: tasks/hst_mass_base_light_modes.md
- updated: '2026-10-17'
- next: 'Phase 1 via Mind start_dev: --lens-light linear|fixed|both, both rows from one chain, --export-prepared
- for both modes, library revisions and SLURM job id in every row, A100 and ral jax_cpu submit scripts against
- a private pinned checkout; no compute until phase 1 ships and phase 2 is authorised.'
- evidence: https://github.com/PyAutoLabs/autolens_inference/pull/23
+ updated: '2026-10-18'
+ next: 'A100 rows of 412535 and 412537 (both light modes, seeds 0-1) read 2026-10-18; jax_gpu sparse fixed-light mass_total[1] diverged on both seeds (Cortex observation, human reading pending). 412539 (jax_cpu dense) has the seed-1 linear row; 412541 (jax_cpu sparse) and 412565 (numba_cpu sparse) RUNNING in source_pix[1] at ~16:55Z; 412563 (numba_cpu dense) FAILED (pool-worker SIGSEGV, not OOM), rerun with --cores 1 or NUMBA_THREADING_LAYER=workqueue needs a human decision. Pull and read 412539, 412541 and 412565 when they finish.'
+ evidence: https://github.com/PyAutoLabs/autolens_inference/pull/26
+ checkin_report: checkins/2026-10-18T170000Z.md
- id: hst_mass_samplers_wave1
campaign: hst-imaging-mass-samplers
title: 'HST imaging mass stage: sampler comparison, wave 1'
- status: needs-slicing
+ status: blocked
priority: normal
path: tasks/hst_mass_samplers_wave1.md
- updated: '2026-10-17'
- next: Depends on the frozen mass_total[1] problems (both light modes, A100 and jax_cpu) from the pinned base
- in hst_mass_base_light_modes; candidate samplers from the experiments runner (nautilus, emcee, nuts, smc) and
- the promoted PyAutoFit catalogue (NSS, DynestyStatic/Dynamic, Zeus) are not yet sliced.
- evidence: https://github.com/PyAutoLabs/autolens_inference/pull/23
+ updated: '2026-10-18'
+ next: 'Fix merged as autolens_inference#34 (ca505be, 2026-10-17T19:25Z): row writer keyed by dotted path, --attempt, --sampler-filter, NUTS/SMC deferred, GPU submit defaults attempt 2 nautilus+emcee --array=0-11%4 with no dependency. Attempt 2 NOT submitted; blocked on git pull of autolens_inference on the RAL login node (at b91ce9c) to ca505be and human authorisation. Attempt-1 NUTS/SMC rows exist on RAL as failed: RESOURCE_EXHAUSTED (385.32 GiB / 48.45 GiB); no nautilus/emcee rows.'
+ evidence: https://github.com/PyAutoLabs/autolens_inference/pull/34
+ checkin_report: checkins/2026-10-18T170000Z.md
diff --git a/checkins/2026-10-18T170000Z.md b/checkins/2026-10-18T170000Z.md
new file mode 100644
index 0000000..b917596
--- /dev/null
+++ b/checkins/2026-10-18T170000Z.md
@@ -0,0 +1,111 @@
+# Inference check-in — 2026-10-18T17:00:00Z
+
+Previous: checkins/2026-10-17T111853Z.md
+Scope: all open campaigns; optional user focus "Pull and read all results in would expect stuff like numba to have begun by now?"
+User decisions: none. No compute was authorised in this chat; nothing was submitted, cancelled or requeued.
+
+## Sources inspected
+
+| Source | Commit / query time | Evidence | Availability |
+|---|---|---|---|
+| RAL scheduler (`sacct`/`squeue`) for 412535, 412537, 412539, 412541, 412543, 412563, 412565 | 2026-10-18 ~16:55Z | queried by the session architect over SSH | verified |
+| RAL result rows and stage outputs | 2026-10-18 ~16:55Z | `results/` of the autolens_inference RAL checkout (rows fetched directly) | verified |
+| autolens_inference RAL checkout | `b91ce9c` | before the wave-1 fix | verified; behind main |
+| autolens_inference main | `ca505be` (autolens_inference#34 merged 2026-10-17T19:25Z) | https://github.com/PyAutoLabs/autolens_inference/pull/34 | verified |
+| Laptop `hpc/sync pull` of output/ | started 2026-10-18 ~16:20Z | still transferring (~300 kB/s) | incomplete; rows read from RAL instead |
+| Release pin | 2026.10.17.1 | rows carry `release_pin` 2026.10.17.1 | verified |
+| PyAutoCortex `projects/autolens_inference.md` | origin/main at sweep time | observation entry added through a Cortex PR (see handoff) | verified |
+| autofit_inference, graphical_ep, analytic_gaussian, ep_toy_gaussian, slope_hierarchy_scale, ic50_workspace, point-source records | not re-inspected this sweep | ledger state retained | not re-inspected |
+
+## Campaign sweep
+
+| Campaign | Change since previous | Coordination state | Job evidence | Blocker | Next bounded step |
+|---|---|---|---|---|---|
+| hst-imaging-mass-samplers | 412535 and 412537 (A100 dense/sparse) COMPLETED and all 8 rows read; 412539 seed 1 linear row written; numba dense 412563 FAILED (pool-worker SIGSEGV); wave-1 fix #34 merged; sparse fixed-light mass stage diverged on both seeds | running | sacct 2026-10-18 ~16:55Z: 412535, 412537 COMPLETED 0:0; 412539, 412541, 412565 RUNNING; 412563 FAILED 1:0; 412543 attempt 1 FAILED/cancelled | Sparse fixed-light divergence unexplained; wave 1 attempt 2 needs RAL `git pull` to `ca505be` and human authorisation; numba dense rerun needs a human decision | Human reads the divergence observation (Cortex) and decides on a Mind bug task |
+| slam-hst-base | No new evidence inspected | superseded | not queried | Unchanged | Unchanged |
+| slam-hst-delaunay | No new evidence inspected | superseded | not queried | Unchanged | Unchanged |
+| point-source-admission | No new evidence inspected | needs-decision | not queried | Unchanged | Unchanged |
+| graphical-ep | No new evidence inspected | needs-slicing | not queried | Unchanged | Unchanged |
+| analytic-gaussian | No new evidence inspected | blocked | not queried | Unchanged | Unchanged |
+| ep-toy-gaussian | No new evidence inspected | needs-decision | not queried | Unchanged | Unchanged |
+| slope-hierarchy-scale | No new evidence inspected | blocked | not queried | Unchanged | Unchanged |
+| ic50-scale | No new evidence inspected | blocked | not queried | Unchanged | Unchanged |
+| gaussian-x3-search-benchmark | No new evidence inspected | running | not queried | Unchanged | Unchanged (B5c-2b) |
+
+### hst-imaging-mass-samplers job detail (sacct/squeue 2026-10-18 ~16:55Z)
+
+| Job | Leg | State | Elapsed / MaxRSS | Rows |
+|---|---|---|---|---|
+| 412535 | jax_gpu dense (gpu) | COMPLETED 0:0 both seeds | 01:00:33 / 00:58:35, 12.8 GB | both light modes, both seeds, read |
+| 412537 | jax_gpu sparse (gpu) | COMPLETED 0:0 both seeds | 01:44:19 / 01:36:15, 7.5 GB | both light modes, both seeds, read |
+| 412539 | jax_cpu dense (ral, 8 CPU) | RUNNING since 2026-10-17T17:46Z (23 h) | — | seed 1 linear row (complete; mass_total[1] finished 15:05Z), now on fixed-light mass_total[1]; seed 0 in linear mass_total[1] since 08:37Z (~28 300 calls, f_live 0.036) |
+| 412541 | jax_cpu sparse (ral, 8 CPU) | RUNNING 23 h | — | none; both seeds in source_pix[1] since 2026-10-17T18:47Z (~33 300–33 500 calls, f_live 0.018–0.022, N_eff ~3 100; the A100 sparse source_pix[1] ended at ~31 000–33 000 calls) |
+| 412563 | numba_cpu dense (ral, 8 CPU, 48 GB) | FAILED 1:0 both seeds after 00:00:57 | MaxRSS 356 MB | `failed: SearchException ...` rows carrying only the shared source_lp[1] stage |
+| 412565 | numba_cpu sparse (ral, 8 CPU, 64 GB) | RUNNING 21 h | — | none; both seeds in source_pix[1] (~28 360 calls, f_live 0.036 / 0.134) |
+| 412543 | GPU wave 1 attempt 1 | 18 started arms FAILED, arms 18–23 cancelled (recorded 2026-10-17) | — | 8 NUTS/SMC rows `failed: RESOURCE_EXHAUSTED` (385.32 GiB NUTS, 48.45 GiB SMC); no nautilus/emcee rows |
+
+412539 seed-0 stage mtimes: source_lp 19:59Z (10-17), source_pix[1] 05:25Z, source_pix[2] 07:04Z, light[1] 08:37Z, mass_total[1] 08:37Z (10-18).
+
+On the user's question: the numba legs did begin. The in-job wait for the shared jax_cpu dense source_lp[1] passed (it completed 18:58Z on 10-17). The sparse leg 412565 has been sampling source_pix[1] for 21 h. The dense leg 412563 started source_pix[1] at 19:59Z and its first likelihood batch killed two Nautilus pool workers with SIGSEGV (exit -11). MaxRSS was 356 MB, so this was not an out-of-memory kill. PyAutoFit's `_check_workers_alive` then failed the fit with a SearchException. The sparse leg runs the same pool without crashing, so the fault is specific to the dense numba likelihood under forked workers.
+
+### Base rows read (status complete, release_pin 2026.10.17.1; stage `wall_s` in s: source_lp / source_pix[1] / source_pix[2] / light[1] / mass_total[1])
+
+| Leg / seed | Stage walls | mass_total[1] linear: worst abs Δσ, log Z | mass_total[1] fixed: worst abs Δσ, log Z |
+|---|---|---|---|
+| jax_gpu dense / 0 | 715 / 701 / 181 / 234 / linear 559, fixed 490 | 0.66 (slope), 31601.9 | 0.24, 31601.0 |
+| jax_gpu dense / 1 | 712 / 733 / 130 / 220 / linear 525, fixed 450 | 2.92 (einstein_radius), 2.65 (gamma_2), 2.63 (slope); 31591.7 | 0.34, 31592.7 |
+| jax_gpu sparse / 0 | 715 / 1786 / 298 / 222 / linear 827, fixed 1625 | 1.22, 31596.5 | DIVERGED: log Z −43567.1, max logL −43501.7 |
+| jax_gpu sparse / 1 | 720 / 1649 / 226 / 197 / linear 812, fixed 1397 | 0.96, 31598.4 | DIVERGED: log Z −43201.6 |
+| jax_cpu dense / 1 | 3411 / 38642 / 4042 / 5703 / linear 24129 | 1.29 (einstein_radius), 31593.0, 22 020 evals | not yet run |
+
+The jax_cpu dense seed-1 linear chain took about 21 h against about 40 min on the A100. The two light modes share stages 1–4, so a per-mode chain wall is stages 1–4 plus that mode's mass_total[1]; the two mass_total[1] walls must not be summed into one chain.
+
+The sparse fixed-light divergence: at seed 0 the fit used 48 700 evaluations and its posterior pinned at slope 2.964±0.0004, einstein_radius 2.493, gamma_1 0.0994, ell_comps_0 0.59 (truth_delta_sigma slope 2250, einstein_radius 114, gamma_1 289). Seed 1 diverged the same way: slope 2.963±0.0002, einstein_radius 2.427, gamma_1 0.0998, ell_comps_0 0.57 (truth_delta_sigma slope 6275). The max logL of about −43 500 compares with about +31 649 for the linear mode on the same stages 1–4.
+
+## Coverage and limits
+
+- `status: complete` is execution completion only. It does not establish convergence or scientific acceptance. The divergent sparse fixed-light rows are `complete` too.
+- numba_cpu dense has no measurement: both seeds failed at the first source_pix[1] batch.
+- jax_cpu dense has one row (seed 1, linear). jax_cpu sparse and numba_cpu sparse have none yet. No CPU fixed-light row exists.
+- Wave 1 has no nautilus/emcee rows: the attempt-1 row writer crashed. The NUTS/SMC rows record only the OOM.
+- The laptop `hpc/sync pull` of output/ was incomplete at the time of the sweep. Rows were read on RAL, and sample archives were not inspected locally.
+- The RAL checkout is at `b91ce9c`, before the wave-1 fix `ca505be`.
+- Six campaigns outside autolens_inference were not re-inspected, and their ledger rows are unchanged.
+- The sparse fixed-light divergence is recorded in Cortex as an observation awaiting the human's reading. This report draws no conclusion from it.
+
+## Decisions and handoff
+
+Proposals (not authorised):
+
+1. File a Mind bug task for the sparse fixed-light divergence and reproduce it on the laptop from the exported prepared problem, before any CPU or sampler wave uses the sparse fixed-light problem.
+2. Wave 1 attempt 2 (nautilus+emcee, `--attempt 2`, `--array=0-11%4`, no dependency; NUTS/SMC deferred): `git pull` autolens_inference on the RAL login node to `ca505be`, then submit. Needs human authorisation.
+3. numba_cpu dense rerun: either `--cores 1` (serial numba, where Nautilus runs no pool) or `NUMBA_THREADING_LAYER=workqueue` set before the fork. Needs a human decision.
+4. Wait for 412539, 412541 and 412565 to finish (jax_cpu sparse source_pix[1] is close to its end), then pull them.
+
+Authorised actions this sweep: this report, the campaigns.yaml update and the Cortex observation entry. No Mind issue was opened.
+
+Compute submitted: none.
+
+## Addendum 2026-10-18 (after the sweep)
+User decision (verbatim, inference chat): "lets just drop dense numba we would never use it."
+Recorded as decisions/hst-cleanslate-drop-numba-dense.md; proposal 3 above is withdrawn.
+The campaign's CPU base is now jax_cpu dense, jax_cpu sparse and numba_cpu sparse. No compute submitted.
+
+## Addendum 2026-10-18 (evening, human-directed housekeeping)
+- `hpc/sync pull` of autolens_inference completed on the laptop (second attempt; the first was
+ cut at 20 % by a 25-minute wrapper timeout, so never wrap a pull in a short timeout).
+ `.cortex/pull.json` records 6 live checkpoints (412539, 412541, 412565).
+- Human decision (verbatim): "Science/inference folder ... should just be a clean slate ... I think
+ not having archive is ok. I guess all this stuff can go somewhere safe on RAL but out of the way".
+ Done with the human's go: on RAL, `output/slam` (2026-09-24 chains) and `output/archive` moved to
+ `/mnt/ral/jnightin/inference/aside/autolens_inference_pre_cleanslate/`; the project's `output/`
+ now holds only `cleanslate_2026_10_17`, `prepared`, `experiments` and `experiment_sources` (the
+ last two are the cs1017 wave-1 arms). The laptop mirror of that clean-slate tree is
+ `/mnt/c/Users/Jammy/Science/inference/autolens_inference/output` (1.3 GB, file-for-file equal to
+ the checkout's clean-slate output); the checkout's `output/` becomes a symlink to it. The pull
+ root stays the checkout so `results/` and `logs/` keep landing there.
+- Flattening `cleanslate_2026_10_17/` is deferred: 412539, 412541 and 412565 write into it, every
+ row's `output_path` and every clean-slate submit names it. Proposed Mind task once they finish.
+- The pulled clean-slate rows (jax_gpu dense/sparse both modes, jax_cpu dense seed 1 linear,
+ numba dense failed rows, 8 failed NUTS/SMC experiment rows) are untracked in the autolens_inference
+ checkout; committing them is the project's own PR, not done here.
diff --git a/dashboard.html b/dashboard.html
index 1242ae0..1f8db47 100644
--- a/dashboard.html
+++ b/dashboard.html
@@ -369,7 +369,7 @@
- PyAutoInsight: https://github.com/PyAutoLabs/PyAutoInsight
- autolens_inference: https://github.com/PyAutoLabs/autolens_inference
- autofit_inference: https://github.com/PyAutoLabs/autofit_inference
Opened 2026-10-17 as a fresh start by human decision: the base is re-measured from scratch on PyAuto release 2026.10.17.1 (PyAutoLens tag v2026.10.17.1, released 2026-10-17) in fresh output trees; the 2026-09-24 rows and both earlier campaigns are superseded history.
Phase 1 via Mind start_dev: lens-light mode as a run variant (--lens-light linear|fixed|both, fixed = ordinary profiles with intensities from the light[1] fit), both rows from one chain, --export-prepared for both modes, library revisions and SLURM job id recorded in every row, submit scripts for A100 dense/sparse and ral jax_cpu dense/sparse against a private pinned checkout; then phase 2 compute by separate authorisation.
Wave 2 closed (B5b-2, 2026-10-15, autofit_inference#12) — RAL 409930 pulled, 12,750 rows admitted at the pin df6d218f10cd with 0 missing; NSS 50-step leads both evidence rankings; the catalogue ranks only legs with success rate >= 50 % over >= 20 attempts (post-hoc presentation convention); fit records published as inference-summary@3 (101 record shards).
B5c-2b (draft/research/autofit_inference/search_ext_b5c2b_scored_wave_submit.md): realisations 2-3 and their exact references, the correlated_nd@2 scored-wave manifest (83 cells x 50 seeds x 3 realisations, 12 450 runs) and its probe for 8 probe-first cells, then the submit on RAL --partition=ral from a private pinned copy with the A2 memory requests; correlated_nd@2 is frozen (autofit_inference 44c5e77f1fb0); the family is catalogued apart from gaussian_x3 and ranks nothing across families.
Verify the release containing Galaxy#634 and Lens#754 before changing priors; retain zero-centre gradient witness. Carried into hst-imaging-mass-samplers phase 1 as a prior-identity question for the frozen model.
Pilot stopped at 223 of 520 runs and is not restarted; freeze gaussian_x3@2 from its 17-row calibration before any scored run; the deferred cells move to scored wave 2 (after A2 and A4).
Phase 1 via Mind start_dev: --lens-light linear|fixed|both, both rows from one chain, --export-prepared for both modes, library revisions and SLURM job id in every row, A100 and ral jax_cpu submit scripts against a private pinned checkout; no compute until phase 1 ships and phase 2 is authorised.
Depends on the frozen mass_total[1] problems (both light modes, A100 and jax_cpu) from the pinned base in hst_mass_base_light_modes; candidate samplers from the experiments runner (nautilus, emcee, nuts, smc) and the promoted PyAutoFit catalogue (NSS, DynestyStatic/Dynamic, Zeus) are not yet sliced.
inference-summary@3: the records are held in one file per protocol and leg, each verified against the index's sha256 at the captured commit. Counts are the producer's assertions, shown without ranking; each shard file holds every record's diagnostics, timings and provenance.
Targets, seeds and stages are retained. No best-seed selection, cross-target ranking or automatic convergence threshold. Declared groups do not themselves certify parity.
No certified comparison pairs supplied by this producer.
Decision History
No decisions recorded yet.
Sampler candidates
Literature expectations, not measured PyAuto recommendations. Review a candidate and copy its investigation prompt.