insight: hst_mass_samplers_wave1 sliced and shipped (autolens_inference#28)

campaigns.yaml

@@ -1,6 +1,6 @@
 version: 1
-last_checkin: '2026-10-17T11:18:53Z'
-last_checkin_report: checkins/2026-10-17T111853Z.md
+last_checkin: '2026-10-18T17:00:00Z'
+last_checkin_report: checkins/2026-10-18T170000Z.md
 campaigns:
 - id: slam-hst-base
   title: HST SLaM base backend parity
@@ -35,23 +35,17 @@ campaigns:
   project: autolens_inference
   checkin_report: checkins/2026-10-17T111853Z.md
 - id: hst-imaging-mass-samplers
-  title: 'HST imaging, clean slate: Nautilus base on A100 and jax_cpu at a pinned release, mass stage with linear
+  title: 'HST imaging, clean slate: Nautilus base on A100, jax_cpu and numba_cpu at a pinned release, mass stage with linear
     lens light and with it fixed, then other samplers on the mass stage'
-  status: needs-slicing
-  updated: '2026-10-17'
-  next: 'Phase 1 via Mind start_dev: lens-light mode as a run variant (--lens-light linear|fixed|both, fixed = ordinary
-    profiles with intensities from the light[1] fit), both rows from one chain, --export-prepared for both modes,
-    library revisions and SLURM job id recorded in every row, submit scripts for A100 dense/sparse and ral jax_cpu
-    dense/sparse against a private pinned checkout; then phase 2 compute by separate authorisation.'
-  recent_progress: 'Opened 2026-10-17 as a fresh start by human decision: the base is re-measured from scratch on
-    PyAuto release 2026.10.17.1 (PyAutoLens tag v2026.10.17.1, released 2026-10-17) in fresh output trees; the
-    2026-09-24 rows and both earlier campaigns are superseded history.'
-  blockers: Phase 1 runner change not started (Mind start_dev pending); no pinned private checkout exists on RAL
-    yet; compute not authorised until phase 1 ships.
-  job_status: none submitted
-  evidence: https://github.com/PyAutoLabs/autolens_inference/pull/23
+  status: running
+  updated: '2026-10-18'
+  next: '(1) Human decision on filing a Mind bug task for the jax_gpu sparse fixed-light mass_total[1] divergence (both seeds) and reproducing it on the laptop from the exported prepared problem, before any CPU or sampler wave uses the sparse fixed-light problem. (2) Human authorisation for GPU wave 1 attempt 2 (nautilus+emcee, --array=0-11%4) after git pull of autolens_inference on the RAL login node to ca505be. (3) numba_cpu dense DROPPED by human decision 2026-10-18 (decisions/hst-cleanslate-drop-numba-dense.md): no rerun. (4) Wait for 412539, 412541 and 412565 to finish (jax_cpu sparse source_pix[1] near its end), then pull and read. (5) Then start_dev the filed Mind prompt draft/refactor/autolens_inference/flatten_cleanslate_output_layout.md (Mind 734e1bf8; human 2026-10-18: flat output/autolens/imaging/... layout, no dated folder; blocked until those arrays finish).'
+  recent_progress: 'Read 2026-10-18 (checkins/2026-10-18T170000Z.md). A100 base complete: 412535 (dense) and 412537 (sparse) rows for both light modes and seeds read. Dense fixed-light mass_total[1] is the best-recovered row (worst abs delta-sigma 0.24 / 0.34). Sparse fixed-light mass_total[1] diverged on both seeds (log Z -43567 / -43202, max logL ~-43500 vs ~+31649 linear; slope pinned at 2.96); observation recorded in Cortex, human reading pending. jax_cpu dense seed 1 linear row written (chain ~21 h vs ~40 min on A100; worst 1.29 sigma). numba_cpu dense 412563 FAILED at the first source_pix[1] batch (two Nautilus pool workers SIGSEGV, MaxRSS 356 MB, not OOM). Wave-1 fix merged as autolens_inference#34 (ca505be) 2026-10-17T19:25Z; attempt 2 not submitted.'
+  blockers: 'Sparse fixed-light mass-stage divergence unexplained (needs reproduction); wave 1 attempt 2 needs RAL checkout pull (b91ce9c -> ca505be) and human authorisation; numba_cpu dense dropped 2026-10-18 (human decision), no longer a blocker.'
+  job_status: 'Read 2026-10-18 ~16:55Z (sacct/squeue by the architect), pin 2026.10.17.1. 412535 (jax_gpu dense) and 412537 (jax_gpu sparse) COMPLETED 0:0 both seeds. 412539 (jax_cpu dense) RUNNING 23 h: seed 1 linear row written, seed 1 on fixed mass_total[1], seed 0 in linear mass_total[1]. 412541 (jax_cpu sparse) RUNNING 23 h, both seeds in source_pix[1], no row. 412563 (numba_cpu dense) FAILED 1:0 both seeds after 00:00:57 (pool-worker SIGSEGV; failed rows). 412565 (numba_cpu sparse) RUNNING 21 h, both seeds in source_pix[1], no row. 412543 wave 1 attempt 1: 18 arms FAILED, 18-23 cancelled; attempt 2 not submitted.'
+  evidence: https://github.com/PyAutoLabs/autolens_inference/pull/34
   project: autolens_inference
-  checkin_report: checkins/2026-10-17T111853Z.md
+  checkin_report: checkins/2026-10-18T170000Z.md
 - id: point-source-admission
   title: Point-source sampler admission
   status: needs-decision
@@ -231,22 +225,20 @@ tasks:
   campaign: hst-imaging-mass-samplers
   title: 'HST imaging mass stage, clean slate: Nautilus base on A100 and jax_cpu at release 2026.10.17.1, linear
     and fixed lens light'
-  status: needs-slicing
+  status: running
   priority: high
   path: tasks/hst_mass_base_light_modes.md
-  updated: '2026-10-17'
-  next: 'Phase 1 via Mind start_dev: --lens-light linear|fixed|both, both rows from one chain, --export-prepared
-    for both modes, library revisions and SLURM job id in every row, A100 and ral jax_cpu submit scripts against
-    a private pinned checkout; no compute until phase 1 ships and phase 2 is authorised.'
-  evidence: https://github.com/PyAutoLabs/autolens_inference/pull/23
+  updated: '2026-10-18'
+  next: 'A100 rows of 412535 and 412537 (both light modes, seeds 0-1) read 2026-10-18; jax_gpu sparse fixed-light mass_total[1] diverged on both seeds (Cortex observation, human reading pending). 412539 (jax_cpu dense) has the seed-1 linear row; 412541 (jax_cpu sparse) and 412565 (numba_cpu sparse) RUNNING in source_pix[1] at ~16:55Z; 412563 (numba_cpu dense) FAILED (pool-worker SIGSEGV, not OOM), rerun with --cores 1 or NUMBA_THREADING_LAYER=workqueue needs a human decision. Pull and read 412539, 412541 and 412565 when they finish.'
+  evidence: https://github.com/PyAutoLabs/autolens_inference/pull/26
+  checkin_report: checkins/2026-10-18T170000Z.md
 - id: hst_mass_samplers_wave1
   campaign: hst-imaging-mass-samplers
   title: 'HST imaging mass stage: sampler comparison, wave 1'
-  status: needs-slicing
+  status: blocked
   priority: normal
   path: tasks/hst_mass_samplers_wave1.md
-  updated: '2026-10-17'
-  next: Depends on the frozen mass_total[1] problems (both light modes, A100 and jax_cpu) from the pinned base
-    in hst_mass_base_light_modes; candidate samplers from the experiments runner (nautilus, emcee, nuts, smc) and
-    the promoted PyAutoFit catalogue (NSS, DynestyStatic/Dynamic, Zeus) are not yet sliced.
-  evidence: https://github.com/PyAutoLabs/autolens_inference/pull/23
+  updated: '2026-10-18'
+  next: 'Fix merged as autolens_inference#34 (ca505be, 2026-10-17T19:25Z): row writer keyed by dotted path, --attempt, --sampler-filter, NUTS/SMC deferred, GPU submit defaults attempt 2 nautilus+emcee --array=0-11%4 with no dependency. Attempt 2 NOT submitted; blocked on git pull of autolens_inference on the RAL login node (at b91ce9c) to ca505be and human authorisation. Attempt-1 NUTS/SMC rows exist on RAL as failed: RESOURCE_EXHAUSTED (385.32 GiB / 48.45 GiB); no nautilus/emcee rows.'
+  evidence: https://github.com/PyAutoLabs/autolens_inference/pull/34
+  checkin_report: checkins/2026-10-18T170000Z.md

checkins/2026-10-18T170000Z.md

@@ -0,0 +1,111 @@
+# Inference check-in — 2026-10-18T17:00:00Z
+
+Previous: checkins/2026-10-17T111853Z.md
+Scope: all open campaigns; optional user focus "Pull and read all results in would expect stuff like numba to have begun by now?"
+User decisions: none. No compute was authorised in this chat; nothing was submitted, cancelled or requeued.
+
+## Sources inspected
+
+| Source | Commit / query time | Evidence | Availability |
+|---|---|---|---|
+| RAL scheduler (`sacct`/`squeue`) for 412535, 412537, 412539, 412541, 412543, 412563, 412565 | 2026-10-18 ~16:55Z | queried by the session architect over SSH | verified |
+| RAL result rows and stage outputs | 2026-10-18 ~16:55Z | `results/` of the autolens_inference RAL checkout (rows fetched directly) | verified |
+| autolens_inference RAL checkout | `b91ce9c` | before the wave-1 fix | verified; behind main |
+| autolens_inference main | `ca505be` (autolens_inference#34 merged 2026-10-17T19:25Z) | https://github.com/PyAutoLabs/autolens_inference/pull/34 | verified |
+| Laptop `hpc/sync pull` of output/ | started 2026-10-18 ~16:20Z | still transferring (~300 kB/s) | incomplete; rows read from RAL instead |
+| Release pin | 2026.10.17.1 | rows carry `release_pin` 2026.10.17.1 | verified |
+| PyAutoCortex `projects/autolens_inference.md` | origin/main at sweep time | observation entry added through a Cortex PR (see handoff) | verified |
+| autofit_inference, graphical_ep, analytic_gaussian, ep_toy_gaussian, slope_hierarchy_scale, ic50_workspace, point-source records | not re-inspected this sweep | ledger state retained | not re-inspected |
+
+## Campaign sweep
+
+| Campaign | Change since previous | Coordination state | Job evidence | Blocker | Next bounded step |
+|---|---|---|---|---|---|
+| hst-imaging-mass-samplers | 412535 and 412537 (A100 dense/sparse) COMPLETED and all 8 rows read; 412539 seed 1 linear row written; numba dense 412563 FAILED (pool-worker SIGSEGV); wave-1 fix #34 merged; sparse fixed-light mass stage diverged on both seeds | running | sacct 2026-10-18 ~16:55Z: 412535, 412537 COMPLETED 0:0; 412539, 412541, 412565 RUNNING; 412563 FAILED 1:0; 412543 attempt 1 FAILED/cancelled | Sparse fixed-light divergence unexplained; wave 1 attempt 2 needs RAL `git pull` to `ca505be` and human authorisation; numba dense rerun needs a human decision | Human reads the divergence observation (Cortex) and decides on a Mind bug task |
+| slam-hst-base | No new evidence inspected | superseded | not queried | Unchanged | Unchanged |
+| slam-hst-delaunay | No new evidence inspected | superseded | not queried | Unchanged | Unchanged |
+| point-source-admission | No new evidence inspected | needs-decision | not queried | Unchanged | Unchanged |
+| graphical-ep | No new evidence inspected | needs-slicing | not queried | Unchanged | Unchanged |
+| analytic-gaussian | No new evidence inspected | blocked | not queried | Unchanged | Unchanged |
+| ep-toy-gaussian | No new evidence inspected | needs-decision | not queried | Unchanged | Unchanged |
+| slope-hierarchy-scale | No new evidence inspected | blocked | not queried | Unchanged | Unchanged |
+| ic50-scale | No new evidence inspected | blocked | not queried | Unchanged | Unchanged |
+| gaussian-x3-search-benchmark | No new evidence inspected | running | not queried | Unchanged | Unchanged (B5c-2b) |
+
+### hst-imaging-mass-samplers job detail (sacct/squeue 2026-10-18 ~16:55Z)
+
+| Job | Leg | State | Elapsed / MaxRSS | Rows |
+|---|---|---|---|---|
+| 412535 | jax_gpu dense (gpu) | COMPLETED 0:0 both seeds | 01:00:33 / 00:58:35, 12.8 GB | both light modes, both seeds, read |
+| 412537 | jax_gpu sparse (gpu) | COMPLETED 0:0 both seeds | 01:44:19 / 01:36:15, 7.5 GB | both light modes, both seeds, read |
+| 412539 | jax_cpu dense (ral, 8 CPU) | RUNNING since 2026-10-17T17:46Z (23 h) | — | seed 1 linear row (complete; mass_total[1] finished 15:05Z), now on fixed-light mass_total[1]; seed 0 in linear mass_total[1] since 08:37Z (~28 300 calls, f_live 0.036) |
+| 412541 | jax_cpu sparse (ral, 8 CPU) | RUNNING 23 h | — | none; both seeds in source_pix[1] since 2026-10-17T18:47Z (~33 300–33 500 calls, f_live 0.018–0.022, N_eff ~3 100; the A100 sparse source_pix[1] ended at ~31 000–33 000 calls) |
+| 412563 | numba_cpu dense (ral, 8 CPU, 48 GB) | FAILED 1:0 both seeds after 00:00:57 | MaxRSS 356 MB | `failed: SearchException ...` rows carrying only the shared source_lp[1] stage |
+| 412565 | numba_cpu sparse (ral, 8 CPU, 64 GB) | RUNNING 21 h | — | none; both seeds in source_pix[1] (~28 360 calls, f_live 0.036 / 0.134) |
+| 412543 | GPU wave 1 attempt 1 | 18 started arms FAILED, arms 18–23 cancelled (recorded 2026-10-17) | — | 8 NUTS/SMC rows `failed: RESOURCE_EXHAUSTED` (385.32 GiB NUTS, 48.45 GiB SMC); no nautilus/emcee rows |
+
+412539 seed-0 stage mtimes: source_lp 19:59Z (10-17), source_pix[1] 05:25Z, source_pix[2] 07:04Z, light[1] 08:37Z, mass_total[1] 08:37Z (10-18).
+
+On the user's question: the numba legs did begin. The in-job wait for the shared jax_cpu dense source_lp[1] passed (it completed 18:58Z on 10-17). The sparse leg 412565 has been sampling source_pix[1] for 21 h. The dense leg 412563 started source_pix[1] at 19:59Z and its first likelihood batch killed two Nautilus pool workers with SIGSEGV (exit -11). MaxRSS was 356 MB, so this was not an out-of-memory kill. PyAutoFit's `_check_workers_alive` then failed the fit with a SearchException. The sparse leg runs the same pool without crashing, so the fault is specific to the dense numba likelihood under forked workers.
+
+### Base rows read (status complete, release_pin 2026.10.17.1; stage `wall_s` in s: source_lp / source_pix[1] / source_pix[2] / light[1] / mass_total[1])
+
+| Leg / seed | Stage walls | mass_total[1] linear: worst abs Δσ, log Z | mass_total[1] fixed: worst abs Δσ, log Z |
+|---|---|---|---|
+| jax_gpu dense / 0 | 715 / 701 / 181 / 234 / linear 559, fixed 490 | 0.66 (slope), 31601.9 | 0.24, 31601.0 |
+| jax_gpu dense / 1 | 712 / 733 / 130 / 220 / linear 525, fixed 450 | 2.92 (einstein_radius), 2.65 (gamma_2), 2.63 (slope); 31591.7 | 0.34, 31592.7 |
+| jax_gpu sparse / 0 | 715 / 1786 / 298 / 222 / linear 827, fixed 1625 | 1.22, 31596.5 | DIVERGED: log Z −43567.1, max logL −43501.7 |
+| jax_gpu sparse / 1 | 720 / 1649 / 226 / 197 / linear 812, fixed 1397 | 0.96, 31598.4 | DIVERGED: log Z −43201.6 |
+| jax_cpu dense / 1 | 3411 / 38642 / 4042 / 5703 / linear 24129 | 1.29 (einstein_radius), 31593.0, 22 020 evals | not yet run |
+
+The jax_cpu dense seed-1 linear chain took about 21 h against about 40 min on the A100. The two light modes share stages 1–4, so a per-mode chain wall is stages 1–4 plus that mode's mass_total[1]; the two mass_total[1] walls must not be summed into one chain.
+
+The sparse fixed-light divergence: at seed 0 the fit used 48 700 evaluations and its posterior pinned at slope 2.964±0.0004, einstein_radius 2.493, gamma_1 0.0994, ell_comps_0 0.59 (truth_delta_sigma slope 2250, einstein_radius 114, gamma_1 289). Seed 1 diverged the same way: slope 2.963±0.0002, einstein_radius 2.427, gamma_1 0.0998, ell_comps_0 0.57 (truth_delta_sigma slope 6275). The max logL of about −43 500 compares with about +31 649 for the linear mode on the same stages 1–4.
+
+## Coverage and limits
+
+- `status: complete` is execution completion only. It does not establish convergence or scientific acceptance. The divergent sparse fixed-light rows are `complete` too.
+- numba_cpu dense has no measurement: both seeds failed at the first source_pix[1] batch.
+- jax_cpu dense has one row (seed 1, linear). jax_cpu sparse and numba_cpu sparse have none yet. No CPU fixed-light row exists.
+- Wave 1 has no nautilus/emcee rows: the attempt-1 row writer crashed. The NUTS/SMC rows record only the OOM.
+- The laptop `hpc/sync pull` of output/ was incomplete at the time of the sweep. Rows were read on RAL, and sample archives were not inspected locally.
+- The RAL checkout is at `b91ce9c`, before the wave-1 fix `ca505be`.
+- Six campaigns outside autolens_inference were not re-inspected, and their ledger rows are unchanged.
+- The sparse fixed-light divergence is recorded in Cortex as an observation awaiting the human's reading. This report draws no conclusion from it.
+
+## Decisions and handoff
+
+Proposals (not authorised):
+
+1. File a Mind bug task for the sparse fixed-light divergence and reproduce it on the laptop from the exported prepared problem, before any CPU or sampler wave uses the sparse fixed-light problem.
+2. Wave 1 attempt 2 (nautilus+emcee, `--attempt 2`, `--array=0-11%4`, no dependency; NUTS/SMC deferred): `git pull` autolens_inference on the RAL login node to `ca505be`, then submit. Needs human authorisation.
+3. numba_cpu dense rerun: either `--cores 1` (serial numba, where Nautilus runs no pool) or `NUMBA_THREADING_LAYER=workqueue` set before the fork. Needs a human decision.
+4. Wait for 412539, 412541 and 412565 to finish (jax_cpu sparse source_pix[1] is close to its end), then pull them.
+
+Authorised actions this sweep: this report, the campaigns.yaml update and the Cortex observation entry. No Mind issue was opened.
+
+Compute submitted: none.
+
+## Addendum 2026-10-18 (after the sweep)
+User decision (verbatim, inference chat): "lets just drop dense numba we would never use it."
+Recorded as decisions/hst-cleanslate-drop-numba-dense.md; proposal 3 above is withdrawn.
+The campaign's CPU base is now jax_cpu dense, jax_cpu sparse and numba_cpu sparse. No compute submitted.
+
+## Addendum 2026-10-18 (evening, human-directed housekeeping)
+- `hpc/sync pull` of autolens_inference completed on the laptop (second attempt; the first was
+  cut at 20 % by a 25-minute wrapper timeout, so never wrap a pull in a short timeout).
+  `.cortex/pull.json` records 6 live checkpoints (412539, 412541, 412565).
+- Human decision (verbatim): "Science/inference folder ... should just be a clean slate ... I think
+  not having archive is ok. I guess all this stuff can go somewhere safe on RAL but out of the way".
+  Done with the human's go: on RAL, `output/slam` (2026-09-24 chains) and `output/archive` moved to
+  `/mnt/ral/jnightin/inference/aside/autolens_inference_pre_cleanslate/`; the project's `output/`
+  now holds only `cleanslate_2026_10_17`, `prepared`, `experiments` and `experiment_sources` (the
+  last two are the cs1017 wave-1 arms). The laptop mirror of that clean-slate tree is
+  `/mnt/c/Users/Jammy/Science/inference/autolens_inference/output` (1.3 GB, file-for-file equal to
+  the checkout's clean-slate output); the checkout's `output/` becomes a symlink to it. The pull
+  root stays the checkout so `results/` and `logs/` keep landing there.
+- Flattening `cleanslate_2026_10_17/` is deferred: 412539, 412541 and 412565 write into it, every
+  row's `output_path` and every clean-slate submit names it. Proposed Mind task once they finish.
+- The pulled clean-slate rows (jax_gpu dense/sparse both modes, jax_cpu dense seed 1 linear,
+  numba dense failed rows, 8 failed NUTS/SMC experiment rows) are untracked in the autolens_inference
+  checkout; committing them is the project's own PR, not done here.