insight: hst-imaging-mass-samplers 2026-10-19 clean-up record (cancel, flat layout, wave-1 attempt 1 aside)

campaigns.yaml

@@ -37,15 +37,15 @@ campaigns:
 - id: hst-imaging-mass-samplers
   title: 'HST imaging, clean slate: Nautilus base on A100, jax_cpu and numba_cpu at a pinned release, mass stage with linear
     lens light and with it fixed, then other samplers on the mass stage'
-  status: running
-  updated: '2026-10-18'
-  next: '(1) Human decision on filing a Mind bug task for the jax_gpu sparse fixed-light mass_total[1] divergence (both seeds) and reproducing it on the laptop from the exported prepared problem, before any CPU or sampler wave uses the sparse fixed-light problem. (2) Human authorisation for GPU wave 1 attempt 2 (nautilus+emcee, --array=0-11%4) after git pull of autolens_inference on the RAL login node to ca505be. (3) numba_cpu dense DROPPED by human decision 2026-10-18 (decisions/hst-cleanslate-drop-numba-dense.md): no rerun. (4) Wait for 412539, 412541 and 412565 to finish (jax_cpu sparse source_pix[1] near its end), then pull and read. (5) Then start_dev the filed Mind prompt draft/refactor/autolens_inference/flatten_cleanslate_output_layout.md (Mind 734e1bf8; human 2026-10-18: flat output/autolens/imaging/... layout, no dated folder; blocked until those arrays finish).'
-  recent_progress: 'Read 2026-10-18 (checkins/2026-10-18T170000Z.md). A100 base complete: 412535 (dense) and 412537 (sparse) rows for both light modes and seeds read. Dense fixed-light mass_total[1] is the best-recovered row (worst abs delta-sigma 0.24 / 0.34). Sparse fixed-light mass_total[1] diverged on both seeds (log Z -43567 / -43202, max logL ~-43500 vs ~+31649 linear; slope pinned at 2.96); observation recorded in Cortex, human reading pending. jax_cpu dense seed 1 linear row written (chain ~21 h vs ~40 min on A100; worst 1.29 sigma). numba_cpu dense 412563 FAILED at the first source_pix[1] batch (two Nautilus pool workers SIGSEGV, MaxRSS 356 MB, not OOM). Wave-1 fix merged as autolens_inference#34 (ca505be) 2026-10-17T19:25Z; attempt 2 not submitted.'
-  blockers: 'Sparse fixed-light mass-stage divergence unexplained (needs reproduction); wave 1 attempt 2 needs RAL checkout pull (b91ce9c -> ca505be) and human authorisation; numba_cpu dense dropped 2026-10-18 (human decision), no longer a blocker.'
-  job_status: 'Read 2026-10-18 ~16:55Z (sacct/squeue by the architect), pin 2026.10.17.1. 412535 (jax_gpu dense) and 412537 (jax_gpu sparse) COMPLETED 0:0 both seeds. 412539 (jax_cpu dense) RUNNING 23 h: seed 1 linear row written, seed 1 on fixed mass_total[1], seed 0 in linear mass_total[1]. 412541 (jax_cpu sparse) RUNNING 23 h, both seeds in source_pix[1], no row. 412563 (numba_cpu dense) FAILED 1:0 both seeds after 00:00:57 (pool-worker SIGSEGV; failed rows). 412565 (numba_cpu sparse) RUNNING 21 h, both seeds in source_pix[1], no row. 412543 wave 1 attempt 1: 18 arms FAILED, 18-23 cancelled; attempt 2 not submitted.'
-  evidence: https://github.com/PyAutoLabs/autolens_inference/pull/34
+  status: needs-decision
+  updated: '2026-10-19'
+  next: 'Nothing is queued (scheduler empty after the human-directed cancel 2026-10-19 ~15:51Z). Each step needs human authorisation: (1) Confirm keeping output/autolens/prepared/ (agent proposal, not yet confirmed: wave.py needs output/autolens/prepared/<config>/seed_<n>/<variant>/manifest.json for every sampler wave and the divergence repro needs it; only re-running the base mass stage regenerates it). (2) File a Mind bug task and run a laptop repro of the jax_gpu sparse fixed-light mass_total[1] divergence (both seeds) before any CPU or sampler wave uses the sparse fixed-light problem. (3) Rerun the cancelled sparse legs under the flat layout: jax_cpu sparse seed 1 from source_pix[2], jax_cpu sparse lightfixed mass_total[1] (both seeds), numba_cpu sparse lightfixed mass_total[1] (both seeds). (4) GPU wave 1 attempt 2 (nautilus+emcee, --array=0-11%4). The RAL checkout is now at c3ca4de, which contains the #34 fix, and the attempt-1 arms are archived. (5) Commit the clean-slate rows through an autolens_inference PR. numba_cpu dense stays dropped (decisions/hst-cleanslate-drop-numba-dense.md).'
+  recent_progress: '2026-10-19 (checkins/2026-10-19T171637Z.md, focused clean-up): 412539 (jax_cpu dense) COMPLETED. On the human''s decision, 412541 (jax_cpu sparse) and 412565 (numba_cpu sparse) were cancelled with partial stages left on disk. jax_cpu sparse: seed 0 has slam_base complete; seed 1 has source_lp[1] and source_pix[1] complete. numba sparse: slam_base complete on both seeds. All lightfixed mass_total[1] stages are incomplete. autolens_inference#36 (flatten, closes #35) merged as c3ca4de 2026-10-19T15:53:33Z. flatten_cleanslate.py --apply ran on RAL (28 moves, 55 rewrites) and on the laptop mirror (28 moves, 38 rewrites). Stage output now lives at output/autolens/imaging/hst/slam/<variant>/<config>/seed_<n>/<stage>/, prepared problems at output/autolens/prepared/<config>/seed_<n>/<variant>/ and rows at results/slam/imaging/hst/<variant>/<config>/. RAL and laptop each hold 1974 files with identical paths and sizes. The wave-1 attempt-1 material (output/experiments, output/experiment_sources, results/searches/experiments) was moved aside to /mnt/ral/jnightin/inference/aside/autolens_inference_cleanslate_wave1_attempt1/ on RAL and to /mnt/c/Users/Jammy/Science/inference/aside/autolens_inference_cleanslate_wave1_attempt1/ on the laptop. Earlier (2026-10-18): A100 base complete; the sparse fixed-light mass_total[1] diverged on both seeds (Cortex observation, human reading pending).'
+  blockers: 'Sparse fixed-light mass-stage divergence unexplained (needs reproduction). Reruns of the cancelled sparse legs and wave 1 attempt 2 need human authorisation. Keeping prepared/ awaits human confirmation. The RAL checkout pull blocker is cleared (c3ca4de).'
+  job_status: 'Read 2026-10-19 15:49:05Z (squeue/sacct on RAL by the architect), pin 2026.10.17.1. 412539 (jax_cpu dense) COMPLETED: 412539_0 ended 2026-10-18T23:46:01 (elapsed 1-05:59:53), 412539_1 ended 2026-10-18T20:43:11 (elapsed 1-02:57:03; elapsed from sacct ~17:25Z, Cortex PR #73 e3ab356); rows not read. 412541 (jax_cpu sparse) and 412565 (numba_cpu sparse) were RUNNING; the human had them scancelled ~15:51Z (sacct CANCELLED, elapsed 1-23:04:43 / 1-20:52:11). The queue was empty afterwards. Earlier: 412535 and 412537 (A100) COMPLETED; 412563 (numba_cpu dense) FAILED (dropped); 412543 wave 1 attempt 1, from sacct 2026-10-19 ~17:25Z (Cortex PR #73 e3ab356): _0.._17 FAILED 2026-10-17 (ends 18:51:38-20:32:26Z, elapsed 00:04:15-00:47:10), _18 CANCELLED after 00:04:00, _[19-23] CANCELLED unstarted at 20:01:03Z; output and rows archived in aside/autolens_inference_cleanslate_wave1_attempt1. Compute submitted 2026-10-19: none.'
+  evidence: https://github.com/PyAutoLabs/autolens_inference/pull/36
   project: autolens_inference
-  checkin_report: checkins/2026-10-18T170000Z.md
+  checkin_report: checkins/2026-10-19T171637Z.md
 - id: point-source-admission
   title: Point-source sampler admission
   status: needs-decision
@@ -225,20 +225,20 @@ tasks:
   campaign: hst-imaging-mass-samplers
   title: 'HST imaging mass stage, clean slate: Nautilus base on A100 and jax_cpu at release 2026.10.17.1, linear
     and fixed lens light'
-  status: running
+  status: needs-decision
   priority: high
   path: tasks/hst_mass_base_light_modes.md
-  updated: '2026-10-18'
-  next: 'A100 rows of 412535 and 412537 (both light modes, seeds 0-1) read 2026-10-18; jax_gpu sparse fixed-light mass_total[1] diverged on both seeds (Cortex observation, human reading pending). 412539 (jax_cpu dense) has the seed-1 linear row; 412541 (jax_cpu sparse) and 412565 (numba_cpu sparse) RUNNING in source_pix[1] at ~16:55Z; 412563 (numba_cpu dense) FAILED (pool-worker SIGSEGV, not OOM), rerun with --cores 1 or NUMBA_THREADING_LAYER=workqueue needs a human decision. Pull and read 412539, 412541 and 412565 when they finish.'
-  evidence: https://github.com/PyAutoLabs/autolens_inference/pull/26
-  checkin_report: checkins/2026-10-18T170000Z.md
+  updated: '2026-10-19'
+  next: 'A100 legs complete; jax_gpu sparse fixed-light mass_total[1] diverged (Cortex observation, human reading pending). 412539 (jax_cpu dense) COMPLETED 2026-10-18, elapsed 1-05:59:53 / 1-02:57:03 (sacct 2026-10-19 ~17:25Z, Cortex PR #73 e3ab356); its rows were not re-read in the 2026-10-19 record. 412541 (jax_cpu sparse) and 412565 (numba_cpu sparse) were CANCELLED 2026-10-19 ~15:51Z on the human''s decision. Completed at cancel: jax_cpu sparse seed 0 slam_base; jax_cpu sparse seed 1 source_lp[1] and source_pix[1]; numba sparse slam_base on both seeds. Every lightfixed mass_total[1] is incomplete. The partial stage dirs stay under output/autolens/imaging/hst/slam/<variant>/<config>/seed_<n>/. A rerun of the remainder under the flat layout needs human authorisation. numba_cpu dense dropped (2026-10-18 decision).'
+  evidence: https://github.com/PyAutoLabs/autolens_inference/pull/36
+  checkin_report: checkins/2026-10-19T171637Z.md
 - id: hst_mass_samplers_wave1
   campaign: hst-imaging-mass-samplers
   title: 'HST imaging mass stage: sampler comparison, wave 1'
-  status: blocked
+  status: needs-decision
   priority: normal
   path: tasks/hst_mass_samplers_wave1.md
-  updated: '2026-10-18'
-  next: 'Fix merged as autolens_inference#34 (ca505be, 2026-10-17T19:25Z): row writer keyed by dotted path, --attempt, --sampler-filter, NUTS/SMC deferred, GPU submit defaults attempt 2 nautilus+emcee --array=0-11%4 with no dependency. Attempt 2 NOT submitted; blocked on git pull of autolens_inference on the RAL login node (at b91ce9c) to ca505be and human authorisation. Attempt-1 NUTS/SMC rows exist on RAL as failed: RESOURCE_EXHAUSTED (385.32 GiB / 48.45 GiB); no nautilus/emcee rows.'
+  updated: '2026-10-19'
+  next: 'Fix merged as autolens_inference#34; the RAL checkout was fast-forwarded to c3ca4de (includes #34 and the #36 flat layout) on 2026-10-19, so the pull blocker is cleared. Attempt 2 (nautilus+emcee, --array=0-11%4, NUTS/SMC deferred) is NOT submitted and needs human authorisation. It reads output/autolens/prepared/<config>/seed_0/<variant>/manifest.json. Attempt-1 material (output/experiments 341M, output/experiment_sources 76M, the 8 failed NUTS/SMC rows from results/searches/experiments) was moved aside 2026-10-19 on the human''s decision to /mnt/ral/jnightin/inference/aside/autolens_inference_cleanslate_wave1_attempt1/ (RAL) and /mnt/c/Users/Jammy/Science/inference/aside/autolens_inference_cleanslate_wave1_attempt1/ (laptop), so attempt 2 starts clean.'
   evidence: https://github.com/PyAutoLabs/autolens_inference/pull/34
-  checkin_report: checkins/2026-10-18T170000Z.md
+  checkin_report: checkins/2026-10-19T171637Z.md