github-actions · Sep 24, 2026 · Sep 24, 2026
diff --git a/projects/ep_toy_gaussian.md b/projects/ep_toy_gaussian.md
@@ -5,16 +5,16 @@ Issue: ep_toy_gaussian#1
 
 ## Now
 
-n50_seed42 is on RAL: NUTS + one-by-one as 342639, EP x3 on identical data as 342640_[0-2]. The one-by-one leg finished (50/50 in 319 s); the EP repeats are turning per-factor Dynesty fits.
-Next: pull and run scripts/compare.py --sample n50_seed42; settle first whether a prior factor with only NO_CHANGE rows counts as zero SUCCESS updates.
+Wave 1 (n50_seed42: 342639 + 342640_[0-2]) failed on infrastructure — NUTS OOM at the 8 GB cap, EP x3 EMFILE (fixed upstream by PyAutoFit#1632/#1634, RAL mirror now a73684012). The collapse question at N=50 is unanswered. Only n5_smoke speaks so far: EP RECOVER 3/3 (parent sigma 10.14/9.96/9.83 vs truth 10, no collapse, no BIASED-TIGHT), NUTS 49.89/11.14. Next: wave 2 needs a human go, a NEW sample name (e.g. n50_seed42_w2 — never re-run into an existing sample directory) and the NUTS SBATCH memory raised above 8 GB.
 
 ## Runs
 
-- 342639 — open — ral — 2026-09-10 — nuts_vs_ep_x3_n50: one_by_one + BlackJAX NUTS, partition ral
-- 342640 — open — ral — 2026-09-10 — nuts_vs_ep_x3_n50: EP x3 on identical data, array 0-2
 
 ## Log
 
+- 2026-09-24 — note — per-search overhead profile on this toy (2026-09-24, laptop, N=5, max_steps=3): 2.45 s per Dynesty factor search, ~1.47 s dynesty run_nested + ~0.98 s autofit wrapper (plots, a redundant second run_nested pass, samples writes); inside run_nested instance_from_vector is 57 % of the likelihood callback for a 1-parameter model. Speed-up prompt filed in the Mind: draft/refactor/autofit/ep_factor_search_wrapper_overhead.md
+- 2026-09-24 — run — 342640 failed — wall 0:14: nuts_vs_ep_x3_n50: EP x3 on identical data, array 0-2 — all three array tasks [0-2] CRASH OSError [Errno 24] Too many open files after ~126 Dynesty factor fits each (wall ~860-880 s), no ep_history.csv; results/n50_seed42/comparison.md scores WITNESS NOT MET 0/7 on infrastructure, not science. Cause: RAL mirror PyAutoFit e354dbb6b built a Pool(1) per single-core Dynesty fit (~+8 fds per fit vs 1024 soft ulimit); fixed by PyAutoFit#1632 (merged 2026-09-15) + #1634 (merged 2026-09-17); RAL mirror now a73684012 (2026-09-19) contains both; local check 2026-09-24 on a73684012: fd count flat at 7 across 15 EP factor searches (N=5, max_steps=3)
+- 2026-09-24 — run — 342639 failed: nuts_vs_ep_x3_n50: one_by_one + BlackJAX NUTS, partition ral — one_by_one completed 50/50 in 319 s (naive parent 52.470 ± 1.260, deconvolved scatter 3.142); NUTS leg OUT_OF_MEMORY at the 8 GB SBATCH cap during window adaptation, no nuts.json
 - 2026-09-10 — run — 342640 submitted: nuts_vs_ep_x3_n50 — EP x3 on identical data, array 0-2
 - 2026-09-10 — run — 342639 submitted: nuts_vs_ep_x3_n50 — one_by_one + BlackJAX NUTS, partition ral
 - 2026-09-10 — note — question: Do three EP fits agree with NUTS at N=50 [archive/tasks/ep_toy_gaussian/nuts_vs_ep_x3_n50.md]
diff --git a/projects/slope_hierarchy_scale.md b/projects/slope_hierarchy_scale.md
@@ -5,17 +5,18 @@ Issue: slope_hierarchy_scale#2
 
 ## Now
 
-N=25 scale-up: graphical array 342348_[0-24] and joint fit 342350_0 still open; EP witness rerun 343299 (MAX_STEPS=2, --use_cpu on the decoupled script from slope_hierarchy_scale#4) submitted 2026-09-15 after 342410 died at ~EP step 3 on LLVM compile memory. Next: pull 343299 and confirm the first factor_step ran JAX-vectorised with no pool; the full EP arm waits on the PyAutoFit compile-memory fix (draft/bug/autofit/ep_re_jit_compiles_the_vmapped_likelihood.md).
+343299 (EP witness rerun, MAX_STEPS=2) completed: dataset factors all SUCCESS and JAX-vectorised with one compile per factor search, but the hierarchical factor never updated (0/50 SUCCESS: 27 BAD_PROJECTION + 23 FAILURE), so the parent came back as the prior. The EP arm cannot estimate the slope scatter with the Laplace projection; the moment-matching cure (draft/feature/autofit/ep_hierarchical_scatter_moment_matching.md) is the gate — human decision. Still open: graphical array 342348_[0-24], joint fit 342350_0, EP arm 342351_0.
 
 ## Runs
 
 - 342348_[0-24] — open — gpu — 2026-09-08 — n25_scale_up: one lens per array task, hpc/batch_gpu/submit_one_by_one, sample_n25_seed42
 - 342350_0 — open — gpu — 2026-09-08 — n25_scale_up: joint hierarchical NUTS fit, hpc/batch_gpu/submit_graphical
 - 342351_0 — open — ral — 2026-09-08 — n25_scale_up: EP arm on the CPU partition, hpc/batch_cpu/submit_ep, max_steps 12; RAL PyAutoFit mirror 68ff9bd57 predates PyAutoFit#1580, so this arm runs without the stale-mask fixed-point fix
-- 343299 — open — ral — 2026-09-15 — n25_scale_up: EP witness rerun on the decoupled --use_cpu script (slope_hierarchy_scale#4), JAX vectorised likelihood on the CPU backend, no pool, MAX_STEPS=2 via sbatch --export; stale 342410 output parked as ep_dead_342410
 
 ## Log
 
+- 2026-09-24 — note — 343299 finding: BAD_PROJECTION = Hessian at the mode not finite or not negative-definite (scale parameter driven to a limit); FAILURE = line search failed and the mean field was handed back unchanged. This is the Laplace-on-scatter caveat (autofit/graphical/README.md §3.5, analytic_gaussian leg B) at 100 % on the lensing model: per-lens slope widths 0.002-0.05 are far tighter than the scatter prior, so the tilted density in sigma sits at the boundary.
+- 2026-09-24 — run — 343299 finished — wall 5:20: n25_scale_up: EP witness rerun on the decoupled --use_cpu script (slope_hierarchy_scale#4), JAX vectorised likelihood on the CPU backend, no pool, MAX_STEPS=2 via sbatch --export; stale 342410 output parked as ep_dead_342410 — COMPLETED 2026-09-16 02:03 BST, MAX_STEPS=2, 25 lenses, JAX-on-CPU, no pool: 50 'jit compiling vectorized' lines = one compile per factor search (2 steps x 25), confirming draft/refactor/autofit/ep_analysis_level_compile_cache.md. Every dataset factor SUCCESS (2/2 each) BUT the hierarchical factor is fully STALE: HierarchicalFactor0 50 rows = 27 BAD_PROJECTION + 23 FAILURE, zero SUCCESS (PriorFactor239 and PriorFactor263 also stale); library STALE FACTORS warning fired. Parent recovery printed exactly the prior: mean 2.0000 ± 0.7071, sigma 0.5000 ± 0.3536 (prior TG(2,1) / TG(0.5,0.5); truth sigma 0.1). EP output local at output/sample_n25_seed42/ep/expectation_propagation/a7ec60f07f4261afcb98b2adef484199/ (ep_history.csv, ep_diagnostics.results)
 - 2026-09-15 — run — 343299 submitted: n25_scale_up: EP witness rerun on the decoupled --use_cpu script (slope_hierarchy_scale#4), JAX vectorised likelihood on the CPU backend, no pool, MAX_STEPS=2 via sbatch --export; stale 342410 output parked as ep_dead_342410
 - 2026-09-15 — run — 342410 failed — wall 11:05: n25_scale_up: EP arm rerun on the JAX-on-CPU path (no multiprocessing pool); replaces 342351_0 — JAX-on-CPU path, no pool: 76 factor searches (~3 EP steps of 25) then LLVM JIT section memory exhausted 64 GB; aborted 2026-09-10 09:29 (slope_hierarchy_scale#3 phase; compile-memory bug filed against PyAutoFit)
 - 2026-09-09 — run — 342410 submitted: n25_scale_up — EP arm rerun on the JAX-on-CPU path (no multiprocessing pool); replaces 342351_0