From 4573cf46a9022c512e2f927f79c273eb07b2b792 Mon Sep 17 00:00:00 2001 From: Jammy2211 Date: Thu, 24 Sep 2026 14:59:20 +0200 Subject: [PATCH] =?UTF-8?q?cortex:=20EP=20check-in=202026-09-24=20?= =?UTF-8?q?=E2=80=94=20ep=5Ftoy=5Fgaussian=20wave=201=20failed,=20slope=5F?= =?UTF-8?q?hierarchy=5Fscale=20343299=20stale?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - ep_toy_gaussian: 342639 failed (one_by_one 50/50; NUTS OOM at 8 GB), 342640_[0-2] failed (EMFILE, fixed by PyAutoFit#1632/#1634); overhead profile note; Now = wave 2 awaits a human go under a new sample name. - slope_hierarchy_scale: 343299 finished (wall 5:20) with the hierarchical factor 0/50 SUCCESS (27 BAD_PROJECTION + 23 FAILURE); Now = moment-matching cure is the gate. Co-Authored-By: Claude Fable 5.1 --- projects/ep_toy_gaussian.md | 8 ++++---- projects/slope_hierarchy_scale.md | 5 +++-- 2 files changed, 7 insertions(+), 6 deletions(-) diff --git a/projects/ep_toy_gaussian.md b/projects/ep_toy_gaussian.md index 7043f6a..9a89208 100644 --- a/projects/ep_toy_gaussian.md +++ b/projects/ep_toy_gaussian.md @@ -5,16 +5,16 @@ Issue: ep_toy_gaussian#1 ## Now -n50_seed42 is on RAL: NUTS + one-by-one as 342639, EP x3 on identical data as 342640_[0-2]. The one-by-one leg finished (50/50 in 319 s); the EP repeats are turning per-factor Dynesty fits. -Next: pull and run scripts/compare.py --sample n50_seed42; settle first whether a prior factor with only NO_CHANGE rows counts as zero SUCCESS updates. +Wave 1 (n50_seed42: 342639 + 342640_[0-2]) failed on infrastructure — NUTS OOM at the 8 GB cap, EP x3 EMFILE (fixed upstream by PyAutoFit#1632/#1634, RAL mirror now a73684012). The collapse question at N=50 is unanswered. Only n5_smoke speaks so far: EP RECOVER 3/3 (parent sigma 10.14/9.96/9.83 vs truth 10, no collapse, no BIASED-TIGHT), NUTS 49.89/11.14. Next: wave 2 needs a human go, a NEW sample name (e.g. n50_seed42_w2 — never re-run into an existing sample directory) and the NUTS SBATCH memory raised above 8 GB. ## Runs -- 342639 — open — ral — 2026-09-10 — nuts_vs_ep_x3_n50: one_by_one + BlackJAX NUTS, partition ral -- 342640 — open — ral — 2026-09-10 — nuts_vs_ep_x3_n50: EP x3 on identical data, array 0-2 ## Log +- 2026-09-24 — note — per-search overhead profile on this toy (2026-09-24, laptop, N=5, max_steps=3): 2.45 s per Dynesty factor search, ~1.47 s dynesty run_nested + ~0.98 s autofit wrapper (plots, a redundant second run_nested pass, samples writes); inside run_nested instance_from_vector is 57 % of the likelihood callback for a 1-parameter model. Speed-up prompt filed in the Mind: draft/refactor/autofit/ep_factor_search_wrapper_overhead.md +- 2026-09-24 — run — 342640 failed — wall 0:14: nuts_vs_ep_x3_n50: EP x3 on identical data, array 0-2 — all three array tasks [0-2] CRASH OSError [Errno 24] Too many open files after ~126 Dynesty factor fits each (wall ~860-880 s), no ep_history.csv; results/n50_seed42/comparison.md scores WITNESS NOT MET 0/7 on infrastructure, not science. Cause: RAL mirror PyAutoFit e354dbb6b built a Pool(1) per single-core Dynesty fit (~+8 fds per fit vs 1024 soft ulimit); fixed by PyAutoFit#1632 (merged 2026-09-15) + #1634 (merged 2026-09-17); RAL mirror now a73684012 (2026-09-19) contains both; local check 2026-09-24 on a73684012: fd count flat at 7 across 15 EP factor searches (N=5, max_steps=3) +- 2026-09-24 — run — 342639 failed: nuts_vs_ep_x3_n50: one_by_one + BlackJAX NUTS, partition ral — one_by_one completed 50/50 in 319 s (naive parent 52.470 ± 1.260, deconvolved scatter 3.142); NUTS leg OUT_OF_MEMORY at the 8 GB SBATCH cap during window adaptation, no nuts.json - 2026-09-10 — run — 342640 submitted: nuts_vs_ep_x3_n50 — EP x3 on identical data, array 0-2 - 2026-09-10 — run — 342639 submitted: nuts_vs_ep_x3_n50 — one_by_one + BlackJAX NUTS, partition ral - 2026-09-10 — note — question: Do three EP fits agree with NUTS at N=50 [archive/tasks/ep_toy_gaussian/nuts_vs_ep_x3_n50.md] diff --git a/projects/slope_hierarchy_scale.md b/projects/slope_hierarchy_scale.md index c0823d0..778be52 100644 --- a/projects/slope_hierarchy_scale.md +++ b/projects/slope_hierarchy_scale.md @@ -5,17 +5,18 @@ Issue: slope_hierarchy_scale#2 ## Now -N=25 scale-up: graphical array 342348_[0-24] and joint fit 342350_0 still open; EP witness rerun 343299 (MAX_STEPS=2, --use_cpu on the decoupled script from slope_hierarchy_scale#4) submitted 2026-09-15 after 342410 died at ~EP step 3 on LLVM compile memory. Next: pull 343299 and confirm the first factor_step ran JAX-vectorised with no pool; the full EP arm waits on the PyAutoFit compile-memory fix (draft/bug/autofit/ep_re_jit_compiles_the_vmapped_likelihood.md). +343299 (EP witness rerun, MAX_STEPS=2) completed: dataset factors all SUCCESS and JAX-vectorised with one compile per factor search, but the hierarchical factor never updated (0/50 SUCCESS: 27 BAD_PROJECTION + 23 FAILURE), so the parent came back as the prior. The EP arm cannot estimate the slope scatter with the Laplace projection; the moment-matching cure (draft/feature/autofit/ep_hierarchical_scatter_moment_matching.md) is the gate — human decision. Still open: graphical array 342348_[0-24], joint fit 342350_0, EP arm 342351_0. ## Runs - 342348_[0-24] — open — gpu — 2026-09-08 — n25_scale_up: one lens per array task, hpc/batch_gpu/submit_one_by_one, sample_n25_seed42 - 342350_0 — open — gpu — 2026-09-08 — n25_scale_up: joint hierarchical NUTS fit, hpc/batch_gpu/submit_graphical - 342351_0 — open — ral — 2026-09-08 — n25_scale_up: EP arm on the CPU partition, hpc/batch_cpu/submit_ep, max_steps 12; RAL PyAutoFit mirror 68ff9bd57 predates PyAutoFit#1580, so this arm runs without the stale-mask fixed-point fix -- 343299 — open — ral — 2026-09-15 — n25_scale_up: EP witness rerun on the decoupled --use_cpu script (slope_hierarchy_scale#4), JAX vectorised likelihood on the CPU backend, no pool, MAX_STEPS=2 via sbatch --export; stale 342410 output parked as ep_dead_342410 ## Log +- 2026-09-24 — note — 343299 finding: BAD_PROJECTION = Hessian at the mode not finite or not negative-definite (scale parameter driven to a limit); FAILURE = line search failed and the mean field was handed back unchanged. This is the Laplace-on-scatter caveat (autofit/graphical/README.md §3.5, analytic_gaussian leg B) at 100 % on the lensing model: per-lens slope widths 0.002-0.05 are far tighter than the scatter prior, so the tilted density in sigma sits at the boundary. +- 2026-09-24 — run — 343299 finished — wall 5:20: n25_scale_up: EP witness rerun on the decoupled --use_cpu script (slope_hierarchy_scale#4), JAX vectorised likelihood on the CPU backend, no pool, MAX_STEPS=2 via sbatch --export; stale 342410 output parked as ep_dead_342410 — COMPLETED 2026-09-16 02:03 BST, MAX_STEPS=2, 25 lenses, JAX-on-CPU, no pool: 50 'jit compiling vectorized' lines = one compile per factor search (2 steps x 25), confirming draft/refactor/autofit/ep_analysis_level_compile_cache.md. Every dataset factor SUCCESS (2/2 each) BUT the hierarchical factor is fully STALE: HierarchicalFactor0 50 rows = 27 BAD_PROJECTION + 23 FAILURE, zero SUCCESS (PriorFactor239 and PriorFactor263 also stale); library STALE FACTORS warning fired. Parent recovery printed exactly the prior: mean 2.0000 ± 0.7071, sigma 0.5000 ± 0.3536 (prior TG(2,1) / TG(0.5,0.5); truth sigma 0.1). EP output local at output/sample_n25_seed42/ep/expectation_propagation/a7ec60f07f4261afcb98b2adef484199/ (ep_history.csv, ep_diagnostics.results) - 2026-09-15 — run — 343299 submitted: n25_scale_up: EP witness rerun on the decoupled --use_cpu script (slope_hierarchy_scale#4), JAX vectorised likelihood on the CPU backend, no pool, MAX_STEPS=2 via sbatch --export; stale 342410 output parked as ep_dead_342410 - 2026-09-15 — run — 342410 failed — wall 11:05: n25_scale_up: EP arm rerun on the JAX-on-CPU path (no multiprocessing pool); replaces 342351_0 — JAX-on-CPU path, no pool: 76 factor searches (~3 EP steps of 25) then LLVM JIT section memory exhausted 64 GB; aborted 2026-09-10 09:29 (slope_hierarchy_scale#3 phase; compile-memory bug filed against PyAutoFit) - 2026-09-09 — run — 342410 submitted: n25_scale_up — EP arm rerun on the JAX-on-CPU path (no multiprocessing pool); replaces 342351_0