diff --git a/campaigns.yaml b/campaigns.yaml index ec0606c..a5a1d61 100644 --- a/campaigns.yaml +++ b/campaigns.yaml @@ -1,6 +1,6 @@ version: 1 -last_checkin: '2026-10-17T11:18:53Z' -last_checkin_report: checkins/2026-10-17T111853Z.md +last_checkin: '2026-10-18T17:00:00Z' +last_checkin_report: checkins/2026-10-18T170000Z.md campaigns: - id: slam-hst-base title: HST SLaM base backend parity @@ -35,23 +35,17 @@ campaigns: project: autolens_inference checkin_report: checkins/2026-10-17T111853Z.md - id: hst-imaging-mass-samplers - title: 'HST imaging, clean slate: Nautilus base on A100 and jax_cpu at a pinned release, mass stage with linear + title: 'HST imaging, clean slate: Nautilus base on A100, jax_cpu and numba_cpu at a pinned release, mass stage with linear lens light and with it fixed, then other samplers on the mass stage' - status: needs-slicing - updated: '2026-10-17' - next: 'Phase 1 via Mind start_dev: lens-light mode as a run variant (--lens-light linear|fixed|both, fixed = ordinary - profiles with intensities from the light[1] fit), both rows from one chain, --export-prepared for both modes, - library revisions and SLURM job id recorded in every row, submit scripts for A100 dense/sparse and ral jax_cpu - dense/sparse against a private pinned checkout; then phase 2 compute by separate authorisation.' - recent_progress: 'Opened 2026-10-17 as a fresh start by human decision: the base is re-measured from scratch on - PyAuto release 2026.10.17.1 (PyAutoLens tag v2026.10.17.1, released 2026-10-17) in fresh output trees; the - 2026-09-24 rows and both earlier campaigns are superseded history.' - blockers: Phase 1 runner change not started (Mind start_dev pending); no pinned private checkout exists on RAL - yet; compute not authorised until phase 1 ships. - job_status: none submitted - evidence: https://github.com/PyAutoLabs/autolens_inference/pull/23 + status: running + updated: '2026-10-18' + next: '(1) Human decision on filing a Mind bug task for the jax_gpu sparse fixed-light mass_total[1] divergence (both seeds) and reproducing it on the laptop from the exported prepared problem, before any CPU or sampler wave uses the sparse fixed-light problem. (2) Human authorisation for GPU wave 1 attempt 2 (nautilus+emcee, --array=0-11%4) after git pull of autolens_inference on the RAL login node to ca505be. (3) numba_cpu dense DROPPED by human decision 2026-10-18 (decisions/hst-cleanslate-drop-numba-dense.md): no rerun. (4) Wait for 412539, 412541 and 412565 to finish (jax_cpu sparse source_pix[1] near its end), then pull and read. (5) Then start_dev the filed Mind prompt draft/refactor/autolens_inference/flatten_cleanslate_output_layout.md (Mind 734e1bf8; human 2026-10-18: flat output/autolens/imaging/... layout, no dated folder; blocked until those arrays finish).' + recent_progress: 'Read 2026-10-18 (checkins/2026-10-18T170000Z.md). A100 base complete: 412535 (dense) and 412537 (sparse) rows for both light modes and seeds read. Dense fixed-light mass_total[1] is the best-recovered row (worst abs delta-sigma 0.24 / 0.34). Sparse fixed-light mass_total[1] diverged on both seeds (log Z -43567 / -43202, max logL ~-43500 vs ~+31649 linear; slope pinned at 2.96); observation recorded in Cortex, human reading pending. jax_cpu dense seed 1 linear row written (chain ~21 h vs ~40 min on A100; worst 1.29 sigma). numba_cpu dense 412563 FAILED at the first source_pix[1] batch (two Nautilus pool workers SIGSEGV, MaxRSS 356 MB, not OOM). Wave-1 fix merged as autolens_inference#34 (ca505be) 2026-10-17T19:25Z; attempt 2 not submitted.' + blockers: 'Sparse fixed-light mass-stage divergence unexplained (needs reproduction); wave 1 attempt 2 needs RAL checkout pull (b91ce9c -> ca505be) and human authorisation; numba_cpu dense dropped 2026-10-18 (human decision), no longer a blocker.' + job_status: 'Read 2026-10-18 ~16:55Z (sacct/squeue by the architect), pin 2026.10.17.1. 412535 (jax_gpu dense) and 412537 (jax_gpu sparse) COMPLETED 0:0 both seeds. 412539 (jax_cpu dense) RUNNING 23 h: seed 1 linear row written, seed 1 on fixed mass_total[1], seed 0 in linear mass_total[1]. 412541 (jax_cpu sparse) RUNNING 23 h, both seeds in source_pix[1], no row. 412563 (numba_cpu dense) FAILED 1:0 both seeds after 00:00:57 (pool-worker SIGSEGV; failed rows). 412565 (numba_cpu sparse) RUNNING 21 h, both seeds in source_pix[1], no row. 412543 wave 1 attempt 1: 18 arms FAILED, 18-23 cancelled; attempt 2 not submitted.' + evidence: https://github.com/PyAutoLabs/autolens_inference/pull/34 project: autolens_inference - checkin_report: checkins/2026-10-17T111853Z.md + checkin_report: checkins/2026-10-18T170000Z.md - id: point-source-admission title: Point-source sampler admission status: needs-decision @@ -231,22 +225,20 @@ tasks: campaign: hst-imaging-mass-samplers title: 'HST imaging mass stage, clean slate: Nautilus base on A100 and jax_cpu at release 2026.10.17.1, linear and fixed lens light' - status: needs-slicing + status: running priority: high path: tasks/hst_mass_base_light_modes.md - updated: '2026-10-17' - next: 'Phase 1 via Mind start_dev: --lens-light linear|fixed|both, both rows from one chain, --export-prepared - for both modes, library revisions and SLURM job id in every row, A100 and ral jax_cpu submit scripts against - a private pinned checkout; no compute until phase 1 ships and phase 2 is authorised.' - evidence: https://github.com/PyAutoLabs/autolens_inference/pull/23 + updated: '2026-10-18' + next: 'A100 rows of 412535 and 412537 (both light modes, seeds 0-1) read 2026-10-18; jax_gpu sparse fixed-light mass_total[1] diverged on both seeds (Cortex observation, human reading pending). 412539 (jax_cpu dense) has the seed-1 linear row; 412541 (jax_cpu sparse) and 412565 (numba_cpu sparse) RUNNING in source_pix[1] at ~16:55Z; 412563 (numba_cpu dense) FAILED (pool-worker SIGSEGV, not OOM), rerun with --cores 1 or NUMBA_THREADING_LAYER=workqueue needs a human decision. Pull and read 412539, 412541 and 412565 when they finish.' + evidence: https://github.com/PyAutoLabs/autolens_inference/pull/26 + checkin_report: checkins/2026-10-18T170000Z.md - id: hst_mass_samplers_wave1 campaign: hst-imaging-mass-samplers title: 'HST imaging mass stage: sampler comparison, wave 1' - status: needs-slicing + status: blocked priority: normal path: tasks/hst_mass_samplers_wave1.md - updated: '2026-10-17' - next: Depends on the frozen mass_total[1] problems (both light modes, A100 and jax_cpu) from the pinned base - in hst_mass_base_light_modes; candidate samplers from the experiments runner (nautilus, emcee, nuts, smc) and - the promoted PyAutoFit catalogue (NSS, DynestyStatic/Dynamic, Zeus) are not yet sliced. - evidence: https://github.com/PyAutoLabs/autolens_inference/pull/23 + updated: '2026-10-18' + next: 'Fix merged as autolens_inference#34 (ca505be, 2026-10-17T19:25Z): row writer keyed by dotted path, --attempt, --sampler-filter, NUTS/SMC deferred, GPU submit defaults attempt 2 nautilus+emcee --array=0-11%4 with no dependency. Attempt 2 NOT submitted; blocked on git pull of autolens_inference on the RAL login node (at b91ce9c) to ca505be and human authorisation. Attempt-1 NUTS/SMC rows exist on RAL as failed: RESOURCE_EXHAUSTED (385.32 GiB / 48.45 GiB); no nautilus/emcee rows.' + evidence: https://github.com/PyAutoLabs/autolens_inference/pull/34 + checkin_report: checkins/2026-10-18T170000Z.md diff --git a/checkins/2026-10-18T170000Z.md b/checkins/2026-10-18T170000Z.md new file mode 100644 index 0000000..b917596 --- /dev/null +++ b/checkins/2026-10-18T170000Z.md @@ -0,0 +1,111 @@ +# Inference check-in — 2026-10-18T17:00:00Z + +Previous: checkins/2026-10-17T111853Z.md +Scope: all open campaigns; optional user focus "Pull and read all results in would expect stuff like numba to have begun by now?" +User decisions: none. No compute was authorised in this chat; nothing was submitted, cancelled or requeued. + +## Sources inspected + +| Source | Commit / query time | Evidence | Availability | +|---|---|---|---| +| RAL scheduler (`sacct`/`squeue`) for 412535, 412537, 412539, 412541, 412543, 412563, 412565 | 2026-10-18 ~16:55Z | queried by the session architect over SSH | verified | +| RAL result rows and stage outputs | 2026-10-18 ~16:55Z | `results/` of the autolens_inference RAL checkout (rows fetched directly) | verified | +| autolens_inference RAL checkout | `b91ce9c` | before the wave-1 fix | verified; behind main | +| autolens_inference main | `ca505be` (autolens_inference#34 merged 2026-10-17T19:25Z) | https://github.com/PyAutoLabs/autolens_inference/pull/34 | verified | +| Laptop `hpc/sync pull` of output/ | started 2026-10-18 ~16:20Z | still transferring (~300 kB/s) | incomplete; rows read from RAL instead | +| Release pin | 2026.10.17.1 | rows carry `release_pin` 2026.10.17.1 | verified | +| PyAutoCortex `projects/autolens_inference.md` | origin/main at sweep time | observation entry added through a Cortex PR (see handoff) | verified | +| autofit_inference, graphical_ep, analytic_gaussian, ep_toy_gaussian, slope_hierarchy_scale, ic50_workspace, point-source records | not re-inspected this sweep | ledger state retained | not re-inspected | + +## Campaign sweep + +| Campaign | Change since previous | Coordination state | Job evidence | Blocker | Next bounded step | +|---|---|---|---|---|---| +| hst-imaging-mass-samplers | 412535 and 412537 (A100 dense/sparse) COMPLETED and all 8 rows read; 412539 seed 1 linear row written; numba dense 412563 FAILED (pool-worker SIGSEGV); wave-1 fix #34 merged; sparse fixed-light mass stage diverged on both seeds | running | sacct 2026-10-18 ~16:55Z: 412535, 412537 COMPLETED 0:0; 412539, 412541, 412565 RUNNING; 412563 FAILED 1:0; 412543 attempt 1 FAILED/cancelled | Sparse fixed-light divergence unexplained; wave 1 attempt 2 needs RAL `git pull` to `ca505be` and human authorisation; numba dense rerun needs a human decision | Human reads the divergence observation (Cortex) and decides on a Mind bug task | +| slam-hst-base | No new evidence inspected | superseded | not queried | Unchanged | Unchanged | +| slam-hst-delaunay | No new evidence inspected | superseded | not queried | Unchanged | Unchanged | +| point-source-admission | No new evidence inspected | needs-decision | not queried | Unchanged | Unchanged | +| graphical-ep | No new evidence inspected | needs-slicing | not queried | Unchanged | Unchanged | +| analytic-gaussian | No new evidence inspected | blocked | not queried | Unchanged | Unchanged | +| ep-toy-gaussian | No new evidence inspected | needs-decision | not queried | Unchanged | Unchanged | +| slope-hierarchy-scale | No new evidence inspected | blocked | not queried | Unchanged | Unchanged | +| ic50-scale | No new evidence inspected | blocked | not queried | Unchanged | Unchanged | +| gaussian-x3-search-benchmark | No new evidence inspected | running | not queried | Unchanged | Unchanged (B5c-2b) | + +### hst-imaging-mass-samplers job detail (sacct/squeue 2026-10-18 ~16:55Z) + +| Job | Leg | State | Elapsed / MaxRSS | Rows | +|---|---|---|---|---| +| 412535 | jax_gpu dense (gpu) | COMPLETED 0:0 both seeds | 01:00:33 / 00:58:35, 12.8 GB | both light modes, both seeds, read | +| 412537 | jax_gpu sparse (gpu) | COMPLETED 0:0 both seeds | 01:44:19 / 01:36:15, 7.5 GB | both light modes, both seeds, read | +| 412539 | jax_cpu dense (ral, 8 CPU) | RUNNING since 2026-10-17T17:46Z (23 h) | — | seed 1 linear row (complete; mass_total[1] finished 15:05Z), now on fixed-light mass_total[1]; seed 0 in linear mass_total[1] since 08:37Z (~28 300 calls, f_live 0.036) | +| 412541 | jax_cpu sparse (ral, 8 CPU) | RUNNING 23 h | — | none; both seeds in source_pix[1] since 2026-10-17T18:47Z (~33 300–33 500 calls, f_live 0.018–0.022, N_eff ~3 100; the A100 sparse source_pix[1] ended at ~31 000–33 000 calls) | +| 412563 | numba_cpu dense (ral, 8 CPU, 48 GB) | FAILED 1:0 both seeds after 00:00:57 | MaxRSS 356 MB | `failed: SearchException ...` rows carrying only the shared source_lp[1] stage | +| 412565 | numba_cpu sparse (ral, 8 CPU, 64 GB) | RUNNING 21 h | — | none; both seeds in source_pix[1] (~28 360 calls, f_live 0.036 / 0.134) | +| 412543 | GPU wave 1 attempt 1 | 18 started arms FAILED, arms 18–23 cancelled (recorded 2026-10-17) | — | 8 NUTS/SMC rows `failed: RESOURCE_EXHAUSTED` (385.32 GiB NUTS, 48.45 GiB SMC); no nautilus/emcee rows | + +412539 seed-0 stage mtimes: source_lp 19:59Z (10-17), source_pix[1] 05:25Z, source_pix[2] 07:04Z, light[1] 08:37Z, mass_total[1] 08:37Z (10-18). + +On the user's question: the numba legs did begin. The in-job wait for the shared jax_cpu dense source_lp[1] passed (it completed 18:58Z on 10-17). The sparse leg 412565 has been sampling source_pix[1] for 21 h. The dense leg 412563 started source_pix[1] at 19:59Z and its first likelihood batch killed two Nautilus pool workers with SIGSEGV (exit -11). MaxRSS was 356 MB, so this was not an out-of-memory kill. PyAutoFit's `_check_workers_alive` then failed the fit with a SearchException. The sparse leg runs the same pool without crashing, so the fault is specific to the dense numba likelihood under forked workers. + +### Base rows read (status complete, release_pin 2026.10.17.1; stage `wall_s` in s: source_lp / source_pix[1] / source_pix[2] / light[1] / mass_total[1]) + +| Leg / seed | Stage walls | mass_total[1] linear: worst abs Δσ, log Z | mass_total[1] fixed: worst abs Δσ, log Z | +|---|---|---|---| +| jax_gpu dense / 0 | 715 / 701 / 181 / 234 / linear 559, fixed 490 | 0.66 (slope), 31601.9 | 0.24, 31601.0 | +| jax_gpu dense / 1 | 712 / 733 / 130 / 220 / linear 525, fixed 450 | 2.92 (einstein_radius), 2.65 (gamma_2), 2.63 (slope); 31591.7 | 0.34, 31592.7 | +| jax_gpu sparse / 0 | 715 / 1786 / 298 / 222 / linear 827, fixed 1625 | 1.22, 31596.5 | DIVERGED: log Z −43567.1, max logL −43501.7 | +| jax_gpu sparse / 1 | 720 / 1649 / 226 / 197 / linear 812, fixed 1397 | 0.96, 31598.4 | DIVERGED: log Z −43201.6 | +| jax_cpu dense / 1 | 3411 / 38642 / 4042 / 5703 / linear 24129 | 1.29 (einstein_radius), 31593.0, 22 020 evals | not yet run | + +The jax_cpu dense seed-1 linear chain took about 21 h against about 40 min on the A100. The two light modes share stages 1–4, so a per-mode chain wall is stages 1–4 plus that mode's mass_total[1]; the two mass_total[1] walls must not be summed into one chain. + +The sparse fixed-light divergence: at seed 0 the fit used 48 700 evaluations and its posterior pinned at slope 2.964±0.0004, einstein_radius 2.493, gamma_1 0.0994, ell_comps_0 0.59 (truth_delta_sigma slope 2250, einstein_radius 114, gamma_1 289). Seed 1 diverged the same way: slope 2.963±0.0002, einstein_radius 2.427, gamma_1 0.0998, ell_comps_0 0.57 (truth_delta_sigma slope 6275). The max logL of about −43 500 compares with about +31 649 for the linear mode on the same stages 1–4. + +## Coverage and limits + +- `status: complete` is execution completion only. It does not establish convergence or scientific acceptance. The divergent sparse fixed-light rows are `complete` too. +- numba_cpu dense has no measurement: both seeds failed at the first source_pix[1] batch. +- jax_cpu dense has one row (seed 1, linear). jax_cpu sparse and numba_cpu sparse have none yet. No CPU fixed-light row exists. +- Wave 1 has no nautilus/emcee rows: the attempt-1 row writer crashed. The NUTS/SMC rows record only the OOM. +- The laptop `hpc/sync pull` of output/ was incomplete at the time of the sweep. Rows were read on RAL, and sample archives were not inspected locally. +- The RAL checkout is at `b91ce9c`, before the wave-1 fix `ca505be`. +- Six campaigns outside autolens_inference were not re-inspected, and their ledger rows are unchanged. +- The sparse fixed-light divergence is recorded in Cortex as an observation awaiting the human's reading. This report draws no conclusion from it. + +## Decisions and handoff + +Proposals (not authorised): + +1. File a Mind bug task for the sparse fixed-light divergence and reproduce it on the laptop from the exported prepared problem, before any CPU or sampler wave uses the sparse fixed-light problem. +2. Wave 1 attempt 2 (nautilus+emcee, `--attempt 2`, `--array=0-11%4`, no dependency; NUTS/SMC deferred): `git pull` autolens_inference on the RAL login node to `ca505be`, then submit. Needs human authorisation. +3. numba_cpu dense rerun: either `--cores 1` (serial numba, where Nautilus runs no pool) or `NUMBA_THREADING_LAYER=workqueue` set before the fork. Needs a human decision. +4. Wait for 412539, 412541 and 412565 to finish (jax_cpu sparse source_pix[1] is close to its end), then pull them. + +Authorised actions this sweep: this report, the campaigns.yaml update and the Cortex observation entry. No Mind issue was opened. + +Compute submitted: none. + +## Addendum 2026-10-18 (after the sweep) +User decision (verbatim, inference chat): "lets just drop dense numba we would never use it." +Recorded as decisions/hst-cleanslate-drop-numba-dense.md; proposal 3 above is withdrawn. +The campaign's CPU base is now jax_cpu dense, jax_cpu sparse and numba_cpu sparse. No compute submitted. + +## Addendum 2026-10-18 (evening, human-directed housekeeping) +- `hpc/sync pull` of autolens_inference completed on the laptop (second attempt; the first was + cut at 20 % by a 25-minute wrapper timeout, so never wrap a pull in a short timeout). + `.cortex/pull.json` records 6 live checkpoints (412539, 412541, 412565). +- Human decision (verbatim): "Science/inference folder ... should just be a clean slate ... I think + not having archive is ok. I guess all this stuff can go somewhere safe on RAL but out of the way". + Done with the human's go: on RAL, `output/slam` (2026-09-24 chains) and `output/archive` moved to + `/mnt/ral/jnightin/inference/aside/autolens_inference_pre_cleanslate/`; the project's `output/` + now holds only `cleanslate_2026_10_17`, `prepared`, `experiments` and `experiment_sources` (the + last two are the cs1017 wave-1 arms). The laptop mirror of that clean-slate tree is + `/mnt/c/Users/Jammy/Science/inference/autolens_inference/output` (1.3 GB, file-for-file equal to + the checkout's clean-slate output); the checkout's `output/` becomes a symlink to it. The pull + root stays the checkout so `results/` and `logs/` keep landing there. +- Flattening `cleanslate_2026_10_17/` is deferred: 412539, 412541 and 412565 write into it, every + row's `output_path` and every clean-slate submit names it. Proposed Mind task once they finish. +- The pulled clean-slate rows (jax_gpu dense/sparse both modes, jax_cpu dense seed 1 linear, + numba dense failed rows, 8 failed NUTS/SMC experiment rows) are untracked in the autolens_inference + checkout; committing them is the project's own PR, not done here. diff --git a/dashboard.html b/dashboard.html index 1242ae0..1f8db47 100644 --- a/dashboard.html +++ b/dashboard.html @@ -369,7 +369,7 @@ - PyAutoInsight: https://github.com/PyAutoLabs/PyAutoInsight - autolens_inference: https://github.com/PyAutoLabs/autolens_inference - autofit_inference: https://github.com/PyAutoLabs/autofit_inference
Last updated 2026-10-19 19:12:20 UTC
↻ Update

-

Last check-in: 2026-10-17T11:18:53Z.

Active campaigns

CampaignRecent progressNext steps
HST imaging, clean slate: Nautilus base on A100 and jax_cpu at a pinned release, mass stage with linear lens light and with it fixed, then other samplers on the mass stageOpened 2026-10-17 as a fresh start by human decision: the base is re-measured from scratch on PyAuto release 2026.10.17.1 (PyAutoLens tag v2026.10.17.1, released 2026-10-17) in fresh output trees; the 2026-09-24 rows and both earlier campaigns are superseded history.Phase 1 via Mind start_dev: lens-light mode as a run variant (--lens-light linear|fixed|both, fixed = ordinary profiles with intensities from the light[1] fit), both rows from one chain, --export-prepared for both modes, library revisions and SLURM job id recorded in every row, submit scripts for A100 dense/sparse and ral jax_cpu dense/sparse against a private pinned checkout; then phase 2 compute by separate authorisation.
Point-source sampler admission2 October project record reports five Nautilus seeds; no gradient-sampler admission measurement.Assess a bounded gradient-sampler admission proposal with the human; no new job is authorized.
Graphical and EP inference coordinationMind campaign map and two scoping contracts preserved verbatim; implementation work remains in Mind.Reconcile current Cortex observations and shipped fixes, then select one bounded phase.
Analytic Gaussian ensemble and control calibrationCortex records accepted wave-1 baseline and 30 September independent-review caveats.Use Cortex planned diagnostic on passing, edge and worst seeds; retain the human conclusion there.
N=50 NUTS versus three EP repeatsCortex records failed wave 1 (EMFILE/OOM), wave-2 preparation and no submission.Present the existing wave-2 proposal with current dependency evidence for decision.
Hierarchical slope recovery at scaleCortex retains graphical baseline, post-processing OOM and stale EP hierarchical factor; Mind records later moments fix.Verify the moments fix release and project mirror; recover missing summaries from existing samples before a proposed rerun.
IC50 EP and graphical scale ladderCortex records N=5/10/25 results, failed N=50 EP rung and width-interpretation caveat.Check implementation close-outs, then propose a seeded ladder with appropriate output and mean-field diagnostics.
PyAutoFit search benchmark on gaussian_x3 (autofit_inference)Wave 2 closed (B5b-2, 2026-10-15, autofit_inference#12) — RAL 409930 pulled, 12,750 rows admitted at the pin df6d218f10cd with 0 missing; NSS 50-step leads both evidence rankings; the catalogue ranks only legs with success rate >= 50 % over >= 20 attempts (post-hoc presentation convention); fit records published as inference-summary@3 (101 record shards).B5c-2b (draft/research/autofit_inference/search_ext_b5c2b_scored_wave_submit.md): realisations 2-3 and their exact references, the correlated_nd@2 scored-wave manifest (83 cells x 50 seeds x 3 realisations, 12 450 runs) and its probe for 8 probe-first cells, then the submit on RAL --partition=ral from a private pinned copy with the A2 memory requests; correlated_nd@2 is frozen (autofit_inference 44c5e77f1fb0); the family is catalogued apart from gaussian_x3 and ranks nothing across families.

Active tasks

TaskCampaignStatusPriorityNext steps
Drop the 1e-3 prior centring for shear / multipole / ell_comps in the benchmarkslam-hst-baseblockednormalVerify the release containing Galaxy#634 and Lens#754 before changing priors; retain zero-centre gradient witness. Carried into hst-imaging-mass-samplers phase 1 as a prior-identity question for the frozen model.
EP campaign — phase map for the 2026 Q3 graphical/EP pushgraphical-epneeds-decisionhighReconcile current Cortex records and human gates; this campaign map is never issued or bulk scheduled.
Expectation Propagation Scale-Up — Scopinggraphical-epneeds-slicinghighReconcile completed wrapper work and Pulse profiling ownership before selecting a bounded remaining inference phase.
Graphical Model Scale-Up — Scopinggraphical-epneeds-slicinghighReconcile current Cortex scale results and library dependencies; select one bounded inference phase with a witness.
NUTS/HMC trial on validated likelihoodspoint-source-admissionneeds-decisionnormalAn unscoped idea, not permission to submit: reconcile point-source admission with imaging and weak-lensing candidate targets.
gaussian_x3 search benchmark — wave 1 (the pilot)gaussian-x3-search-benchmarkblockednormalPilot stopped at 223 of 520 runs and is not restarted; freeze gaussian_x3@2 from its 17-row calibration before any scored run; the deferred cells move to scored wave 2 (after A2 and A4).
gaussian_x3 search benchmark — wave 2 (scored)gaussian-x3-search-benchmarkrunninghighWave 2 is closed and judged (B5b-2); the fit snapshot is captured from main under inference-summary@3 after the B5b-2 PRs merge.
HST imaging mass stage, clean slate: Nautilus base on A100 and jax_cpu at release 2026.10.17.1, linear and fixed lens lighthst-imaging-mass-samplersneeds-slicinghighPhase 1 via Mind start_dev: --lens-light linear|fixed|both, both rows from one chain, --export-prepared for both modes, library revisions and SLURM job id in every row, A100 and ral jax_cpu submit scripts against a private pinned checkout; no compute until phase 1 ships and phase 2 is authorised.
HST imaging mass stage: sampler comparison, wave 1hst-imaging-mass-samplersneeds-slicingnormalDepends on the frozen mass_total[1] problems (both light modes, A100 and jax_cpu) from the pinned base in hst_mass_base_light_modes; candidate samplers from the experiments runner (nautilus, emcee, nuts, smc) and the promoted PyAutoFit catalogue (NSS, DynestyStatic/Dynamic, Zeus) are not yet sliced.

Ledger

Inference Results

PyAutoFit

Integrity: ok · Freshness: freshness policy unspecified · evidence 2026-10-15T16:38:44Z · Scientific qualification: 5129/12973 accepted by producer; remaining assessments shown below

Project dashboard · Ingest receipt

Record shards

inference-summary@3: the records are held in one file per protocol and leg, each verified against the index's sha256 at the captured commit. Counts are the producer's assertions, shown without ranking; each shard file holds every record's diagnostics, timings and provenance.

ProtocolShardRecordsExecutionConvergenceAcceptanceEvidence
gaussian_x3@1gaussian_x3@1/gaussian_x3_blend/bfgs/default/local_jax_cpu_fp64_cache_warm10complete 2, failed 8not_assessed 10not_assessed 10records
gaussian_x3@1gaussian_x3@1/gaussian_x3_blend/bfgs/default/local_numpy_fp6410failed 10not_assessed 10rejected 10records
gaussian_x3@1gaussian_x3@1/gaussian_x3_blend/blackjax_nuts/warmup_200_samples_500/local_jax_cpu_fp64_cache_warm2failed 2not_converged 2not_assessed 2records
gaussian_x3@1gaussian_x3@1/gaussian_x3_blend/blackjax_nuts_warm/warmup_200_samples_500/local_jax_cpu_fp64_cache_warm1failed 1not_converged 1not_assessed 1records
gaussian_x3@1gaussian_x3@1/gaussian_x3_blend/dynesty_dynamic/default/local_jax_cpu_fp64_cache_warm2complete 2not_assessed 2not_assessed 2records
gaussian_x3@1gaussian_x3@1/gaussian_x3_blend/dynesty_dynamic/default/local_numpy_fp642complete 2not_assessed 2accepted 2records
gaussian_x3@1gaussian_x3@1/gaussian_x3_blend/dynesty_static/nlive_200/local_jax_cpu_fp64_cache_warm2complete 2not_assessed 2not_assessed 2records
gaussian_x3@1gaussian_x3@1/gaussian_x3_blend/dynesty_static/nlive_200/local_numpy_fp642complete 2not_assessed 2rejected 2records
gaussian_x3@1gaussian_x3@1/gaussian_x3_blend/emcee/walkers_50_steps_2000/local_jax_cpu_fp64_cache_warm2complete 1, failed 1not_converged 2not_assessed 2records
gaussian_x3@1gaussian_x3@1/gaussian_x3_blend/emcee/walkers_50_steps_2000/local_numpy_fp642complete 2not_converged 2rejected 2records
gaussian_x3@1gaussian_x3@1/gaussian_x3_blend/lbfgs/default/local_jax_cpu_fp64_cache_warm10complete 1, failed 9not_assessed 10not_assessed 10records
gaussian_x3@1gaussian_x3@1/gaussian_x3_blend/lbfgs/default/local_numpy_fp6410complete 2, failed 8not_assessed 10rejected 10records
gaussian_x3@1gaussian_x3@1/gaussian_x3_blend/multi_start_adabelief/starts_48_steps_300/local_jax_cpu_fp64_cache_warm10complete 10not_assessed 10not_assessed 10records
gaussian_x3@1gaussian_x3@1/gaussian_x3_blend/multi_start_adam/starts_48_steps_300/local_jax_cpu_fp64_cache_warm10complete 10not_assessed 10not_assessed 10records
gaussian_x3@1gaussian_x3@1/gaussian_x3_blend/multi_start_lion/starts_48_steps_300/local_jax_cpu_fp64_cache_warm10complete 10not_assessed 10not_assessed 10records
gaussian_x3@1gaussian_x3@1/gaussian_x3_blend/multi_start_prodigy/starts_48_steps_300/local_jax_cpu_fp64_cache_warm10complete 10not_assessed 10not_assessed 10records
gaussian_x3@1gaussian_x3@1/gaussian_x3_blend/nautilus/n_live_100/local_jax_cpu_fp64_cache_warm2complete 2converged 2not_assessed 2records
gaussian_x3@1gaussian_x3@1/gaussian_x3_blend/nautilus/n_live_100/local_numpy_fp642complete 2converged 2accepted 2records
gaussian_x3@1gaussian_x3@1/gaussian_x3_blend/nautilus/n_live_200/local_jax_cpu_fp64_cache_warm2complete 2converged 2not_assessed 2records
gaussian_x3@1gaussian_x3@1/gaussian_x3_blend/nautilus/n_live_200/local_numpy_fp642complete 1, failed 1converged 1, not_converged 1accepted 1, rejected 1records
gaussian_x3@1gaussian_x3@1/gaussian_x3_blend/nautilus/n_live_400/local_jax_cpu_fp64_cache_warm2complete 2converged 2not_assessed 2records
gaussian_x3@1gaussian_x3@1/gaussian_x3_blend/nautilus/n_live_400/local_numpy_fp642complete 2converged 2accepted 2records
gaussian_x3@1gaussian_x3@1/gaussian_x3_blend/smc/default/local_jax_cpu_fp64_cache_warm2complete 2not_assessed 2not_assessed 2records
gaussian_x3@1gaussian_x3@1/gaussian_x3_blend/zeus/walkers_50_steps_2000/local_jax_cpu_fp64_cache_warm1failed 1not_converged 1not_assessed 1records
gaussian_x3@1gaussian_x3@1/gaussian_x3_blend/zeus/walkers_50_steps_2000/local_numpy_fp642complete 1, failed 1not_converged 2rejected 2records
gaussian_x3@1gaussian_x3@1/gaussian_x3_separated/bfgs/default/local_jax_cpu_fp64_cache_warm10complete 10not_assessed 10rejected 10records
gaussian_x3@1gaussian_x3@1/gaussian_x3_separated/bfgs/default/local_numpy_fp6410complete 10not_assessed 10rejected 10records
gaussian_x3@1gaussian_x3@1/gaussian_x3_separated/blackjax_nuts/warmup_200_samples_500/local_jax_cpu_fp64_cache_warm2complete 2not_converged 2rejected 2records
gaussian_x3@1gaussian_x3@1/gaussian_x3_separated/blackjax_nuts_warm/warmup_200_samples_500/local_jax_cpu_fp64_cache_warm1complete 1not_converged 1accepted 1records
gaussian_x3@1gaussian_x3@1/gaussian_x3_separated/dynesty_dynamic/default/local_jax_cpu_fp64_cache_warm2complete 2not_assessed 2accepted 1, rejected 1records
gaussian_x3@1gaussian_x3@1/gaussian_x3_separated/dynesty_dynamic/default/local_numpy_fp642complete 2not_assessed 2accepted 1, rejected 1records
gaussian_x3@1gaussian_x3@1/gaussian_x3_separated/dynesty_static/nlive_200/local_jax_cpu_fp64_cache_warm2complete 2not_assessed 2rejected 2records
gaussian_x3@1gaussian_x3@1/gaussian_x3_separated/dynesty_static/nlive_200/local_numpy_fp642complete 2not_assessed 2accepted 1, rejected 1records
gaussian_x3@1gaussian_x3@1/gaussian_x3_separated/emcee/walkers_50_steps_2000/local_jax_cpu_fp64_cache_warm2complete 2not_converged 2rejected 2records
gaussian_x3@1gaussian_x3@1/gaussian_x3_separated/emcee/walkers_50_steps_2000/local_numpy_fp642complete 2not_converged 2rejected 2records
gaussian_x3@1gaussian_x3@1/gaussian_x3_separated/lbfgs/default/local_jax_cpu_fp64_cache_warm10complete 10not_assessed 10rejected 10records
gaussian_x3@1gaussian_x3@1/gaussian_x3_separated/lbfgs/default/local_numpy_fp6410complete 10not_assessed 10rejected 10records
gaussian_x3@1gaussian_x3@1/gaussian_x3_separated/multi_start_adabelief/starts_48_steps_300/local_jax_cpu_fp64_cache_warm10complete 10not_assessed 10rejected 10records
gaussian_x3@1gaussian_x3@1/gaussian_x3_separated/multi_start_adam/starts_48_steps_300/local_jax_cpu_fp64_cache_warm10complete 10not_assessed 10rejected 10records
gaussian_x3@1gaussian_x3@1/gaussian_x3_separated/multi_start_lion/starts_48_steps_300/local_jax_cpu_fp64_cache_warm10complete 10not_assessed 10rejected 10records
gaussian_x3@1gaussian_x3@1/gaussian_x3_separated/multi_start_prodigy/starts_48_steps_300/local_jax_cpu_fp64_cache_warm10complete 10not_assessed 10accepted 10records
gaussian_x3@1gaussian_x3@1/gaussian_x3_separated/nautilus/n_live_100/local_jax_cpu_fp64_cache_warm2complete 2converged 2accepted 2records
gaussian_x3@1gaussian_x3@1/gaussian_x3_separated/nautilus/n_live_100/local_numpy_fp642complete 2converged 2accepted 2records
gaussian_x3@1gaussian_x3@1/gaussian_x3_separated/nautilus/n_live_200/local_jax_cpu_fp64_cache_warm2complete 2converged 2accepted 2records
gaussian_x3@1gaussian_x3@1/gaussian_x3_separated/nautilus/n_live_200/local_numpy_fp642complete 2converged 2accepted 2records
gaussian_x3@1gaussian_x3@1/gaussian_x3_separated/nautilus/n_live_400/local_jax_cpu_fp64_cache_warm2complete 2converged 2accepted 2records
gaussian_x3@1gaussian_x3@1/gaussian_x3_separated/nautilus/n_live_400/local_numpy_fp642complete 2converged 2accepted 2records
gaussian_x3@1gaussian_x3@1/gaussian_x3_separated/smc/default/local_jax_cpu_fp64_cache_warm1complete 1not_assessed 1rejected 1records
gaussian_x3@1gaussian_x3@1/gaussian_x3_separated/zeus/walkers_50_steps_2000/local_jax_cpu_fp64_cache_warm1failed 1not_converged 1rejected 1records
gaussian_x3@1gaussian_x3@1/gaussian_x3_separated/zeus/walkers_50_steps_2000/local_numpy_fp642complete 2not_converged 2accepted 2records
gaussian_x3@2gaussian_x3@2/gaussian_x3_blend/bfgs/default/ral_jax_cpu_fp64_cache_warm250failed 250not_assessed 250rejected 250records
gaussian_x3@2gaussian_x3@2/gaussian_x3_blend/bfgs/default/ral_numpy_fp64250failed 250not_assessed 250rejected 250records
gaussian_x3@2gaussian_x3@2/gaussian_x3_blend/blackjax_nuts/warmup_200_samples_500/ral_jax_cpu_fp64_cache_warm250failed 250not_converged 250rejected 250records
gaussian_x3@2gaussian_x3@2/gaussian_x3_blend/dynesty_dynamic/default/ral_jax_cpu_fp64_cache_warm250complete 250not_assessed 250accepted 59, rejected 191records
gaussian_x3@2gaussian_x3@2/gaussian_x3_blend/dynesty_dynamic/default/ral_numpy_fp64250complete 250not_assessed 250accepted 59, rejected 191records
gaussian_x3@2gaussian_x3@2/gaussian_x3_blend/dynesty_static/nlive_200/ral_jax_cpu_fp64_cache_warm250complete 250not_assessed 250accepted 36, rejected 214records
gaussian_x3@2gaussian_x3@2/gaussian_x3_blend/dynesty_static/nlive_200/ral_numpy_fp64250complete 250not_assessed 250accepted 36, rejected 214records
gaussian_x3@2gaussian_x3@2/gaussian_x3_blend/emcee/walkers_50_steps_2000/ral_jax_cpu_fp64_cache_warm250complete 250not_converged 250rejected 250records
gaussian_x3@2gaussian_x3@2/gaussian_x3_blend/emcee/walkers_50_steps_2000/ral_numpy_fp64250complete 250not_converged 250rejected 250records
gaussian_x3@2gaussian_x3@2/gaussian_x3_blend/lbfgs/default/ral_jax_cpu_fp64_cache_warm250complete 40, failed 210not_assessed 250rejected 250records
gaussian_x3@2gaussian_x3@2/gaussian_x3_blend/lbfgs/default/ral_numpy_fp64250complete 40, failed 210not_assessed 250rejected 250records
gaussian_x3@2gaussian_x3@2/gaussian_x3_blend/multi_start_adabelief/starts_48_steps_300/ral_jax_cpu_fp64_cache_warm250complete 250not_assessed 250rejected 250records
gaussian_x3@2gaussian_x3@2/gaussian_x3_blend/multi_start_adam/starts_48_steps_300/ral_jax_cpu_fp64_cache_warm250complete 250not_assessed 250rejected 250records
gaussian_x3@2gaussian_x3@2/gaussian_x3_blend/multi_start_lion/starts_48_steps_300/ral_jax_cpu_fp64_cache_warm250complete 250not_assessed 250rejected 250records
gaussian_x3@2gaussian_x3@2/gaussian_x3_blend/multi_start_prodigy/starts_48_steps_300/ral_jax_cpu_fp64_cache_warm250complete 250not_assessed 250accepted 152, rejected 98records
gaussian_x3@2gaussian_x3@2/gaussian_x3_blend/nautilus/n_live_100/ral_jax_cpu_fp64_cache_warm250complete 250converged 250accepted 250records
gaussian_x3@2gaussian_x3@2/gaussian_x3_blend/nautilus/n_live_100/ral_numpy_fp64250complete 250converged 250accepted 250records
gaussian_x3@2gaussian_x3@2/gaussian_x3_blend/nautilus/n_live_200/ral_jax_cpu_fp64_cache_warm250complete 250converged 250accepted 250records
gaussian_x3@2gaussian_x3@2/gaussian_x3_blend/nautilus/n_live_200/ral_numpy_fp64250complete 250converged 250accepted 250records
gaussian_x3@2gaussian_x3@2/gaussian_x3_blend/nautilus/n_live_400/ral_jax_cpu_fp64_cache_warm250complete 250converged 250accepted 250records
gaussian_x3@2gaussian_x3@2/gaussian_x3_blend/nautilus/n_live_400/ral_numpy_fp64250complete 250converged 250accepted 250records
gaussian_x3@2gaussian_x3@2/gaussian_x3_blend/nss/n_live_200_mcmc_steps_50/ral_jax_cpu_fp64_cache_warm250complete 250not_assessed 250accepted 158, rejected 92records
gaussian_x3@2gaussian_x3@2/gaussian_x3_blend/smc/default/ral_jax_cpu_fp64_cache_warm250complete 250not_assessed 250rejected 250records
gaussian_x3@2gaussian_x3@2/gaussian_x3_blend/zeus/walkers_50_steps_2000/ral_jax_cpu_fp64_cache_warm250complete 250not_converged 250rejected 250records
gaussian_x3@2gaussian_x3@2/gaussian_x3_blend/zeus/walkers_50_steps_2000/ral_numpy_fp64250complete 250not_converged 250rejected 250records
gaussian_x3@2gaussian_x3@2/gaussian_x3_separated/bfgs/default/ral_jax_cpu_fp64_cache_warm250complete 250not_assessed 250rejected 250records
gaussian_x3@2gaussian_x3@2/gaussian_x3_separated/bfgs/default/ral_numpy_fp64250complete 250not_assessed 250rejected 250records
gaussian_x3@2gaussian_x3@2/gaussian_x3_separated/blackjax_nuts/warmup_200_samples_500/ral_jax_cpu_fp64_cache_warm250complete 250not_converged 250rejected 250records
gaussian_x3@2gaussian_x3@2/gaussian_x3_separated/blackjax_nuts_warm/warmup_200_samples_500/ral_jax_cpu_fp64_cache_warm250complete 250converged 21, not_converged 229accepted 250records
gaussian_x3@2gaussian_x3@2/gaussian_x3_separated/dynesty_dynamic/default/ral_jax_cpu_fp64_cache_warm250complete 250not_assessed 250accepted 145, rejected 105records
gaussian_x3@2gaussian_x3@2/gaussian_x3_separated/dynesty_dynamic/default/ral_numpy_fp64250complete 250not_assessed 250accepted 145, rejected 105records
gaussian_x3@2gaussian_x3@2/gaussian_x3_separated/dynesty_static/nlive_200/ral_jax_cpu_fp64_cache_warm250complete 250not_assessed 250accepted 103, rejected 147records
gaussian_x3@2gaussian_x3@2/gaussian_x3_separated/dynesty_static/nlive_200/ral_numpy_fp64250complete 250not_assessed 250accepted 103, rejected 147records
gaussian_x3@2gaussian_x3@2/gaussian_x3_separated/emcee/walkers_50_steps_2000/ral_jax_cpu_fp64_cache_warm250complete 250not_converged 250rejected 250records
gaussian_x3@2gaussian_x3@2/gaussian_x3_separated/emcee/walkers_50_steps_2000/ral_numpy_fp64250complete 250not_converged 250rejected 250records
gaussian_x3@2gaussian_x3@2/gaussian_x3_separated/lbfgs/default/ral_jax_cpu_fp64_cache_warm250complete 250not_assessed 250rejected 250records
gaussian_x3@2gaussian_x3@2/gaussian_x3_separated/lbfgs/default/ral_numpy_fp64250complete 250not_assessed 250rejected 250records
gaussian_x3@2gaussian_x3@2/gaussian_x3_separated/multi_start_adabelief/starts_48_steps_300/ral_jax_cpu_fp64_cache_warm250complete 250not_assessed 250rejected 250records
gaussian_x3@2gaussian_x3@2/gaussian_x3_separated/multi_start_adam/starts_48_steps_300/ral_jax_cpu_fp64_cache_warm250complete 250not_assessed 250rejected 250records
gaussian_x3@2gaussian_x3@2/gaussian_x3_separated/multi_start_lion/starts_48_steps_300/ral_jax_cpu_fp64_cache_warm250complete 250not_assessed 250rejected 250records
gaussian_x3@2gaussian_x3@2/gaussian_x3_separated/multi_start_prodigy/starts_48_steps_300/ral_jax_cpu_fp64_cache_warm250complete 250not_assessed 250accepted 250records
gaussian_x3@2gaussian_x3@2/gaussian_x3_separated/nautilus/n_live_100/ral_jax_cpu_fp64_cache_warm250complete 250converged 250accepted 250records
gaussian_x3@2gaussian_x3@2/gaussian_x3_separated/nautilus/n_live_100/ral_numpy_fp64250complete 250converged 250accepted 250records
gaussian_x3@2gaussian_x3@2/gaussian_x3_separated/nautilus/n_live_200/ral_jax_cpu_fp64_cache_warm250complete 250converged 250accepted 250records
gaussian_x3@2gaussian_x3@2/gaussian_x3_separated/nautilus/n_live_200/ral_numpy_fp64250complete 250converged 250accepted 250records
gaussian_x3@2gaussian_x3@2/gaussian_x3_separated/nautilus/n_live_400/ral_jax_cpu_fp64_cache_warm250complete 250converged 250accepted 250records
gaussian_x3@2gaussian_x3@2/gaussian_x3_separated/nautilus/n_live_400/ral_numpy_fp64250complete 250converged 250accepted 250records
gaussian_x3@2gaussian_x3@2/gaussian_x3_separated/nss/n_live_200_mcmc_steps_50/ral_jax_cpu_fp64_cache_warm250complete 250not_assessed 250accepted 210, rejected 40records
gaussian_x3@2gaussian_x3@2/gaussian_x3_separated/smc/default/ral_jax_cpu_fp64_cache_warm250complete 250not_assessed 250rejected 250records
gaussian_x3@2gaussian_x3@2/gaussian_x3_separated/zeus/walkers_50_steps_2000/ral_jax_cpu_fp64_cache_warm250complete 250not_converged 250accepted 190, rejected 60records
gaussian_x3@2gaussian_x3@2/gaussian_x3_separated/zeus/walkers_50_steps_2000/ral_numpy_fp64250complete 250not_converged 250accepted 198, rejected 52records

Comparisons and parity groups

Targets, seeds and stages are retained. No best-seed selection, cross-target ranking or automatic convergence threshold. Declared groups do not themselves certify parity.

No certified comparison pairs supplied by this producer.

Decision History

No decisions recorded yet.

Sampler candidates

Literature expectations, not measured PyAuto recommendations. Review a candidate and copy its investigation prompt.

dynesty

dynamic nested sampling · reviewed 2026-10-08

Paper ↗ · Official code ↗

Expected uses

  • Evidence and posterior baseline with adjustable effort allocation.

Limitations

  • Prior transforms and likelihood-contour exploration need validation; no universal dimensional scaling advantage.

Integration: integrated: DynestyStatic and DynestyDynamic

Benchmark evidence: not_assessed: this literature review establishes no benchmark result

Investigation prompt
Nautilus

neural-network importance nested sampling · reviewed 2026-10-08

Paper ↗ · Official code ↗

Expected uses

  • Evidence and posterior estimation for expensive gradient-free likelihoods.

Limitations

  • Learned proposal coverage and importance-weight stability need checking on unseen modes.

Integration: integrated: Nautilus

Benchmark evidence: not_assessed: this literature review establishes no benchmark result

Investigation prompt