Skip to content

campaigns: likelihood-lifecycle row updated to the phase-4 state #91

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Jammy2211 merged 1 commit into from
Nov 5, 2026
Merged

campaigns: likelihood-lifecycle row updated to the phase-4 state #91

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 15 additions & 13 deletions campaigns.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,19 +149,21 @@ campaigns:
- id: likelihood-lifecycle
title: Likelihood lifecycle phase 2 (per-dataset-type runtime and breakdown cells on the pinned release)
status: active
updated: '2026-10-30'
next: 'Phase 2 sub-campaigns 2a-2g on pin 2026.10.25.1 (epic in Mind draft/research/autolens_profiling/likelihood_lifecycle_epic.md).
2a imaging, 2b interferometer and 2c point source rows committed (#461, #483, #480); 2b/2c quiet RAL
reruns in flight on their own branches. 2d multi_galaxy+group, 2e cluster, 2f multi_dataset+datacube and
2g weak: batch 0 ran 2026-10-29 (RAL 413272-413274/413292-413293, A100 413294-413297; all 56 legs exit 0,
every parity witness match, jax 0.11.2 with the eigen flag on every CPU leg). Walls commit
autolens_profiling#487 sets batch 1 at 0:30-1:30 (human /prm pending). 2g rows (the campaign''s rows
under D9) in autolens_profiling#488, stacked on #487. Human ruling 2026-10-30: loaded RAL nodes accepted
(batch-0 RAL rows had loadavg 95-109); batch 1 runs on shared ral nodes with no quiet gate. 2d driver
fixed to keep the eigen flag off GPU legs (#487, 830314e9). Batch 1 submitted 2026-10-30 (human go) from
autolens_profiling_p2dg at 830314e9 (#487 head, unmerged): RAL 413445-413452 (ral arrays), A100
413453-413459 (one job each). Next: verify batch 1, then PR B for 2d/2e/2f.'
evidence: https://github.com/PyAutoLabs/autolens_profiling/pull/487
updated: '2026-11-05'
next: 'Phases 2, 3 and 6 complete (day_zero 2026.10.25.1: 480 records / 116 cells; day_zero_gradient: 10 records).
Phase 4 (inference wall-time validation, autolens_inference -> walltime-validation@1) in compute. Runners for all
20 likelihoods merged (autolens_inference#55-#58); 56 imaging/hst rows on 2026.10.17.1 adopted (human ruling
2026-11-02). GPU batch 413966-413982 (34 legs) all COMPLETED; CPU batch 414141-414180 submitted 2026-11-05 after
the RAL outage (XLA eigen workaround exported on the validation legs): 22 legs COMPLETED with rows, 19 still running
at 16:36 GMT (numba pixelized legs and the 1-core sweep leg). delaunay_nn numba CPU failed (414150, GNU OpenMP
fork) -> fixed by PyAutoArray#677 (pending release) and rerun as 414189 on the pin plus that fix. Preview of the
exporter on the pulled rows (not committed): pixelized A100 ratios measured/predicted 1.2-1.4 (interferometer
sparse 2.4-2.7); cheap likelihoods (weak, point source, cluster, MGE on A100) at 5-10 ms/eval, far above the
per-call prediction, so sampler overhead dominates; rectangular/hst numba cores sweep relative to 2 cores is
1.82x/3.21x/4.27x/6.56x at 4/8/16/32 cores (parallel_efficiency awaits the 1-core leg 414141). Parity not assessed
on any new row. Next: when the CPU legs finish, 4e ingest (rows + exporter + Pulse lens-walltime refresh) via
start_dev; the conductor judges ratios and the cores cap.'
evidence: https://github.com/PyAutoLabs/autolens_inference/pull/58
tasks:
- id: setup_baseline_campaign
campaign: setup-baseline
Expand Down
2 changes: 1 addition & 1 deletion dashboard.html
View file
Open in desktop

Large diffs are not rendered by default.

6 changes: 3 additions & 3 deletions dashboard.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# PyAutoPulse — profiling dashboard

<!-- pulse-campaigns:0df98084819c074752f8e1899e1f75b939fab981778ea01ac8f75d9f4fdbca13 -->
<!-- pulse-campaigns:401bb4c933eb8eba42ee640b28a680895c3e1ab2249fb31905e51d85562f36bd -->
## Profiling Check In

<details><summary>Full check-in prompt</summary>
Expand Down Expand Up @@ -175,9 +175,9 @@ Reviewed 2026-10-17. Parked unchanged. No new campaign evidence; existing C2/lib

<details><summary>Likelihood lifecycle phase 2 (per-dataset-type runtime and breakdown cells on the pinned release) — active</summary>

Reviewed 2026-10-30. Phase 2 sub-campaigns 2a-2g on pin 2026.10.25.1 (epic in Mind draft/research/autolens_profiling/likelihood_lifecycle_epic.md). 2a imaging, 2b interferometer and 2c point source rows committed (#461, #483, #480); 2b/2c quiet RAL reruns in flight on their own branches. 2d multi_galaxy+group, 2e cluster, 2f multi_dataset+datacube and 2g weak: batch 0 ran 2026-10-29 (RAL 413272-413274/413292-413293, A100 413294-413297; all 56 legs exit 0, every parity witness match, jax 0.11.2 with the eigen flag on every CPU leg). Walls commit autolens_profiling#487 sets batch 1 at 0:30-1:30 (human /prm pending). 2g rows (the campaign&#x27;s rows under D9) in autolens_profiling#488, stacked on #487. Human ruling 2026-10-30: loaded RAL nodes accepted (batch-0 RAL rows had loadavg 95-109); batch 1 runs on shared ral nodes with no quiet gate. 2d driver fixed to keep the eigen flag off GPU legs (#487, 830314e9). Batch 1 submitted 2026-10-30 (human go) from autolens_profiling_p2dg at 830314e9 (#487 head, unmerged): RAL 413445-413452 (ral arrays), A100 413453-413459 (one job each). Next: verify batch 1, then PR B for 2d/2e/2f.
Reviewed 2026-11-05. Phases 2, 3 and 6 complete (day_zero 2026.10.25.1: 480 records / 116 cells; day_zero_gradient: 10 records). Phase 4 (inference wall-time validation, autolens_inference -&gt; walltime-validation@1) in compute. Runners for all 20 likelihoods merged (autolens_inference#55-#58); 56 imaging/hst rows on 2026.10.17.1 adopted (human ruling 2026-11-02). GPU batch 413966-413982 (34 legs) all COMPLETED; CPU batch 414141-414180 submitted 2026-11-05 after the RAL outage (XLA eigen workaround exported on the validation legs): 22 legs COMPLETED with rows, 19 still running at 16:36 GMT (numba pixelized legs and the 1-core sweep leg). delaunay_nn numba CPU failed (414150, GNU OpenMP fork) -&gt; fixed by PyAutoArray#677 (pending release) and rerun as 414189 on the pin plus that fix. Preview of the exporter on the pulled rows (not committed): pixelized A100 ratios measured/predicted 1.2-1.4 (interferometer sparse 2.4-2.7); cheap likelihoods (weak, point source, cluster, MGE on A100) at 5-10 ms/eval, far above the per-call prediction, so sampler overhead dominates; rectangular/hst numba cores sweep relative to 2 cores is 1.82x/3.21x/4.27x/6.56x at 4/8/16/32 cores (parallel_efficiency awaits the 1-core leg 414141). Parity not assessed on any new row. Next: when the CPU legs finish, 4e ingest (rows + exporter + Pulse lens-walltime refresh) via start_dev; the conductor judges ratios and the cores cap.

[Campaign evidence](https://github.com/PyAutoLabs/autolens_profiling/pull/487)
[Campaign evidence](https://github.com/PyAutoLabs/autolens_inference/pull/58)

### Active tasks

Expand Down
Loading