docs: persist evening profiling campaign check-in 2026-10-17

checkins/2026-10-17.md

@@ -0,0 +1,90 @@
+# Profiling check-in — 2026-10-17
+
+Reviewed at 2026-10-17T15:12:59Z; morning check-in 2026-10-17T11:09:26Z (Pulse PR #66, unmerged at this
+check-in: the human's `!` merge commands did not take effect, so this update is stacked on it).
+Review time and board generation are not measurement times. This check-in records verified
+outcomes and the human's decisions of 2026-10-17 (chat). It authorizes no compute, default or
+baseline change, campaign freeze, scientific acceptance, release or merge.
+
+## Human decisions 2026-10-17 (chat)
+
+These are recorded in the ledger `next:` texts and task check-in sections. They are not new
+decision records unless stated.
+
+1. Interferometer CPU work is exhaustive and closed. Curvature preload parked: "we will come back to it soon".
+2. Interferometer GPU levers: "file wiki research but not make source code changes for now". No library prompt for JAX FFT pruning; campaign parked at knowledge-complete.
+3. DelaunayNN: "file in research to come back to in the future". New campaign `delaunay-nn-cpu`, parked; the ~1.5× target and two hazards recorded as future work, not filed.
+4. Point-source source-plane speed-ups "not worth pursuing again, file as research". Campaign closed as research.
+5. Point-source image plane: "just research to file". Campaign parked; A100 numbers recorded as a reference measurement; phase-2 inference fit measurement dropped.
+6. Imaging phase 2 (A100 production over-sampling re-measure): undecided → needs-decision.
+
+New decision record: [`direct-conv-symmetric-gate`](../decisions/direct-conv-symmetric-gate.md),
+capturing the human decision of 2026-10-16 on [PyAutoArray#635](https://github.com/PyAutoLabs/PyAutoArray/issues/635)
+(ymin column ordering, default gate 0.5 → 0.7).
+
+## Verified availability
+
+- All PRs below are OPEN, CI green and unmerged. Every Heart read was STALE ("release validation incomplete: no rehearsal for current source"); no override was used anywhere.
+- Mind: all related tasks are awaiting-merge. Cortex run log pushed for RAL jobs 412495/412498.
+- Measurements were made by the project workers today; this check-in did not re-run them.
+
+## Campaign review
+
+| Campaign | State | Change / next bounded step |
+|---|---|---|
+| Interferometer likelihood and streaming | parked | Decisions 1–2. L2 symmetric `direct_conv` PR-open (PyAutoArray#636 + record #425): real Aris RAL 1.49× / 1.53× / 1.58× at 1,459 / 2,577 / 5,746 vertices (pred 1.48× / 1.50× / 1.53×), logL to round-off, peak RSS 43 GB without NUFFT chunking; gain over the old default route at 1,459 only ~1.02× (FFT 15.32 s). RAL FFT/full direct_conv 0.68 measured vs 0.89 predicted: constants refit deferred. A100 FFT levers (#433): pruned FFT 1.08× / 1.18× / 1.41× whole likelihood at alma / alma_high / jvla, F exact to 1.7e-15; pixel-scale coarsening shifts log evidence +1,310 / +1,807 / +2,757 nats (fails 0.5-nat bar). Streaming (#435): 1e8 vis CPU 707.5 s / 1.78 GB RSS, A100 157.3 s / 1.48 GB device, compile ~3 s; in-memory fails at 2.5e5 (RAL 10 GB cap) and 4e6 (A100). Memo after-measurement (#435, job 412526): Delaunay iid 1.918 [1.887, 1.925] vs 2.17 in #332; 15 solves 1.018 [1.013, 1.023]; walk 0.46; rect iid 0.887 / 0.987; rect walk 0.15; FoM identical — an after-measurement, not a fix. |
+| Point-source source plane | complete | Decision 4. autolens_inference#29 + autolens_profiling#432: MultiStartAdam forward gradients, 48 starts, 5 seeds, 2026.10.17.1 wheels (jax 0.11.2), fp64. Default 300 steps: 4/5 seeds hit the limit, two 1.4σ off; 3000 steps: all converge into the Nautilus logL range, ≤ 0.8σ. Likelihood+gradient ≤ 3.7 % of fit wall; per step 1.7 ms CPU / 4.4 ms A100 (0.24 / 0.35 ms likelihood+grad); A100 slower end to end than 8 CPU cores. Runtime cell medians A100 0.233 ms (single_jit 0.347), RAL CPU 0.147 ms (0.177), pins pass. Blackjax stays parked; Adam step-limit is a PyAutoFit search-settings note, not a task. |
+| Point-source image plane · A100 | parked | Decision 5. #434, job 412519 euclid-ral-gpu-2 exclusive, A100 80GB, jax 0.10.2, fp64: single call 0.934 ms (pre-#768 0.910), vmap-256 0.0125 ms (0.0120), compile 6.54 s; forward gradient finite 16/16, equal to reverse to 1.6e-11 (was NaN); reverse/likelihood 1.889; FD 4/6 (same points as before); logL bit-exact; within-run MDI 1.50 %; +2.7 % not attributed (different node/date). A100 per-step breakdown rows unusable (11/43 negative). |
+| Production imaging and likelihood breakdown | active | Production over-sampling phase 1 PR-open (#431, smoke 8/8): `--over-sampling {flat1,production}` on the 4 JAX Delaunay/DelaunayNN cells; euclid [4,4,2]@[0.1,0.3] and hst [4,2,2]@[0.1,0.3], sub 4 where source S/N > 3 else 2; HST 2,022 / 15,361 px at sub 4, Euclid 1,057 / 3,841. Findings: HST flat-1 adapt_split pin fails on unmodified main (29155.000 vs 29140.296, 5e-4); numba HST threshold never reaches sub 4; rectangular JAX cells uncovered. Phase 2 A100: needs-decision (decision 6). |
+| DelaunayNN mesh on the CPU path (new) | parked | Decision 3. PyAutoArray#637 (issue #629, pending-release, CI 3 legs green): RAL compute-10-4 exclusive, 1 thread, A-B-B-A ×2, 1,030 vertices / 15,380 px: NN sub4 0.569 → 0.392 s (2.20× Delaunay, was 3.00×); NN sub2 0.235 → 0.189 s (1.75×, was 2.10×); Delaunay sub4 0.189 → 0.178 s; logL bit-identical; adaptive signals 158 → 6.5 ms; unique_mappings 22.5 → 17.0 ms; Sibson ~0.16 s (~41 %). ~1.5× target not met; fixes the size-0 Sibson NaN edge case. |
+| Measurement reliability and profiling tools | active | Merge-order note added (below). #417/#418 open; no new measurements. |
+| Setup baseline, Point-source image CPU, Cluster PointSolver, Linear-solver, PyAutoFit, Critical curves, Certified solver | unchanged | Carried from the morning check-in; no new evidence this evening. |
+
+## Task changes
+
+| Task | State | Note |
+|---|---|---|
+| [interferometer_large_mask_aris](../tasks/interferometer_large_mask_aris.md) | active | L2 at PR-open; Knowledge (f) L2 row, (g) bullet and (h) added. Open only until #636/#425 merge and release. |
+| [interferometer_w_tilde_fft_size_levers](../tasks/interferometer_w_tilde_fft_size_levers.md) | parked | A100 research filed (#433); no library prompt. |
+| [interferometer_streaming_scaling](../tasks/interferometer_streaming_scaling.md) | parked | 1e8 CPU/A100 rows at PR-open (#435). |
+| [interferometer_nnls_memo_scattered_stream_guard](../tasks/interferometer_nnls_memo_scattered_stream_guard.md) | parked | After-measurement at PR-open (#435); Nautilus-replay witness and varied-seed rounds owed as research. |
+| [interferometer_fixed_mapper_curvature_preload](../tasks/interferometer_fixed_mapper_curvature_preload.md) | parked | Decision 1. |
+| [point_source_source_plane_chi_squared_speed](../tasks/point_source_source_plane_chi_squared_speed.md) | complete | Closed as research. |
+| [runtime_cell_single_jit_gpu_warmup](../tasks/runtime_cell_single_jit_gpu_warmup.md) | complete | Fresh medians (#432). |
+| [point_source_image_plane_gpu_breakdown](../tasks/point_source_image_plane_gpu_breakdown.md) | parked | Reference measurement (#434). |
+| [imaging_production_over_sampling](../tasks/imaging_production_over_sampling.md) | needs-decision | Phase 1 PR-open (#431); phase 2 undecided. |
+| [delaunay_nn_cpu_imaging](../tasks/delaunay_nn_cpu_imaging.md) (new) | parked | #637 at PR-open. |
+
+## Open PRs (the human's to merge or close)
+
+| PR | Content |
+|---|---|
+| [PyAutoArray#636](https://github.com/PyAutoLabs/PyAutoArray/pull/636) | L2 symmetric direct_conv (head 62a1e3ac, pending-release) |
+| [PyAutoArray#637](https://github.com/PyAutoLabs/PyAutoArray/pull/637) | DelaunayNN padded-row and size-0 Sibson fixes (pending-release) |
+| [autolens_profiling#425](https://github.com/PyAutoLabs/autolens_profiling/pull/425) | L2 results note |
+| [autolens_profiling#431](https://github.com/PyAutoLabs/autolens_profiling/pull/431) | Imaging production over-sampling phase 1 |
+| [autolens_profiling#432](https://github.com/PyAutoLabs/autolens_profiling/pull/432) | Point-source source-plane gradient fit share + runtime medians |
+| [autolens_profiling#433](https://github.com/PyAutoLabs/autolens_profiling/pull/433) | A100 FFT size levers (titles still name the dropped CPU constants) |
+| [autolens_profiling#434](https://github.com/PyAutoLabs/autolens_profiling/pull/434) | Point-source image-plane A100 reference measurement |
+| [autolens_profiling#435](https://github.com/PyAutoLabs/autolens_profiling/pull/435) | Interferometer streaming 1e8 rows + memo after-measurement |
+| [autolens_inference#29](https://github.com/PyAutoLabs/autolens_inference/pull/29) | MultiStartAdam point-source source-plane fits |
+| [PyAutoPulse#66](https://github.com/PyAutoLabs/PyAutoPulse/pull/66) | Morning check-in (this update is stacked on it) |
+
+**Merge order.** The six autolens_profiling PRs (#425, #431, #432, #433, #434, #435) all regenerate
+the shared catalogue/dashboard, so each merge after the first needs a rebase and re-render before the
+next. Suggested order: #425, #432, #431, #433, #434, #435. Library PRs #636 and #637 are file-disjoint
+and independent; merge #425 after #636.
+
+## Unavailable or unverified
+
+- No release contains #636 or #637; all "speed-up" figures are PR-open measurements, not released behaviour or scientific acceptance.
+- RAL jax 0.10.2 (used for the A100 image-plane and FFT-lever jobs) is outside 2026.10.17.1's declared jax range; source SHAs are the provenance.
+- Owed research, not tasks: memo Nautilus-replay witness, varied-seed rounds, optional 2e8 streaming rows, RAL route-constant refit.
+
+## RAL cleanup owed (recorded, not deleted)
+
+- `/mnt/ral/jnightin/scratch/fft_levers_20261017/` (abandoned partial clones)
+- `/mnt/ral/jnightin/scratch/fft_levers_a100_20261017/` (partial extract)
+- `/mnt/ral/jnightin/scratch/fft_levers_stack.tgz`, `fft_levers_jobs/`, `fft_levers_results/` (results JSON is committed in #433)
+- `/mnt/ral/jnightin/profiling/autolens_profiling_wt/ps-image-admission_stale_*`
+- Worker private clones under `profiling/` and `inference/` are kept for reuse.