Repository navigation
profiling: split and re-baseline the numba curvature-matrix F breakdown (#505) - #189
Merged
Merged
Conversation
…kdowns post-#462 (#505) Step 0 of PyAutoArray#505: F was 78 % of an HST numba-CPU evaluation but was timed as a single step, so the lever could not be picked from the split. Harness (pixelization_numba.py, delaunay_numba.py) — three accessors added ahead of the F step, one per block of the new per-block helpers in PyAutoArray `imaging_numba/sparse.py`: F: mapper x mapper block [sparse-op] F: mapper x linear-func block [dense conv] F: linear-func x linear-func [BLAS] Design of the split: the helpers are plain (uncached) methods, so touching them does not prime the `curvature_matrix` cached property — the F step that follows recomputes all three and then mirrors + adds the diagonal. Reporting all four raw would double-count F and inflate `total_step_by_step` by ~2x F, breaking comparability with the pre-instrumentation artifacts. So the F row is reported as a residual, F residual = t(curvature_matrix) - (t_block_1 + t_block_2 + t_block_3) i.e. mirror + diag-add + assembly overhead, and the four F rows sum to exactly the un-split F step, which each result JSON now also records directly as `curvature_matrix_f_total_s` alongside a `curvature_matrix_f_split_note`. The residual is a difference of averages and can read slightly negative at the ~1e-2 s level; it is recorded as measured, never clipped. n_repeats, pins and output paths unchanged. Artifacts: all four cells re-run from scratch (not hand-copied) with OMP_NUM_THREADS=1, AUTOARRAY_NUMBA_OPERATED_MEMO=0 — the committed *_v2026.8.17.1 files pre-dated PyAutoArray#462. Adds the previously-missing hst `_rtu` variant. Result (seconds/evaluation): the dense mapper x linear-func convolution dominates F on every cell — 0.953 of 1.275 s (hst bilinear), 0.198 of 0.284 s (euclid), 0.929 of 1.250 s (hst RTU), 0.899 of 1.055 s (Delaunay-1250), i.e. 70-85 %. It does not scale with the source mesh, only with image pixels. The FFT lever is confirmed as the phase-1 target. Two re-baseline surprises, both written up in results/notes/numba_curvature_matrix_f_split.md: - hst rectangular RTU is an 8.5 s cell, not 1.3 s — "mapper sparse triplets" costs 7.88 s there (0.006 s on Bilinear). - Delaunay-1250 is now 1.455 s, not 3.22 s (NNLS warm-start memo + #462). Pins PASSED: hst bilinear 27661.910133664103, hst RTU 27180.704715696862, hst Delaunay 29090.52721044813. euclid has no pin by design (6213.306873885871). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SqrSVGPrFcUB1vvDsoTw3n
…block (#505) PyAutoArray#505 steps 1-2 landed: the redundant passes over F are gone and the mapper x linear-func block's dense sliding-window correlation now runs as a batched FFT convolution with the reversed PSF, with only the scatter in numba. All four breakdown cells re-run from scratch (OMP_NUM_THREADS=1, AUTOARRAY_NUMBA_OPERATED_MEMO=0, n_repeats 10), plus a same-session re-measurement of the step-0 commit for the three paired cells, because this host carries ~20-30 % session-to-session variance and the step-0 table was measured in a different session. Seconds per evaluation: | cell | eval | F | mapper x l-func | |-------------------|----------------|----------------|-----------------| | hst bilinear | 1.562 -> 0.595 | 1.195 -> 0.359 | 0.858 -> 0.054 | | euclid bilinear | 0.349 -> 0.249 | 0.256 -> 0.097 | 0.182 -> 0.023 | | hst Delaunay-1250 | 1.367 -> 0.758 | 1.077 -> 0.184 | 0.854 -> 0.052 | | hst RTU | (8.501) -> 7.479 | (1.250) -> 0.334 | (0.929) -> 0.061 | ~16x on the block, 3.3x on F (5.9x on Delaunay) and 2.6x on the whole HST evaluation. RTU is unpaired: it is an 8.5 s cell dominated by its 7.1 s "mapper sparse triplets" step and GPU-only by the 2026-08-28 decision, so it was re-run once for currency. F is no longer the dominant term at HST resolution — the mapper x mapper sparse-operator block (0.277 s) and the MGE operated mapping matrix (0.224 s) now lead, and the mapper x linear-func block has fallen from 72 % of F to 15 %. Pins PASSED on all three pinned cells (hst bilinear 27661.91013366411, hst RTU 27180.70471569685, hst Delaunay 29090.527210448134); euclid, unpinned, is unchanged at 6213.306873885871. The `F: mapper x linear-func block` step label changes from `[dense conv]` to `[FFT conv + scatter]`, which is what it now times. Nothing else about the step list, the residual design or the pins changed. results/notes/numba_curvature_matrix_f_split.md gains the steps 1-2 section. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fr6iJ5T1RDARWfxttCuGkK
…lit (#505) Runs autolens_workspace's cpu_fast_modeling.py first fit against canonical PyAutoArray main and against the task worktree, at number_of_cores=8 and serially, to confirm the single-thread F win survives multiprocessing and that the new FFT path introduces no implicit thread-pool oversubscription. Pool of 8 improves 9% and the single process 7%; the parallel speed-up ratio is flat (2.52x -> 2.57x on 8 cores), so no oversubscription. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fr6iJ5T1RDARWfxttCuGkK
Merged
4 tasks done
`build_readme.py --check` (the lint job's dashboard-idempotence leg) failed on PR #189: the re-baselined numba breakdown artifacts changed three headline rows that the generated tables index. Regenerated with `python scripts/misc/tooling/build_readme.py`. Only this task's three cells move - imaging/delaunay_numba hst, imaging/pixelization_numba euclid and hst - so no other claimant's rows are touched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fr6iJ5T1RDARWfxttCuGkK
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Profiling side of PyAutoArray#505 — the numba CPU-path speed-up of the curvature
matrix
Fat HST resolution. Adds per-sub-block timing ofFto the numbalikelihood-breakdown harnesses, re-baselines the breakdown artifacts (they were
stale since #462), and records the full before/after method and results.
The harness split
Fintomapper x mapper,mapper x linear-funcandlinear-func x linear-funcsteps, which is what identified the densesliding-window correlation in the mapper x linear-func block as 72 % of
Fand55 % of the whole HST evaluation — the finding the library change acts on.
Headline (s per evaluation,
OMP_NUM_THREADS=1, memo off, n_repeats 10; thebefore column is a fresh re-measurement taken back-to-back with the after run,
since this host carries 20-30 % session-to-session variance):
The notes also record a Nautilus pool run: the same workspace fit at
number_of_cores=8against canonical PyAutoArraymainand against the featurebranch, plus a serial control, confirming the single-thread win survives
multiprocessing (pool +9 %, serial +7 %, parallel speed-up ratio flat at
2.52x -> 2.57x on 8 cores — no FFT/BLAS thread oversubscription).
Only this task's claimed file sets are touched (
scripts/imaging/likelihood_breakdown/,results/breakdown/imaging/,results/notes/), committed with explicitpathspecs per the parallel claim with
nuts-warm-start-driver-and-a100-probe.Upstream PR
PyAutoLabs/PyAutoArray#506 — PyAutoLabs/PyAutoArray#506
Library-first merge gate: this PR must NOT merge until PyAutoArray#506 is
MERGED. The re-baselined artifacts and the "after" columns were measured
against that branch.
Scripts Changed
scripts/imaging/likelihood_breakdown/pixelization_numba.py— split theFstep into
mapper x mapper,mapper x linear-funcandlinear-func x linear-funcsub-steps. The mapper x linear-func label was[dense conv]and is now[FFT conv + scatter], since that is what it nowtimes. Step list, residual design and pins otherwise unchanged.
scripts/imaging/likelihood_breakdown/delaunay_numba.py— same sub-blocksplit for the Delaunay cell.
Results regenerated
results/breakdown/imaging/pixelization_numba_breakdown_hst_v2026.8.17.1.{json,png}results/breakdown/imaging/pixelization_numba_breakdown_hst_v2026.8.17.1_rtu.{json,png}results/breakdown/imaging/pixelization_numba_breakdown_euclid_v2026.8.17.1.{json,png}results/breakdown/imaging/delaunay_numba_breakdown_hst_v2026.8.17.1.{json,png}results/notes/numba_curvature_matrix_f_split.md— method, the step-0baseline, the paired before/after table, the "where the time went" analysis,
the correctness evidence, and the pool-run section.
Test Plan
bilinear 27661.91013366411 (pin 27661.910133665442), hst RTU
27180.70471569685 (pin 27180.704715698186), hst Delaunay
29090.527210448134 (pin 29090.527192092646). euclid has no pin; it
measured 6213.306873885871, unchanged to every recorded digit.
Fagrees with the pre-change sliding-window result to 3e-18 relativeon euclid and hst.
wiped between runs,
autoarray.__file__asserted on both sides.out; against
mainthe harness will report the "before" column.Heart readiness — human RED override
pyauto-heart readinessgraded RED score 45 at ship time (ts2026-08-28T15:02:11Z). The diff repairs none of the named reasons, so this is aplain human override, not the corrective-PR exception. Verbatim:
Human authorisation, 2026-08-28, verbatim:
Scope: push + PR-open for this task only. Merge and release stay human, and
the override does not extend to any other task. The RED is pre-existing on
mainand unrelated — every failing script is on the JAX likelihood path or inautofit, in other repos.Generated by the PyAutoLabs agent workflow.