Skip to content

profiling: split and re-baseline the numba curvature-matrix F breakdown (#505) - #189

Merged
Jammy2211 merged 4 commits into
mainfrom
feature/numba-hst-curvature-matrix-speedup
Aug 28, 2026
Merged

Jammy2211 merged 4 commits into
mainfrom
feature/numba-hst-curvature-matrix-speedup

Conversation

@Jammy2211

Copy link
Copy Markdown
Contributor

Summary

Profiling side of PyAutoArray#505 — the numba CPU-path speed-up of the curvature
matrix F at HST resolution. Adds per-sub-block timing of F to the numba
likelihood-breakdown harnesses, re-baselines the breakdown artifacts (they were
stale since #462), and records the full before/after method and results.

The harness split F into mapper x mapper, mapper x linear-func and
linear-func x linear-func steps, which is what identified the dense
sliding-window correlation in the mapper x linear-func block as 72 % of F and
55 % of the whole HST evaluation — the finding the library change acts on.

Headline (s per evaluation, OMP_NUM_THREADS=1, memo off, n_repeats 10; the
before column is a fresh re-measurement taken back-to-back with the after run,
since this host carries 20-30 % session-to-session variance):

cell eval F total mapper x mapper mapper x l-func
hst, rectangular bilinear 1.562 -> 0.595 1.195 -> 0.359 0.295 -> 0.277 0.858 -> 0.054
euclid, rectangular bilinear 0.349 -> 0.249 0.256 -> 0.097 0.060 -> 0.065 0.182 -> 0.023
hst, Delaunay-1250 1.367 -> 0.758 1.077 -> 0.184 0.122 -> 0.123 0.854 -> 0.052
hst, rectangular RTU (8.501) -> 7.479 (1.250) -> 0.334 (0.304) -> 0.249 (0.929) -> 0.061

The notes also record a Nautilus pool run: the same workspace fit at
number_of_cores=8 against canonical PyAutoArray main and against the feature
branch, plus a serial control, confirming the single-thread win survives
multiprocessing (pool +9 %, serial +7 %, parallel speed-up ratio flat at
2.52x -> 2.57x on 8 cores — no FFT/BLAS thread oversubscription).

Only this task's claimed file sets are touched (scripts/imaging/likelihood_breakdown/,
results/breakdown/imaging/, results/notes/), committed with explicit
pathspecs per the parallel claim with nuts-warm-start-driver-and-a100-probe.

Upstream PR

PyAutoLabs/PyAutoArray#506 — PyAutoLabs/PyAutoArray#506

Library-first merge gate: this PR must NOT merge until PyAutoArray#506 is
MERGED.
The re-baselined artifacts and the "after" columns were measured
against that branch.

Scripts Changed

  • scripts/imaging/likelihood_breakdown/pixelization_numba.py — split the F
    step into mapper x mapper, mapper x linear-func and
    linear-func x linear-func sub-steps. The mapper x linear-func label was
    [dense conv] and is now [FFT conv + scatter], since that is what it now
    times. Step list, residual design and pins otherwise unchanged.
  • scripts/imaging/likelihood_breakdown/delaunay_numba.py — same sub-block
    split for the Delaunay cell.

Results regenerated

  • results/breakdown/imaging/pixelization_numba_breakdown_hst_v2026.8.17.1.{json,png}
  • results/breakdown/imaging/pixelization_numba_breakdown_hst_v2026.8.17.1_rtu.{json,png}
  • results/breakdown/imaging/pixelization_numba_breakdown_euclid_v2026.8.17.1.{json,png}
  • results/breakdown/imaging/delaunay_numba_breakdown_hst_v2026.8.17.1.{json,png}
  • results/notes/numba_curvature_matrix_f_split.md — method, the step-0
    baseline, the paired before/after table, the "where the time went" analysis,
    the correctness evidence, and the pool-run section.

Test Plan

  • All 3 pinned log-likelihoods PASSED on the re-baselined harness: hst
    bilinear 27661.91013366411 (pin 27661.910133665442), hst RTU
    27180.70471569685 (pin 27180.704715698186), hst Delaunay
    29090.527210448134 (pin 29090.527192092646). euclid has no pin; it
    measured 6213.306873885871, unchanged to every recorded digit.
  • F agrees with the pre-change sliding-window result to 3e-18 relative
    on euclid and hst.
  • Pool run reproduced twice per configuration with the output directory
    wiped between runs, autoarray.__file__ asserted on both sides.
  • Reviewer: the "after" numbers only reproduce with PyAutoArray#506 checked
    out; against main the harness will report the "before" column.

Heart readiness — human RED override

pyauto-heart readiness graded RED score 45 at ship time (ts
2026-08-28T15:02:11Z). The diff repairs none of the named reasons, so this is a
plain human override, not the corrective-PR exception. Verbatim:

red_reasons:
  "release validation FAILED (stage integrate)"
yellow_reasons:
  "workspace validation not passing (2 failed, cloud#33179766004: autolens_test scripts/imaging/rectangular_mge.py, autolens_test scripts/imaging/rectangular_mge_rtu.py)"
  "manifest drift: session-start hooks (generated) — 32 mismatch(es) vs PyAutoMind/repos.yaml"

Human authorisation, 2026-08-28, verbatim:

I authorize things to override the heart RED.

Scope: push + PR-open for this task only. Merge and release stay human, and
the override does not extend to any other task.
The RED is pre-existing on
main and unrelated — every failing script is on the JAX likelihood path or in
autofit, in other repos.

Generated by the PyAutoLabs agent workflow.

Jammy2211 and others added 3 commits August 28, 2026 15:32
…kdowns post-#462 (#505)

Step 0 of PyAutoArray#505: F was 78 % of an HST numba-CPU evaluation but was
timed as a single step, so the lever could not be picked from the split.

Harness (pixelization_numba.py, delaunay_numba.py) — three accessors added
ahead of the F step, one per block of the new per-block helpers in
PyAutoArray `imaging_numba/sparse.py`:

  F: mapper x mapper block        [sparse-op]
  F: mapper x linear-func block   [dense conv]
  F: linear-func x linear-func    [BLAS]

Design of the split: the helpers are plain (uncached) methods, so touching them
does not prime the `curvature_matrix` cached property — the F step that follows
recomputes all three and then mirrors + adds the diagonal. Reporting all four
raw would double-count F and inflate `total_step_by_step` by ~2x F, breaking
comparability with the pre-instrumentation artifacts. So the F row is reported
as a residual,

    F residual = t(curvature_matrix) - (t_block_1 + t_block_2 + t_block_3)

i.e. mirror + diag-add + assembly overhead, and the four F rows sum to exactly
the un-split F step, which each result JSON now also records directly as
`curvature_matrix_f_total_s` alongside a `curvature_matrix_f_split_note`. The
residual is a difference of averages and can read slightly negative at the
~1e-2 s level; it is recorded as measured, never clipped.

n_repeats, pins and output paths unchanged.

Artifacts: all four cells re-run from scratch (not hand-copied) with
OMP_NUM_THREADS=1, AUTOARRAY_NUMBA_OPERATED_MEMO=0 — the committed
*_v2026.8.17.1 files pre-dated PyAutoArray#462. Adds the previously-missing
hst `_rtu` variant.

Result (seconds/evaluation): the dense mapper x linear-func convolution
dominates F on every cell — 0.953 of 1.275 s (hst bilinear), 0.198 of 0.284 s
(euclid), 0.929 of 1.250 s (hst RTU), 0.899 of 1.055 s (Delaunay-1250), i.e.
70-85 %. It does not scale with the source mesh, only with image pixels. The
FFT lever is confirmed as the phase-1 target.

Two re-baseline surprises, both written up in
results/notes/numba_curvature_matrix_f_split.md:
- hst rectangular RTU is an 8.5 s cell, not 1.3 s — "mapper sparse triplets"
  costs 7.88 s there (0.006 s on Bilinear).
- Delaunay-1250 is now 1.455 s, not 3.22 s (NNLS warm-start memo + #462).

Pins PASSED: hst bilinear 27661.910133664103, hst RTU 27180.704715696862,
hst Delaunay 29090.52721044813. euclid has no pin by design (6213.306873885871).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SqrSVGPrFcUB1vvDsoTw3n
…block (#505)

PyAutoArray#505 steps 1-2 landed: the redundant passes over F are gone and the
mapper x linear-func block's dense sliding-window correlation now runs as a
batched FFT convolution with the reversed PSF, with only the scatter in numba.

All four breakdown cells re-run from scratch (OMP_NUM_THREADS=1,
AUTOARRAY_NUMBA_OPERATED_MEMO=0, n_repeats 10), plus a same-session
re-measurement of the step-0 commit for the three paired cells, because this
host carries ~20-30 % session-to-session variance and the step-0 table was
measured in a different session. Seconds per evaluation:

| cell              | eval           | F              | mapper x l-func |
|-------------------|----------------|----------------|-----------------|
| hst bilinear      | 1.562 -> 0.595 | 1.195 -> 0.359 | 0.858 -> 0.054  |
| euclid bilinear   | 0.349 -> 0.249 | 0.256 -> 0.097 | 0.182 -> 0.023  |
| hst Delaunay-1250 | 1.367 -> 0.758 | 1.077 -> 0.184 | 0.854 -> 0.052  |
| hst RTU           | (8.501) -> 7.479 | (1.250) -> 0.334 | (0.929) -> 0.061 |

~16x on the block, 3.3x on F (5.9x on Delaunay) and 2.6x on the whole HST
evaluation. RTU is unpaired: it is an 8.5 s cell dominated by its 7.1 s "mapper
sparse triplets" step and GPU-only by the 2026-08-28 decision, so it was re-run
once for currency.

F is no longer the dominant term at HST resolution — the mapper x mapper
sparse-operator block (0.277 s) and the MGE operated mapping matrix (0.224 s)
now lead, and the mapper x linear-func block has fallen from 72 % of F to 15 %.

Pins PASSED on all three pinned cells (hst bilinear 27661.91013366411, hst RTU
27180.70471569685, hst Delaunay 29090.527210448134); euclid, unpinned, is
unchanged at 6213.306873885871.

The `F: mapper x linear-func block` step label changes from `[dense conv]` to
`[FFT conv + scatter]`, which is what it now times. Nothing else about the step
list, the residual design or the pins changed.

results/notes/numba_curvature_matrix_f_split.md gains the steps 1-2 section.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fr6iJ5T1RDARWfxttCuGkK
…lit (#505)

Runs autolens_workspace's cpu_fast_modeling.py first fit against canonical
PyAutoArray main and against the task worktree, at number_of_cores=8 and
serially, to confirm the single-thread F win survives multiprocessing and that
the new FFT path introduces no implicit thread-pool oversubscription.

Pool of 8 improves 9% and the single process 7%; the parallel speed-up ratio is
flat (2.52x -> 2.57x on 8 cores), so no oversubscription.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fr6iJ5T1RDARWfxttCuGkK
`build_readme.py --check` (the lint job's dashboard-idempotence leg) failed on
PR #189: the re-baselined numba breakdown artifacts changed three headline rows
that the generated tables index.

Regenerated with `python scripts/misc/tooling/build_readme.py`. Only this task's
three cells move - imaging/delaunay_numba hst, imaging/pixelization_numba euclid
and hst - so no other claimant's rows are touched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fr6iJ5T1RDARWfxttCuGkK
@Jammy2211
Jammy2211 merged commit b3fa632 into main Aug 28, 2026
1 check passed
@Jammy2211
Jammy2211 deleted the feature/numba-hst-curvature-matrix-speedup branch August 28, 2026 21:16
@Jammy2211 Jammy2211 removed the pending-release PR queued for the next release build label Oct 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant