Repository navigation
perf: skip Sibson row padding in adaptive signals and unique mappings (DelaunayNN imaging) #629
Description
Overview
Follow-up to #623 (#624 numba Sibson kernel, #625 routing tests). On HST-scale imaging at sub_size 4, DelaunayNN is still ~2.7× Delaunay per likelihood (0.78 s vs 0.29 s, 1030 vertices, threads=1) because two steps work over its 32-wide padded Sibson rows: adaptive_pixel_signals_from (0.16 s vs 0.01 s, np.add.at over all 32 slots) and data_slim_to_pixelization_unique_from (0.30 s vs 0.10 s). Also fixes a pre-existing Sibson edge case where a degenerate row ends with size == 0, silently dropping its NaN so the evidence comes out finite and wrong. Tracked in a new PyAutoPulse speed-up campaign (queued — see Plan).
Heart RED override (development only) — authorised by the live human on 2026-10-15 for this task (AskUserQuestion "Yes, override for this task"). Reasons at the time, verbatim: "Heart: RED — Release RED; monitoring RED · 10/100 · 415 unresolved (updated 7h ago)"; "[red] release validation FAILED (stage integrate)"; "[red] PyAutoHands/workflows/Python Version Matrix: failure"; "[red] PyAutoHeart/workflows/Release Integrate: failure"; "[red] PyAutoMind/workflows/Lifecycle Drift: failure"; "[red] PyAutoArray/Tests: red"; "[red] PyAutoArray/Tests: red"; "[red] autofit_workspace_test/Smoke Tests: red"; "[red] autogalaxy_workspace_test/Smoke Tests: red"; "[red] autolens_workspace_test/Smoke Tests: red"; "[red] CI wall-clock: 26 gates · slowest PyAutoBrain Nightly Release 63m · 5 hang events"; "[red] Release readiness: release validation FAILED (stage integrate)"; "[red] autofit_test: red"; "[red] Release validation: NOT release_ready — v2026.10.15.1.dev81701 profile=release (2026-10-15T07:04:34+00:00)"; "[red] validation_report/stages/integrate: fail"; "[red] PyAutoArray: red"; "[red] autolens_inference: red"; "[red] autolens_profiling: red"; "[red] Worktree drift: 5 orphan / 0 missing / 4 dirty"; "[red] euclid_strong_lens_modeling_pipeline: red".
Queued (2026-10-15): PyAutoArray is claimed by
interferometer-sparse-data-vector(#626, --auto launch) and PyAutoPulse byral-category-paths(PyAutoCortex#67). Registered in Mindplanned.md; starts when PyAutoArray frees.
Plan
- Make the NumPy branch of
adaptive_pixel_signals_fromskip padding: compact to real entries and usenp.bincountinstead ofnp.add.at; JAX branch and the normalisation tail untouched. - Trim
data_slim_to_pixelization_unique_from: reset only thepix_checkentries a pixel touched (helps every mesh), profile the per-entry overhead; aprangeparallel variant only if it measurably pays and stays bit-identical. - Make failed Sibson rows keep
size ≥ 1so NaN propagates (reference, numba kernel and JAX alike) — deliberate behaviour change: finite-wrong evidence → NaN. - Tests: parity of both functions vs current on 32-wide padded DelaunayNN rows (Delaunay results bit-identical), JAX/NumPy parity for adaptive signals, a degenerate
size == 0row now gives NaN. - Re-time imaging Delaunay vs DelaunayNN at sub 2 and 4 (threads=1, plus multi-thread if parallelised) and the Aris interferometer as a regression check. Target ≈2× Delaunay at sub 4 (not 1.5×) — the Sibson step (0.21 s) and the extra real neighbours (13–15 vs 3) are genuine work.
- File the
sub_slim_indexes_for_pix_indexpadding hazard (abstract.py:222-229 appends -1 padding to the last pixel's list) as its own Mind prompt. - Log and monitor the work in a new PyAutoPulse campaign
delaunay-nn-cpu(tasks for feat: numba Sibson kernel so DelaunayNN matches Delaunay speed on CPU #623 and this issue), once PyAutoPulse is free.
Tier: undeclared (judge) — merge mode: human /prm.
Detailed implementation plan
Affected Repositories
- PyAutoArray (primary)
- PyAutoPulse (campaign ledger only:
campaigns.yamlrows +tasks/files)
Branch Survey
| Repository | Current Branch | Dirty? |
|---|---|---|
| ./array/PyAutoArray | main (a84f724) | clean — claimed by interferometer-sparse-data-vector |
| ./organs/PyAutoPulse | main | clean — claimed by ral-category-paths |
Suggested branch: feature/delaunaynn-wide-row-overhead · worktree ~/Code/PyAutoLabs-wt/delaunaynn-wide-row-overhead/
Implementation Steps
autoarray/inversion/mappers/mapper_util.py:6-113adaptive_pixel_signals_from, NumPy branch:valid = arange(B) < size[:, None];np.bincount(pix_idx[valid], weights=(w*signal)[valid], minlength=pixels)and counts likewise, replacing the twonp.add.at(L69-74) and the dump-to-slot-pixelstrick; must match current output bit-for-bit or to 1e-15. JAX.at[].addbranch and the safe-denominator/grad tail (L91-113) untouched. Callers: regularization/adapt.py:339, adapt_power.py:103, adapt_split_power.py:98, adapt_split_zeroth_power.py:100, brightness_zeroth.py:130, matern_adapt_kernel.py:134, matern_adapt_power_kernel.py:106 (viaMapper.pixel_signals_from, abstract.py:470-516).autoarray/inversion/mappers/mapper_numba_util.py:7-100data_slim_to_pixelization_unique_from(@numba_util.jit, serial): replace the per-data-pixelpix_check[:] = -1(L74, O(data_pixels × pix_pixels)) with resetting only the touched ids; profile the inner loop (output buffers are(data_pixels, max(pix_sizes) * sub_size²)≈ 240 wide at sub 4); optionalnumba.njit(parallel=True)prangeover data pixels (numba_util.jit cannot express parallel) only if measured and identical. Outputs (−1/0 padding,pix_lengths) unchanged. Consumers: imaging_numba/sparse.py:199-202, 245-247, 389-391, 422-442, 623-625, 721-723.- Sibson failed-row size:
sibson.py:509-555after thefailedmask →size = where(failed & (size == 0), 1, size); same in_sibson_numba.py:323-357; JAX path shares_sibson_single_from_tablesviaxp. - Tests:
test_mapper_util.py:260(+test_linear_obj.py:5) and:341-551with a new 32-wide padded DelaunayNN case; imaging numba-vs-mapping DelaunayNN parity (test_factory.py:385); extend NaN-route tests (test_factory.py:428,test_interferometer_numba.py) with a constructed degeneratesize == 0row;test_sibson_numba.pynumba-vs-reference parity incl. the new size rule. - Gates: full test_autoarray, nojax-leg emulation, DelaunayNN workspace_test scripts, independent review; timing table in PR body.
- Pulse (when free): new campaign
delaunay-nn-cpuincampaigns.yaml+tasks/delaunaynn_numba_sibson.md(complete; perf: numba Sibson kernel so DelaunayNN matches Delaunay speed on CPU (#623 phase 1) #624/test: pin DelaunayNN routing to the numba likelihood paths (#623 phase 2) #625) +tasks/delaunaynn_wide_row_imaging_overhead.md(this issue), via a small Pulse PR, thenbin/pyauto-pulse board --offline+check --offline.
Key Files
autoarray/inversion/mappers/mapper_util.py,autoarray/inversion/mappers/mapper_numba_util.pyautoarray/inversion/mesh/interpolator/sibson.py,_sibson_numba.pyautoarray/inversion/inversion/imaging_numba/sparse.py(consumer)PyAutoPulse/campaigns.yaml,PyAutoPulse/tasks/
Original Prompt
Click to expand starting prompt
Speed up unique_mappings and adaptive_pixel_signals for wide DelaunayNN rows (imaging)
Type: feature
Target: autoarray
Repos:
- PyAutoArray
Themes: - numba-cpu
- imaging
- delaunay-nn
Difficulty: small
Autonomy: supervised
Priority: low
Status: draft
Follow-up from PyAutoArray#623 phase 2 (measured 2026-10-15, laptop, threads=1, load 3-5).
After the numba Sibson kernel (#624), DelaunayNN on HST-scale imaging is still ~2.7x
Delaunay at sub_size 4 (0.78 s vs 0.29 s per log_likelihood_function, 1030 vertices,
15,380 pixels; sub 2: 0.32-0.48 s vs 0.31-0.38 s). Profile of the DelaunayNN extra cost:
- Sibson interpolation of 245k queries: 0.21 s
mapper.unique_mappings(mapper_numba_util.data_slim_to_pixelization_unique_from): 0.30 s vs 0.10 s Delaunayadaptive_pixel_signals_from: 0.16 s vs 0.01 s — annp.add.atover the 32-wide (SIBSON_MAX_NEIGHBORS) rows, most slots empty (13-15 non-zeros max)
Fix ideas: iterate only pix_sizes[i] entries per row (skip padding) in both, numba or
compacted-CSR np.bincount for adaptive_pixel_signals. Acceptance: imaging DelaunayNN
within ~1.5x Delaunay at sub 4 with bit-/1e-12 parity; Delaunay unchanged.
Timing scripts: session scratchpad 0b91c774.../scratchpad/phase2/imaging_timing.py (temporary).
Also (from the #625 independent review, pre-existing): sibson.py ~L518-555 can in principle emit a failed
row with size == 0 whose NaN weights sit only in padding; every route reads the first size entries, so
the evidence comes out finite (and wrong) instead of NaN. Decide whether failed rows should keep size >= 1
so NaN propagates; add cavity-overflow and degenerate cases to the NaN-route tests.
Activity
Jammy2211 commentedon Oct 17, 2026
Library PR created: #637
Implemented on
feature/delaunaynn-wide-row-overhead(58a044e). The live-entry numba kernel and bincount fallback for adaptive signals, the touched-only reset in unique mappings and failed Sibson rows kept at size ≥ 1 are all bit-identical on the Delaunay and DelaunayNN likelihoods, apart from the intended size-0 NaN fix.RAL acceptance timing — job 412525,
euclid-ral-compute-10-4,--partition=ral --exclusive, threads=1 (NUMBA/OMP/OPENBLAS/MKL=1),use_jax=False, load 0.4–1.0 (the benchmark process itself). Private checkout (/mnt/ral/jnightin/scratch/dnn_imaging_20261017, shared mirror untouched): PyAutoNerves/Fit/Galaxy/Lens at2026.10.17.1(7f3055b / 4b183e22 / 35100def / c90eceb0); PyAutoArray A =e59d5fa9(2026.10.17.1 = main), B =58a044eb(this PR). HST-scale simulated imaging: 150×150 @ 0.05″, r=3.5″ mask (15,380 pixels), 11×11 PSF, Hilbert 1000 + 30 edge = 1030 vertices, AdaptSplit, sparse operator →InversionImagingSparseNumba. Order A-B-B-A × 2 blocks per case (4 processes per variant), 10 timedlog_likelihood_functioncalls per process after 2 warm-up calls (compile excluded); value = median over processes of per-process medians; step times = wrapped-module medians.34898.34096822383)34840.82970999481)34891.78161640761)34833.08741739975)Spread within each variant ≤ ±2 % (sub 4 NN B 0.380–0.396 s). Table width 32, live entries ≤ 13 (mean 7.2) per DelaunayNN row.
Against the acceptance line: the padded-row overhead is gone (adaptive signals 24× faster, now ~1.6 % of a DelaunayNN likelihood), but DelaunayNN is still 2.20× Delaunay at sub 4, so the draft's ~1.5× is not met (the issue plan's revised ≈2× target is close but also slightly above). What remains is genuine work, not padding: the Sibson interpolation itself (~0.16 s at threads=1, ~41 % of the call) and the curvature build over the wider unique-mapping rows (up to 43 unique pixels per data pixel vs 27 for Delaunay). Neither is in this PR's scope.
Remaining for the human:
/prmon #637 (judge tier). Follow-ups if the ~1.5× target is still wanted: Sibson cost at threads=1, and the curvature build over the wider unique rows.🤖 Generated with Claude Code
Jammy2211 commentedon Oct 17, 2026
Shipped
Merged 2026-10-17: #637 — adaptive pixel signals and unique mappings visit only live Sibson entries (bit-identical outputs), and failed Sibson rows keep size ≥ 1 so NaN evidence propagates. RAL A/B (job 412525): DelaunayNN sub 4 0.569 → 0.392 s (2.20× Delaunay, was 3.00×); the ~1.5× target is not met — the remainder is Sibson interpolation and wider curvature rows, outside this PR.
Pending release: PyAutoArray#637 ships with the next PyAutoArray release. Human decision 2026-10-17: further DelaunayNN speed-up filed as research.