[Jammy2211]

Summary

A100 re-measure of the DelaunayNN Sibson launch-latency cuts — PyAutoArray issue #532, Phase A, shipped in PyAutoArray #533. This PR carries the profiling-side artifacts: the new --split-setup split-Sibson stage and Sibson provenance in the breakdown cell, the five submit scripts, and the ten job units' results plus the written note.

What was measured. All ten job units ran on the same node (euclid-ral-gpu-2) inside one 28-minute window, so this is a same-session A/B, not a comparison against a recorded baseline:

• control = private PyAutoArray checkout at the branch's merge base d7c96762
• feature = 9e9a7d50 (Phase A at the old chunk 256, for the same-chunk A/B) and the shipped e5d31d74 at the new SIBSON_QUERY_CHUNK = 4096 default
• chunk sweep 512/1024/2048/4096 via PYAUTO_SIBSON_QUERY_CHUNK, with peak VRAM from a 2 s nvidia-smi sampler
• runtime cell (likelihood_runtime/imaging/delaunay_nn) on control, feature@256 and shipped

Jobs 342321–342325, 342329, 342330. Cell: HST / 3.5" mask / Hilbert-1500 image mesh / MGE-60 lens light / Isothermal + ExternalShear / ConstantSplit, fp64, dense inversion, A100 80GB PCIe. Breakdown flags --split-setup --vmap-batch 16.

The shared /mnt/ral/jnightin/PyAuto install was deliberately untouched — the subhalo-validation and Euclid DR1 CPU runs (342299, 342311, 342314) were live on it throughout. Every leg instead prepends a private PyAutoArray checkout to the PYTHONPATH activate.sh exports, and each job prints autoarray.__file__ plus the checkout's git rev-parse HEAD into its SLURM log. Recorded xla_flags are identical on all ten units (--xla_disable_hlo_passes=constant_folding --xla_gpu_autotune_level=0).

Headline — control vs the shipped feature

ms per likelihood call; vmap/16 is jax.jit(jax.vmap(fn)) at batch 16, reported as batch time / 16.

Step	control	shipped	Δ	control vmap/16	shipped vmap/16
Ray-trace data grid	0.169	0.169	−0.001	—	—
Ray-trace mesh grid	0.161	0.160	−0.001	—	—
Lens light images (pre-PSF)	0.138	0.133	−0.006	—	—
Blurred image (PSF convolution)	0.854	0.807	−0.047	—	—
Profile-subtracted image	0.138	0.123	−0.016	—	—
Inversion setup (steps 5–8 combined)	111.552	27.482	−84.071	20.364	17.674
Data vector (D)	0.348	0.336	−0.012	—	—
Curvature matrix (F)	4.804	4.812	+0.008	—	—
Regularization matrix (H)	45.103	13.768	−31.335	12.622	7.491
Regularized reconstruction	30.820	30.736	−0.084	—	—
Mapped recon + log evidence	2.269	2.191	−0.078	—	—
Total step-by-step (unbatched)	196.357	80.716	−115.641	—	—

Four-way setup split and the prefixes:

Piece	control	shipped	Δ	control vmap/16	shipped vmap/16
Border relocation	1.526	1.079	−0.447	0.092	0.064
Triangulation + interpolation	97.274	13.365	−83.909	11.603	8.879
Mapping matrix	−0.909	0.102	+1.011	0.074	−2.403
Blurred mapping matrix (PSF)	8.943	8.353	−0.590	10.705	10.808
Interpolator prefix (params→step 6)	98.800	14.444	−84.356	11.695	8.944
Split-Sibson prefix (params→split mappings)	134.159	14.242	−119.917	14.432	6.434
params→H prefix (the witness)	143.903	28.212	−115.691 (5.10×)	24.317	16.435 (1.48×)

The ±2 ms "Mapping matrix" sign flips are the documented by-difference artifact. The shipped leg's interpolator-prefix vmap number (8.944 ms) is that run's first prefix and carries warm-up; its strict superset in the same run reads 6.434 ms and the sweep's own 4096 row 6.398 ms — read the Sibson share at vmap 16 as ≈ 6.4 ms per call.

Whole-likelihood runtime cell

	control	feature @256	shipped @4096	Δ (control → shipped)
Full pipeline (single JIT)	201.320	136.082	76.322	−125.0 (2.64×)
vmap batch 16, per call	58.022	54.209	52.618	−5.4 (1.10×)

The unbatched saving (−125.0 ms) exceeds the params→H prefix drop (−115.7 ms) by the fusion slack expected when the whole pipeline is one program; the batched saving is small for the same reason as #531 — at batch 16 the dense linear algebra dominates.

Pins

EXPECTED_LOG_EVIDENCE_HST = 29144.581944 passed unchanged on every leg — both breakdown A/B legs, all four sweep chunks, the shipped re-measure, and pinned_drift: [] on all four runtime rows. The chunk is bit-neutral by construction and the sweep proves it end-to-end: five different lax.map block sizes, one log evidence.

The witness — half met

• Unbatched half (params→H ≤ 60 ms): MET. 143.90 → 28.21 ms at the shipped default (24.66 ms on the sweep's own 4096 row); met at every chunk ≥ 512.
• Batched half (≤ 11 ms per call at vmap 16): NOT met. 24.32 → 16.44 ms.

The new split-Sibson stage says why, and it is not the Sibson loops. At vmap 16 and the shipped chunk the 16.44 ms decomposes as:

piece	per call @vmap 16	comparison
Sibson (locate + both interpolation passes)	~6.4 ms	Delaunay's whole interpolator prefix: 5.07 ms
split-point Sibson residual after concatenation	~0 ms	(folded into the row above)
H, ConstantSplit assembly	~10.0 ms	Delaunay's H row: 0.07 ms

The assembly row reads 9.9–10.2 ms at every chunk and on the control — it is untouched by Phase A and cannot be touched by any change to the Sibson chunking. After Phase A roughly 60 % of the per-call params→H prefix is the 33-wide split-stencil regularization assembly, not the Sibson loops. That is why the ≤ 11 ms batched witness could not be met: 10 ms of it was never Sibson's to give. Unbatched the split is the other way round (14.44 ms interpolator prefix vs 13.97 ms assembly out of 28.21 ms) because the 33-wide block build amortises under vmap while the Sibson loops do not.

The chunk sweep and the 4096 default

PYAUTO_SIBSON_QUERY_CHUNK swept on the feature checkout; peak VRAM is a 2 s sampler floor on the true peak.

chunk	params→H unbatched	params→H per call @vmap 16	Sibson share @vmap 16	ConstantSplit assembly @vmap 16	peak sampled VRAM	pin
256 (9e9a7d50, 342322)	88.287	23.072	10.488	10.152	41,495 MiB	PASS
512 (342323_0)	52.205	18.243	8.309	10.014	41,503 MiB	PASS
1024 (342323_1)	35.324	19.567	7.062	12.557	41,503 MiB	PASS
2048 (342323_2)	27.218	19.127	6.602	10.058	41,503 MiB	PASS
4096 (342323_3)	24.664	16.451	6.398	10.028	41,503 MiB	PASS
4096 shipped (e5d31d74, 342329)	28.212	16.435	6.434	10.001	not sampled	PASS
control 256 (342321)	143.903	24.317	11.695	9.885	41,495 MiB	PASS

The 256 row is the same-chunk A/B: it isolates the unroll + single concatenated pass from the chunk change (143.903 → 88.287 ms unbatched, 1.63×). Everything below it is the chunk doing the rest.

Ruling: 4096 ships. The pre-registered rule (plan A.3) was "the largest of 512/1024/2048/4096 whose peak VRAM at vmap 16 stays under ~50 % of the 80 GB and which is fastest per call; if 4096 wins on both, ship 2048 unless the margin is > 15 %". 4096 is fastest on both readings, but the VRAM clause turned out uninformative and was therefore not used to break the tie: the ~41.5 GiB plateau (50.7 % of 81,920 MiB) is identical at every chunk and on the control leg, and the GPU read 0 MiB with no other process at the start of every job — so the plateau is this cell's vmap-16 dense inversion block, not the cavity intermediates. A 16× change of chunk moves it by 8 MiB. The guard arithmetic says why there is room: (C, 3, 2) fp64 intermediates at ~25 kB per query per lane over 4096 queries × 16 lanes is ~1.6 GB, ~2 % of an 80 GB card. With the VRAM leg vacuous the decision rests on speed alone — 4096 is 14.0 % faster per call at vmap 16 than 2048 (16.451 vs 19.127) and 9.4 % faster unbatched, a margin the "ship 2048 unless > 15 %" clause would have caught had the VRAM clause been live. PYAUTO_SIBSON_QUERY_CHUNK is kept as the documented escape hatch for a smaller GPU or a much larger cell.

Phase B — the numbers, not the decision

Left to the human; recorded here so the call is made on measured figures.

• Phase B as planned (cavity fori_loop → lax.while_loop early exit) targets the Sibson share only. Its headroom at vmap 16 is ~6.4 ms per call; observed cavity occupancy is 11 of the 32 fixed trips, and the plan's own estimate was ~20 % of the remaining Sibson cost — ≈ 1.3 ms per call at vmap 16, ≈ 3 ms unbatched.
• The ConstantSplit assembly is ~10.0 ms per call at vmap 16 — 1.6× the entire Sibson share and 143× Delaunay's equivalent H row. It is a 33-wide split-stencil block-diagonal build (reg_split_from) and nothing in Phase A or B touches it.
• For scale: the whole likelihood at vmap 16 is 52.6 ms per call, so Phase B's ~1.3 ms is ~2.5 % of a production evaluation and the assembly's 10.0 ms is ~19 %.

So the bigger per-call lever is a ConstantSplit-assembly rework, not the cavity early exit — by roughly 8× on the batched number production pays. Phase B remains the cheaper, lower-risk change (one loop, stop_gradient boundary already scoped, no numerical reordering) and is the only one of the two with a written plan; the assembly rework has no prompt yet. Phase C (loop-free k-ring cavity) is now the smallest of the three levers on the batched number and should be re-costed against these figures before it is planned.

Scripts Changed

• scripts/imaging/likelihood_breakdown/delaunay_nn.py — adds the --split-setup split-Sibson stage: a prefix stopping right after InterpolatorDelaunayNN._mappings_sizes_weights_split that returns all three split arrays alongside the step-6 outputs (a strict superset of prefix 6, a strict subset of prefix 11 — return fewer and XLA dead-code-eliminates the walk it exists to measure), yielding the "Split-point Sibson" and "H, ConstantSplit assembly" rows by the same prefix-difference arithmetic. Adds sibson_provenance() and a sibson block in the result JSON (effective SIBSON_QUERY_CHUNK / caps as imported, the mesh's own query_chunk, and the raw PYAUTO_SIBSON_QUERY_CHUNK override) plus split_sibson_prefix_s next to interpolator_prefix_s and regularization_matrix_prefix_s. The step list, steps, setup_split and every other pre-existing JSON field are unchanged, so older result JSONs stay comparable row-for-row. Docstring records the new 4096 chunk default.
• scripts/imaging/likelihood_runtime/delaunay_nn.py — docstring only: the recorded Sibson pins now read SIBSON_QUERY_CHUNK = 4096 (raised from 256 by PyAutoArray#533) with PYAUTO_SIBSON_QUERY_CHUNK named as the import-time override.

Submit scripts (new): hpc/batch_gpu/submit_breakdown_imaging_delaunay_nn_a100_hst_fp64_{launch_latency_control,launch_latency,chunk_sweep} and hpc/batch_gpu/submit_runtime_imaging_delaunay_nn_a100_hst_fp64_{control,launch_latency}. The #531 file submit_breakdown_imaging_delaunay_nn_a100_hst_fp64_control is untouched by this PR.

Results Added

• results/notes/delaunay_nn_launch_latency.md — the full written record (provenance table, revision table, every table above, the Phase B discussion)
• results/breakdown/imaging/delaunay_nn_hpc_a100_fp64_launch_latency_control.{json,png}
• results/breakdown/imaging/delaunay_nn_hpc_a100_fp64_launch_latency.{json,png} (shipped e5d31d74, chunk 4096)
• results/breakdown/imaging/delaunay_nn_hpc_a100_fp64_launch_latency_chunk256.{json,png} (9e9a7d50, the same-chunk A/B)
• results/breakdown/imaging/delaunay_nn_hpc_a100_fp64_chunk{512,1024,2048,4096}.{json,png}
• results/runtime/imaging/delaunay_nn/delaunay_nn_hpc_a100_fp64_launch_latency{_control,,_chunk256}.json

The 2026-09-07 ..._walk_early_exit rows in delaunay_walk_early_exit.md are left untouched; the control leg here supersedes them for any comparison against this branch (its merge base moved bcd15cd9 → d7c96762, and this checkout adds the split-point Sibson cut, so the row sets are not interchangeable).

Upstream PR

• Library: sibson: cut DelaunayNN kernel launches — gated candidate unroll, single concatenated pass, chunk as memory guard PyAutoArray#533 (commits 9e9a7d50 + e5d31d74, default chunk 4096)
• Workspace: delaunay_nn: assert single-pass parity, chunk invariance and the chunk env override autolens_workspace_test#307
• Issue: perf: DelaunayNN Sibson — unroll candidate loop, single concatenated pass, chunk as memory guard PyAutoArray#532

Library-first gate. These results were taken with the feature library on a private checkout, so they are valid regardless of merge order; the breakdown cell's new stage runs against either library (the split-Sibson prefix is defined by the interpolator API, not by the Phase A change). Merge after PyAutoArray#533 like any pending-release workspace PR.

Test Plan

• ruff check . — all checks passed
• ruff format --check . — 206 files already formatted
• scripts/misc/wall/check_submits.py --check — 55 submits with a wall-basis contract, 0 failing, 105 scanned
• scripts/misc/tooling/build_readme.py --check — 67 versioned artifacts scanned, all READMEs unchanged
• All 13 delaunay_nn result JSONs load
• hpc/sync.conf (gitignored) not staged; #531's ..._fp64_control submit not in the diff

Generated by the PyAutoLabs agent workflow.

🤖 Generated with Claude Code

https://claude.ai/code/session_01B5HT8dp7sWc9qDhZp6moGr