profiling: phase-2 instrumentation and the measured result (#507)

README.md

@@ -62,7 +62,7 @@ Results are framed by **astronomy instrument** (HST, Euclid, JWST, …) rather t
 | `imaging/delaunay` | hst | hpc_a100_mp | dense (mapping) | 96.8 ms | v2026.7.6.649 |
 | `imaging/delaunay` | hst | hpc_a100_mp | sparse (w-tilde) | 95.5 ms | v2026.7.6.649 |
 | `imaging/delaunay_numba` | euclid | local_cpu_fp64 | sparse (numba) | 1.19 s | v2026.8.17.1 |
-| `imaging/delaunay_numba` | hst | local_cpu_fp64 | sparse (numba) | 679.2 ms | v2026.8.17.1 |
+| `imaging/delaunay_numba` | hst | local_cpu_fp64 | sparse (numba) | 482.4 ms | v2026.8.17.1 |
 | `imaging/mge` | hst | local_cpu_fp64 | dense (mapping) | 179.5 ms | v2026.7.6.649 |
 | `imaging/mge` | hst | hpc_a100_fp64 | dense (mapping) | 7.8 ms | v2026.7.6.649 |
 | `imaging/pixelization` | hst | local_cpu_fp64 | dense (mapping) | 8.65 s | v2026.7.6.649 |
@@ -71,8 +71,8 @@ Results are framed by **astronomy instrument** (HST, Euclid, JWST, …) rather t
 | `imaging/pixelization` | hst | hpc_a100_fp64 | sparse (w-tilde) | 57.9 ms | v2026.7.6.649 |
 | `imaging/pixelization` | hst | hpc_a100_mp | dense (mapping) | 56.4 ms | v2026.7.6.649 |
 | `imaging/pixelization` | hst | hpc_a100_mp | sparse (w-tilde) | 55.5 ms | v2026.7.6.649 |
-| `imaging/pixelization_numba` | euclid | local_cpu_fp64 | sparse (numba) | 288.6 ms | v2026.8.17.1 |
-| `imaging/pixelization_numba` | hst | local_cpu_fp64 | sparse (numba) | 691.5 ms | v2026.8.17.1 |
+| `imaging/pixelization_numba` | euclid | local_cpu_fp64 | sparse (numba) | 124.3 ms | v2026.8.17.1 |
+| `imaging/pixelization_numba` | hst | local_cpu_fp64 | sparse (numba) | 304.5 ms | v2026.8.17.1 |
 <!-- END auto-table:headline -->
 
 The tables above are auto-generated by `scripts/misc/tooling/build_readme.py` from the artifacts under [`results/`](./results/README.md) — never edit them by hand; run `python scripts/misc/tooling/build_readme.py` after a profiling run and commit the result (CI checks idempotence via `--check`). Narrative context — per-cell "where to optimize next" recommendations and the mp-vs-fp64 verdicts — lives in [`scripts/misc/likelihood_runtime/OPTIMIZATION_NOTES.md`](./scripts/misc/likelihood_runtime/OPTIMIZATION_NOTES.md).

results/breakdown/imaging/delaunay_numba_breakdown_hst_v2026.8.17.1.json

@@ -25,33 +25,68 @@
     "omp_num_threads": "1"
   },
   "steps": {
-    "FitImaging construct": 7.37200001367455e-05,
-    "Blurred image (FFT convolve)": 0.010467500000095243,
-    "Profile subtracted image": 5.2989999994679236e-05,
-    "Inversion build (trace+Delaunay+mapper)": 0.02722744000016064,
-    "PSF-weighted data [numba]": 0.008432200000015655,
-    "MGE operated mapping matrix (60 funcs)": 0.2072877299999618,
-    "Mapper sparse triplets": 0.018328900000051364,
-    "Data vector D [numba]": 0.004825830000118003,
-    "F: mapper\u00d7mapper block [sparse-op]": 0.12305467000014687,
-    "F: mapper\u00d7linear-func block [FFT conv + scatter]": 0.0515464300000076,
-    "F: linear-func\u00d7linear-func block [BLAS]": 0.01916231999989577,
-    "Curvature matrix F [residual: mirror + diag-add]": -0.010250559999940373,
-    "Regularization matrix H (ConstantSplit)": 0.0681292599999324,
-    "F + H": 0.002749279999989085,
-    "Reconstruction solve (BLAS)": 0.03926761000002443,
-    "Mapped reconstruction [numba+FFT]": 0.0034941299999900364,
-    "Model data + chi^2": 0.003210120000039751,
-    "log det (F+H) [Cholesky]": 0.02755216999998993,
-    "log det H [Cholesky]": 0.024576129999877593,
-    "Regularization term s'Hs": 0.0007812499999545252,
-    "Log evidence (figure of merit)": 0.0492773199999192
+    "FitImaging construct": 7.01600003594649e-05,
+    "Blurred image (FFT convolve)": 0.009483080000063638,
+    "Profile subtracted image": 5.181000033189775e-05,
+    "Inversion build (trace+Delaunay+mapper)": 0.023930310000287135,
+    "PSF-weighted data [numba]": 0.008010229999672446,
+    "MGE: image_2d_from(grid) x60 [numpy]": 0.022226080000109505,
+    "MGE: image_2d_from(blurring_grid) x60 [numpy]": 0.01153678000009677,
+    "MGE: batched PSF convolution [real-space np]": 0.05661098999953538,
+    "MGE operated mapping matrix [residual: dict assembly]": 3.100000085395904e-05,
+    "Mapper sparse triplets": 0.017609500000071422,
+    "Data vector D [numba]": 0.004609609999715758,
+    "F: mapper\u00d7mapper block [sparse-op]": 0.06782932999958576,
+    "F: mapper\u00d7linear-func block [FFT conv + scatter]": 0.04873490999980277,
+    "F: linear-func\u00d7linear-func block [BLAS]": 0.017525710000336404,
+    "Curvature matrix F [residual: mirror + diag-add]": -0.005954410000049398,
+    "Regularization matrix H (ConstantSplit)": 0.06016452000058052,
+    "F + H": 0.001769830000193906,
+    "Reconstruction solve (BLAS)": 0.0378300099999251,
+    "Mapped reconstruction [numba+FFT]": 0.0035184800002753037,
+    "Model data + chi^2": 0.0034454800004823484,
+    "log det (F+H) [Cholesky]": 0.02494014000021707,
+    "log det H [Cholesky]": 0.023139239999909477,
+    "Regularization term s'Hs": 0.0006984200001170394,
+    "Log evidence (figure of merit)": 0.04456750999979704
+  },
+  "total_step_by_step": 0.4823787200022708,
+  "curvature_matrix_f_total_s": 0.12813553999967553,
+  "mge_operated_mapping_matrix_total_s": 0.09040485000059562,
+  "mge_operated_mapping_matrix_split_note": "The four 'MGE: ...' / 'MGE operated mapping matrix' rows sum to mge_operated_mapping_matrix_total_s, the raw un-split cost of the inversion.linear_func_operated_mapping_matrix_dict step. The three piece rows re-run, uncached, the three parts of LightProfileLinearObjFuncList.operated_mapping_matrix_override (60 profile images on the data grid, 60 on the blurring grid, one batched real-space PSF convolution); the residual row is the step minus those three. mge_split_reproduces_step records that the three pieces reconstruct the step's output bit-identically. PyAutoArray#507 step 0.",
+  "mge_split_reproduces_step": true,
+  "mge_blurring_stack_path": "shared_geometry",
+  "geometry": {
+    "psf_shape_native": [
+      21,
+      21
+    ],
+    "image_pixels_masked": 15361,
+    "psf_precision_pairs_stored": 11080573,
+    "psf_precision_lengths_sum": 11080573,
+    "psf_precision_lengths_mean": 721.3445088210403,
+    "psf_precision_lengths_max": 841,
+    "mappers": [
+      {
+        "params": 1250,
+        "data_to_pix_unique_shape": [
+          15361,
+          3
+        ],
+        "pix_lengths_mean": 1.5523077924614284,
+        "pix_lengths_max": 3,
+        "pix_lengths_sum": 23845,
+        "mapper_mapper_inner_ops": 32691733,
+        "two_stage_stage_1_scatter_ops": 18187061,
+        "two_stage_stage_2_axpy_ops": 29806250,
+        "two_stage_clear_ops": 19201250,
+        "two_stage_total_ops": 67194561
+      }
+    ]
   },
-  "total_step_by_step": 0.6792464400003608,
-  "curvature_matrix_f_total_s": 0.18351286000010986,
   "curvature_matrix_f_split_note": "The four 'F: ...' / 'Curvature matrix F' rows sum to curvature_matrix_f_total_s, the raw un-split cost of the inversion.curvature_matrix step. The three block rows are direct measurements of the (uncached) per-block helpers in imaging_numba/sparse.py; the 'Curvature matrix F [residual: ...]' row is t(curvature_matrix) minus those three, i.e. the global mirror, the no-regularization diagonal add and assembly overhead, and it carries the combined noise of the four timings. PyAutoArray#505 step 0.",
-  "direct_log_likelihood_function_per_call": 0.7576715100000001,
-  "warmup_incl_numba_compile_s": 4.814360099999703,
+  "direct_log_likelihood_function_per_call": 0.4267223799999556,
+  "warmup_incl_numba_compile_s": 3.621436499999618,
   "log_likelihood": 29090.527210448134,
   "pinned_expected": 29090.527192092646,
   "pinned_drift": []

results/breakdown/imaging/pixelization_numba_breakdown_euclid_v2026.8.17.1.json

@@ -25,33 +25,68 @@
     "omp_num_threads": "1"
   },
   "steps": {
-    "FitImaging construct": 7.546999995611259e-05,
-    "Blurred image (FFT convolve)": 0.008001979999926334,
-    "Profile subtracted image": 3.681000011965807e-05,
-    "Inversion build (trace+mesh+mapper)": 0.01710581000006641,
-    "PSF-weighted data [numba]": 0.0026192299999820536,
-    "MGE operated mapping matrix (60 funcs)": 0.10346844000009696,
-    "Mapper sparse triplets": 0.0016650700000354845,
-    "Data vector D [numba]": 0.001205819999859159,
-    "F: mapper\u00d7mapper block [sparse-op]": 0.06530645000011645,
-    "F: mapper\u00d7linear-func block [FFT conv + scatter]": 0.022554969999964668,
-    "F: linear-func\u00d7linear-func block [BLAS]": 0.004474150000078225,
-    "Curvature matrix F [residual: mirror + diag-add]": 0.004506459999856813,
-    "Regularization matrix H": 0.004038749999972424,
-    "F + H": 0.0012899899999865739,
-    "Reconstruction solve (BLAS)": 0.0099663199999668,
-    "Mapped reconstruction [numba+FFT]": 0.0015609000000040396,
-    "Model data + chi^2": 0.001559590000033495,
-    "log det (F+H) [Cholesky]": 0.010603099999934784,
-    "log det H [Cholesky]": 0.00950673000002098,
-    "Regularization term s'Hs": 0.0004289500000595581,
-    "Log evidence (figure of merit)": 0.018604910000021846
+    "FitImaging construct": 6.200999996508472e-05,
+    "Blurred image (FFT convolve)": 0.0056353500001932845,
+    "Profile subtracted image": 2.655000043887412e-05,
+    "Inversion build (trace+mesh+mapper)": 0.012572869999530667,
+    "PSF-weighted data [numba]": 0.002042729999811854,
+    "MGE: image_2d_from(grid) x60 [numpy]": 0.007493010000325739,
+    "MGE: image_2d_from(blurring_grid) x60 [numpy]": 0.006900779999705264,
+    "MGE: batched PSF convolution [real-space np]": 0.015458829999624868,
+    "MGE operated mapping matrix [residual: dict assembly]": -0.0011304999994536033,
+    "Mapper sparse triplets": 0.001228639999862935,
+    "Data vector D [numba]": 0.0007602999996379368,
+    "F: mapper\u00d7mapper block [sparse-op]": 0.017563919999702193,
+    "F: mapper\u00d7linear-func block [FFT conv + scatter]": 0.014607230000365234,
+    "F: linear-func\u00d7linear-func block [BLAS]": 0.0033264399999097805,
+    "Curvature matrix F [residual: mirror + diag-add]": 0.00015849000028538007,
+    "Regularization matrix H": 0.0016086299998278263,
+    "F + H": 0.0006751000000804197,
+    "Reconstruction solve (BLAS)": 0.0067057800002658045,
+    "Mapped reconstruction [numba+FFT]": 0.001095799999711744,
+    "Model data + chi^2": 0.0012144199999966077,
+    "log det (F+H) [Cholesky]": 0.007371300000340852,
+    "log det H [Cholesky]": 0.006293040000127803,
+    "Regularization term s'Hs": 0.00035453999989840666,
+    "Log evidence (figure of merit)": 0.012255779999759398
+  },
+  "total_step_by_step": 0.12428103999991434,
+  "curvature_matrix_f_total_s": 0.03565608000026259,
+  "mge_operated_mapping_matrix_total_s": 0.028722120000202266,
+  "mge_operated_mapping_matrix_split_note": "The four 'MGE: ...' / 'MGE operated mapping matrix' rows sum to mge_operated_mapping_matrix_total_s, the raw un-split cost of the inversion.linear_func_operated_mapping_matrix_dict step. The three piece rows re-run, uncached, the three parts of LightProfileLinearObjFuncList.operated_mapping_matrix_override (60 profile images on the data grid, 60 on the blurring grid, one batched real-space PSF convolution); the residual row is the step minus those three. mge_split_reproduces_step records that the three pieces reconstruct the step's output bit-identically. PyAutoArray#507 step 0.",
+  "mge_split_reproduces_step": true,
+  "mge_blurring_stack_path": "shared_geometry",
+  "geometry": {
+    "psf_shape_native": [
+      21,
+      21
+    ],
+    "image_pixels_masked": 3841,
+    "psf_precision_pairs_stored": 2318869,
+    "psf_precision_lengths_sum": 2318869,
+    "psf_precision_lengths_mean": 603.7149179901068,
+    "psf_precision_lengths_max": 841,
+    "mappers": [
+      {
+        "params": 784,
+        "data_to_pix_unique_shape": [
+          3841,
+          4
+        ],
+        "pix_lengths_mean": 4.0,
+        "pix_lengths_max": 4,
+        "pix_lengths_sum": 15364,
+        "mapper_mapper_inner_ops": 37101904,
+        "two_stage_stage_1_scatter_ops": 9275476,
+        "two_stage_stage_2_axpy_ops": 12045376,
+        "two_stage_clear_ops": 3011344,
+        "two_stage_total_ops": 24332196
+      }
+    ]
   },
-  "total_step_by_step": 0.28857990000005884,
-  "curvature_matrix_f_total_s": 0.09684203000001616,
   "curvature_matrix_f_split_note": "The four 'F: ...' / 'Curvature matrix F' rows sum to curvature_matrix_f_total_s, the raw un-split cost of the inversion.curvature_matrix step. The three block rows are direct measurements of the (uncached) per-block helpers in imaging_numba/sparse.py; the 'Curvature matrix F [residual: ...]' row is t(curvature_matrix) minus those three, i.e. the global mirror, the no-regularization diagonal add and assembly overhead, and it carries the combined noise of the four timings. PyAutoArray#505 step 0.",
-  "direct_log_likelihood_function_per_call": 0.24940190999996048,
-  "warmup_incl_numba_compile_s": 2.1857472000001508,
+  "direct_log_likelihood_function_per_call": 0.11088282000000618,
+  "warmup_incl_numba_compile_s": 1.5452468999992561,
   "log_likelihood": 6213.306873885871,
   "pinned_expected": null,
   "pinned_drift": []

results/breakdown/imaging/pixelization_numba_breakdown_hst_v2026.8.17.1.json

@@ -25,33 +25,68 @@
     "omp_num_threads": "1"
   },
   "steps": {
-    "FitImaging construct": 8.33399999919493e-05,
-    "Blurred image (FFT convolve)": 0.01084737999999561,
-    "Profile subtracted image": 6.784000001971435e-05,
-    "Inversion build (trace+mesh+mapper)": 0.026095339999892532,
-    "PSF-weighted data [numba]": 0.008763999999928274,
-    "MGE operated mapping matrix (60 funcs)": 0.22401575999997475,
-    "Mapper sparse triplets": 0.005336110000007466,
-    "Data vector D [numba]": 0.0052595000000565054,
-    "F: mapper\u00d7mapper block [sparse-op]": 0.27679772000001324,
-    "F: mapper\u00d7linear-func block [FFT conv + scatter]": 0.05443964000005508,
-    "F: linear-func\u00d7linear-func block [BLAS]": 0.021487699999988764,
-    "Curvature matrix F [residual: mirror + diag-add]": 0.0066316599999936,
-    "Regularization matrix H": 0.0018421100000068692,
-    "F + H": 0.0008287500000278669,
-    "Reconstruction solve (BLAS)": 0.00876641000008931,
-    "Mapped reconstruction [numba+FFT]": 0.003643519999968703,
-    "Model data + chi^2": 0.0037507700000787734,
-    "log det (F+H) [Cholesky]": 0.008669489999920187,
-    "log det H [Cholesky]": 0.007879109999976208,
-    "Regularization term s'Hs": 0.00032316000001628706,
-    "Log evidence (figure of merit)": 0.015974480000113544
+    "FitImaging construct": 6.414999952539802e-05,
+    "Blurred image (FFT convolve)": 0.00926681000037206,
+    "Profile subtracted image": 4.731000008177944e-05,
+    "Inversion build (trace+mesh+mapper)": 0.021709480000026815,
+    "PSF-weighted data [numba]": 0.007878119999440969,
+    "MGE: image_2d_from(grid) x60 [numpy]": 0.02258933999983128,
+    "MGE: image_2d_from(blurring_grid) x60 [numpy]": 0.010614800000075775,
+    "MGE: batched PSF convolution [real-space np]": 0.046522589999585764,
+    "MGE operated mapping matrix [residual: dict assembly]": -0.003177729998969883,
+    "Mapper sparse triplets": 0.004005030000007537,
+    "Data vector D [numba]": 0.003737139999975625,
+    "F: mapper\u00d7mapper block [sparse-op]": 0.0751765200000591,
+    "F: mapper\u00d7linear-func block [FFT conv + scatter]": 0.045131479999690784,
+    "F: linear-func\u00d7linear-func block [BLAS]": 0.017325829999936104,
+    "Curvature matrix F [residual: mirror + diag-add]": -0.0015314400001443518,
+    "Regularization matrix H": 0.0013350499997613952,
+    "F + H": 0.0006081399998947746,
+    "Reconstruction solve (BLAS)": 0.007613869999840972,
+    "Mapped reconstruction [numba+FFT]": 0.002937240000028396,
+    "Model data + chi^2": 0.003083939999669383,
+    "log det (F+H) [Cholesky]": 0.007987400000092747,
+    "log det H [Cholesky]": 0.007959590000427851,
+    "Regularization term s'Hs": 0.00029186999981902775,
+    "Log evidence (figure of merit)": 0.01328899000036472
+  },
+  "total_step_by_step": 0.30446551999939403,
+  "curvature_matrix_f_total_s": 0.13610238999954163,
+  "mge_operated_mapping_matrix_total_s": 0.07654900000052293,
+  "mge_operated_mapping_matrix_split_note": "The four 'MGE: ...' / 'MGE operated mapping matrix' rows sum to mge_operated_mapping_matrix_total_s, the raw un-split cost of the inversion.linear_func_operated_mapping_matrix_dict step. The three piece rows re-run, uncached, the three parts of LightProfileLinearObjFuncList.operated_mapping_matrix_override (60 profile images on the data grid, 60 on the blurring grid, one batched real-space PSF convolution); the residual row is the step minus those three. mge_split_reproduces_step records that the three pieces reconstruct the step's output bit-identically. PyAutoArray#507 step 0.",
+  "mge_split_reproduces_step": true,
+  "mge_blurring_stack_path": "shared_geometry",
+  "geometry": {
+    "psf_shape_native": [
+      21,
+      21
+    ],
+    "image_pixels_masked": 15361,
+    "psf_precision_pairs_stored": 11080573,
+    "psf_precision_lengths_sum": 11080573,
+    "psf_precision_lengths_mean": 721.3445088210403,
+    "psf_precision_lengths_max": 841,
+    "mappers": [
+      {
+        "params": 784,
+        "data_to_pix_unique_shape": [
+          15361,
+          4
+        ],
+        "pix_lengths_mean": 4.0,
+        "pix_lengths_max": 4,
+        "pix_lengths_sum": 61444,
+        "mapper_mapper_inner_ops": 177289168,
+        "two_stage_stage_1_scatter_ops": 44322292,
+        "two_stage_stage_2_axpy_ops": 48172096,
+        "two_stage_clear_ops": 12043024,
+        "two_stage_total_ops": 104537412
+      }
+    ]
   },
-  "total_step_by_step": 0.6915037900001153,
-  "curvature_matrix_f_total_s": 0.3593567200000507,
   "curvature_matrix_f_split_note": "The four 'F: ...' / 'Curvature matrix F' rows sum to curvature_matrix_f_total_s, the raw un-split cost of the inversion.curvature_matrix step. The three block rows are direct measurements of the (uncached) per-block helpers in imaging_numba/sparse.py; the 'Curvature matrix F [residual: ...]' row is t(curvature_matrix) minus those three, i.e. the global mirror, the no-regularization diagonal add and assembly overhead, and it carries the combined noise of the four timings. PyAutoArray#505 step 0.",
-  "direct_log_likelihood_function_per_call": 0.5949823200000083,
-  "warmup_incl_numba_compile_s": 4.023859600000378,
+  "direct_log_likelihood_function_per_call": 0.2970425999999861,
+  "warmup_incl_numba_compile_s": 3.0814270000009856,
   "log_likelihood": 27661.91013366411,
   "pinned_expected": 27661.910133665442,
   "pinned_drift": []