diff --git a/hpc/batch_gpu/submit_breakdown_imaging_delaunay_a100_hst_fp64_control b/hpc/batch_gpu/submit_breakdown_imaging_delaunay_a100_hst_fp64_control new file mode 100644 index 000000000..8912f5ecc --- /dev/null +++ b/hpc/batch_gpu/submit_breakdown_imaging_delaunay_a100_hst_fp64_control @@ -0,0 +1,85 @@ +#!/bin/bash -l +# +# A100 likelihood-breakdown profile: imaging/delaunay x hst x fp64 — PyAutoArray#531 +# A/B leg: **control**. +# +# Verification run for the JAX Delaunay walk early-exit change +# (PyAutoArray issue #530 / PR #531: lax.while_loop early exit, seed argmin +# chunked, single concatenated locate). Both legs of the A/B run in the same +# session on the same node class with the same everything except the +# PyAutoArray checkout prepended to PYTHONPATH: +# +# control -> bcd15cd9 (the branch's merge base = PyAutoArray main) +# feature -> 6152bdf0 (feature/delaunay-walk-early-exit) +# +# The shared /mnt/ral/jnightin/PyAuto/PyAutoArray install is NOT touched — long +# science runs (subhalo validation, Euclid DR1) share it. The private checkouts +# live under /mnt/ral/jnightin/PyAuto_wt/delaunay-walk-early-exit/ and are +# prepended to the PYTHONPATH that $AP_ROOT/activate.sh exports, so only +# PyAutoArray is overridden and Nerves/Fit/Galaxy/Lens stay on the shared tree. +# +# WITNESS. The feature concatenates the data-grid and split-point walks into a +# single locate call, so the split walk now runs *inside* the "Triangulation + +# interpolation" prefix; before, XLA dead-code-eliminated it there and its cost +# was charged to the "Regularization matrix (H)" row (a prefix difference). +# Compare the **params->H prefix** (regularization_matrix_prefix_s, equivalently +# Tri+interp + H) across the two legs — the H row alone can go ~0 or negative. +# +# --split-setup four-way split of the inversion-setup block +# --vmap-batch 16 batched re-timing at the Nautilus reference n_batch + +#SBATCH -J bd_delaunay_ctrl +#SBATCH --partition=gpu +#SBATCH --gres=gpu:1 +#SBATCH --ntasks=1 +#SBATCH --cpus-per-task=4 +#SBATCH --mem=64gb +#SBATCH --time=12:00:00 +#SBATCH -o output/output.%A.out +#SBATCH -e error/error.%A.err + +export AP_ROOT="$(cd "${SLURM_SUBMIT_DIR:-$(dirname "$0")}/../.." && pwd)" +echo "AP_ROOT=$AP_ROOT" +source $AP_ROOT/activate.sh + +# --- PyAutoArray A/B override (private checkout, shared install untouched) --- +AA_ROOT=/mnt/ral/jnightin/PyAuto_wt/delaunay-walk-early-exit/PyAutoArray_control +export PYTHONPATH=$AA_ROOT:$PYTHONPATH + +export JAX_PLATFORM_NAME=cuda +export JAX_PLATFORMS=cuda,cpu +export XLA_PYTHON_CLIENT_PREALLOCATE=false +export JAX_ENABLE_X64=True +export XLA_FLAGS="--xla_disable_hlo_passes=constant_folding" +export NUMBA_CACHE_DIR=/tmp/numba_cache +export MPLCONFIGDIR=/tmp/matplotlib + +nvidia-smi + +echo "==========================================" +date +echo "Cell: likelihood_breakdown/imaging/delaunay" +echo "Instrument: hst" +echo "Precision: fp64" +echo "A/B leg: control" +echo "Flags: --split-setup --vmap-batch 16 --config-name hpc_a100_fp64_control_2026_09_07" + +echo "=== source revisions ===" +printf '%-14s %s\n' "PyAutoArray(AB)" "$(git -C $AA_ROOT rev-parse HEAD 2>&1)" +for r in PyAutoNerves PyAutoFit PyAutoGalaxy PyAutoLens; do + printf '%-14s %s\n' "$r" "$(git -C /mnt/ral/jnightin/PyAuto/$r rev-parse HEAD 2>&1)" +done +echo "AP_ROOT rev: $(git -C $AP_ROOT rev-parse HEAD 2>&1)" + +cd $AP_ROOT + +echo "=== import provenance ===" +python3 -c "import autoarray, autolens; print('autoarray.__file__ =', autoarray.__file__); print('autoarray.__version__ =', autoarray.__version__); print('autolens.__file__ =', autolens.__file__)" + +python3 -u scripts/imaging/likelihood_breakdown/delaunay.py \ + --config-name hpc_a100_fp64_control_2026_09_07 \ + --split-setup \ + --vmap-batch 16 + +echo "Finished." +date diff --git a/hpc/batch_gpu/submit_breakdown_imaging_delaunay_a100_hst_fp64_walk_early_exit b/hpc/batch_gpu/submit_breakdown_imaging_delaunay_a100_hst_fp64_walk_early_exit new file mode 100644 index 000000000..e83849d9c --- /dev/null +++ b/hpc/batch_gpu/submit_breakdown_imaging_delaunay_a100_hst_fp64_walk_early_exit @@ -0,0 +1,85 @@ +#!/bin/bash -l +# +# A100 likelihood-breakdown profile: imaging/delaunay x hst x fp64 — PyAutoArray#531 +# A/B leg: **walk_early_exit**. +# +# Verification run for the JAX Delaunay walk early-exit change +# (PyAutoArray issue #530 / PR #531: lax.while_loop early exit, seed argmin +# chunked, single concatenated locate). Both legs of the A/B run in the same +# session on the same node class with the same everything except the +# PyAutoArray checkout prepended to PYTHONPATH: +# +# control -> bcd15cd9 (the branch's merge base = PyAutoArray main) +# feature -> 6152bdf0 (feature/delaunay-walk-early-exit) +# +# The shared /mnt/ral/jnightin/PyAuto/PyAutoArray install is NOT touched — long +# science runs (subhalo validation, Euclid DR1) share it. The private checkouts +# live under /mnt/ral/jnightin/PyAuto_wt/delaunay-walk-early-exit/ and are +# prepended to the PYTHONPATH that $AP_ROOT/activate.sh exports, so only +# PyAutoArray is overridden and Nerves/Fit/Galaxy/Lens stay on the shared tree. +# +# WITNESS. The feature concatenates the data-grid and split-point walks into a +# single locate call, so the split walk now runs *inside* the "Triangulation + +# interpolation" prefix; before, XLA dead-code-eliminated it there and its cost +# was charged to the "Regularization matrix (H)" row (a prefix difference). +# Compare the **params->H prefix** (regularization_matrix_prefix_s, equivalently +# Tri+interp + H) across the two legs — the H row alone can go ~0 or negative. +# +# --split-setup four-way split of the inversion-setup block +# --vmap-batch 16 batched re-timing at the Nautilus reference n_batch + +#SBATCH -J bd_delaunay_walk +#SBATCH --partition=gpu +#SBATCH --gres=gpu:1 +#SBATCH --ntasks=1 +#SBATCH --cpus-per-task=4 +#SBATCH --mem=64gb +#SBATCH --time=12:00:00 +#SBATCH -o output/output.%A.out +#SBATCH -e error/error.%A.err + +export AP_ROOT="$(cd "${SLURM_SUBMIT_DIR:-$(dirname "$0")}/../.." && pwd)" +echo "AP_ROOT=$AP_ROOT" +source $AP_ROOT/activate.sh + +# --- PyAutoArray A/B override (private checkout, shared install untouched) --- +AA_ROOT=/mnt/ral/jnightin/PyAuto_wt/delaunay-walk-early-exit/PyAutoArray_feature +export PYTHONPATH=$AA_ROOT:$PYTHONPATH + +export JAX_PLATFORM_NAME=cuda +export JAX_PLATFORMS=cuda,cpu +export XLA_PYTHON_CLIENT_PREALLOCATE=false +export JAX_ENABLE_X64=True +export XLA_FLAGS="--xla_disable_hlo_passes=constant_folding" +export NUMBA_CACHE_DIR=/tmp/numba_cache +export MPLCONFIGDIR=/tmp/matplotlib + +nvidia-smi + +echo "==========================================" +date +echo "Cell: likelihood_breakdown/imaging/delaunay" +echo "Instrument: hst" +echo "Precision: fp64" +echo "A/B leg: walk_early_exit" +echo "Flags: --split-setup --vmap-batch 16 --config-name hpc_a100_fp64_walk_early_exit" + +echo "=== source revisions ===" +printf '%-14s %s\n' "PyAutoArray(AB)" "$(git -C $AA_ROOT rev-parse HEAD 2>&1)" +for r in PyAutoNerves PyAutoFit PyAutoGalaxy PyAutoLens; do + printf '%-14s %s\n' "$r" "$(git -C /mnt/ral/jnightin/PyAuto/$r rev-parse HEAD 2>&1)" +done +echo "AP_ROOT rev: $(git -C $AP_ROOT rev-parse HEAD 2>&1)" + +cd $AP_ROOT + +echo "=== import provenance ===" +python3 -c "import autoarray, autolens; print('autoarray.__file__ =', autoarray.__file__); print('autoarray.__version__ =', autoarray.__version__); print('autolens.__file__ =', autolens.__file__)" + +python3 -u scripts/imaging/likelihood_breakdown/delaunay.py \ + --config-name hpc_a100_fp64_walk_early_exit \ + --split-setup \ + --vmap-batch 16 + +echo "Finished." +date diff --git a/hpc/batch_gpu/submit_breakdown_imaging_delaunay_nn_a100_hst_fp64_control b/hpc/batch_gpu/submit_breakdown_imaging_delaunay_nn_a100_hst_fp64_control new file mode 100644 index 000000000..b69c894e7 --- /dev/null +++ b/hpc/batch_gpu/submit_breakdown_imaging_delaunay_nn_a100_hst_fp64_control @@ -0,0 +1,85 @@ +#!/bin/bash -l +# +# A100 likelihood-breakdown profile: imaging/delaunay_nn x hst x fp64 — PyAutoArray#531 +# A/B leg: **control**. +# +# Verification run for the JAX Delaunay walk early-exit change +# (PyAutoArray issue #530 / PR #531: lax.while_loop early exit, seed argmin +# chunked, single concatenated locate). Both legs of the A/B run in the same +# session on the same node class with the same everything except the +# PyAutoArray checkout prepended to PYTHONPATH: +# +# control -> bcd15cd9 (the branch's merge base = PyAutoArray main) +# feature -> 6152bdf0 (feature/delaunay-walk-early-exit) +# +# The shared /mnt/ral/jnightin/PyAuto/PyAutoArray install is NOT touched — long +# science runs (subhalo validation, Euclid DR1) share it. The private checkouts +# live under /mnt/ral/jnightin/PyAuto_wt/delaunay-walk-early-exit/ and are +# prepended to the PYTHONPATH that $AP_ROOT/activate.sh exports, so only +# PyAutoArray is overridden and Nerves/Fit/Galaxy/Lens stay on the shared tree. +# +# WITNESS. The feature concatenates the data-grid and split-point walks into a +# single locate call, so the split walk now runs *inside* the "Triangulation + +# interpolation" prefix; before, XLA dead-code-eliminated it there and its cost +# was charged to the "Regularization matrix (H)" row (a prefix difference). +# Compare the **params->H prefix** (regularization_matrix_prefix_s, equivalently +# Tri+interp + H) across the two legs — the H row alone can go ~0 or negative. +# +# --split-setup four-way split of the inversion-setup block +# --vmap-batch 16 batched re-timing at the Nautilus reference n_batch + +#SBATCH -J bd_delnn_ctrl +#SBATCH --partition=gpu +#SBATCH --gres=gpu:1 +#SBATCH --ntasks=1 +#SBATCH --cpus-per-task=4 +#SBATCH --mem=64gb +#SBATCH --time=12:00:00 +#SBATCH -o output/output.%A.out +#SBATCH -e error/error.%A.err + +export AP_ROOT="$(cd "${SLURM_SUBMIT_DIR:-$(dirname "$0")}/../.." && pwd)" +echo "AP_ROOT=$AP_ROOT" +source $AP_ROOT/activate.sh + +# --- PyAutoArray A/B override (private checkout, shared install untouched) --- +AA_ROOT=/mnt/ral/jnightin/PyAuto_wt/delaunay-walk-early-exit/PyAutoArray_control +export PYTHONPATH=$AA_ROOT:$PYTHONPATH + +export JAX_PLATFORM_NAME=cuda +export JAX_PLATFORMS=cuda,cpu +export XLA_PYTHON_CLIENT_PREALLOCATE=false +export JAX_ENABLE_X64=True +export XLA_FLAGS="--xla_disable_hlo_passes=constant_folding" +export NUMBA_CACHE_DIR=/tmp/numba_cache +export MPLCONFIGDIR=/tmp/matplotlib + +nvidia-smi + +echo "==========================================" +date +echo "Cell: likelihood_breakdown/imaging/delaunay_nn" +echo "Instrument: hst" +echo "Precision: fp64" +echo "A/B leg: control" +echo "Flags: --split-setup --vmap-batch 16 --config-name hpc_a100_fp64_control_2026_09_07" + +echo "=== source revisions ===" +printf '%-14s %s\n' "PyAutoArray(AB)" "$(git -C $AA_ROOT rev-parse HEAD 2>&1)" +for r in PyAutoNerves PyAutoFit PyAutoGalaxy PyAutoLens; do + printf '%-14s %s\n' "$r" "$(git -C /mnt/ral/jnightin/PyAuto/$r rev-parse HEAD 2>&1)" +done +echo "AP_ROOT rev: $(git -C $AP_ROOT rev-parse HEAD 2>&1)" + +cd $AP_ROOT + +echo "=== import provenance ===" +python3 -c "import autoarray, autolens; print('autoarray.__file__ =', autoarray.__file__); print('autoarray.__version__ =', autoarray.__version__); print('autolens.__file__ =', autolens.__file__)" + +python3 -u scripts/imaging/likelihood_breakdown/delaunay_nn.py \ + --config-name hpc_a100_fp64_control_2026_09_07 \ + --split-setup \ + --vmap-batch 16 + +echo "Finished." +date diff --git a/hpc/batch_gpu/submit_breakdown_imaging_delaunay_nn_a100_hst_fp64_walk_early_exit b/hpc/batch_gpu/submit_breakdown_imaging_delaunay_nn_a100_hst_fp64_walk_early_exit new file mode 100644 index 000000000..25becb213 --- /dev/null +++ b/hpc/batch_gpu/submit_breakdown_imaging_delaunay_nn_a100_hst_fp64_walk_early_exit @@ -0,0 +1,85 @@ +#!/bin/bash -l +# +# A100 likelihood-breakdown profile: imaging/delaunay_nn x hst x fp64 — PyAutoArray#531 +# A/B leg: **walk_early_exit**. +# +# Verification run for the JAX Delaunay walk early-exit change +# (PyAutoArray issue #530 / PR #531: lax.while_loop early exit, seed argmin +# chunked, single concatenated locate). Both legs of the A/B run in the same +# session on the same node class with the same everything except the +# PyAutoArray checkout prepended to PYTHONPATH: +# +# control -> bcd15cd9 (the branch's merge base = PyAutoArray main) +# feature -> 6152bdf0 (feature/delaunay-walk-early-exit) +# +# The shared /mnt/ral/jnightin/PyAuto/PyAutoArray install is NOT touched — long +# science runs (subhalo validation, Euclid DR1) share it. The private checkouts +# live under /mnt/ral/jnightin/PyAuto_wt/delaunay-walk-early-exit/ and are +# prepended to the PYTHONPATH that $AP_ROOT/activate.sh exports, so only +# PyAutoArray is overridden and Nerves/Fit/Galaxy/Lens stay on the shared tree. +# +# WITNESS. The feature concatenates the data-grid and split-point walks into a +# single locate call, so the split walk now runs *inside* the "Triangulation + +# interpolation" prefix; before, XLA dead-code-eliminated it there and its cost +# was charged to the "Regularization matrix (H)" row (a prefix difference). +# Compare the **params->H prefix** (regularization_matrix_prefix_s, equivalently +# Tri+interp + H) across the two legs — the H row alone can go ~0 or negative. +# +# --split-setup four-way split of the inversion-setup block +# --vmap-batch 16 batched re-timing at the Nautilus reference n_batch + +#SBATCH -J bd_delnn_walk +#SBATCH --partition=gpu +#SBATCH --gres=gpu:1 +#SBATCH --ntasks=1 +#SBATCH --cpus-per-task=4 +#SBATCH --mem=64gb +#SBATCH --time=12:00:00 +#SBATCH -o output/output.%A.out +#SBATCH -e error/error.%A.err + +export AP_ROOT="$(cd "${SLURM_SUBMIT_DIR:-$(dirname "$0")}/../.." && pwd)" +echo "AP_ROOT=$AP_ROOT" +source $AP_ROOT/activate.sh + +# --- PyAutoArray A/B override (private checkout, shared install untouched) --- +AA_ROOT=/mnt/ral/jnightin/PyAuto_wt/delaunay-walk-early-exit/PyAutoArray_feature +export PYTHONPATH=$AA_ROOT:$PYTHONPATH + +export JAX_PLATFORM_NAME=cuda +export JAX_PLATFORMS=cuda,cpu +export XLA_PYTHON_CLIENT_PREALLOCATE=false +export JAX_ENABLE_X64=True +export XLA_FLAGS="--xla_disable_hlo_passes=constant_folding" +export NUMBA_CACHE_DIR=/tmp/numba_cache +export MPLCONFIGDIR=/tmp/matplotlib + +nvidia-smi + +echo "==========================================" +date +echo "Cell: likelihood_breakdown/imaging/delaunay_nn" +echo "Instrument: hst" +echo "Precision: fp64" +echo "A/B leg: walk_early_exit" +echo "Flags: --split-setup --vmap-batch 16 --config-name hpc_a100_fp64_walk_early_exit" + +echo "=== source revisions ===" +printf '%-14s %s\n' "PyAutoArray(AB)" "$(git -C $AA_ROOT rev-parse HEAD 2>&1)" +for r in PyAutoNerves PyAutoFit PyAutoGalaxy PyAutoLens; do + printf '%-14s %s\n' "$r" "$(git -C /mnt/ral/jnightin/PyAuto/$r rev-parse HEAD 2>&1)" +done +echo "AP_ROOT rev: $(git -C $AP_ROOT rev-parse HEAD 2>&1)" + +cd $AP_ROOT + +echo "=== import provenance ===" +python3 -c "import autoarray, autolens; print('autoarray.__file__ =', autoarray.__file__); print('autoarray.__version__ =', autoarray.__version__); print('autolens.__file__ =', autolens.__file__)" + +python3 -u scripts/imaging/likelihood_breakdown/delaunay_nn.py \ + --config-name hpc_a100_fp64_walk_early_exit \ + --split-setup \ + --vmap-batch 16 + +echo "Finished." +date diff --git a/hpc/batch_gpu/submit_runtime_imaging_delaunay_a100_hst_fp64_control b/hpc/batch_gpu/submit_runtime_imaging_delaunay_a100_hst_fp64_control new file mode 100644 index 000000000..61d510348 --- /dev/null +++ b/hpc/batch_gpu/submit_runtime_imaging_delaunay_a100_hst_fp64_control @@ -0,0 +1,50 @@ +#!/bin/bash -l +# +# A100 likelihood-runtime profile: imaging/delaunay x hst x fp64 — PyAutoArray#531 +# A/B leg: **control**. Single-JIT whole-likelihood cost, the end-to-end companion +# to submit_breakdown_imaging_delaunay_a100_hst_fp64_control. +# +# Same private-checkout PYTHONPATH override as the breakdown A/B legs; the +# shared /mnt/ral/jnightin/PyAuto install is untouched. Phase A (--vmap-probe) +# is skipped: only the full single-JIT + vmap timing is wanted here. + +#SBATCH -J rt_delaunay_control +#SBATCH --partition=gpu +#SBATCH --gres=gpu:1 +#SBATCH --ntasks=1 +#SBATCH --cpus-per-task=4 +#SBATCH --mem=64gb +#SBATCH --time=3:00:00 +#SBATCH -o output/output.%A.out +#SBATCH -e error/error.%A.err + +export AP_ROOT="$(cd "${SLURM_SUBMIT_DIR:-$(dirname "$0")}/../.." && pwd)" +echo "AP_ROOT=$AP_ROOT" +source $AP_ROOT/activate.sh + +AA_ROOT=/mnt/ral/jnightin/PyAuto_wt/delaunay-walk-early-exit/PyAutoArray_control +export PYTHONPATH=$AA_ROOT:$PYTHONPATH + +export JAX_PLATFORM_NAME=cuda +export JAX_PLATFORMS=cuda,cpu +export XLA_PYTHON_CLIENT_PREALLOCATE=false +export JAX_ENABLE_X64=True +export NUMBA_CACHE_DIR=/tmp/numba_cache +export MPLCONFIGDIR=/tmp/matplotlib + +nvidia-smi +echo "==========================================" +date +echo "Cell: likelihood_runtime/imaging/delaunay" +echo "A/B leg: control" +printf 'PyAutoArray(AB) %s\n' "$(git -C $AA_ROOT rev-parse HEAD 2>&1)" + +cd $AP_ROOT +python3 -c "import autoarray; print('autoarray.__file__ =', autoarray.__file__)" + +python3 -u scripts/imaging/likelihood_runtime/delaunay.py \ + --instrument hst \ + --config-name hpc_a100_fp64_control_2026_09_07 + +echo "Finished." +date diff --git a/hpc/batch_gpu/submit_runtime_imaging_delaunay_a100_hst_fp64_walk_early_exit b/hpc/batch_gpu/submit_runtime_imaging_delaunay_a100_hst_fp64_walk_early_exit new file mode 100644 index 000000000..07a07c696 --- /dev/null +++ b/hpc/batch_gpu/submit_runtime_imaging_delaunay_a100_hst_fp64_walk_early_exit @@ -0,0 +1,50 @@ +#!/bin/bash -l +# +# A100 likelihood-runtime profile: imaging/delaunay x hst x fp64 — PyAutoArray#531 +# A/B leg: **walk_early_exit**. Single-JIT whole-likelihood cost, the end-to-end companion +# to submit_breakdown_imaging_delaunay_a100_hst_fp64_walk_early_exit. +# +# Same private-checkout PYTHONPATH override as the breakdown A/B legs; the +# shared /mnt/ral/jnightin/PyAuto install is untouched. Phase A (--vmap-probe) +# is skipped: only the full single-JIT + vmap timing is wanted here. + +#SBATCH -J rt_delaunay_walk_early_exit +#SBATCH --partition=gpu +#SBATCH --gres=gpu:1 +#SBATCH --ntasks=1 +#SBATCH --cpus-per-task=4 +#SBATCH --mem=64gb +#SBATCH --time=3:00:00 +#SBATCH -o output/output.%A.out +#SBATCH -e error/error.%A.err + +export AP_ROOT="$(cd "${SLURM_SUBMIT_DIR:-$(dirname "$0")}/../.." && pwd)" +echo "AP_ROOT=$AP_ROOT" +source $AP_ROOT/activate.sh + +AA_ROOT=/mnt/ral/jnightin/PyAuto_wt/delaunay-walk-early-exit/PyAutoArray_feature +export PYTHONPATH=$AA_ROOT:$PYTHONPATH + +export JAX_PLATFORM_NAME=cuda +export JAX_PLATFORMS=cuda,cpu +export XLA_PYTHON_CLIENT_PREALLOCATE=false +export JAX_ENABLE_X64=True +export NUMBA_CACHE_DIR=/tmp/numba_cache +export MPLCONFIGDIR=/tmp/matplotlib + +nvidia-smi +echo "==========================================" +date +echo "Cell: likelihood_runtime/imaging/delaunay" +echo "A/B leg: walk_early_exit" +printf 'PyAutoArray(AB) %s\n' "$(git -C $AA_ROOT rev-parse HEAD 2>&1)" + +cd $AP_ROOT +python3 -c "import autoarray; print('autoarray.__file__ =', autoarray.__file__)" + +python3 -u scripts/imaging/likelihood_runtime/delaunay.py \ + --instrument hst \ + --config-name hpc_a100_fp64_walk_early_exit + +echo "Finished." +date diff --git a/results/breakdown/imaging/delaunay_hpc_a100_fp64_control_2026_09_07.json b/results/breakdown/imaging/delaunay_hpc_a100_fp64_control_2026_09_07.json new file mode 100644 index 000000000..52fc37f7d --- /dev/null +++ b/results/breakdown/imaging/delaunay_hpc_a100_fp64_control_2026_09_07.json @@ -0,0 +1,56 @@ +{ + "autolens_version": "2026.8.17.1", + "device": { + "backend": "gpu", + "device": "cuda:0", + "xla_flags": "--xla_disable_hlo_passes=constant_folding --xla_gpu_autotune_level=0", + "omp_num_threads": null, + "cpu_count": 124, + "nvidia_smi": "NVIDIA A100 80GB PCIe, 41495 MiB, 81920 MiB" + }, + "instrument": "hst", + "configuration": { + "pixel_scale_arcsec": 0.05, + "mask_radius_arcsec": 3.5, + "image_pixels_masked": 15361, + "over_sampled_pixels": 17980, + "delaunay_vertices": 1500, + "edge_zeroed_pixels": 0, + "inversion_path": "dense" + }, + "steps": { + "Ray-trace data grid": 0.00018841889686882496, + "Ray-trace mesh grid": 0.00022632270120084285, + "Lens light images (pre-PSF)": 0.00016110208816826342, + "Blurred image (PSF convolution)": 0.0008522381074726581, + "Profile-subtracted image": 0.0001577841816470027, + "Inversion setup (steps 5-8 combined)": 0.04465006028767675, + "Data vector (D)": 0.0003422010922804475, + "Curvature matrix (F)": 0.0048084759153425695, + "Regularization matrix (H)": 0.013555067591369152, + "Regularized reconstruction": 0.032281293999403715, + "Mapped recon + log evidence": 0.0022403292823582888 + }, + "total_step_by_step": 0.09946329414378852, + "regularization_matrix_prefix_s": 0.045148767484351995, + "interpolator_prefix_s": 0.03159369989298284, + "setup_split": { + "Border relocation": 0.0010311831021681428, + "Triangulation + interpolation": 0.0305625167908147, + "Mapping matrix": 0.0003326961072161738, + "Blurred mapping matrix (PSF)": 0.00805002460256219 + }, + "vmap_batch": 16, + "steps_vmap_per_call": { + "Inversion setup (steps 5-8 combined)": 0.015451028637471609, + "Regularization matrix (H)": 0.0007413830753648654 + }, + "setup_split_vmap": { + "Border relocation": 9.938294388120995e-05, + "Triangulation + interpolation": 0.006714640074642376, + "Mapping matrix": -1.10918190330319e-06, + "Blurred mapping matrix (PSF)": 0.008398903276247439 + }, + "regularization_matrix_prefix_vmap_per_call_s": 0.007555406093888451, + "interpolator_prefix_vmap_per_call_s": 0.006814023018523585 +} \ No newline at end of file diff --git a/results/breakdown/imaging/delaunay_hpc_a100_fp64_control_2026_09_07.png b/results/breakdown/imaging/delaunay_hpc_a100_fp64_control_2026_09_07.png new file mode 100644 index 000000000..70ad1182e Binary files /dev/null and b/results/breakdown/imaging/delaunay_hpc_a100_fp64_control_2026_09_07.png differ diff --git a/results/breakdown/imaging/delaunay_hpc_a100_fp64_walk_early_exit.json b/results/breakdown/imaging/delaunay_hpc_a100_fp64_walk_early_exit.json new file mode 100644 index 000000000..4c741d58d --- /dev/null +++ b/results/breakdown/imaging/delaunay_hpc_a100_fp64_walk_early_exit.json @@ -0,0 +1,56 @@ +{ + "autolens_version": "2026.8.17.1", + "device": { + "backend": "gpu", + "device": "cuda:0", + "xla_flags": "--xla_disable_hlo_passes=constant_folding --xla_gpu_autotune_level=0", + "omp_num_threads": null, + "cpu_count": 124, + "nvidia_smi": "NVIDIA A100 80GB PCIe, 41495 MiB, 81920 MiB" + }, + "instrument": "hst", + "configuration": { + "pixel_scale_arcsec": 0.05, + "mask_radius_arcsec": 3.5, + "image_pixels_masked": 15361, + "over_sampled_pixels": 17980, + "delaunay_vertices": 1500, + "edge_zeroed_pixels": 0, + "inversion_path": "dense" + }, + "steps": { + "Ray-trace data grid": 0.00018229398410767316, + "Ray-trace mesh grid": 0.00017150109633803367, + "Lens light images (pre-PSF)": 0.00013617018703371286, + "Blurred image (PSF convolution)": 0.00084638730622828, + "Profile-subtracted image": 0.00015273820608854293, + "Inversion setup (steps 5-8 combined)": 0.020429736003279686, + "Data vector (D)": 0.0003239401150494814, + "Curvature matrix (F)": 0.004797589988447726, + "Regularization matrix (H)": 0.00022505368106067163, + "Regularized reconstruction": 0.032512539788149296, + "Mapped recon + log evidence": 0.0022666632896289228 + }, + "total_step_by_step": 0.06204461364541203, + "regularization_matrix_prefix_s": 0.007227654289454222, + "interpolator_prefix_s": 0.00700260060839355, + "setup_split": { + "Border relocation": 0.0010324133094400168, + "Triangulation + interpolation": 0.005970187298953534, + "Mapping matrix": 0.0001954708946868772, + "Blurred mapping matrix (PSF)": 0.011927697691135108 + }, + "vmap_batch": 16, + "steps_vmap_per_call": { + "Inversion setup (steps 5-8 combined)": 0.013805706561834085, + "Regularization matrix (H)": 6.95051683578642e-05 + }, + "setup_split_vmap": { + "Border relocation": 6.248771242098883e-05, + "Triangulation + interpolation": 0.005006508050428238, + "Mapping matrix": 0.00012145250511821394, + "Blurred mapping matrix (PSF)": 0.008343454381974879 + }, + "regularization_matrix_prefix_vmap_per_call_s": 0.0051385009312070904, + "interpolator_prefix_vmap_per_call_s": 0.005068995762849226 +} \ No newline at end of file diff --git a/results/breakdown/imaging/delaunay_hpc_a100_fp64_walk_early_exit.png b/results/breakdown/imaging/delaunay_hpc_a100_fp64_walk_early_exit.png new file mode 100644 index 000000000..12fda66d2 Binary files /dev/null and b/results/breakdown/imaging/delaunay_hpc_a100_fp64_walk_early_exit.png differ diff --git a/results/breakdown/imaging/delaunay_nn_hpc_a100_fp64_control_2026_09_07.json b/results/breakdown/imaging/delaunay_nn_hpc_a100_fp64_control_2026_09_07.json new file mode 100644 index 000000000..8ccc4958d --- /dev/null +++ b/results/breakdown/imaging/delaunay_nn_hpc_a100_fp64_control_2026_09_07.json @@ -0,0 +1,56 @@ +{ + "autolens_version": "2026.8.17.1", + "device": { + "backend": "gpu", + "device": "cuda:0", + "xla_flags": "--xla_disable_hlo_passes=constant_folding --xla_gpu_autotune_level=0", + "omp_num_threads": null, + "cpu_count": 124, + "nvidia_smi": "NVIDIA A100 80GB PCIe, 41495 MiB, 81920 MiB" + }, + "instrument": "hst", + "configuration": { + "pixel_scale_arcsec": 0.05, + "mask_radius_arcsec": 3.5, + "image_pixels_masked": 15361, + "over_sampled_pixels": 17980, + "delaunay_vertices": 1500, + "edge_zeroed_pixels": 0, + "inversion_path": "dense" + }, + "steps": { + "Ray-trace data grid": 0.00017087310552597046, + "Ray-trace mesh grid": 0.0001385691110044718, + "Lens light images (pre-PSF)": 0.00014830308500677348, + "Blurred image (PSF convolution)": 0.0008490002015605569, + "Profile-subtracted image": 0.00014221209567040206, + "Inversion setup (steps 5-8 combined)": 0.1354279973078519, + "Data vector (D)": 0.0003226821077987552, + "Curvature matrix (F)": 0.004821999906562268, + "Regularization matrix (H)": 0.05546664176508785, + "Regularized reconstruction": 0.030796876596286892, + "Mapped recon + log evidence": 0.002287936117500067 + }, + "total_step_by_step": 0.23057309139985588, + "regularization_matrix_prefix_s": 0.17897086418233812, + "interpolator_prefix_s": 0.12350422241725027, + "setup_split": { + "Border relocation": 0.0010540070012211799, + "Triangulation + interpolation": 0.12245021541602909, + "Mapping matrix": -0.0006168675143271712, + "Blurred mapping matrix (PSF)": 0.00909734289161862 + }, + "vmap_batch": 16, + "steps_vmap_per_call": { + "Inversion setup (steps 5-8 combined)": 0.022263527681934646, + "Regularization matrix (H)": 0.012741508361068556 + }, + "setup_split_vmap": { + "Border relocation": 6.409913767129183e-05, + "Triangulation + interpolation": 0.01595128248154651, + "Mapping matrix": -0.002431770932162179, + "Blurred mapping matrix (PSF)": 0.010330748069100082 + }, + "regularization_matrix_prefix_vmap_per_call_s": 0.02875688998028636, + "interpolator_prefix_vmap_per_call_s": 0.016015381619217804 +} \ No newline at end of file diff --git a/results/breakdown/imaging/delaunay_nn_hpc_a100_fp64_control_2026_09_07.png b/results/breakdown/imaging/delaunay_nn_hpc_a100_fp64_control_2026_09_07.png new file mode 100644 index 000000000..490f7e5c2 Binary files /dev/null and b/results/breakdown/imaging/delaunay_nn_hpc_a100_fp64_control_2026_09_07.png differ diff --git a/results/breakdown/imaging/delaunay_nn_hpc_a100_fp64_walk_early_exit.json b/results/breakdown/imaging/delaunay_nn_hpc_a100_fp64_walk_early_exit.json new file mode 100644 index 000000000..9f8a9c50b --- /dev/null +++ b/results/breakdown/imaging/delaunay_nn_hpc_a100_fp64_walk_early_exit.json @@ -0,0 +1,56 @@ +{ + "autolens_version": "2026.8.17.1", + "device": { + "backend": "gpu", + "device": "cuda:0", + "xla_flags": "--xla_disable_hlo_passes=constant_folding --xla_gpu_autotune_level=0", + "omp_num_threads": null, + "cpu_count": 124, + "nvidia_smi": "NVIDIA A100 80GB PCIe, 41495 MiB, 81920 MiB" + }, + "instrument": "hst", + "configuration": { + "pixel_scale_arcsec": 0.05, + "mask_radius_arcsec": 3.5, + "image_pixels_masked": 15361, + "over_sampled_pixels": 17980, + "delaunay_vertices": 1500, + "edge_zeroed_pixels": 0, + "inversion_path": "dense" + }, + "steps": { + "Ray-trace data grid": 0.00018018099945038557, + "Ray-trace mesh grid": 0.00015555520076304674, + "Lens light images (pre-PSF)": 0.00013933321461081504, + "Blurred image (PSF convolution)": 0.0012038412038236856, + "Profile-subtracted image": 0.00014065420255064963, + "Inversion setup (steps 5-8 combined)": 0.11148364611435682, + "Data vector (D)": 0.00031830610241740943, + "Curvature matrix (F)": 0.004833393800072372, + "Regularization matrix (H)": 0.04566282599698751, + "Regularized reconstruction": 0.030601281695999204, + "Mapped recon + log evidence": 0.002243989403359592 + }, + "total_step_by_step": 0.1969630079343915, + "regularization_matrix_prefix_s": 0.1447891157120466, + "interpolator_prefix_s": 0.0991262897150591, + "setup_split": { + "Border relocation": 0.0010661459062248468, + "Triangulation + interpolation": 0.09806014380883425, + "Mapping matrix": -0.0019123112084344085, + "Blurred mapping matrix (PSF)": 0.009209847101010388 + }, + "vmap_batch": 16, + "steps_vmap_per_call": { + "Inversion setup (steps 5-8 combined)": 0.022944040725997185, + "Regularization matrix (H)": 0.01274863845028449 + }, + "setup_split_vmap": { + "Border relocation": 6.460213771788403e-05, + "Triangulation + interpolation": 0.011610259737062733, + "Mapping matrix": 0.002601451036753133, + "Blurred mapping matrix (PSF)": 0.008355711201147642 + }, + "regularization_matrix_prefix_vmap_per_call_s": 0.024423500325065108, + "interpolator_prefix_vmap_per_call_s": 0.011674861874780618 +} \ No newline at end of file diff --git a/results/breakdown/imaging/delaunay_nn_hpc_a100_fp64_walk_early_exit.png b/results/breakdown/imaging/delaunay_nn_hpc_a100_fp64_walk_early_exit.png new file mode 100644 index 000000000..3cc60b4ed Binary files /dev/null and b/results/breakdown/imaging/delaunay_nn_hpc_a100_fp64_walk_early_exit.png differ diff --git a/results/notes/delaunay_walk_early_exit.md b/results/notes/delaunay_walk_early_exit.md new file mode 100644 index 000000000..c7ad27691 --- /dev/null +++ b/results/notes/delaunay_walk_early_exit.md @@ -0,0 +1,206 @@ +# JAX Delaunay walk — early exit + single concatenated locate (A100, 2026-09-07) + +PyAutoArray issue #530 / PR #531, branch `feature/delaunay-walk-early-exit` (`6152bdf0`), +against its merge base `bcd15cd9` (PyAutoArray `main`). + +## What changed + +The JAX point locator `pix_indexes_delaunay_walk_from` replaced its fixed-trip +`jax.lax.fori_loop` of `DELAUNAY_WALK_STEPS = 128` with a `jax.lax.while_loop` that exits +as soon as every query is located or outside the hull (typical resolution is under ten +steps), and stopped pushing the walk itself through the `DELAUNAY_LOCATE_CHUNK = 1024` +`lax.map` — only the nearest-vertex seed argmin is still chunked, which is all the chunking +was ever for (bounding the `(chunk, N)` intermediate under `vmap`). `jax_delaunay` now +locates the data grid and the 6,000 `ConstantSplit` cross points in **one** concatenated +call instead of two. Point location is integer-valued, so the walk's float inputs are +wrapped in `stop_gradient` (`while_loop` has no reverse-mode rule); every differentiable +quantity is still computed from the traced arrays. + +## Verdict + +On the A100, the HST / Hilbert-1500 / MGE-60 / `ConstantSplit` Delaunay imaging likelihood's +**params→H prefix — the aggregate that is invariant to the attribution shift below — falls +45.15 ms → 7.23 ms unbatched (−37.9 ms, 6.25×)** and 7.56 → 5.14 ms per call at `vmap` 16 +(−2.4 ms, 1.47×). The witness asked for a ≥ 18 ms unbatched drop; the measured drop is +2.1× that. Whole-likelihood single-JIT runtime goes **93.46 → 60.36 ms (1.55×)** and the +step-by-step sum 99.46 → 62.05 ms. Both eager log-evidence pins held exactly, on both cells. +The DelaunayNN sibling, which shares the walk but not the concatenation, improves more +modestly (params→H prefix 178.97 → 144.79 ms, 1.24×). + +## Provenance + +| Job | Cell | Leg | Node | Start (2026-09-07) | Elapsed | State | +|---|---|---|---|---|---:|---| +| 342315 | `likelihood_breakdown/imaging/delaunay` | control | `euclid-ral-gpu-2` | 23:22:51 | 1:57 | COMPLETED | +| 342316 | `likelihood_breakdown/imaging/delaunay` | feature | `euclid-ral-gpu-2` | 23:24:49 | 1:47 | COMPLETED | +| 342317 | `likelihood_breakdown/imaging/delaunay_nn` | control | `euclid-ral-gpu-2` | 23:26:37 | 2:29 | COMPLETED | +| 342318 | `likelihood_breakdown/imaging/delaunay_nn` | feature | `euclid-ral-gpu-2` | 23:29:07 | 2:28 | COMPLETED | +| 342319 | `likelihood_runtime/imaging/delaunay` | control | `euclid-ral-gpu-2` | 23:33:58 | 1:02 | COMPLETED | +| 342320 | `likelihood_runtime/imaging/delaunay` | feature | `euclid-ral-gpu-2` | 23:35:01 | 1:03 | COMPLETED | + +- Breakdown flags `--split-setup --vmap-batch 16`; runtime cell full timing only (no + `--vmap-probe`), `vmap` batch 16 in both legs. `PyAutoLens 2026.8.17.1`, fp64 + (`JAX_ENABLE_X64=True`), dense inversion path, NVIDIA A100 80GB PCIe. +- Recorded `xla_flags` on every row: `--xla_disable_hlo_passes=constant_folding + --xla_gpu_autotune_level=0` — identical across the six jobs, so these rows are + comparable with the 2026-09-05 rows in `delaunay_nn_breakdown.md` and **not** with the + 2026-07-10 rows in `preopt_breakdown_baseline.md`. +- **All six jobs ran on the same node in one 14-minute window**, so the A/B is a + same-session comparison, not a comparison against a recorded baseline. + +### The A/B differs in PyAutoArray and nothing else + +| Repo | Revision | +|---|---| +| PyAutoArray — control leg | `bcd15cd9d599c2e3fb7a7c3407d362fb39507c53` | +| PyAutoArray — feature leg | `6152bdf0506229dce36666db8b6b9bef636985fe` (`feature/delaunay-walk-early-exit`) | +| PyAutoNerves | `fc9c474ba36cca0aaf0d40351e65e791b99f9a35` | +| PyAutoFit | `f6a9915045550ac5fb6930c6076cd9242d014f1e` | +| PyAutoGalaxy | `6d216c151c914b2ce79af18fbc0348a6fc5425d7` | +| PyAutoLens | `146a3d7254418eac964edd8b89ada7281ac778e7` | +| autolens_profiling (`AP_ROOT`) | `d51b94ae9bdd612c0607ea27fc9a04b25a0f4418` | + +The **shared** `/mnt/ral/jnightin/PyAuto` install was deliberately not touched — the +subhalo-validation and Euclid DR1 CPU runs (`342299`, `342301`, `342311`, `342314`) were +live on it throughout. Both legs instead prepend a private PyAutoArray checkout to the +`PYTHONPATH` that `activate.sh` exports: +`/mnt/ral/jnightin/PyAuto_wt/delaunay-walk-early-exit/PyAutoArray_{control,feature}`, and +each job prints `autoarray.__file__` and the checkout's `git rev-parse HEAD` into its +SLURM log. Note the shared install sits at `e36a5af4`, **eight commits behind** the +branch's merge base, which is why the control leg is a private `bcd15cd9` checkout rather +than the shared tree: running the control on the shared install would have folded eight +unrelated commits into the difference. Submits: +`hpc/batch_gpu/submit_{breakdown_imaging_delaunay,breakdown_imaging_delaunay_nn,runtime_imaging_delaunay}_a100_hst_fp64_{control,walk_early_exit}`, +run from `/mnt/ral/jnightin/autolens_profiling_wt/delaunay-walk-early-exit` (detached at +`d51b94a`). + +### Pins + +`EXPECTED_LOG_EVIDENCE_HST` passed **unchanged on every leg** — `29110.920858` for +Delaunay (342315/342316, and `pinned_drift: []` on the two runtime rows) and +`29144.581944` for DelaunayNN (342317/342318). The mapping is bit-identical to the eager +NumPy reference on both sides of the change. + +## Delaunay — control vs feature + +All values ms per likelihood call. `vmap/16` is `jax.jit(jax.vmap(fn))` over a params +pytree broadcast to batch 16, reported as batch time / 16; only the combined +inversion-setup block, the four `--split-setup` prefixes and the params→H prefix are +re-timed under `vmap`, so the remaining rows read `—` there rather than zero. + +| Step | control | feature | Δ | control vmap/16 | feature vmap/16 | +|---|---:|---:|---:|---:|---:| +| Ray-trace data grid | 0.188 | 0.182 | −0.006 | — | — | +| Ray-trace mesh grid | 0.226 | 0.172 | −0.054 | — | — | +| Lens light images (pre-PSF) | 0.161 | 0.136 | −0.025 | — | — | +| Blurred image (PSF convolution) | 0.852 | 0.846 | −0.006 | — | — | +| Profile-subtracted image | 0.158 | 0.153 | −0.005 | — | — | +| **Inversion setup (steps 5–8 combined)** | **44.650** | **20.430** | **−24.220** | **15.451** | **13.806** | +| Data vector (D) | 0.342 | 0.324 | −0.018 | — | — | +| Curvature matrix (F) | 4.808 | 4.798 | −0.010 | — | — | +| **Regularization matrix (H)** | **13.555** | **0.225** | **−13.330** | **0.741** | **0.070** | +| Regularized reconstruction | 32.281 | 32.513 | +0.232 | — | — | +| Mapped recon + log evidence | 2.240 | 2.267 | +0.027 | — | — | +| **Total step-by-step (unbatched)** | **99.463** | **62.045** | **−37.418** | — | — | + +### Four-way setup split and the prefixes + +| Piece | control | feature | Δ | control vmap/16 | feature vmap/16 | +|---|---:|---:|---:|---:|---:| +| Border relocation | 1.031 | 1.032 | +0.001 | 0.099 | 0.062 | +| **Triangulation + interpolation** | **30.563** | **5.970** | **−24.593** | **6.715** | **5.007** | +| Mapping matrix | 0.333 | 0.195 | −0.138 | −0.001 | 0.121 | +| Blurred mapping matrix (PSF) | 8.050 | 11.928 | +3.878 | 8.399 | 8.343 | +| *Interpolator prefix (params→step 6)* | *31.594* | *7.003* | *−24.591* | *6.814* | *5.069* | +| **params→H prefix (the witness)** | **45.149** | **7.228** | **−37.921** | **7.555** | **5.139** | + +The **+3.878 ms "Blurred mapping matrix (PSF)" row is a by-difference artifact, not a +regression**: it is prefix 8 minus prefix 7, both of which shrank, and the same row under +`vmap` 16 — where the prefixes are re-measured independently — is flat (8.399 → 8.343 ms). +The combined-block row (44.650 → 20.430 ms) and the whole-likelihood single-JIT runtime +below both fall by more than the sum of the split, so no work moved into the PSF step. + +### Whole-likelihood runtime cell (`likelihood_runtime/imaging/delaunay`) + +| | control | feature | Δ | +|---|---:|---:|---:| +| Full pipeline (single JIT) | 93.463 | 60.355 | **−33.108 (1.55×)** | +| `vmap` batch 16, per call | 41.945 | 39.520 | −2.425 (1.06×) | + +This is the number a production evaluation actually pays. The unbatched saving (−33.1 ms) +tracks the params→H prefix drop (−37.9 ms) to within the fusion slack expected when the +whole pipeline is compiled as one program; the batched saving is small because at batch 16 +the walk was already amortising and the dense linear algebra dominates. + +## The H-row attribution caveat (read this before comparing rows across versions) + +**The H row moved meaning with this change; the params→H prefix did not.** The breakdown +attributes step 11 as `t(params → H) − t(params → interpolator outputs)`. Before #531 a +prefix stopping at step 6 never asked for the `ConstantSplit` split points, so XLA +dead-code-eliminated that second walk out of the interpolator prefix and its whole cost +surfaced in the H row's subtraction. The feature concatenates both walks into one locate +call, so the split walk now runs **inside** the step-6 prefix — inside "Triangulation + +interpolation" — and the subtraction leaves only the regularization assembly. Hence +**H: 13.555 → 0.225 ms**, and a small negative would have been equally legitimate. + +Two consequences: + +1. Comparing the "Triangulation + interpolation" row alone *understates* the change + (30.563 → 5.970 ms is a 5.1× improvement measured across a boundary that moved work + *into* the numerator), and comparing the H row alone *overstates* it. +2. The aggregate to quote across library versions is **params→H (Tri+interp + H)**, i.e. + `regularization_matrix_prefix_s` in the JSON. A short note to that effect has been added + to `scripts/imaging/likelihood_breakdown/delaunay.py`'s module docstring and to the + comment above `_setup_prefix_fn`. + +The NumPy/scipy path (`scipy_delaunay`, the numba likelihood) is untouched by #531 and +still charges the split walk to H. + +## DelaunayNN sibling + +`DelaunayNN` (Sibson natural-neighbour, cap 32) shares the locator but **not** the +concatenation: `sibson.py:522` and `sibson.py:648` still call +`pix_indexes_delaunay_walk_from` twice, once for the data grid and once for the split +points. It therefore gets the early exit and the unchunked walk, but not the single-call +saving — and its H row stays large rather than collapsing. + +| Row | control | feature | Δ | control vmap/16 | feature vmap/16 | +|---|---:|---:|---:|---:|---:| +| Inversion setup (steps 5–8 combined) | 135.428 | 111.484 | −23.944 | 22.264 | 22.944 | +| Triangulation + interpolation | 122.450 | 98.060 | −24.390 | 15.951 | 11.610 | +| Regularization matrix (H) | 55.467 | 45.663 | −9.804 | 12.742 | 12.749 | +| Interpolator prefix (params→step 6) | 123.504 | 99.126 | −24.378 | 16.015 | 11.675 | +| **params→H prefix** | **178.971** | **144.789** | **−34.182 (1.24×)** | **28.757** | **24.424** | +| Total step-by-step (unbatched) | 230.573 | 196.963 | −33.610 | — | — | + +Both DelaunayNN legs held the `29144.581944` pin. The `−0.617` / `−1.912` ms "Mapping +matrix" rows are the same below-noise by-difference artifact documented in +`delaunay_nn_breakdown.md`. + +## Phase 2 recommendation + +**Do not start Phase 2 (static image-plane seed + fan test) on the Delaunay path; extend +the single concatenated locate to the Sibson path first.** The number this rests on: after +#531 the Delaunay params→H prefix is **7.23 ms unbatched and 5.14 ms per call at `vmap` 16** +— about 13% of the 39.5 ms batched per-evaluation cost — so Phase 2's remaining headroom on +this cell is at most a few ms per call, for a change that reaches into Mapper/AdaptImages +plumbing. DelaunayNN, by contrast, still spends **144.8 ms unbatched / 24.4 ms per call at +`vmap` 16** in the same prefix, and the cheapest available cut there is the one Delaunay +already took: concatenate `sibson.py`'s two walk calls into one. That is a much smaller +change than Phase 2 and targets the cell with 20× the remaining cost. + +## Artifacts + +- `results/breakdown/imaging/delaunay_hpc_a100_fp64_control_2026_09_07.{json,png}` +- `results/breakdown/imaging/delaunay_hpc_a100_fp64_walk_early_exit.{json,png}` +- `results/breakdown/imaging/delaunay_nn_hpc_a100_fp64_control_2026_09_07.{json,png}` +- `results/breakdown/imaging/delaunay_nn_hpc_a100_fp64_walk_early_exit.{json,png}` +- `results/runtime/imaging/delaunay/delaunay_hpc_a100_fp64_control_2026_09_07.json` +- `results/runtime/imaging/delaunay/delaunay_hpc_a100_fp64_walk_early_exit.json` +- `hpc/batch_gpu/submit_breakdown_imaging_delaunay_a100_hst_fp64_{control,walk_early_exit}` +- `hpc/batch_gpu/submit_breakdown_imaging_delaunay_nn_a100_hst_fp64_{control,walk_early_exit}` +- `hpc/batch_gpu/submit_runtime_imaging_delaunay_a100_hst_fp64_{control,walk_early_exit}` + +The pre-existing `delaunay_hpc_a100_fp64.json` / `delaunay_nn_hpc_a100_fp64.json` rows +(2026-09-05, job 342277/342278, PyAutoArray `e36a5af4`) are left untouched; the control +legs here supersede them for any comparison against this branch. diff --git a/results/runtime/imaging/delaunay/delaunay_hpc_a100_fp64_control_2026_09_07.json b/results/runtime/imaging/delaunay/delaunay_hpc_a100_fp64_control_2026_09_07.json new file mode 100644 index 000000000..e49ce3611 --- /dev/null +++ b/results/runtime/imaging/delaunay/delaunay_hpc_a100_fp64_control_2026_09_07.json @@ -0,0 +1,30 @@ +{ + "autolens_version": "2026.8.17.1", + "device": { + "backend": "gpu", + "device": "cuda:0", + "xla_flags": "--xla_disable_hlo_passes=constant_folding --xla_gpu_autotune_level=0", + "omp_num_threads": null, + "cpu_count": 124, + "nvidia_smi": "NVIDIA A100 80GB PCIe, 35333 MiB, 81920 MiB" + }, + "instrument": "hst", + "configuration": { + "pixel_scale_arcsec": 0.05, + "mask_radius_arcsec": 3.5, + "image_pixels_masked": 15361, + "over_sampled_pixels": 17980, + "delaunay_vertices": 1500, + "edge_zeroed_pixels": 0, + "inversion_path": "dense" + }, + "full_pipeline_single_jit": 0.09346281460020692, + "vmap": { + "batch_size": 16, + "batch_time": 0.6711140753934159, + "per_call": 0.041944629712088496, + "speedup_vs_single_jit": 2.2 + }, + "pinned_expected": 29110.92085793, + "pinned_drift": [] +} \ No newline at end of file diff --git a/results/runtime/imaging/delaunay/delaunay_hpc_a100_fp64_walk_early_exit.json b/results/runtime/imaging/delaunay/delaunay_hpc_a100_fp64_walk_early_exit.json new file mode 100644 index 000000000..b3fe25cf0 --- /dev/null +++ b/results/runtime/imaging/delaunay/delaunay_hpc_a100_fp64_walk_early_exit.json @@ -0,0 +1,30 @@ +{ + "autolens_version": "2026.8.17.1", + "device": { + "backend": "gpu", + "device": "cuda:0", + "xla_flags": "--xla_disable_hlo_passes=constant_folding --xla_gpu_autotune_level=0", + "omp_num_threads": null, + "cpu_count": 124, + "nvidia_smi": "NVIDIA A100 80GB PCIe, 35333 MiB, 81920 MiB" + }, + "instrument": "hst", + "configuration": { + "pixel_scale_arcsec": 0.05, + "mask_radius_arcsec": 3.5, + "image_pixels_masked": 15361, + "over_sampled_pixels": 17980, + "delaunay_vertices": 1500, + "edge_zeroed_pixels": 0, + "inversion_path": "dense" + }, + "full_pipeline_single_jit": 0.060355040105059744, + "vmap": { + "batch_size": 16, + "batch_time": 0.6323271838016808, + "per_call": 0.03952044898760505, + "speedup_vs_single_jit": 1.5 + }, + "pinned_expected": 29110.92085793, + "pinned_drift": [] +} \ No newline at end of file diff --git a/scripts/imaging/likelihood_breakdown/delaunay.py b/scripts/imaging/likelihood_breakdown/delaunay.py index eb0a4b8ce..848bdfd3d 100644 --- a/scripts/imaging/likelihood_breakdown/delaunay.py +++ b/scripts/imaging/likelihood_breakdown/delaunay.py @@ -66,6 +66,21 @@ therefore returns the step-6 outputs alongside H, making it a strict superset of ``_setup_prefix_fn(6)``. +Since PyAutoArray#531 the H row is a **prefix difference only** and may read +~0 or negative on the JAX path. The early-exit walk locates the data grid and +the ConstantSplit cross points in one concatenated ``lax.while_loop`` call, so +the split-point walk now runs *inside* ``_setup_prefix_fn(6)`` — i.e. inside +the "Triangulation + interpolation" row — where it previously could not (a +prefix stopping at step 6 never asked for the split points, XLA eliminated +them, and their whole cost surfaced in the H row's subtraction). What the +subtraction leaves is therefore only the regularization assembly, which is +small enough for scatter between two nested prefixes to take it below zero. +**The aggregate to compare across library versions is the params->H prefix +itself** — ``regularization_matrix_prefix_s`` in the JSON, equivalently +"Triangulation + interpolation" + the H row — not either row alone. The +NumPy/scipy path is unchanged and still charges the split walk to H. Nothing +in the measurement changed; only where the same work is attributed. + Before 2026-09 this row timed ``jnp.array(inversion.regularization_matrix)`` — a host-to-device copy of the 19.5 MB matrix the *eager NumPy* ``FitImaging`` had already computed. On the A100 that read as ~14.4 ms of PCIe traffic and @@ -782,6 +797,11 @@ def blurred_mm_from_params(params_tree): # interpolator prefix (``upto=6``) is timed **either way**, because step 11 # attributes the H row as t(params -> H) - t(params -> interpolator outputs); # when ``--split-setup`` is on its timing is reused rather than compiled twice. +# +# NOTE (PyAutoArray#531): on the JAX path the split-point walk now rides inside +# prefix 6 (single concatenated locate call), so the step-11 difference is the +# regularization assembly alone and can read ~0/negative. Compare the params->H +# prefix across versions, not the H row. See the module docstring. def _setup_prefix_fn(upto):