Skip to content

fix: magnification latent is inf/0.0 for any pixelized source #726

Closed

Description

@Jammy2211

Overview

The magnification latent in autolens/analysis/latent.py is total_lensed_source_flux_mujy / total_source_flux_mujy. The denominator calls Galaxy.image_2d_from on the converted tracer's source galaxy, and a pixelization-only galaxy returns zeros there (a Pixelization is not a LightProfile). So every pixelized fit gets total_source_flux = 0.0, total_source_flux_mujy = 0.0 and magnification = +inf — dropped as a column by PyAutoFit's NaN guard under current code, or archived as a 0.0 sentinel (9/9 initial_lens_model/vis_pix results). Every Euclid vis_pix / full_model stage therefore has no magnification, while the Sersic control on the same fit reproduces truth.json's 15.146 bit-for-bit — the latent's definition is sound, only the pixelized-source route is broken. Fix: define the source-plane flux of a pixelized source as the inversion's reconstruction integrated with the mesh's exact quadrature areas (areas_for_magnification, barycentric dual areas for Delaunay since PyAutoArray#524 / PR #525), added to the existing light-profile flux.

Plan

  • Add a _pixelized_source_flux(fit, xp) helper to the latent module: for each mapper of the source galaxy in the fit's inversion, sum reconstruction · areas_for_magnification; a mesh geometry without quadrature areas (DelaunayNN / KNN / KNNBarycentric) yields NaN with a once-per-process warning, never a silent zero.
  • total_source_flux and total_source_flux_mujy become light-profile flux (existing path) + pixelized flux; magnification is unchanged in form and becomes finite for pixelized sources.
  • Keep every new line trace-safe: latents run inside the per-sample jax.jit batch path (LatentLens.BATCH_MODE = "jit", xp = analysis._xp).
  • Unit tests in test_autolens/analysis/test_latent.py: Delaunay and rectangular pixelized fits give a finite magnification equal to Σ galaxy image / Σ reconstruction·areas; linear light profile + pixelization sums both contributions; light-profile-only fits are numerically unchanged; a KNN mesh gives NaN.
  • Ship the PyAutoLens PR first (library-first gate). Workspace follow-up in euclid_strong_lens_modeling_pipeline: correct the false "magnification is stage-invariant" docstring, add a test-mode vis_pix latent assertion, and leave a ledger note that archived vis_pix rows carry a 0.0 sentinel and must be re-derived.
  • Human acceptance after both merges: the phase 8 audit's part3b_fit_latent.py construction (Delaunay fit on euclid_dr1_like, magzero 24.6) reports a finite magnification close to the Sersic control's 15.146 — record it on this issue.
Detailed implementation plan

Affected Repositories

  • PyAutoLens (primary — the fix and its tests)
  • PyAutoGalaxy (read-only — Galaxy.image_2d_from semantics are not changed)
  • euclid_strong_lens_modeling_pipeline (workspace follow-up behind the library-first gate)

Work Classification

Both — library PyAutoLens first, workspace follow-up second.

Branch Survey

Surveyed via the GitHub API from a web session (no local worktree).

Repository Current Branch Dirty?
./PyAutoLens main (146a3d7) clean (fresh clone)
./PyAutoGalaxy not modified —
./euclid_strong_lens_modeling_pipeline not in session scope —

Remote branches on PyAutoLens: main, dev_Q, two archive/condemned/*; no existing feature branch for this task. No active.md worktree claim conflicts.

Suggested branch: feature/pixelized-source-magnification-latent

Worktree root: ~/Code/PyAutoLabs-wt/pixelized-source-magnification-latent/ (created later by /start_library; a web session works on the clone directly)

Implementation Steps

  1. autolens/analysis/latent.py — add _pixelized_source_flux(fit, xp=np):
    • inv = fit.inversion; if it is None, or the source galaxy (fit.tracer.galaxies[-1]) carries no Pixelization, return 0.0 (the light-profile path already covers that case).
    • Iterate fit.tracer_to_inversion.mapper_galaxy_dict.items() (autolens/lens/to_inversion.py:394; fit.tracer_to_inversion at autolens/imaging/fit_imaging.py:153) and keep the mappers whose galaxy is the source galaxy. Caveat to verify: the converted tracer's galaxies are new objects — key the lookup on whichever galaxy object actually keys mapper_galaxy_dict (check against galaxy_image_dict's keys, fit_imaging.py:204); do not assume identity with fit.tracer_linear_light_profiles_to_light_profiles.galaxies[-1].
    • For each kept mapper, xp.sum(inv.reconstruction_dict[mapper] * mapper.mesh_geometry.areas_for_magnification) (reconstruction_dict at autoarray/inversion/inversion/abstract.py:667). A mesh geometry with no areas_for_magnification → xp.nan plus a once-per-process warning, following the _maybe_magzero_warn pattern already in the module.
    • Trace safety: reconstruction_dict values are xp arrays; Delaunay dual areas are traced on the JAX path after PyAutoArray#524; rectangular areas_transformed already is. No Python-level branching on traced values.
  2. total_source_flux (latent.py:121-139) and total_source_flux_mujy (latent.py:189-215) — add the pixelized flux to the light-profile sum before the µJy conversion. magnification (latent.py:218-229) is unchanged in form. Update the three docstrings to state the new definition.
  3. test_autolens/analysis/test_latent.py — extend the existing file (NumPy path only, per the repo's JAX rule):
    • FitImaging with a Delaunay pixelized source → magnification finite and equal to Σ galaxy_image_dict[source] / Σ reconstruction·areas_for_magnification.
    • Same with RectangularBilinearAdaptDensity.
    • Linear light profile + pixelization on the source galaxy → both contributions summed.
    • Light-profile-only fit → total_source_flux, total_source_flux_mujy, magnification numerically unchanged from main.
    • A KNN mesh → NaN (and one warning).
  4. ship_library → PR-open. The PR body's API Changes section: changed behaviour — total_source_flux, total_source_flux_mujy and magnification now include pixelized sources (were 0 / inf). State that end-to-end Delaunay accuracy needs PyAutoArray#524 merged (it is, 2026-09-05, still pending release); the unit tests assert internal consistency and hold on either PyAutoArray.
  5. Workspace follow-up (/start_workspace → ship_workspace, behind the library-first gate) in euclid_strong_lens_modeling_pipeline:
    • scripts/tools/diagnose_latent_vis_pix.py:42 docstring — magnification is not stage-invariant; state the new definition.
    • tests/test_compute_latent_variable.py (or test_latent_run_level.py) — assert the vis_pix Delaunay stage's magnification is present and finite when LatentEuclid.variables is evaluated directly on a test-mode Delaunay fit of the committed dataset/simulated/euclid_dr1_like (test-mode CI skips latents at every level, so run-level summaries can only be structural).
    • Ledger note for Mind phase 9 / Cortex phase 4: archived vis_pix rows carry a 0.0 sentinel and must be re-derived; catalogue/scripts/magnitudes.py:305 ingests the latent unchanged.

Key Files

  • autolens/analysis/latent.py — the latent registry; total_source_flux, total_source_flux_mujy, magnification, LatentLens.variables (jit batch path)
  • autolens/lens/to_inversion.py — TracerToInversion.mapper_galaxy_dict (mapper → galaxy)
  • autolens/imaging/fit_imaging.py — tracer_to_inversion, galaxy_image_dict, inversion
  • test_autolens/analysis/test_latent.py — existing latent tests to extend
  • PyAutoGalaxy/autogalaxy/galaxy/galaxy.py:245-278 — Galaxy.image_2d_from zeros for a pixelization-only galaxy (read-only)
  • PyAutoArray/autoarray/inversion/pixelization/mesh_geometry/delaunay.py — areas_for_magnification (dual areas since feat: weak-lensing fit class (FitWeak + plotter) — step 3 #524)

Notes for the implementer

  • The Feature/Bug agents' heuristic sizing scores this "too-large" purely on prompt length and repo count; the declared size is small-medium and the design above is the human-approved plan — do not re-split into phases.
  • This defect is independent of the Delaunay area defect (PyAutoArray#524): the latent never touched areas_for_magnification before, and neither bug masks the other.

Original Prompt

Click to expand starting prompt

PyAutoMind/draft/bug/autolens/magnification_latent_zero_for_pixelized_source.md

magnification latent is inf/0.0 for any pixelized source: Galaxy.image_2d_from returns zeros for a Pixelization-only galaxy

Type: bug
Target: autolens
Repos:

  • PyAutoLens
  • PyAutoGalaxy
  • euclid_strong_lens_modeling_pipeline
    Themes:
  • euclid
  • pixelization
  • latent
    Difficulty: small-medium
    Autonomy: supervised
    Priority: high
    Status: formalised
    Consequence: judge
    Review-minutes: 20
    Unattended: ready
    Epic: euclid-dr1-prep
    Parent: draft/feature/euclid/euclid_dr1_prep_epic.md
    Filed: 2026-09-04

The finding

autolens/analysis/latent.py::magnification (PyAutoLens/autolens/analysis/latent.py:218-229)
is total_lensed_source_flux_mujy / total_source_flux_mujy. The denominator,
total_source_flux (:121-139, via total_source_flux_mujy at :189-215), calls
tracer.galaxies[-1].image_2d_from(grid=fit.dataset.grids.lp) on
fit.tracer_linear_light_profiles_to_light_profiles. Galaxy.image_2d_from
(PyAutoGalaxy/autogalaxy/galaxy/galaxy.py:245-278) returns xp.zeros((grid.shape[0],))
at :278 whenever the galaxy has no non-linear LightProfile — and a Pixelization is
not a LightProfile. So for any pixelized source the source flux is a hard 0.0, then
ab_mag_via_flux_from(0.0, magzero=25) = inf and flux_mujy_via_ab_mag_from(inf) = 0.0
exactly, so magnification = lensed / 0.0 = +inf. PyAutoFit's latent writer
(autofit/non_linear/analysis/latent.py:258-263, the np.isfinite(all_values[:, i]).any()
guard) then drops the whole magnification column.

Confirmed empirically, not only by reading. A real al.FitImaging built on the
pipeline's shipped dataset/simulated/euclid_dr1_like (magzero 24.6) with the truth lens
from truth.json and a Delaunay source with zeroed_pixels=30 — the vis_pix /
full_model configuration — gives total_source_flux = 0.0,
total_source_flux_mujy = 0.0, magnification = inf. The Sersic control (the vis_lp
configuration) on the same fit gives total_source_flux = 0.8679858287,
total_source_flux_mujy = 0.4555254383 and magnification = 15.146299, reproducing
truth.json's 15.146298766165936 bit-for-bit — so the latent's definition is
sound and only the pixelized-source route is broken.

Production evidence: 9/9 archived initial_lens_model/vis_pix latent_summary.json
files record magnification = 0.0 alongside total_source_flux = 0.0 — a meaningless
sentinel that a downstream catalogue would silently ingest — while the matching
initial_lens_model/vis_lp results record real values (6.2727, 10.0782, 18.4297, …). The
vis_pix key set is otherwise complete (9 keys, same as vis_lp), so the NaN-drop
signature does not fire on those particular files; absence of the key is the signature
under current code and 0.0 under whatever version produced the archive. Either way the
Euclid pixelized stages have no magnification.

Note this defect is independent of the Delaunay area defect, now shipped as
complete/2026/09/delaunay-dual-area-magnification.md (PyAutoArray#524, PR #525): the latent
never touches areas_for_magnification at all, and neither bug masks the other.

Impact

Every Euclid pixelized stage — vis_pix and full_model — has no magnification.
catalogue/scripts/magnitudes.py maps latent.magnification into the catalogue at
:305, so the sentinel propagates. The docstring of
scripts/tools/diagnose_latent_vis_pix.py, which claims that the library latents not
depending on the source reconstruction (magnification among them) are "identical
between the two stages", is false — magnification depends on the source model,
which is exactly what changes at vis_pix, and the production numbers (6.27/10.08/18.43
vs 0.0/0.0/0.0) disprove it directly. That docstring must be corrected.

Proposed fix (for the implementer to verify, not to take on trust)

Preferred: for a pixelized source, define the source-plane flux as the reconstruction
integrated with the mesh's exact quadrature areas — the barycentric dual areas for
Delaunay (see the sibling prompt above), the transformed cell areas for rectangular
meshes (whose own correctness is an open lead in the cluster epic's
draft/test/workspaces/mesh_magnification_correctness.md) — so that magnification =
image-plane mapped flux / source-plane integrated flux.

Alternative, if that is judged out of scope for the latent layer: make the latent return
NaN explicitly for pixelized sources, and stop the 0.0 sentinel from reaching the
catalogue. What must not survive is a 0.0 that reads as a measurement.

Tests: a unit test that a pixelization-only galaxy's magnification latent is finite (or
explicitly NaN, per whichever route is chosen), and a Euclid pipeline test-mode assertion
on the vis_pix latent keys so a silently-dropped or sentinel column fails loudly.

Provenance

Proven by the euclid-dr1-prep phase 8 audit, PyAutoArray#522 (audit posted on the issue),
scripts part3_latent_reach.py, part3b_fit_latent.py.

Gate note

This directly affects Cortex phase 4's witness ("catalogue numerics match the 20260623
reference") for the vis_pix magnification column, and Mind phase 9's magnification
layer. Neither can be scored on the vis_pix magnification until this is fixed.

Implementation design (architect, 2026-09-04 — approved plan; execute from this)

Sequence: the PyAutoArray dual-area fix is merged (2026-09-05, PyAutoArray#524 / PR #525,
record complete/2026/09/delaunay-dual-area-magnification.md), so the Delaunay accuracy test may
be written against main in-worktree. /start_dev this prompt → task
pixelized-source-magnification-latent; classification both: library PyAutoLens first
(PyAutoGalaxy read-only), workspace follow-up in euclid_strong_lens_modeling_pipeline behind the
library-first gate.

Facts (do not re-derive): latents run inside a per-sample JAX jit (LatentLens.BATCH_MODE = "jit", autolens/analysis/latent.py:304-336, xp = analysis._xp) — new code must be trace-safe.
total_lensed_source_flux (:106-115) reads fit.galaxy_image_dict[fit.tracer.galaxies[-1]], which
already includes the inversion's mapped image — the numerator is sound. total_source_flux
(:121-139) / total_source_flux_mujy (:189-215) call
fit.tracer_linear_light_profiles_to_light_profiles.galaxies[-1].image_2d_from(...), zeros for a
pixelization-only galaxy (autogalaxy/galaxy/galaxy.py:278). Per-mapper reconstructions:
fit.inversion.reconstruction_dict (autoarray/inversion/inversion/abstract.py:667); mapper →
galaxy: fit.tracer_to_inversion.mapper_galaxy_dict (autolens/lens/to_inversion.py:394;
fit.tracer_to_inversion at fit_imaging.py:184). Caveat: the converted tracer's galaxies are
new objects — key lookups must use whichever galaxy object actually keys mapper_galaxy_dict
(verify against galaxy_image_dict's keys).

Library changes (PyAutoLens):

  1. autolens/analysis/latent.py — add _pixelized_source_flux(fit, xp): inv = fit.inversion;
    None or no Pixelization on the source galaxy → 0.0; else for each (mapper, gal) in
    fit.tracer_to_inversion.mapper_galaxy_dict.items() belonging to the source galaxy,
    xp.sum(inv.reconstruction_dict[mapper] * mapper.mesh_geometry.areas_for_magnification). A mesh
    geometry with no areas_for_magnification (DelaunayNN, KNN, KNNBarycentric) → xp.nan with a
    one-time-per-process warning (no silent zero). total_source_flux and total_source_flux_mujy =
    light-profile flux (existing path) + pixelized flux; magnification unchanged in form.
    Trace-safe: reconstruction_dict values are xp arrays; Delaunay dual areas are xp after the
    autoarray fix; rectangular areas_transformed already is.
  2. Tests in test_autolens/analysis/ (existing latent test file, else test_latent.py): FitImaging
    with a Delaunay pixelized source → magnification finite and equal to
    Σ galaxy_image[-1] / Σ reconstruction·areas; same with RectangularBilinearAdaptDensity; linear
    light profile + pixelization sums both; light-profile-only fit numerically unchanged; a KNN mesh →
    NaN. Numpy path (JAX only if the file already has a JAX pattern).
  3. ship_library → PR-open. API Changes: Changed behaviour — total_source_flux,
    total_source_flux_mujy, magnification now include pixelized sources (were 0 / inf). The PR body
    states that end-to-end Delaunay accuracy needs the autoarray fix merged; these tests assert
    internal consistency and hold on either PyAutoArray.

Workspace follow-up (euclid_strong_lens_modeling_pipeline, /start_workspace → ship_workspace):
4. scripts/tools/diagnose_latent_vis_pix.py:42 docstring — magnification is NOT stage-invariant;
state the new definition.
5. tests/test_compute_latent_variable.py (or test_latent_run_level.py) — assert the vis_pix
Delaunay stage's magnification is present and finite when LatentEuclid.variables is evaluated
directly on a test-mode Delaunay fit of the committed dataset/simulated/euclid_dr1_like (a direct
call — test-mode CI skips latents at every level, so run-level summaries can only be structural).
6. Ledger note for Mind phase 9 / Cortex phase 4: archived vis_pix rows carry a 0.0 sentinel and
must be re-derived; catalogue/scripts/magnitudes.py:305 ingests the latent unchanged.

Acceptance (human, after both merges): the audit's part3b_fit_latent.py construction (Delaunay
fit on euclid_dr1_like, magzero 24.6) reports a finite magnification close to the Sersic
control's 15.146; record it on the issue.

Activity

  1. Jammy2211 commented on Sep 5, 2026

    @Jammy2211
    CollaboratorAuthor

    Library PR Created

    API changes (behaviour only, no signatures): total_source_flux, total_source_flux_mujy and magnification now add the inversion's reconstruction integrated with the mesh's areas_for_magnification for a pixelized source. They were 0.0 / 0.0 / inf for a pixelization-only source and are now finite. Light-profile-only fits are numerically unchanged. The mapper → galaxy lookup uses inversion.linear_obj_galaxy_dict (verified: its galaxy values are the tracer's own objects, identical to the galaxy_image_dict keys), not the uncached fit.tracer_to_inversion property.

    One finding for the record: with the installed autoarray, KNearestNeighbor, KNNBarycentric and DelaunayNN all inherit InterpolatorDelaunay.mesh_geometry, so their geometry does expose areas_for_magnification — none of them hits the NaN branch. The NaN-plus-one-warning path is pinned with a mock mapper instead; whether kNN areas are the right quadrature for those interpolants is out of scope here (see draft/test/workspaces/mesh_magnification_correctness.md).

    Tests: 31 passed in test_latent.py (7 new); full suite 617 passed, 1 xfailed.

    Next: /prm once CI is green, then /start_workspace for the euclid_strong_lens_modeling_pipeline follow-up (docstring correction in scripts/tools/diagnose_latent_vis_pix.py, the test-mode vis_pix latent assertion, and the ledger note that archived vis_pix rows carry a 0.0 sentinel).


    Generated by Claude Code

  2. Jammy2211 commented on Sep 5, 2026

    @Jammy2211
    CollaboratorAuthor

    Status: library merged + corrected, workspace PR open

    Measured (local, PyPI autoarray 2026.9.4.1 = Voronoi areas, 150 Hilbert + 30 edge points): pixelized magnification 10.95, total_source_flux 1.257; Sersic control 15.15 / 0.868. With PyAutoArray main (#524 dual areas) and the pipeline's 500-point mesh the human acceptance number on part3b_fit_latent.py will differ — that step remains open.

    Two findings for the record, not acted on here:

    1. magnification is NaN whenever the regularized reconstruction has net-negative flux, because total_source_flux_mujy routes a negative flux through log10 in the AB-mag conversion. Pre-existing behaviour of the µJy path, unchanged by these PRs; worth a prompt if it shows up on real tiles.
    2. scripts/initial_lens_model.py passes the appended grid length as Delaunay(pixels=…) alongside zeroed_pixels=30; Delaunay.__init__ adds zeroed_pixels to pixels itself, so the zeroed indices land past the end of the mesh and the edge ring is never zeroed. Latents are bit-identical either way on the test scene. Filed as PyAutoMind/draft/bug/euclid/delaunay_edge_ring_never_zeroed.md.

    Generated by Claude Code

  3. Jammy2211 commented on Sep 5, 2026

    @Jammy2211
    CollaboratorAuthor

    Shipped

    • PyAutoLens fix: magnification latent finite for pixelized sources #727 — merged 633c8e0: _pixelized_source_flux integrates the inversion's reconstruction with the mesh's areas_for_magnification; total_source_flux, total_source_flux_mujy and magnification include it, so a pixelization-only source has a finite magnification.
    • PyAutoLens fix: express pixelized source flux per data pixel in the latents #728 — merged 1f5b1e9: the integral is divided by the data pixel area (per-data-pixel convention of every other flux latent); the undivided value was 100× too large at 0.1"/pixel.
    • euclid_strong_lens_modeling_pipeline Tidy up tracer plane / image index swap #51 — merged 7067e59: diagnose_latent_vis_pix.py docstring corrected (magnification is not stage-invariant), and tests/test_compute_latent_variable.py asserts the vis_pix Delaunay magnification is present, finite and positive via a direct LatentEuclid.variables call.

    Both library PRs carry pending-release; the workspace test runs against PyAutoLens main until a release ships.

    Still open, human: the acceptance measurement on part3b_fit_latent.py (Delaunay on euclid_dr1_like, magzero 24.6, 500-point mesh, PyAutoArray main dual areas) — record the number here. Local datum on a 150+30 mesh with PyPI autoarray: 10.95 vs the Sersic control's 15.15. Archived vis_pix results still carry the 0.0 sentinel and must be re-derived (Mind epic item 8, Cortex phase 4 note).

    Mind record: PyAutoMind/complete/2026/09/pixelized-source-magnification-latent.md.


    Generated by Claude Code

  4. Jammy2211 commented on Sep 5, 2026

    @Jammy2211
    CollaboratorAuthor

    Acceptance measurement

    part3b_fit_latent.py construction, reproduced from the pipeline's test fixture: dataset/simulated/euclid_dr1_like (0.1"/px, magzero 24.6 from the header), truth lens, Delaunay source on a Hilbert image mesh + 30 circle-edge points, AdaptSplit regularization. Libraries: PyAutoLens main (#727 + #728), PyAutoArray main de92d09 (#524 barycentric dual areas), PyPI autogalaxy/autofit 2026.9.4.1.

    fit magnification total_source_flux total_lensed_source_flux
    Sersic control (= truth.json 15.1356 within the masked-vs-full-frame convention) 15.146 0.868 13.147
    Delaunay, 150 + 30 points 12.37 1.113 13.770
    Delaunay, 500 + 30 points (production mesh) 10.97 1.289 14.139

    Finite, positive, and the right order — the inf / 0.0 sentinel is gone. Against the Sersic truth the production-mesh value is 28% low. The decomposition says why: the lensed flux is 7.5% higher (the free-form source fits the noisy data more fully than the noiseless truth profile) and the source-plane flux is 48% higher (the regularized reconstruction carries a positive floor across the 10.2 arcsec² hull; NNLS positivity means noise only adds flux). That is the source-model systematic Cortex phase 7 (magnification_robustness) exists to measure, not a latent defect; the mesh areas are the dual areas (0 zeroed cells, Σ = hull area) and the pixel-area convention is right (150 → 500 points moves it by 11%, not 100×).

    Edge-ring double count, verified: as written the pipeline's Delaunay(pixels=530, zeroed_pixels=30) yields mesh.pixels = 560 for a 530-parameter mapper with zeroed indices 530..559 (out of range), so the explicit zeroing is a no-op — but the 30 ring values are exactly zero regardless (no data maps to them; 294 of 530 pixels are zero under the positivity solver), and correcting it to pixels=500 gives a bit-identical solution. Filed at low severity.


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions