Repository navigation
fix: magnification latent is inf/0.0 for any pixelized source #726
Description
Overview
The magnification latent in autolens/analysis/latent.py is total_lensed_source_flux_mujy / total_source_flux_mujy. The denominator calls Galaxy.image_2d_from on the converted tracer's source galaxy, and a pixelization-only galaxy returns zeros there (a Pixelization is not a LightProfile). So every pixelized fit gets total_source_flux = 0.0, total_source_flux_mujy = 0.0 and magnification = +inf — dropped as a column by PyAutoFit's NaN guard under current code, or archived as a 0.0 sentinel (9/9 initial_lens_model/vis_pix results). Every Euclid vis_pix / full_model stage therefore has no magnification, while the Sersic control on the same fit reproduces truth.json's 15.146 bit-for-bit — the latent's definition is sound, only the pixelized-source route is broken. Fix: define the source-plane flux of a pixelized source as the inversion's reconstruction integrated with the mesh's exact quadrature areas (areas_for_magnification, barycentric dual areas for Delaunay since PyAutoArray#524 / PR #525), added to the existing light-profile flux.
Plan
- Add a
_pixelized_source_flux(fit, xp)helper to the latent module: for each mapper of the source galaxy in the fit's inversion, sumreconstruction · areas_for_magnification; a mesh geometry without quadrature areas (DelaunayNN / KNN / KNNBarycentric) yields NaN with a once-per-process warning, never a silent zero. total_source_fluxandtotal_source_flux_mujybecome light-profile flux (existing path) + pixelized flux;magnificationis unchanged in form and becomes finite for pixelized sources.- Keep every new line trace-safe: latents run inside the per-sample
jax.jitbatch path (LatentLens.BATCH_MODE = "jit",xp = analysis._xp). - Unit tests in
test_autolens/analysis/test_latent.py: Delaunay and rectangular pixelized fits give a finitemagnificationequal to Σ galaxy image / Σ reconstruction·areas; linear light profile + pixelization sums both contributions; light-profile-only fits are numerically unchanged; a KNN mesh gives NaN. - Ship the PyAutoLens PR first (library-first gate). Workspace follow-up in
euclid_strong_lens_modeling_pipeline: correct the false "magnification is stage-invariant" docstring, add a test-modevis_pixlatent assertion, and leave a ledger note that archivedvis_pixrows carry a0.0sentinel and must be re-derived. - Human acceptance after both merges: the phase 8 audit's
part3b_fit_latent.pyconstruction (Delaunay fit oneuclid_dr1_like, magzero 24.6) reports a finitemagnificationclose to the Sersic control's 15.146 — record it on this issue.
Detailed implementation plan
Affected Repositories
- PyAutoLens (primary — the fix and its tests)
- PyAutoGalaxy (read-only —
Galaxy.image_2d_fromsemantics are not changed) - euclid_strong_lens_modeling_pipeline (workspace follow-up behind the library-first gate)
Work Classification
Both — library PyAutoLens first, workspace follow-up second.
Branch Survey
Surveyed via the GitHub API from a web session (no local worktree).
| Repository | Current Branch | Dirty? |
|---|---|---|
| ./PyAutoLens | main (146a3d7) |
clean (fresh clone) |
| ./PyAutoGalaxy | not modified | — |
| ./euclid_strong_lens_modeling_pipeline | not in session scope | — |
Remote branches on PyAutoLens: main, dev_Q, two archive/condemned/*; no existing feature branch for this task. No active.md worktree claim conflicts.
Suggested branch: feature/pixelized-source-magnification-latent
Worktree root: ~/Code/PyAutoLabs-wt/pixelized-source-magnification-latent/ (created later by /start_library; a web session works on the clone directly)
Implementation Steps
autolens/analysis/latent.py— add_pixelized_source_flux(fit, xp=np):inv = fit.inversion; if it isNone, or the source galaxy (fit.tracer.galaxies[-1]) carries noPixelization, return0.0(the light-profile path already covers that case).- Iterate
fit.tracer_to_inversion.mapper_galaxy_dict.items()(autolens/lens/to_inversion.py:394;fit.tracer_to_inversionatautolens/imaging/fit_imaging.py:153) and keep the mappers whose galaxy is the source galaxy. Caveat to verify: the converted tracer's galaxies are new objects — key the lookup on whichever galaxy object actually keysmapper_galaxy_dict(check againstgalaxy_image_dict's keys,fit_imaging.py:204); do not assume identity withfit.tracer_linear_light_profiles_to_light_profiles.galaxies[-1]. - For each kept mapper,
xp.sum(inv.reconstruction_dict[mapper] * mapper.mesh_geometry.areas_for_magnification)(reconstruction_dictatautoarray/inversion/inversion/abstract.py:667). A mesh geometry with noareas_for_magnification→xp.nanplus a once-per-process warning, following the_maybe_magzero_warnpattern already in the module. - Trace safety:
reconstruction_dictvalues arexparrays; Delaunay dual areas are traced on the JAX path after PyAutoArray#524; rectangularareas_transformedalready is. No Python-level branching on traced values.
total_source_flux(latent.py:121-139) andtotal_source_flux_mujy(latent.py:189-215) — add the pixelized flux to the light-profile sum before the µJy conversion.magnification(latent.py:218-229) is unchanged in form. Update the three docstrings to state the new definition.test_autolens/analysis/test_latent.py— extend the existing file (NumPy path only, per the repo's JAX rule):FitImagingwith a Delaunay pixelized source →magnificationfinite and equal toΣ galaxy_image_dict[source] / Σ reconstruction·areas_for_magnification.- Same with
RectangularBilinearAdaptDensity. - Linear light profile + pixelization on the source galaxy → both contributions summed.
- Light-profile-only fit →
total_source_flux,total_source_flux_mujy,magnificationnumerically unchanged frommain. - A KNN mesh → NaN (and one warning).
ship_library→ PR-open. The PR body's API Changes section: changed behaviour —total_source_flux,total_source_flux_mujyandmagnificationnow include pixelized sources (were 0 / inf). State that end-to-end Delaunay accuracy needs PyAutoArray#524 merged (it is, 2026-09-05, still pending release); the unit tests assert internal consistency and hold on either PyAutoArray.- Workspace follow-up (
/start_workspace→ship_workspace, behind the library-first gate) ineuclid_strong_lens_modeling_pipeline:scripts/tools/diagnose_latent_vis_pix.py:42docstring —magnificationis not stage-invariant; state the new definition.tests/test_compute_latent_variable.py(ortest_latent_run_level.py) — assert thevis_pixDelaunay stage'smagnificationis present and finite whenLatentEuclid.variablesis evaluated directly on a test-mode Delaunay fit of the committeddataset/simulated/euclid_dr1_like(test-mode CI skips latents at every level, so run-level summaries can only be structural).- Ledger note for Mind phase 9 / Cortex phase 4: archived
vis_pixrows carry a0.0sentinel and must be re-derived;catalogue/scripts/magnitudes.py:305ingests the latent unchanged.
Key Files
autolens/analysis/latent.py— the latent registry;total_source_flux,total_source_flux_mujy,magnification,LatentLens.variables(jit batch path)autolens/lens/to_inversion.py—TracerToInversion.mapper_galaxy_dict(mapper → galaxy)autolens/imaging/fit_imaging.py—tracer_to_inversion,galaxy_image_dict,inversiontest_autolens/analysis/test_latent.py— existing latent tests to extendPyAutoGalaxy/autogalaxy/galaxy/galaxy.py:245-278—Galaxy.image_2d_fromzeros for a pixelization-only galaxy (read-only)PyAutoArray/autoarray/inversion/pixelization/mesh_geometry/delaunay.py—areas_for_magnification(dual areas since feat: weak-lensing fit class (FitWeak + plotter) — step 3 #524)
Notes for the implementer
- The Feature/Bug agents' heuristic sizing scores this "too-large" purely on prompt length and repo count; the declared size is small-medium and the design above is the human-approved plan — do not re-split into phases.
- This defect is independent of the Delaunay area defect (PyAutoArray#524): the latent never touched
areas_for_magnificationbefore, and neither bug masks the other.
Original Prompt
Click to expand starting prompt
PyAutoMind/draft/bug/autolens/magnification_latent_zero_for_pixelized_source.md
magnification latent is inf/0.0 for any pixelized source: Galaxy.image_2d_from returns zeros for a Pixelization-only galaxy
Type: bug
Target: autolens
Repos:
- PyAutoLens
- PyAutoGalaxy
- euclid_strong_lens_modeling_pipeline
Themes: - euclid
- pixelization
- latent
Difficulty: small-medium
Autonomy: supervised
Priority: high
Status: formalised
Consequence: judge
Review-minutes: 20
Unattended: ready
Epic: euclid-dr1-prep
Parent: draft/feature/euclid/euclid_dr1_prep_epic.md
Filed: 2026-09-04
The finding
autolens/analysis/latent.py::magnification (PyAutoLens/autolens/analysis/latent.py:218-229)
is total_lensed_source_flux_mujy / total_source_flux_mujy. The denominator,
total_source_flux (:121-139, via total_source_flux_mujy at :189-215), calls
tracer.galaxies[-1].image_2d_from(grid=fit.dataset.grids.lp) on
fit.tracer_linear_light_profiles_to_light_profiles. Galaxy.image_2d_from
(PyAutoGalaxy/autogalaxy/galaxy/galaxy.py:245-278) returns xp.zeros((grid.shape[0],))
at :278 whenever the galaxy has no non-linear LightProfile — and a Pixelization is
not a LightProfile. So for any pixelized source the source flux is a hard 0.0, then
ab_mag_via_flux_from(0.0, magzero=25) = inf and flux_mujy_via_ab_mag_from(inf) = 0.0
exactly, so magnification = lensed / 0.0 = +inf. PyAutoFit's latent writer
(autofit/non_linear/analysis/latent.py:258-263, the np.isfinite(all_values[:, i]).any()
guard) then drops the whole magnification column.
Confirmed empirically, not only by reading. A real al.FitImaging built on the
pipeline's shipped dataset/simulated/euclid_dr1_like (magzero 24.6) with the truth lens
from truth.json and a Delaunay source with zeroed_pixels=30 — the vis_pix /
full_model configuration — gives total_source_flux = 0.0,
total_source_flux_mujy = 0.0, magnification = inf. The Sersic control (the vis_lp
configuration) on the same fit gives total_source_flux = 0.8679858287,
total_source_flux_mujy = 0.4555254383 and magnification = 15.146299, reproducing
truth.json's 15.146298766165936 bit-for-bit — so the latent's definition is
sound and only the pixelized-source route is broken.
Production evidence: 9/9 archived initial_lens_model/vis_pix latent_summary.json
files record magnification = 0.0 alongside total_source_flux = 0.0 — a meaningless
sentinel that a downstream catalogue would silently ingest — while the matching
initial_lens_model/vis_lp results record real values (6.2727, 10.0782, 18.4297, …). The
vis_pix key set is otherwise complete (9 keys, same as vis_lp), so the NaN-drop
signature does not fire on those particular files; absence of the key is the signature
under current code and 0.0 under whatever version produced the archive. Either way the
Euclid pixelized stages have no magnification.
Note this defect is independent of the Delaunay area defect, now shipped as
complete/2026/09/delaunay-dual-area-magnification.md (PyAutoArray#524, PR #525): the latent
never touches areas_for_magnification at all, and neither bug masks the other.
Impact
Every Euclid pixelized stage — vis_pix and full_model — has no magnification.
catalogue/scripts/magnitudes.py maps latent.magnification into the catalogue at
:305, so the sentinel propagates. The docstring of
scripts/tools/diagnose_latent_vis_pix.py, which claims that the library latents not
depending on the source reconstruction (magnification among them) are "identical
between the two stages", is false — magnification depends on the source model,
which is exactly what changes at vis_pix, and the production numbers (6.27/10.08/18.43
vs 0.0/0.0/0.0) disprove it directly. That docstring must be corrected.
Proposed fix (for the implementer to verify, not to take on trust)
Preferred: for a pixelized source, define the source-plane flux as the reconstruction
integrated with the mesh's exact quadrature areas — the barycentric dual areas for
Delaunay (see the sibling prompt above), the transformed cell areas for rectangular
meshes (whose own correctness is an open lead in the cluster epic's
draft/test/workspaces/mesh_magnification_correctness.md) — so that magnification =
image-plane mapped flux / source-plane integrated flux.
Alternative, if that is judged out of scope for the latent layer: make the latent return
NaN explicitly for pixelized sources, and stop the 0.0 sentinel from reaching the
catalogue. What must not survive is a 0.0 that reads as a measurement.
Tests: a unit test that a pixelization-only galaxy's magnification latent is finite (or
explicitly NaN, per whichever route is chosen), and a Euclid pipeline test-mode assertion
on the vis_pix latent keys so a silently-dropped or sentinel column fails loudly.
Provenance
Proven by the euclid-dr1-prep phase 8 audit, PyAutoArray#522 (audit posted on the issue),
scripts part3_latent_reach.py, part3b_fit_latent.py.
Gate note
This directly affects Cortex phase 4's witness ("catalogue numerics match the 20260623
reference") for the vis_pix magnification column, and Mind phase 9's magnification
layer. Neither can be scored on the vis_pix magnification until this is fixed.
Implementation design (architect, 2026-09-04 — approved plan; execute from this)
Sequence: the PyAutoArray dual-area fix is merged (2026-09-05, PyAutoArray#524 / PR #525,
record complete/2026/09/delaunay-dual-area-magnification.md), so the Delaunay accuracy test may
be written against main in-worktree. /start_dev this prompt → task
pixelized-source-magnification-latent; classification both: library PyAutoLens first
(PyAutoGalaxy read-only), workspace follow-up in euclid_strong_lens_modeling_pipeline behind the
library-first gate.
Facts (do not re-derive): latents run inside a per-sample JAX jit (LatentLens.BATCH_MODE = "jit", autolens/analysis/latent.py:304-336, xp = analysis._xp) — new code must be trace-safe.
total_lensed_source_flux (:106-115) reads fit.galaxy_image_dict[fit.tracer.galaxies[-1]], which
already includes the inversion's mapped image — the numerator is sound. total_source_flux
(:121-139) / total_source_flux_mujy (:189-215) call
fit.tracer_linear_light_profiles_to_light_profiles.galaxies[-1].image_2d_from(...), zeros for a
pixelization-only galaxy (autogalaxy/galaxy/galaxy.py:278). Per-mapper reconstructions:
fit.inversion.reconstruction_dict (autoarray/inversion/inversion/abstract.py:667); mapper →
galaxy: fit.tracer_to_inversion.mapper_galaxy_dict (autolens/lens/to_inversion.py:394;
fit.tracer_to_inversion at fit_imaging.py:184). Caveat: the converted tracer's galaxies are
new objects — key lookups must use whichever galaxy object actually keys mapper_galaxy_dict
(verify against galaxy_image_dict's keys).
Library changes (PyAutoLens):
autolens/analysis/latent.py— add_pixelized_source_flux(fit, xp):inv = fit.inversion;
None or noPixelizationon the source galaxy →0.0; else for each(mapper, gal)in
fit.tracer_to_inversion.mapper_galaxy_dict.items()belonging to the source galaxy,
xp.sum(inv.reconstruction_dict[mapper] * mapper.mesh_geometry.areas_for_magnification). A mesh
geometry with noareas_for_magnification(DelaunayNN, KNN, KNNBarycentric) →xp.nanwith a
one-time-per-process warning (no silent zero).total_source_fluxandtotal_source_flux_mujy=
light-profile flux (existing path) + pixelized flux;magnificationunchanged in form.
Trace-safe:reconstruction_dictvalues arexparrays; Delaunay dual areas arexpafter the
autoarray fix; rectangularareas_transformedalready is.- Tests in
test_autolens/analysis/(existing latent test file, elsetest_latent.py): FitImaging
with a Delaunay pixelized source →magnificationfinite and equal to
Σ galaxy_image[-1] / Σ reconstruction·areas; same withRectangularBilinearAdaptDensity; linear
light profile + pixelization sums both; light-profile-only fit numerically unchanged; a KNN mesh →
NaN. Numpy path (JAX only if the file already has a JAX pattern). ship_library→ PR-open. API Changes: Changed behaviour —total_source_flux,
total_source_flux_mujy,magnificationnow include pixelized sources (were 0 / inf). The PR body
states that end-to-end Delaunay accuracy needs the autoarray fix merged; these tests assert
internal consistency and hold on either PyAutoArray.
Workspace follow-up (euclid_strong_lens_modeling_pipeline, /start_workspace → ship_workspace):
4. scripts/tools/diagnose_latent_vis_pix.py:42 docstring — magnification is NOT stage-invariant;
state the new definition.
5. tests/test_compute_latent_variable.py (or test_latent_run_level.py) — assert the vis_pix
Delaunay stage's magnification is present and finite when LatentEuclid.variables is evaluated
directly on a test-mode Delaunay fit of the committed dataset/simulated/euclid_dr1_like (a direct
call — test-mode CI skips latents at every level, so run-level summaries can only be structural).
6. Ledger note for Mind phase 9 / Cortex phase 4: archived vis_pix rows carry a 0.0 sentinel and
must be re-derived; catalogue/scripts/magnitudes.py:305 ingests the latent unchanged.
Acceptance (human, after both merges): the audit's part3b_fit_latent.py construction (Delaunay
fit on euclid_dr1_like, magzero 24.6) reports a finite magnification close to the Sersic
control's 15.146; record it on the issue.
Activity
Jammy2211 commentedon Sep 5, 2026
Library PR Created
feature/pixelized-source-magnification-latent, labelledpending-release)API changes (behaviour only, no signatures):
total_source_flux,total_source_flux_mujyandmagnificationnow add the inversion's reconstruction integrated with the mesh'sareas_for_magnificationfor a pixelized source. They were0.0/0.0/inffor a pixelization-only source and are now finite. Light-profile-only fits are numerically unchanged. The mapper → galaxy lookup usesinversion.linear_obj_galaxy_dict(verified: its galaxy values are the tracer's own objects, identical to thegalaxy_image_dictkeys), not the uncachedfit.tracer_to_inversionproperty.One finding for the record: with the installed autoarray,
KNearestNeighbor,KNNBarycentricandDelaunayNNall inheritInterpolatorDelaunay.mesh_geometry, so their geometry does exposeareas_for_magnification— none of them hits the NaN branch. The NaN-plus-one-warning path is pinned with a mock mapper instead; whether kNN areas are the right quadrature for those interpolants is out of scope here (seedraft/test/workspaces/mesh_magnification_correctness.md).Tests: 31 passed in
test_latent.py(7 new); full suite 617 passed, 1 xfailed.Next:
/prmonce CI is green, then/start_workspacefor theeuclid_strong_lens_modeling_pipelinefollow-up (docstring correction inscripts/tools/diagnose_latent_vis_pix.py, the test-modevis_pixlatent assertion, and the ledger note that archivedvis_pixrows carry a0.0sentinel).Generated by Claude Code
Jammy2211 commentedon Sep 5, 2026
Status: library merged + corrected, workspace PR open
633c8e0): pixelized source flux = reconstruction integrated withareas_for_magnification.euclid_dr1_likescene the pixelizedmagnificationcame out 1095 — exactly 100× the 10.95 it gives after the fix (Sersic control 15.15). PyAutoLens' 1" fixtures could not see it; fix: express pixelized source flux per data pixel in the latents #728 adds a 0.5"/pixel test that pins the division and a fixed-field-of-view scene whose pixelized magnification is stable across pixel scales.diagnose_latent_vis_pix.pyand the directLatentEuclid.variablestest asserting thevis_pixDelaunay magnification is present, finite and positive.Measured (local, PyPI autoarray 2026.9.4.1 = Voronoi areas, 150 Hilbert + 30 edge points): pixelized
magnification10.95,total_source_flux1.257; Sersic control 15.15 / 0.868. With PyAutoArraymain(#524 dual areas) and the pipeline's 500-point mesh the human acceptance number onpart3b_fit_latent.pywill differ — that step remains open.Two findings for the record, not acted on here:
magnificationis NaN whenever the regularized reconstruction has net-negative flux, becausetotal_source_flux_mujyroutes a negative flux throughlog10in the AB-mag conversion. Pre-existing behaviour of the µJy path, unchanged by these PRs; worth a prompt if it shows up on real tiles.scripts/initial_lens_model.pypasses the appended grid length asDelaunay(pixels=…)alongsidezeroed_pixels=30;Delaunay.__init__addszeroed_pixelstopixelsitself, so the zeroed indices land past the end of the mesh and the edge ring is never zeroed. Latents are bit-identical either way on the test scene. Filed asPyAutoMind/draft/bug/euclid/delaunay_edge_ring_never_zeroed.md.Generated by Claude Code
Jammy2211 commentedon Sep 5, 2026
Shipped
633c8e0:_pixelized_source_fluxintegrates the inversion's reconstruction with the mesh'sareas_for_magnification;total_source_flux,total_source_flux_mujyandmagnificationinclude it, so a pixelization-only source has a finite magnification.1f5b1e9: the integral is divided by the data pixel area (per-data-pixel convention of every other flux latent); the undivided value was 100× too large at 0.1"/pixel.7067e59:diagnose_latent_vis_pix.pydocstring corrected (magnification is not stage-invariant), andtests/test_compute_latent_variable.pyasserts thevis_pixDelaunay magnification is present, finite and positive via a directLatentEuclid.variablescall.Both library PRs carry
pending-release; the workspace test runs against PyAutoLensmainuntil a release ships.Still open, human: the acceptance measurement on
part3b_fit_latent.py(Delaunay oneuclid_dr1_like, magzero 24.6, 500-point mesh, PyAutoArraymaindual areas) — record the number here. Local datum on a 150+30 mesh with PyPI autoarray: 10.95 vs the Sersic control's 15.15. Archivedvis_pixresults still carry the0.0sentinel and must be re-derived (Mind epic item 8, Cortex phase 4 note).Mind record:
PyAutoMind/complete/2026/09/pixelized-source-magnification-latent.md.Generated by Claude Code
Jammy2211 commentedon Sep 5, 2026
Acceptance measurement
part3b_fit_latent.pyconstruction, reproduced from the pipeline's test fixture:dataset/simulated/euclid_dr1_like(0.1"/px, magzero 24.6 from the header), truth lens, Delaunay source on a Hilbert image mesh + 30 circle-edge points,AdaptSplitregularization. Libraries: PyAutoLensmain(#727 + #728), PyAutoArraymainde92d09(#524 barycentric dual areas), PyPI autogalaxy/autofit 2026.9.4.1.Finite, positive, and the right order — the
inf/0.0sentinel is gone. Against the Sersic truth the production-mesh value is 28% low. The decomposition says why: the lensed flux is 7.5% higher (the free-form source fits the noisy data more fully than the noiseless truth profile) and the source-plane flux is 48% higher (the regularized reconstruction carries a positive floor across the 10.2 arcsec² hull; NNLS positivity means noise only adds flux). That is the source-model systematic Cortex phase 7 (magnification_robustness) exists to measure, not a latent defect; the mesh areas are the dual areas (0 zeroed cells, Σ = hull area) and the pixel-area convention is right (150 → 500 points moves it by 11%, not 100×).Edge-ring double count, verified: as written the pipeline's
Delaunay(pixels=530, zeroed_pixels=30)yieldsmesh.pixels = 560for a 530-parameter mapper with zeroed indices 530..559 (out of range), so the explicit zeroing is a no-op — but the 30 ring values are exactly zero regardless (no data maps to them; 294 of 530 pixels are zero under the positivity solver), and correcting it topixels=500gives a bit-identical solution. Filed at low severity.Generated by Claude Code
fix: magnification latent finite for pixelized sources (PyAutoLabs#726)
fix: express pixelized source flux per data pixel in the latents (PyA…