Repository navigation
feat(interferometer): pixelization-only fits, save_attributes and aggregator reload on array-free datasets (streaming phase 2) #638
Description
Overview
Streaming epic phase 2 (Mind epic streaming-visibilities; Discussion https://github.com/orgs/PyAutoLabs/discussions/13). Phase 1 (PyAutoLabs/PyAutoArray#593) added the array-free aa.Interferometer.from_stream / from_sparse_terms dataset (no data, noise_map, uv_wavelengths, transformer). An AnalysisInterferometer search still cannot run on it: save_attributes writes the visibility arrays and the transformer class, the aggregator loader reads dataset.fits HDUs positionally and never re-attaches a sparse operator, and FitInterferometer.profile_visibilities builds Visibilities.zeros(N_vis) through the transformer. This phase makes pixelization-only fits, save_attributes and the aggregator reload work on the array-free dataset, in PyAutoGalaxy and PyAutoLens.
Plan
- Guard
profile_visibilities/profile_subtracted_visibilitiesso a transformer-less dataset with no light profile subtracts nothing (a light profile on it raises a typed error until phase 4); passtransformer=Nonethrough to the inversion. save_attributes: when the dataset is array-free, writedataset.fitsfrom theSparseTerms(EXTNAMEsMASK,NUFFT_PRECISION_OPERATOR,DIRTY_IMAGE,DIRTY_BEAM; scalars and provenance in the header); keep the visibility layout, now EXTNAME-tagged, otherwise; skip thetransformer_classjson without a transformer. One shared helper, used by both repos.- Aggregator loader (autogalaxy, imported by autolens): EXTNAME lookup with a positional fallback for legacy outputs; rebuild via
aa.SparseTerms→aa.Interferometer.from_sparse_termswhen the terms extensions are present. - Tests in both repos incl. a real save → reload round trip; witness =
AnalysisInterferometerend to end (visualization disabled until phase 3) with reloadedlog_evidenceequal at 1e-8.
Detailed implementation plan
Affected Repositories
- PyAutoGalaxy (primary), PyAutoLens (mirror). PyAutoArray unchanged (phase 1 on main).
Branch Survey
| Repository | Current Branch | Dirty? |
|---|---|---|
| ./PyAutoGalaxy | main | clean |
| ./PyAutoLens | main | clean |
Worktree guard clear. Suggested branch: feature/streaming-p2-fit-save-reload
Implementation Steps
autogalaxy/interferometer/fit_interferometer.py:profile_visibilities→Nonewhendataset.transformer is Noneand no galaxy has a light profile; raiseexc.DatasetExceptionif one does;profile_subtracted_visibilities→Nonewhendata is None;inversion_with_datareturns the inversion itself whendata is None.galaxies_to_inversionpassestransformer=Nonethrough (GalaxiesToInversion.transformertolerates it).autogalaxy/interferometer/model/analysis.py: extractinterferometer_hdu_list_from(dataset)— array-free branch writes mask +nufft_precision_operator+dirty_image_native+dirty_beam_nativewith EXTNAMEs and header scalarsSUM_W,DATA_TERM,NOISE_NORM,N_VIS,EPS,TRANSFORMER_CLASS(plus the mask's own header); in-memory branch unchanged except EXTNAMEs;save_attributesuses it and writestransformer_classonly when a transformer exists.autogalaxy/aggregator/interferometer/interferometer.py_interferometer_from:_hdu(hdu_list, name)case-insensitive EXTNAME lookup with positional fallback; terms present →aa.SparseTerms(...)from extensions + header →aa.Interferometer.from_sparse_terms(terms, real_space_mask); else existing path.autolens/interferometer/fit_interferometer.pyandautolens/interferometer/model/analysis.py: mirror (reuse the autogalaxy helper for the HDU list).- Tests: array-free pixelization-only fit parity (numpy + jit) and light-profile typed error in both repos;
save_attributesEXTNAME/header assertions for both dataset kinds; aggregator round trip throughaf.DirectoryPathsintmp_pathwith a minimalfitstub exposing.value(name); red-check the round trip against the unpatched loader. - Witness in the PR body: end-to-end
AnalysisInterferometer(visualization off),dataset.fitsEXTNAME listing and size, reloadedlog_evidenceequal at 1e-8.
Key Files
autogalaxy/interferometer/fit_interferometer.py,autogalaxy/interferometer/model/analysis.py,autogalaxy/aggregator/interferometer/interferometer.pyautolens/interferometer/fit_interferometer.py,autolens/interferometer/model/analysis.pytest_autogalaxy/interferometer/,test_autolens/interferometer/
Original Prompt
Click to expand starting prompt
Streaming phase 2: pixelization-only fits, save_attributes and aggregator reload on array-free datasets
Type: feature
Target: PyAutoGalaxy
Repos:
- PyAutoGalaxy
- PyAutoLens
Themes: - interferometer
- sparse-operator
- memory
Autonomy: supervised
Priority: medium
Status: draft
Epic: streaming-visibilities
Phase: 2
Difficulty: medium
Consequence: judge
Witness: a pixelization-onlyag.FitInterferometerandal.FitInterferometeron anaa.Interferometer.from_streamdataset evaluatefigure_of_merit(numpy andjax.jit) equal to the in-memory sparse fit at rel 1e-8; anAnalysisInterferometerrun on it completes,save_attributeswritesdataset.fitswithSparseTermsFITS extensions (EXTNAME lookup) instead of visibilities, and the aggregator rebuilds the dataset viafrom_sparse_termswith the sparse operator re-attached so the reloaded fit's log_evidence equals the original at 1e-8.
Review-minutes: 8
Unattended: ready
Parent: draft/feature/autoarray/interferometer_from_stream_array_free_dataset.md
Blocked-by: none (phase 1 merged 2026-09-30, PyAutoArray#593)
Source: https://github.com/orgs/PyAutoLabs/discussions/13 phase 2, sliced 2026-09-30 (decisions (b), (d)).
What
profile_visibilities(ag + al) must not formVisibilities.zeros(N_vis)whentransformer is None;galaxies_to_inversion/
tracer_to_inversionaccept a missing transformer (GalaxiesToInversion.transformeralready tolerates it).save_attributes(aginterferometer/model/analysis.py~L216-276; al ~L320-380): when the dataset has no arrays, write
EXTNAMEsMASK,NUFFT_PRECISION_OPERATOR,DIRTY_IMAGE,DIRTY_BEAMand header scalarsSUM_W,DATA_TERM,
NOISE_NORM,N_VIS,EPS,TRANSFORMER_CLASS; keep the visibility layout (now also EXTNAME-tagged) when arrays exist.autogalaxy/aggregator/interferometer/interferometer.py_interferometer_from(autolens imports it): switch from positional
HDUs to EXTNAME lookup; rebuild viafrom_sparse_termswhen the terms extensions are present.- Tests in
test_autogalaxy/interferometer(+ model/analysis save/reload round-trip, which has no test today) and the autolens mirrors.
🤖 Generated with Claude Code
Activity
Jammy2211 commentedon Sep 30, 2026
Heart RED override — development only (recorded per AUTONOMY.md)
release validation FAILED (stage integrate). Unrelated to these branches.feature/streaming-p2-fit-save-reloadon PyAutoGalaxy and PyAutoLens: "Authorize override for feat(interferometer): pixelization-only fits, save_attributes and aggregator reload on array-free datasets (streaming phase 2) #638" — commit, push and the pending-release PRs. Merge remains a separate explicit human command with every required check green. No release, no rehearsal, no claim that these branches repair Heart.test_autogalaxy1284 passed;test_autolens770 passed + 1 xfailed; array-free round trip red-checked against the old loader; witness reload exact; independent Codex gpt-6-astra review: FINDINGS (3) — Feature/config #1 FITS header cards truncate float64 scalars (reproduced: 1.2345678901234567e+20 → 1.23456789012345e+20): fixed in-branch with a losslessSPARSE_TERMS_SCALARSfloat64 HDU (header cards kept as readable copies, loader falls back to them for older files), red-checked; Refactor/autogalaxy #3agg_util.mask_header_fromreadPIXSCAYfor both axes (pre-existing): fixed in-branch with a non-square round-trip test, red-checked; Coordinates #2 database-path searches never registerdataset.fitsviasave_fits(pre-existing, in-memory path too): filed asdraft/bug/autogalaxy/database_paths_dataset_fits_not_registered.md.🤖 Generated with Claude Code
Jammy2211 commentedon Sep 30, 2026
Library PRs created
test_autogalaxy/: 1287 passed; test_autolens/: 770 passed, 1 xfailed
Shipped under the human-authorized Heart RED development override (comment above); merge stays a separate human command via /prm once CI is green (PyAutoGalaxy first).
🤖 Generated with Claude Code
Jammy2211 commentedon Sep 30, 2026
Shipped
interferometer_hdu_list_from(SparseTerms FITS extensions + lossless scalars HDU), EXTNAME aggregator loader with re-attached operator,mask_header_frompixel-scale fix.save_attributesthrough the shared helper.Both
pending-release. Streaming epic phase 2 of 5; phase 3 (visualizer) is next, after which the follow-up on Discussion #13 is due.🤖 Generated with Claude Code