Skip to content

feat(interferometer): pixelization-only fits, save_attributes and aggregator reload on array-free datasets (streaming phase 2) #638

Closed

Description

@Jammy2211

Overview

Streaming epic phase 2 (Mind epic streaming-visibilities; Discussion https://github.com/orgs/PyAutoLabs/discussions/13). Phase 1 (PyAutoLabs/PyAutoArray#593) added the array-free aa.Interferometer.from_stream / from_sparse_terms dataset (no data, noise_map, uv_wavelengths, transformer). An AnalysisInterferometer search still cannot run on it: save_attributes writes the visibility arrays and the transformer class, the aggregator loader reads dataset.fits HDUs positionally and never re-attaches a sparse operator, and FitInterferometer.profile_visibilities builds Visibilities.zeros(N_vis) through the transformer. This phase makes pixelization-only fits, save_attributes and the aggregator reload work on the array-free dataset, in PyAutoGalaxy and PyAutoLens.

Plan

  • Guard profile_visibilities / profile_subtracted_visibilities so a transformer-less dataset with no light profile subtracts nothing (a light profile on it raises a typed error until phase 4); pass transformer=None through to the inversion.
  • save_attributes: when the dataset is array-free, write dataset.fits from the SparseTerms (EXTNAMEs MASK, NUFFT_PRECISION_OPERATOR, DIRTY_IMAGE, DIRTY_BEAM; scalars and provenance in the header); keep the visibility layout, now EXTNAME-tagged, otherwise; skip the transformer_class json without a transformer. One shared helper, used by both repos.
  • Aggregator loader (autogalaxy, imported by autolens): EXTNAME lookup with a positional fallback for legacy outputs; rebuild via aa.SparseTerms → aa.Interferometer.from_sparse_terms when the terms extensions are present.
  • Tests in both repos incl. a real save → reload round trip; witness = AnalysisInterferometer end to end (visualization disabled until phase 3) with reloaded log_evidence equal at 1e-8.
Detailed implementation plan

Affected Repositories

  • PyAutoGalaxy (primary), PyAutoLens (mirror). PyAutoArray unchanged (phase 1 on main).

Branch Survey

Repository Current Branch Dirty?
./PyAutoGalaxy main clean
./PyAutoLens main clean

Worktree guard clear. Suggested branch: feature/streaming-p2-fit-save-reload

Implementation Steps

  1. autogalaxy/interferometer/fit_interferometer.py: profile_visibilities → None when dataset.transformer is None and no galaxy has a light profile; raise exc.DatasetException if one does; profile_subtracted_visibilities → None when data is None; inversion_with_data returns the inversion itself when data is None. galaxies_to_inversion passes transformer=None through (GalaxiesToInversion.transformer tolerates it).
  2. autogalaxy/interferometer/model/analysis.py: extract interferometer_hdu_list_from(dataset) — array-free branch writes mask + nufft_precision_operator + dirty_image_native + dirty_beam_native with EXTNAMEs and header scalars SUM_W, DATA_TERM, NOISE_NORM, N_VIS, EPS, TRANSFORMER_CLASS (plus the mask's own header); in-memory branch unchanged except EXTNAMEs; save_attributes uses it and writes transformer_class only when a transformer exists.
  3. autogalaxy/aggregator/interferometer/interferometer.py _interferometer_from: _hdu(hdu_list, name) case-insensitive EXTNAME lookup with positional fallback; terms present → aa.SparseTerms(...) from extensions + header → aa.Interferometer.from_sparse_terms(terms, real_space_mask); else existing path.
  4. autolens/interferometer/fit_interferometer.py and autolens/interferometer/model/analysis.py: mirror (reuse the autogalaxy helper for the HDU list).
  5. Tests: array-free pixelization-only fit parity (numpy + jit) and light-profile typed error in both repos; save_attributes EXTNAME/header assertions for both dataset kinds; aggregator round trip through af.DirectoryPaths in tmp_path with a minimal fit stub exposing .value(name); red-check the round trip against the unpatched loader.
  6. Witness in the PR body: end-to-end AnalysisInterferometer (visualization off), dataset.fits EXTNAME listing and size, reloaded log_evidence equal at 1e-8.

Key Files

  • autogalaxy/interferometer/fit_interferometer.py, autogalaxy/interferometer/model/analysis.py, autogalaxy/aggregator/interferometer/interferometer.py
  • autolens/interferometer/fit_interferometer.py, autolens/interferometer/model/analysis.py
  • test_autogalaxy/interferometer/, test_autolens/interferometer/

Original Prompt

Click to expand starting prompt

Streaming phase 2: pixelization-only fits, save_attributes and aggregator reload on array-free datasets

Type: feature
Target: PyAutoGalaxy
Repos:

  • PyAutoGalaxy
  • PyAutoLens
    Themes:
  • interferometer
  • sparse-operator
  • memory
    Autonomy: supervised
    Priority: medium
    Status: draft
    Epic: streaming-visibilities
    Phase: 2
    Difficulty: medium
    Consequence: judge
    Witness: a pixelization-only ag.FitInterferometer and al.FitInterferometer on an aa.Interferometer.from_stream dataset evaluate figure_of_merit (numpy and jax.jit) equal to the in-memory sparse fit at rel 1e-8; an AnalysisInterferometer run on it completes, save_attributes writes dataset.fits with SparseTerms FITS extensions (EXTNAME lookup) instead of visibilities, and the aggregator rebuilds the dataset via from_sparse_terms with the sparse operator re-attached so the reloaded fit's log_evidence equals the original at 1e-8.
    Review-minutes: 8
    Unattended: ready
    Parent: draft/feature/autoarray/interferometer_from_stream_array_free_dataset.md
    Blocked-by: none (phase 1 merged 2026-09-30, PyAutoArray#593)

Source: https://github.com/orgs/PyAutoLabs/discussions/13 phase 2, sliced 2026-09-30 (decisions (b), (d)).

What

  1. profile_visibilities (ag + al) must not form Visibilities.zeros(N_vis) when transformer is None; galaxies_to_inversion /
    tracer_to_inversion accept a missing transformer (GalaxiesToInversion.transformer already tolerates it).
  2. save_attributes (ag interferometer/model/analysis.py ~L216-276; al ~L320-380): when the dataset has no arrays, write
    EXTNAMEs MASK, NUFFT_PRECISION_OPERATOR, DIRTY_IMAGE, DIRTY_BEAM and header scalars SUM_W, DATA_TERM,
    NOISE_NORM, N_VIS, EPS, TRANSFORMER_CLASS; keep the visibility layout (now also EXTNAME-tagged) when arrays exist.
  3. autogalaxy/aggregator/interferometer/interferometer.py _interferometer_from (autolens imports it): switch from positional
    HDUs to EXTNAME lookup; rebuild via from_sparse_terms when the terms extensions are present.
  4. Tests in test_autogalaxy/interferometer (+ model/analysis save/reload round-trip, which has no test today) and the autolens mirrors.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JZZksyZ8LTA4LLxoZjQMNF

Activity

  1. Jammy2211 commented on Sep 30, 2026

    @Jammy2211
    CollaboratorAuthor

    Heart RED override — development only (recorded per AUTONOMY.md)

    • Heart verdict at ship: RED, 2026-09-30T12:42:09Z (unchanged since the phase-1 ship). Exact reason string: release validation FAILED (stage integrate). Unrelated to these branches.
    • Authorization: live human, in-session 2026-09-30, for this issue (feat(interferometer): pixelization-only fits, save_attributes and aggregator reload on array-free datasets (streaming phase 2) #638) and branches feature/streaming-p2-fit-save-reload on PyAutoGalaxy and PyAutoLens: "Authorize override for feat(interferometer): pixelization-only fits, save_attributes and aggregator reload on array-free datasets (streaming phase 2) #638" — commit, push and the pending-release PRs. Merge remains a separate explicit human command with every required check green. No release, no rehearsal, no claim that these branches repair Heart.
    • Branch gates passed before the ask: test_autogalaxy 1284 passed; test_autolens 770 passed + 1 xfailed; array-free round trip red-checked against the old loader; witness reload exact; independent Codex gpt-6-astra review: FINDINGS (3) — Feature/config #1 FITS header cards truncate float64 scalars (reproduced: 1.2345678901234567e+20 → 1.23456789012345e+20): fixed in-branch with a lossless SPARSE_TERMS_SCALARS float64 HDU (header cards kept as readable copies, loader falls back to them for older files), red-checked; Refactor/autogalaxy #3 agg_util.mask_header_from read PIXSCAY for both axes (pre-existing): fixed in-branch with a non-square round-trip test, red-checked; Coordinates #2 database-path searches never register dataset.fits via save_fits (pre-existing, in-memory path too): filed as draft/bug/autogalaxy/database_paths_dataset_fits_not_registered.md.

    🤖 Generated with Claude Code

  2. Jammy2211 commented on Sep 30, 2026

    @Jammy2211
    CollaboratorAuthor

    Library PRs created

    test_autogalaxy/: 1287 passed; test_autolens/: 770 passed, 1 xfailed

    Shipped under the human-authorized Heart RED development override (comment above); merge stays a separate human command via /prm once CI is green (PyAutoGalaxy first).

    🤖 Generated with Claude Code

  3. Jammy2211 commented on Sep 30, 2026

    @Jammy2211
    CollaboratorAuthor

    Shipped

    Both pending-release. Streaming epic phase 2 of 5; phase 3 (visualizer) is next, after which the follow-up on Discussion #13 is due.

    🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions