Skip to content

feat(interferometer): visualizer on array-free datasets — natural-weighted dirty panels, inversion subplot fix (streaming phase 3) #596

Closed
#597
Closed
feat(interferometer): visualizer on array-free datasets — natural-weighted dirty panels, inversion subplot fix (streaming phase 3)#596
#597

Description

@Jammy2211
Jammy2211
opened on Sep 30, 2026

Overview

Streaming epic phase 3 (Mind epic streaming-visibilities; Discussion https://github.com/orgs/PyAutoLabs/discussions/13). Phases 1–2 (#593, PyAutoLabs/PyAutoGalaxy#639, PyAutoLabs/PyAutoLens#758) let an array-free aa.Interferometer (no data / noise_map / uv_wavelengths / transformer) fit, save and reload. The go/no-go benchmark (2026-09-30) showed the in-memory sparse path runs out of memory at ~1e6 visibilities on a 16 GB laptop while streaming stays flat at 1.5–1.8 GB to 5e7, so the epic continues. A streamed fit is still usable only with visualization off: the dataset subplot reads six visibility-space quantities that raise, every fit.dirty_* panel goes through the absent transformer, subplot_of_mapper / subplot_mappings escape with InversionException because _recon_array reads the operated dict first (a live bug), and both visualizers' fallback logger(...) is a TypeError.

Plan

  • Branch in place on is_array_free inside the existing plot functions (same names, same filenames): skip visibility-space panels, draw natural-weighted dirty image / beam / model / residual from the sparse terms; in-memory output unchanged.
  • One autoarray helper for the natural dirty model image (W̃·m / Σw via operated_matrix_slim_from); ag/al fit properties model_image_natural, dirty_model_image_natural, dirty_residual_map_natural.
  • Fix _recon_array (fall back to mapped_reconstructed_data_dict without a transformer) and the handlers; fix the two logger(...) calls; fix the stale autolens fits assertion.
  • Tests in all three repos incl. visualize_before_fit + visualize end to end on an array-free fit; witness with visualization on.
Detailed implementation plan

Affected Repositories

  • PyAutoArray (primary), PyAutoGalaxy, PyAutoLens

Branch Survey

Repository Current Branch Dirty?
./PyAutoArray main clean
./PyAutoGalaxy main clean
./PyAutoLens main clean

PyAutoArray is also claimed by raw-pdip-forward-polish (#594, PR #595; NNLS/settings files only — disjoint). Parallel claim approved with the plan 2026-09-30. Suggested branch: feature/streaming-p3-visualizer

Design decisions (human-approved 2026-09-30, on the epic ledger)

Branch in place; in-memory byte-unchanged; skip rather than approximate visibility-space panels; natural-weighted panels from the terms with model image sum(galaxy_image_dict.values()); labels "… (Natural)" and EXTNAMEs DIRTY_IMAGE_NATURAL, DIRTY_BEAM, DIRTY_MODEL_IMAGE_NATURAL, DIRTY_RESIDUAL_MAP_NATURAL; no silent except DatasetException; no new skip switch (filed separately).

Implementation Steps

  1. PyAutoArray autoarray/fit/fit_interferometer.py: dirty_model_image_natural_from(dataset, image).
  2. autoarray/dataset/plot/interferometer_plots.py: array-free branches in subplot_interferometer_dataset, subplot_interferometer_dirty_images (1×2 natural dirty image / beam) and fits_interferometer (guard data reads; natural extensions).
  3. autoarray/fit/plot/fit_interferometer_plots.py: optional model_image kwarg; natural panels on array-free.
  4. autoarray/inversion/plot/inversion_plots.py: _recon_array fallback without a transformer; exc.InversionException in the two handlers.
  5. PyAutoGalaxy: fit properties; subplot_fit, subplot_fit_dirty_images, subplot_fit_real_space, fits_dirty_images array-free branches; visualizer logger.warning.
  6. PyAutoLens: fit properties; subplot_fit, subplot_fit_dirty_images, subplot_fit_interferometer_combined, subplot_fit_real_space, subplot_tracer_from_fit array-free branches; visualizer logger.warning + positions image → dirty_image_natural; stale dirty_images.fits assertion fixed.
  7. Tests in all three repos (array-free plot outputs written; in-memory unchanged; _recon_array red-check; helper vs transformer path at 1e-10; model_image_natural == inversion.mapped_reconstructed_data on pixelization-only fits).
  8. Witness: phase-2 MockSearch setup with visualization ON; png/fits listing for array-free vs in-memory; in-memory fits arrays identical to base.

Key Files

  • autoarray/dataset/plot/interferometer_plots.py, autoarray/fit/plot/fit_interferometer_plots.py, autoarray/inversion/plot/inversion_plots.py, autoarray/fit/fit_interferometer.py
  • autogalaxy/interferometer/{fit_interferometer.py,plot/fit_interferometer_plots.py,model/visualizer.py}
  • autolens/interferometer/{fit_interferometer.py,plot/fit_interferometer_plots.py,model/visualizer.py}

Original Prompt

Click to expand starting prompt

Streaming phase 3: visualizer on array-free datasets (natural-weighted dirty panels, uv panels skipped)

Type: feature
Target: PyAutoArray
Repos:

  • PyAutoArray
  • PyAutoGalaxy
  • PyAutoLens
    Themes:
  • interferometer
  • sparse-operator
  • memory
    Autonomy: supervised
    Priority: medium
    Status: draft
    Epic: streaming-visibilities
    Phase: 3
    Difficulty: medium
    Consequence: glance
    Witness: visualize_before_fit and visualize (ag + al interferometer visualizers) on an array-free fit write every png/fits without raising — uv/visibility panels skipped, dirty image / dirty beam / dirty model / dirty residual present from the terms and labelled "(natural weighting)" — while in-memory output is byte-unchanged; inversion_plots._recon_array no longer performs a forward transform to type-check.
    Review-minutes: 6
    Unattended: ready
    Parent: draft/feature/autoarray/interferometer_from_stream_array_free_dataset.md
    Blocked-by: none (phase 2 merged 2026-09-30)

Source: https://github.com/orgs/PyAutoLabs/discussions/13 phase 2, sliced 2026-09-30 (decision (c)). After this phase ships, post the promised follow-up on the discussion (see epics.md notes).

What

  1. autoarray/dataset/plot/interferometer_plots.py: guard uv/visibility/S-N panels on array presence (precedent: fits_interferometer
    noise_map is not None guards); dirty image / beam from dirty_image_natural / dirty_beam when arrays are absent.
  2. fit/plot/fit_interferometer_plots.py (+ ag/al mirrors): dirty model = W~·image via operated_matrix_slim_from; dirty residual =
    d~ − W~·(M s + i_p); skip normalized-residual / chi-squared dirty panels and the uv subplots when arrays are absent.
  3. inversion/plot/inversion_plots.py _recon_array: replace the forward-transform type probe.
  4. ag/al visualizers: no crash paths; tests mirror test_plotter_interferometer.py with an array-free fixture.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JZZksyZ8LTA4LLxoZjQMNF

Activity

  1. Jammy2211 commented on Sep 30, 2026

    @Jammy2211
    Jammy2211
    on Sep 30, 2026
    CollaboratorAuthor

    Heart RED override — development only (recorded per AUTONOMY.md)

    • Heart verdict at ship: RED, 2026-09-30T18:32:16Z. Exact reason strings: release validation FAILED (stage integrate); PyAutoLens: on branch feature/point-solver-padding-backend (not main); PyAutoLens: 2 uncommitted source change(s) (the last two are canonical-checkout drift from another session, not these branches). Unrelated to this work.
    • Authorization: live human, in-session 2026-09-30, for this issue (feat(interferometer): visualizer on array-free datasets — natural-weighted dirty panels, inversion subplot fix (streaming phase 3) #596) and the three feature/streaming-p3-visualizer branches (PyAutoArray, PyAutoGalaxy, PyAutoLens): "Authorize override for feat(interferometer): visualizer on array-free datasets — natural-weighted dirty panels, inversion subplot fix (streaming phase 3) #596" — commit, push and the pending-release PRs. Merge remains a separate explicit human command with every required check green. No release, no rehearsal, no claim that these branches repair Heart.
    • Branch gates passed before the ask: test_autoarray 1898 passed; test_autogalaxy 1293 passed; test_autolens 774 passed + 1 xfailed; red-checks (inversion-plot fix; array-free tests with source stashed); witness with visualization on; independent Codex gpt-6-astra review: FINDINGS (1) — model_image_natural summed galaxy_image_dict, whose entries are overwritten by a galaxy's linear reconstruction, so a galaxy with both ordinary and linear light lost its ordinary component (reproduced: rel error 1.0 in ag, 1.14/1.31 in al); fixed in-branch as profile_image + inversion.mapped_reconstructed_data, with mixed-galaxy tests against an independent transformer reference from fit.model_data (rel ≤ 2e-14), red-checked.

    🤖 Generated with Claude Code

  2. Jammy2211 commented on Sep 30, 2026

    @Jammy2211
    Jammy2211
    on Sep 30, 2026
    CollaboratorAuthor

    Library PRs created

    Tests: test_autoarray 1898 / test_autogalaxy 1293 / test_autolens 776 + 1 xfailed

    Shipped under the human-authorized Heart RED development override (comment above); merge stays a separate human command via /prm once CI is green (PyAutoArray → PyAutoGalaxy → PyAutoLens).

    🤖 Generated with Claude Code

  3. Jammy2211
    closed this as completedin #597on Sep 30, 2026

  4. Jammy2211 commented on Sep 30, 2026

    @Jammy2211
    Jammy2211
    on Sep 30, 2026
    CollaboratorAuthor

    Shipped

    All pending-release. Streaming epic phase 3 of 5: a streamed (array-free) dataset now fits, saves, reloads and visualizes end to end.

    🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions