Repository navigation
Description
Overview
Phase 5 (last) of the streaming-visibilities epic (Discussion https://github.com/orgs/PyAutoLabs/discussions/13). The streamed accumulator sparse_terms_from_chunks already sums per-chunk SparseTerms; this task (a) proves and exposes that MFS terms are the sum of per-channel terms (sum(list_of_terms) via __radd__, parity vs an in-memory MFS dataset at 1e-12), (b) adds chunk-by-chunk phase-centre shifts (phase_centre=(y, x) arcsec; data multiplied by exp(+2πi(u l0 + v m0)), radians; recorded as provenance so terms with different centres cannot be summed), threaded through Interferometer.from_stream, and (c) adds an array-free datacube example to autolens_workspace that runs under the smoke profile. Library first (PyAutoArray), workspace PR follows.
Plan
- Add an optional
phase_centretosparse_terms_from_chunks, applied to each chunk's visibilities as a unit phase before the dirty image is formed; thread it throughInterferometer.from_stream. - Record
phase_centreasSparseTermsprovenance checked in__add__; add__radd__sosum(terms_list)works. - Tests: per-channel sum == one accumulation == in-memory MFS operator/scalars at rel 1e-12 (numpy and jax); phase-centre accumulation == in-memory result on pre-shifted data at 1e-12; DFT point-source test pins the sign and (y, x) order.
- Workspace: new
scripts/interferometer/features/datacube/modeling_array_free.pystreaming each channel throughfrom_streaminto the existing FactorGraph fit, showing the MFS sum andphase_centre; listed insmoke_tests.txt; notebook regenerated.
Detailed implementation plan
Affected Repositories
- PyAutoArray (primary, library)
- autolens_workspace (workspace, after the library PR)
Branch Survey
| Repository | Current Branch | Dirty? |
|---|---|---|
| array/PyAutoArray | main | clean |
| lens/autolens_workspace | main | clean |
Suggested branch: feature/streaming-p5-cubes-phase-centre (both repos; worktree ~/Code/PyAutoLabs-wt/streaming-p5-cubes-phase-centre/)
Implementation Steps
A. PyAutoArray (array/PyAutoArray) — library PR
autoarray/inversion/inversion/interferometer/inversion_interferometer_util.pySparseTerms(L1879): add provenance fieldphase_centre: Optional[Tuple[float, float]] = None(arcsec, autoarray
(y, x)order likeorigin);__add__(L1944) checks it with the other recorded provenance (mismatch →
exc.InversionException),_recorded(L2009) carries it; add__radd__(self, other)returningselfwhen
other == 0(sosum(terms_list)works) andNotImplementedotherwise.sparse_terms_from_chunks(L2034): new kwargphase_centre: Optional[Tuple[float, float]] = None(arcsec,
(y, x)). After_complex_visibilities_from(data)(L2119) and the σ checks, when set:
l0, m0 = x, y in radians(units.arcsec.to(units.rad)astransformer.py:331does; note the tuple order), then
data = data * np.exp(2j*np.pi*(uv[:,0]*l0 + uv[:,1]*m0)). Kernel / beam / scalars untouched. Record
phase_centrein the chunk provenance. Docstring: what the shift means, the sign, the units, invariance of the
scalars, and that MFS = sum of per-channel terms (__add__,sum).
autoarray/dataset/interferometer/dataset.pyfrom_stream(L328): addphase_centre=Noneand pass it explicitly
(the kwargs are listed by hand at L372-382).apply_sparse_operator_from_chunks(L605) needs no change (forwards
**accumulator_kwargs) — add a docstring line.from_sparse_terms(L252): no change (phase centre is not a mask
property).- Tests
test_autoarray/inversion/inversion/interferometer/test_inversion_interferometer_util.py(helpers
_streaming_inputs(n_visibilities, seed)L1129,_chunks_fromL1153; template
test__sparse_terms_from_chunks__matches_one_shot_…L1161):test__sparse_terms__sum_of_channels_equals_mfs: 3 "channels" from_streaming_inputswith different seeds /
sizes;sum([terms_c for c])(exercises__radd__) vssparse_terms_from_chunksover the concatenated chunks vs
the in-memoryInterferometer(concatenated).apply_sparse_operator()operator kernel,dirty_image,data_term,
noise_normalization,sum_weights,n_visat rel 1e-12; repeat withuse_jax=True.test__sparse_terms_from_chunks__phase_centre__matches_in_memory_on_shifted_data: accumulate with
phase_centre=(m0, l0)vsapply_sparse_operatorondata * exp(+2πi(u l0 + v m0))(radians) at 1e-12, NUFFT and
DFT; kernel / beam / scalars identical to the unshifted run.test__sparse_terms_from_chunks__phase_centre__point_source_recentres(DFT, small mask): visibilities of a point
source at (y, x) = (m0, l0) viaTransformerDFT.visibilities_from; the shifted dirty image's argmax is the
mask-centre pixel (pins the sign and the (y, x) order).__add__with differingphase_centreraises; unrecorded-left keeps right (mirror L1404).test_autoarray/dataset/interferometer/test_dataset.py:from_stream(..., phase_centre=…)carries it into
sparse_terms.phase_centreand equals the pre-shifted in-memory sparse fit'slog_evidence-relevant scalars
(mirror L684);from_sparse_terms(sum(per_channel_terms))builds an array-free dataset whose fit matches the
in-memory MFS sparse fit (reuse_random_interferometer).
B. autolens_workspace (lens/autolens_workspace) — workspace PR, after the library PR
- New
scripts/interferometer/features/datacube/modeling_array_free.py(sibling ofmodeling.py; same mask /
dataset / FactorGraph structure):- For each
channel_*: buildchunksas a generator over the channel's(uv_wavelengths, data, noise_map)fits
(one chunk, or sliced into two to show chunking),dataset = al.Interferometer.from_stream(chunks, real_space_mask=mask, transformer_class=al.TransformerNUFFT); then the sameAnalysisInterferometer/
AnalysisFactor/FactorGraphModelfit asmodeling.py. Prose explains what array-free means, that no
data/noise_map/uv_wavelengthsare held, what the visualizer shows instead (natural dirty images, from P3),
and that ordinary light profiles now work (P4). - A short section:
mfs_terms = sum(dataset_c.sparse_terms for dataset_c in datasets)→
al.Interferometer.from_sparse_terms(mfs_terms, real_space_mask=mask)with a one-channel-model MFS fit, and a
sentence onphase_centre=for re-centring (no fit; illustrate the call on one channel). - Smoke: no
__Env__override (defaults: TEST_MODE 2, SMALL_DATASETS cap, DISABLE_JAX —from_streamalready honours
disable_jax()); build the terms on the same (possibly capped) mask passed tofrom_sparse_terms; do not
hard-codeuse_jax=True. Addscripts/interferometer/features/datacube/modeling_array_free.pyto
smoke_tests.txt. Reusesimulator.pyauto-simulation exactly asmodeling.pydoes.
- For each
scripts/interferometer/features/datacube/README.md: one row for the new script.- Notebook mirror under
notebooks/…via the workspace's generator (resolve the exact command in start_workspace —
generate.pyruns from the workspace root with the project key; see memoryCWD+key,md≠nb). - Run the script under the smoke profile locally (
PYAUTO_TEST_MODE=2 SMALL_DATASETS=1 DISABLE_JAX=1 …) and with
the full profile once; smoke test via/smoke_testbefore/ship_workspace.
Trade-offs / decisions
- API units and order:
phase_centrein arcsec, (y, x) — matchesorigin,centreand every autoarray tuple;
converted to radians inside the accumulator. Documented with the sign convention. - Shift on data, not on the kernel: W̃ and the beam are shift-invariant; shifting the data is one complex multiply
per chunk and keeps everything else bit-identical (asserted in tests). - No change to
__add__semantics beyond the new provenance check;__radd__only for0sosum()works. - Workspace example is additive (new script), leaving
modeling.pyas the in-memory reference. - Out of scope: per-channel models beyond the existing FactorGraph; visualization of per-channel dirty beams;
the profiling campaign (per-chunk JAX recompile) stays in its own draft.
Key Files
autoarray/inversion/inversion/interferometer/inversion_interferometer_util.py—SparseTerms(phase_centre,__radd__),sparse_terms_from_chunks(phase_centre=)autoarray/dataset/interferometer/dataset.py—Interferometer.from_stream(phase_centre=)test_autoarray/inversion/inversion/interferometer/test_inversion_interferometer_util.py,test_autoarray/dataset/interferometer/test_dataset.pyautolens_workspace/scripts/interferometer/features/datacube/modeling_array_free.py(new),README.md,smoke_tests.txt, notebook mirror
Verification
pytest test_autoarray/inversion/inversion/interferometer/test_inversion_interferometer_util.py test_autoarray/dataset/interferometer/test_dataset.py, then fulltest_autoarray(baseline after P4: 1903).- Witness (prompt header): sum of per-channel
SparseTerms== MFS terms at rel 1e-12 (numpy and jax);
phase_centreaccumulation == in-memory result on data shifted byexp(2πi(u l0 + v m0))at 1e-12; the
autolens_workspacedatacube example gains an array-free variant that runs under the smoke profile. - Workspace:
modeling_array_free.pyruns under the smoke profile locally and in the workspace smoke CI on the
same-named branch; notebook regenerated; full-profile run completes once.
Original Prompt
Click to expand starting prompt
Streaming phase 5: per-channel cubes and phase-centre shifts in sparse_terms_from_chunks
Type: feature
Target: PyAutoArray
Repos:
- PyAutoArray
- autolens_workspace
Themes: - interferometer
- sparse-operator
- memory
Autonomy: supervised
Priority: medium
Status: draft
Epic: streaming-visibilities
Phase: 5
Difficulty: small
Consequence: glance
Witness: summing per-channelSparseTermsequals the MFS terms accumulated over all channels at rel 1e-12 (numpy and jax);sparse_terms_from_chunks(..., phase_centre=(l0, m0))applied per chunk equals the in-memory result on data shifted byexp(2πi(u l0 + v m0))at 1e-12; theautolens_workspacedatacube example gains an array-free variant that runs under the smoke profile.
Review-minutes: 4
Unattended: ready
Parent: draft/feature/autoarray/interferometer_from_stream_array_free_dataset.md
Blocked-by: none (phase 3 merged 2026-09-30)
Source: https://github.com/orgs/PyAutoLabs/discussions/13 phase 2, sliced 2026-09-30. No phase-centre code exists today; SparseTerms.__add__ exists and is tested; datacube examples live in autolens_workspace/scripts/interferometer/features/datacube/.
Activity
Jammy2211 commentedon Oct 1, 2026
Library PR Created (streaming P5)
SparseTerms.phase_centreprovenance +__radd__;sparse_terms_from_chunks(phase_centre=)/Interferometer.from_stream(phase_centre=); sum of per-channel terms == MFS (1e-12, numpy + JAX brute-force leg); DFT point-source test pins the sign and (y, x) order.phase_centrepersisted through thedataset.fitsround trip (PyAutoLens reuses the ag writer/loader).Both labelled
pending-release; merge order Array → Galaxy (separate human/prm). Next:/start_workspace→ the array-free datacube exampleautolens_workspace/scripts/interferometer/features/datacube/modeling_array_free.py(smoke-listed), shipped under the same grant once its full-profile run and smoke test pass.Heart RED development override (human-authorized, 2026-10-01)
Live human authorization in the CLI session (Fable 5.1), quoted verbatim: "Yes: library now, workspace when its run passes" for this task. Exact RED reasons at ship time (
pyauto-heart readiness):None relate to this branch. Scope: push + PR-open only (library now; workspace PR under the same grant on a passing run); merge via separate
/prmon green checks; not a release.Branch gates
phase_centre, reloadedNonesummed with anything) → persisted in PyAutoGalaxy#643; F2 (high, introduced —apply_sparse_operator_from_chunks(phase_centre=)shifted the operator but not the retained data; sparse χ² 0 vs residual 2) → typed raise; F4 (medium —data_termfrom unshifted data while the dirty image used shifted data, Δχ² 1800 at σr≈σi) → shift applied before every data-dependent term; also the JAX test leg made real (spy) and__add__checkstransformer_class_name. F3/F5 pre-existing (NUFFT ignores mask origin; mask-compatibility gap) →draft/bug/autoarray/sparse_terms_nufft_origin_and_mask_compatibility.md.modeling_array_free.pysmoke exit 0 (6 s),modeling.pyregression exit 0, navigator/catalogue/sizes/black checks OK; full-profile run in progress.Deviations from the plan
phase_centre=(0.0, 0.0)(notNone) so shifted + unshifted sums raise;None= not recorded (hand-built / pre-change files).phase_centredemo re-centres on the dirty-image arc peak (lensed emission), not the source-plane centre.Jammy2211 commentedon Oct 1, 2026
Workspace PR Created (streaming P5)
scripts/interferometer/features/datacube/modeling_array_free.py(+ generated notebook, README row,smoke_tests.txtentry, navigator catalogue). Streams each channel throughfrom_stream(2 chunks/channel) into the existing FactorGraph fit; MFS section viasum()of per-channelSparseTerms(n_vis760 asserted; channel-0 evidence identical to the in-memory dataset at the true model);phase_centredemo re-centres a channel's dirty-image peak on the origin with its value preserved.Gates: full-profile run exit 0 (~13 min,
n_live=50);pyauto-heart smoke autolens --root <worktree>: 42/42 scripts PASS, 2/2 notebooks PASS (modeling_array_free.py6.4 s);check_navigator/check_search_memory/check_sizes/blackOK. Opened under the same human RED override grant ("Yes: library now, workspace when its run passes"); merge via/prmafter PyAutoArray#601 and PyAutoGalaxy#643 (library-first), and thephase_centre/sum()sections need the released library (release-gate).Follow-ups filed:
draft/bug/autoarray/interferometer_log_evidence_large_regularization_coefficient_cholesky.md(search's max-likelihood coefficient ≈ 9.7e5 makeslog_evidenceraise/NaN eagerly on both dataset types);scripts/multi_dataset/plot.pyleaves an untrackeddataset.fitsin the workspace root on every smoke run (not this branch).Jammy2211 commentedon Oct 1, 2026
Shipped
Merged 2026-10-01 via human
/prmwith every CI leg green, mergeability CLEAN, library-first:The two library PRs remain pending release (labelled
pending-release; last tag 2026.9.27.2); the workspace example'sphase_centreandsum()sections need that release (release-gate). Mind record:complete/2026/10/streaming-p5-cubes-phase-centre.md. This closes the development phases of thestreaming-visibilitiesepic; the Discussion #13 follow-up post is the remaining item. Follow-ups filed:draft/bug/autoarray/sparse_terms_nufft_origin_and_mask_compatibility.md,draft/bug/autoarray/interferometer_log_evidence_large_regularization_coefficient_cholesky.md.