Repository navigation
research: interferometer streaming vs in-memory scaling (phase 1, CPU) #368
Description
Overview
Phase 1 of the interferometer streaming-scaling campaign (contract: PyAutoPulse tasks/interferometer_streaming_scaling.md). It ports the 2026-09-30 scratchpad benchmark into versioned cells. The cells measure Interferometer.from_stream accumulation time and peak RSS against N_vis and chunk size. They push the in-memory apply_sparse_operator path to failure under a 10 GB address-space cap and check log_evidence parity. All runs are CPU only, on the local machine. Streaming epic phases 3-5 have already shipped (PyAutoArray #589/#593/#601, released 2026.10.2.1), so the result is recorded as release evidence, not as a go/no-go.
Plan
- Add three cells under
scripts/interferometer/streaming_scaling/(accumulate, in_memory, parity). They use_profile_cli.py, and each measurement runs in a fresh child process with an RLIMIT_AS cap and a timeout. - Run accumulate at N_vis {1e6, 4e6, 1.6e7} x chunk {4096, 65536}. Run 5e7 and 1e8 only if the total wall time stays under about 45 min.
- Run in_memory upward until the first failure under the cap, and run parity at 4e6.
- Write JSON rows and a PNG to
results/streaming_scaling/. - Write the wiki page
wiki/campaigns/interferometer_streaming.md: the table, the 2e8 extrapolation, and a cProfile top-10 if any rate exceeds 5 s per 1e6 vis. Add awiki/index.mdrow. - Regenerate the README and dashboard, run lint and pytest, and open the PR. The PR is not merged.
- Timing, memory, correctness and hardware are reported separately. No speed score is computed.
Tier: glance — merge mode: held for human /prm (orchestrator instruction: open the PR, do not merge)
Detailed implementation plan
Affected Repositories
- autolens_profiling (primary)
Branch Survey
| Repository | Current Branch | Dirty? |
|---|---|---|
| lens/autolens_profiling | main | untracked dataset/abell_1201/ (another task's) |
Suggested branch: feature/interferometer-streaming-scaling
Implementation Steps
scripts/interferometer/streaming_scaling/_stream_common.py. It provides seeded synthetic per-chunk visibilities, a 400-px circular mask at 0.05"/pix, andTransformerNUFFT. It also provides the child runner:multiprocessingspawn,resource.setrlimit(RLIMIT_AS)and a timeout, with peak RSS taken fromru_maxrss.accumulate.py. It timesInterferometer.from_streamat each (N_vis, chunk) and records s per 1e6 vis and the linearity fit.in_memory.py. It runsInterferometer(...).apply_sparse_operator()at increasing N until the first failure, recording the MemoryError or kill as the result.parity.py. It compares the log_evidence of a 20x20 rectangular sparse inversion, streamed against in-memory, at 4e6.- Write the results to
results/streaming_scaling/(JSON + PNG). Update the wiki page and the index row, then runbuild_readme.pyandbuild_dashboard.py. - Lint: check_wiki, check_results_layout, check_submits, ruff, pytest.
Key Files
_profile_cli.py— the shared CLI/JSON helperwiki/campaigns/_template.md— the template for the wiki page
Original Prompt
Click to expand starting prompt
See PyAutoMind active/interferometer_streaming_scaling.md and PyAutoPulse tasks/interferometer_streaming_scaling.md. The human request was: "do the proposed priority order stuff, all of it".
Activity
Jammy2211 commentedon Oct 4, 2026
Phase 1 runs complete locally (worktree
~/Code/PyAutoLabs-wt/interferometer-streaming-scaling, branchfeature/interferometer-streaming-scaling, uncommitted). from_stream at chunk 65536 runs at about 10 s per 1e6 vis (5e7 in 504 s, 1.69 GB peak RSS). The in-memory path first fails at 1e6 under the 10 GB RLIMIT_AS cap, on both arms. Streamed and in-memory log_evidence agree to 1.2e-9 nats at 5e5; parity at 4e6 cannot be measured because the in-memory arm runs out of memory there. The 1e8 row was skipped on the wall budget. All lints pass and pytest passes (1018). Parked at the ship gate:pyauto-heart readinessreturns RED with 'release validation FAILED (stage integrate)'. That failure is not caused by this task. Committing, pushing and opening the PR need a human Heart-RED override (AUTONOMY.md).Merge pull request #375 from PyAutoLabs/feature/interferometer-stream…
Jammy2211 commentedon Oct 4, 2026
Shipped
Phase 1 of the interferometer streaming-scaling campaign adds the CPU cells under
scripts/interferometer/streaming_scaling/, the rows inresults/streaming_scaling/andwiki/campaigns/interferometer_streaming.md.Remainder (to re-file):
transformer.image_from(1568 B/vis transient), a library candidate;Jammy2211 commentedon Oct 17, 2026
Reopened 2026-10-17: phase 2, RAL rows on the released 2026.10.17.1 stack
Human-authorized compute (2026-10-17, "Do all that, I authorise all"). Measurement only: no library edits (PyAutoArray#635 owns the interferometer library files). Mind prompt
active/interferometer_streaming_ral_a100_rows.md; Pulse tasktasks/interferometer_streaming_scaling.md.High-level plan
2026.10.17.1tags, from a private library checkout (never the shared mirror; noHPCPullPyAuto).--partition=ralonly, 8 threads): the full streaming grid plus the 1e8 row skipped in phase 1, and the in-memory and parity cells on the same node, so the two paths sit side by side on matched hardware.--gres=gpu:1 --exclusive): the same three cells, fp64, with device peak memory recorded beside host peak RSS.results/streaming_scaling/, extend the campaign wiki (phase 2 row, tables, journal), regenerate catalogue/dashboard.Detailed plan
scripts/interferometer/delaunay/_streaming.py+ the three cells:--hardwaretag (defaultlocal_cpu, so phase-1 names are unchanged) →<cell>_<hardware>_v<version>.{json,png}; the child recordscompile_s/n_compiles(jax.monitoring duration events) andexec_s = wall_s - compile_s, the JAX backend and x64 flag, and on GPU the devicepeak_bytes_in_use/bytes_limit(withXLA_PYTHON_CLIENT_PREALLOCATE=false).--cap-gb 0(already supported) disablesRLIMIT_AS, which CUDA cannot run under.hpc/batch_cpu/submit_streaming_scaling_ral_cpu_fp64andhpc/batch_gpu/submit_streaming_scaling_a100_fp64with WALL-BASIS blocks, an import guard asserting the private 2026.10.17.1 checkout, and the provenance header (node, loadavg, SHAs).1e6,4e6,1.6e7× chunks65536,4096+ extra5e7:65536,1e8:65536(budget large enough to run 1e8); in_memory at the phase-1 10 GB cap (comparable) and uncapped to the SLURM--mem(where the real host wall is); parity at 5e5 and 4e6 (uncapped). A100: the same grid with noRLIMIT_AS.build_dashboard --check),ship_workspace; judge CI once; no merge.Branch:
feature/interferometer-streaming-memo-after(shared with the memo after-measurement issue, one PR).Claim note:
autolens_profilingis also claimed byhst-mge-pilot-qualification(#418, shipped, awaiting merge, touches onlyscripts/misc/jax_compile/*.md); no file overlap.results: streaming scaling phase 2 (RAL CPU 1e8 + A100) and fnnls mem…
Jammy2211 commentedon Oct 17, 2026
Measured on RAL at 2026.10.17.1 and shipped as PR #435 (awaiting human /prm). The tables, hardware, versions and caveats are in the PR body and the campaign wiki. Heart at ship: STALE, "release validation incomplete: no rehearsal for current source" (not RED, no override). RAL jobs: 412522 (ral CPU streaming), 412523 (A100 streaming, euclid-ral-gpu-1 exclusive), 412526 (memo after-measurement; 412521 crashed on the harness's float(None) gate read after PyAutoArray#632, fixed in 1e98f87).
Jammy2211 commentedon Oct 17, 2026
Shipped
PR #435 merged 2026-10-17: streaming-scaling phase 2. 1e8 visibilities streamed (fp64, chunk 65536): RAL CPU (8490H, 8 threads) 707.5 s wall / 3.1 s compile / 1.78 GB RSS; A100 157.3 s / 3.6 s / 1.48 GB device; RSS flat from 1e6 to 1e8; compile constant (~3 s). In-memory: CPU 10 GB cap fails at 2.5e5, uncapped 1.6e7 at 119.3 GB; A100 fails at 4e6 (46.2 GB). Parity 1.2e-9 nats at 5e5, 6.1e-8 at 4e6. Human decision 2026-10-17: interferometer campaign parked at knowledge-complete, no source changes (chunked transformer.image_from candidate not filed). Optional 2e8 rows remain research.