perf: reuse an already-correctly-capped dataset instead of re-simulating it

[Jammy2211]

Summary

Follow-up to PyAutoNerves#153 / #474, which added the SMALLDAT stamp and deliberately left this branch alone.

should_simulate's PYAUTO_SMALL_DATASETS=1 branch deleted and re-simulated every dataset unconditionally. That was correct when written, and the docstring said exactly why: it "cannot know the capped dataset on disk was produced by the SAME cap, so it always regenerates." The stamp removes that limitation, so the branch was paying for a full simulation pass per dataset per smoke run — across ~253 should_simulate call sites in autolens_workspace — for datasets that were already correct.

Reuse now requires positive evidence on both counts: stamped SMALLDAT = T and measuring exactly SMALL_DATASETS_SHAPE_NATIVE.

Why the second half is not optional

Dropping it would be worse than not doing this at all, and it is why this wasn't the one-liner it looked like.

SMALLDAT = T means "capped at whatever the cap was when this file was written", not "capped at today's cap". If SMALL_DATASETS_SHAPE_NATIVE is ever changed, every dataset already on disk goes on claiming T at the old size. Reusing on the stamp alone would then silently feed stale, wrong-sized data to a run that asked for the new cap — the same class of silent-stale-dataset bug the stamp was introduced to prevent, reintroduced through the opposite branch. Pinned by test.

This is the mirror image of _stamp_contradicted_by_shape on the full-resolution branch, and it exists for the same reason: the stamp records the writer's environment, not a measured property of the data. Neither branch may treat it as unfalsifiable.

Behaviour matrix

dataset on disk, entering a capped run	before	after
capped at the current cap — the common case	re-simulate	reuse
stamped T but at a different cap	re-simulate	re-simulate
unstamped legacy capped dataset	re-simulate	re-simulate
full resolution (SMALLDAT = F)	re-simulate	re-simulate
interferometer (n_vis, 2)	re-simulate	re-simulate
no readable top-level data.fits	re-simulate	re-simulate

Only the first row changes. Everything else keeps today's behaviour, because reuse requires positive evidence and everything else fails to provide it.

Interferometer datasets deliberately never qualify. Their data.fits is (n_visibilities, 2) — shape fixed by the committed uv file, unchanged by the cap — so shape cannot corroborate their stamp. Rather than trust the stamp alone for precisely the family whose corruption is invisible, they keep regenerating every run. That decision is written into the docstring rather than left to fall out of the code.

API Changes

None — internal changes only.

should_simulate's signature is unchanged. The only behavioural change is skipping a re-simulation that produced a byte-equivalent dataset.

Test Plan

• python -m pytest test_autoarray/ — 1106 passed, 64 skipped, 0 failed
• Green with PYAUTO_SMALL_DATASETS=1 exported and unset; tree clean both ways
• Reuse verified end-to-end against a real capped write
• Different-cap dataset stamped T is regenerated — the trap above
• Unstamped legacy, full-resolution, interferometer, and no-data.fits cases all regenerate
• The pre-existing test asserting unconditional deletion encoded the limitation being removed; rewritten, with a case per trap

Full API Changes (for automation & release notes)

Removed / Renamed / Changed Signature

None.

Added (private)

• autoarray.util.dataset_util._is_capped_at_the_current_cap(dataset_path) — True only when data.fits is stamped SMALLDAT = T and its shape is exactly SMALL_DATASETS_SHAPE_NATIVE.

Changed Behaviour

• should_simulate(dataset_path) — the PYAUTO_SMALL_DATASETS=1 branch keeps a dataset already capped to the current cap instead of deleting it. Full-resolution branch untouched.

Migration

None.

Generated by the PyAutoLabs agent workflow.

---

Generated by Claude Code