Repository navigation
fix: write dataset.fits once, always, to image/ (drop fits_dataset knob) #608
Description
Overview
Since the 2026-07-08 aggregator fix (PyAutoGalaxy #479, PyAutoLens #574, issue #478) every imaging and interferometer search writes dataset.fits twice with identical contents: files/dataset.fits unconditionally from save_attributes via paths.save_fits, and image/dataset.fits from the plotter interface gated on visualize/plots.yaml dataset.fits_dataset. Spotted in a Euclid vis_lp output folder; it is generic library behaviour. Decision: one unconditional write, in image/, and drop the fits_dataset config knob.
Safe because autofit.aggregator.SearchOutput._outputs_by_suffix already scans both files/ and image/ for .fits, so fit.value(name="dataset") and every *Agg loader keep working, and autolens_workspace/scripts/guides/results/start_here.py already reads image/dataset.fits.
Plan
- Move the unconditional
dataset.fitswrite in eachsave_attributes(Galaxy + Lens imaging, Galaxy + Lens interferometer) frompaths.save_fits(→files/) to a direct write intopaths.image_path. - Delete the
fits_dataset-gated plotter writes in the imaging and interferometer plotters. - Remove the
fits_datasetkey and its comment from the packagedplots.yamlin PyAutoGalaxy and PyAutoLens. - Update the
save_attributesregression tests to assert theimage/path; drop the plotter tests'dataset.fitsassertions. - Workspace follow-up (library-first gate): remove
fits_datasetfromautolens_workspace/autogalaxy_workspaceconfig/visualize/plots.yamland any prose mentioning it; regenerate affected notebooks. - No PyAutoFit change (repo is claimed by two other tasks). Known trade-off:
DatabasePaths.save_fitsno longer receivesdataset— the pre-July behaviour, not a new gap; an image-folder-awaresave_fitsin PyAutoFit can follow if the DB route ever needs it.
Detailed implementation plan
Affected Repositories
- PyAutoGalaxy (primary)
- PyAutoLens
- autolens_workspace (follow-up)
- autogalaxy_workspace (follow-up)
Branch Survey
| Repository | Current Branch | Dirty? |
|---|---|---|
| ./PyAutoGalaxy | main | clean |
| ./PyAutoLens | main | clean |
| ./autolens_workspace | main | clean |
| ./autogalaxy_workspace | main | clean |
| ./PyAutoFit | main | clean (claimed by two other tasks; deliberately not touched) |
Suggested branch: feature/dataset-fits-image-only
Implementation Steps
- PyAutoGalaxy
autogalaxy/imaging/model/analysis.pysave_attributes: keep building the HDU list viahdu_list_for_output_from, replacepaths.save_fits(name="dataset", ...)withhdu_list.writeto(paths.image_path / "dataset.fits", overwrite=True); rewrite the comment (files→image; no mention of a gated plotter write). - PyAutoGalaxy
autogalaxy/interferometer/model/analysis.pysave_attributes: same. - PyAutoGalaxy
autogalaxy/imaging/model/plotter.pyimaging(): delete theshould_plot("fits_dataset")block; drop thehdu_list_for_output_fromimport if unused. Same inautogalaxy/interferometer/model/plotter.py. Leaveautogalaxy/ellipse/model/plotter.py(already a single unconditionalimage/write). - PyAutoGalaxy
autogalaxy/config/visualize/plots.yaml: removedataset.fits_datasetand its comment line; keepfits_adapt_images. - PyAutoGalaxy tests:
test_autogalaxy/analysis/analysis/test_analysis.py::test__save_attributes__dataset_fits_output_for_aggregatorassertspaths.image_path / "dataset.fits";test_autogalaxy/imaging/model/test_plotter_imaging.py::test__imagingandtest_autogalaxy/interferometer/model/test_plotter_interferometer.pydrop thedataset.fitsread; add an interferometersave_attributestest mirroring the imaging one if none exists. - PyAutoLens: same rewrite in
autolens/imaging/model/analysis.pyandautolens/interferometer/model/analysis.pysave_attributes; removefits_datasetfromautolens/config/visualize/plots.yaml;test_autolens/analysis/analysis/test_analysis_dataset.py::test__save_attributes__dataset_fits_output_for_aggregatorasserts theimage/path; grepautolens/for any lens-sidefits_datasetplotter write ortest_plotter_*assertingdataset.fitsand treat the same way. - Workspaces (after the library PRs): remove
fits_datasetkey + comment fromautolens_workspace/config/visualize/plots.yamlandautogalaxy_workspace/config/visualize/plots.yaml; grepscripts/forfits_datasetprose and update; regenerate affected notebooks per each workspace's convention.
Key Files
PyAutoGalaxy/autogalaxy/imaging/model/analysis.py—save_attributeswriterPyAutoGalaxy/autogalaxy/interferometer/model/analysis.py—save_attributeswriterPyAutoGalaxy/autogalaxy/imaging/model/plotter.py,.../interferometer/model/plotter.py— gated plotter writes to deletePyAutoGalaxy/autogalaxy/config/visualize/plots.yaml,PyAutoLens/autolens/config/visualize/plots.yaml—fits_datasetkeyPyAutoLens/autolens/imaging/model/analysis.py,.../interferometer/model/analysis.py—save_attributeswritersPyAutoFit/autofit/aggregator/search_output.py—_outputs_by_suffix(read-only: confirmsimage/is scanned)
Verification
- Unit tests above green in both library worktrees.
- End-to-end: run
autolens_workspace/scripts/imaging/modeling/start_here.pyunder the smoke profile from the worktree; the search output has exactly onedataset.fits, underimage/, none underfiles/. - Aggregator round-trip:
af.Aggregator.from_directory(...)on that output, thenal.agg.ImagingAgg(agg).dataset_gen_from()yields the dataset.
Original Prompt
Click to expand starting prompt
dataset.fits written twice per search (files/ and image/): write once, always, to image/
Type: bug
Target: autogalaxy
Repos:
- PyAutoGalaxy
- PyAutoLens
- autolens_workspace
- autogalaxy_workspace
Themes: - output
- aggregator
- hygiene
Difficulty: small
Autonomy: safe
Priority: medium
Status: draft
Consequence: judge
Review-minutes: 15
Unattended: ready
Filed: 2026-09-08
Since PyAutoGalaxy#479 / PyAutoLens#574 (2026-07-08, fix for #478) every imaging and
interferometer search writes dataset.fits twice: AnalysisImaging.save_attributes /
AnalysisInterferometer.save_attributes write files/dataset.fits unconditionally via
paths.save_fits, and the plotter interface writes an identical image/dataset.fits
gated on visualize/plots.yaml dataset.fits_dataset. Spotted in a
euclid_strong_lens_modeling_pipeline vis_lp output folder; it is generic library
behaviour, not a pipeline one.
User request (verbatim): "I do not want duplication and I want dataset.fits to always be in
image, so I guess the unconditional writing should just put it there? To be honest it sounds
like we pretty much dont allow a user to not output the dataset.fits, which is fine but yeah
avoid duplication."
Facts
autofit.aggregator.SearchOutput._outputs_by_suffixalready scans bothfiles/and
image/for.fits, sofit.value(name="dataset")resolvesimage/dataset.fitswith
no aggregator change.autolens_workspace/scripts/guides/results/start_here.pyalready documents and reads
image/dataset.fits—image/is the user-facing location.- The ellipse plotter writes
image/dataset.fitsunconditionally and has nofiles/
twin; it is not duplicated today. DatabasePaths.save_fitsstores fits in the database. Writing directly to
paths.image_path(as the plotters already do) means the database route no longer
receivesdataset— which is the pre-July behaviour, not a new gap.
Deliverable
- Single unconditional
image/dataset.fitswrite insave_attributes(Galaxy + Lens
imaging; Lens interferometer; Galaxy interferometer if it has the same pair). - Remove the
fits_dataset-gated plotter writes and thefits_datasetkey from the
packagedplots.yaml(PyAutoGalaxy, PyAutoLens) and the workspace copies. - Update the two
save_attributesregression tests to assertimage/dataset.fits, and
the plotter tests that assert the gated write.
Activity
Jammy2211 commentedon Sep 8, 2026
Plan addendum (human, 2026-09-08): since
dataset.fitsis now always written byAnalysisImaging/AnalysisInterferometer,save_attributesshould skip the write whenimage/dataset.fitsalready exists (e.g. a resumed search), to save the time of rewriting it. Implement as an existence check beforewriteto; the regression tests should cover both the fresh write and the skip.Jammy2211 commentedon Sep 8, 2026
Library PR Created
Both
pending-release. Tests: PyAutoGalaxy 1182 passed; PyAutoLens 624 passed, 1 xfailed. E2E: a real fit writes exactly onedataset.fits, underimage/, none underfiles/;af.Aggregator.from_directory+ImagingAgg(...).dataset_gen_from()round-trips it.API-change summary:
visualize/plots.yamldataset.fits_datasetis removed (the file is always written, once, toimage/);dataset.fitsis no longer written tofiles/. The aggregator needs no change (it already scansimage/).Workspace impact: option (ii) parity — the key is dead, not breaking. Copies live in 9 downstream repos. This task removes it from
autolens_workspace,autogalaxy_workspace,autolens_workspace_test(5 files, 2 CRLF) andautogalaxy_workspace_test(4 files). The other five copies (euclid pipeline, both assistants, HowToLens, HowToGalaxy) are filed as a low-priority Mind hygiene draft. Next:/start_workspace→/ship_workspace.Jammy2211 commentedon Sep 8, 2026
Smoke test results (workspace ship)
Run with the task worktree stack active (
source $WT/activate.sh) viapyauto-heart smoke autogalaxy autolens autolens_test, after the config-onlyworkspace commits were pushed on
feature/dataset-fits-image-only.autogalaxy_workspaceautolens_workspaceautolens_workspace_testCounts are scripts + notebooks: autogalaxy 14 scripts + 2 notebooks, autolens
37 scripts + 2 notebooks, autolens_test 29 scripts (no notebook leg).
autogalaxy_workspace_testhas no Heart smoke key, so it is not covered here;its change is the same config-only edit.
No failures. Notably the two scripts named in the current Heart YELLOW
reasons both passed on this branch:
autolens_workspace/scripts/multi_dataset/modeling.py— PASS (16.4s)autolens_workspace_test/scripts/imaging/jax_likelihood/delaunay.py— PASS (41.0s)Every script that reads
config/visualize/plots.yamlran clean with thedataset.fits_datasetkey removed, confirming nothing in the workspaces stilldepends on it.
Failure tracebacks
None — the run exited 0 with zero failures across all three workspaces.
Jammy2211 commentedon Sep 8, 2026
/prm: five PRs green on every leg; autogalaxy_workspace_test#120 red on both smoke legs —
scripts/imaging/visualization/visualization.pyassertsPlotterImaging.imaging()writesdataset.fits(run 34258035876,FileNotFoundError: scripts/imaging/images/visualization/dataset.fits). Workspace CI clones the library chain on the same-named branch, so it saw the new plotter. The autolens_workspace_test imaging + interferometer twins assert the same and are only green because they are disabled in that repo's smoke list. Missed by the impact grep (fits_dataset, notdataset.fits). Fixing all three on the task branches; nothing merged yet.Jammy2211 commentedon Sep 8, 2026
Shipped
dataset.fitsis now written once per search, always, to the search'simage/folder byAnalysisImaging.save_attributes/AnalysisInterferometer.save_attributes(skipped when it already exists, so a resumed search does not rewrite it). Thefits_dataset-gated plotter writes and thefits_datasetkey inconfig/visualize/plots.yamlare removed. The aggregator is unchanged (it already scansimage/).Merged:
dataset.fits)Verification: PyAutoGalaxy 1182 passed, PyAutoLens 624 passed (1 xfailed); E2E fit → exactly one
dataset.fitsunderimage/, aggregator round-trip OK;pyauto-heart smoke autogalaxy autolens autolens_test84/84; all PR CI legs green.Not done here: five more
plots.yamlcopies (euclid pipeline, both assistants, HowToLens, HowToGalaxy) still carry the dead key — harmless, filed as a Mind hygiene draft. Known trade-off: the database route no longer receivesdatasetfromsave_attributes(pre-#479 behaviour).