perf: retro-optimize committed markdown PNGs; decide ellipse/modeling render

[Jammy2211]

Overview

Two deferred items from the batch-2a markdown-renderings rollout (autolens_workspace#264). Investigation before planning corrected the prompt's framing on both:

PNG size. optimize_pngs() already exists in PyAutoHands/autohands/generate_markdown.py:240 and runs after every extraction (:379) — it is forward-only by construction and never touches previously committed images. So the leftover is not "add an optimize step"; it is a one-off retro pass over already-committed PNGs. pngquant is the wrong tool: reusing optimize_pngs keeps retro-optimized files byte-identical to what a future re-render produces.

Measured across the six workspaces: 405 of 429 committed markdown PNGs are still RGBA (unoptimized); the 24 P-mode ones were rendered after PR #200. Running a 25-file sample through the existing algorithm: 88.4 MB → ~23 MB (26.1%) — roughly 65 MB saved, no quality loss beyond the 256-colour quantize already shipping.

Repo	PNGs	RGBA (to fix)	markdown/
autolens_workspace	207	196	61 M
autogalaxy_workspace	122	115	22 M
HowToGalaxy	57	57	3.9 M
HowToFit	31	31	1.5 M
autofit_workspace	6	6	568 K
HowToLens	94	0	3.1 M (already done)

Caveat on the record: this shrinks the working tree and future clone-of-tip, not .git history — the old blobs stay in the pack. No history rewrite is on the table.

ellipse/modeling. autogalaxy_workspace/scripts/ellipse/modeling.py runs 22 sequential fits (2 x 10 DynestyStatic + 2 Drawer, lines 445-583), split across two notebook cells by the """ section markers — so it is two multi-hour cells, not one. There is currently no output/ellipse* in the workspace, so a render starts from zero. Rather than gamble on a bumped per-cell timeout, pre-run the script outside nbconvert so the fits land in output/, then let the render resume them near-instantly (the module docstring guarantees PyAutoFit completed-run resume). The max_minutes bump becomes a safety margin rather than a bet.

Plan

• Add a --optimize-only mode to generate_markdown.py that reuses the existing optimize_pngs() over already-committed markdown/**/*_files/, and run it across all six workspaces — one deterministic ~65 MB win.
• No dpi knob: quantize already delivers 74%, and a dpi drop would degrade every future render's readability for a smaller marginal gain.
• For ellipse/modeling.py, measure one fit first; if the extrapolated 22-fit cost is under a 2-hour ceiling, pre-run the script to populate output/, then render via the resume path. Above the ceiling, leave the page out (as the prompt permits) and replace the yaml's "tracked as a follow-up" comment with a settled decision recording the measured cost.
• Ship library-first (PyAutoHands), then the workspace commits.

Detailed implementation plan

Work Classification

Library (PyAutoHands tool change first; workspace commits follow).

Affected Repositories

• PyAutoLabs/PyAutoHands (primary)
• PyAutoLabs/autolens_workspace
• PyAutoLabs/autogalaxy_workspace
• PyAutoLabs/autofit_workspace
• PyAutoLabs/HowToGalaxy
• PyAutoLabs/HowToFit
• PyAutoLabs/HowToLens (expected near-no-op)

Branch Survey

Repository	Current Branch	Dirty?
./PyAutoHands	main	clean
./autogalaxy_workspace	main	clean
./autolens_workspace	main	clean
./autofit_workspace	main	clean
./HowToGalaxy	main	clean
./HowToFit	main	clean
./HowToLens	main	clean

worktree_list_claimed empty; worktree_check_conflict exit 0 (no conflict). Unregistered-work warning: autogalaxy_workspace has a local feature/stored-sample-reconstruction-guard branch, not checked out and not claimed — not a conflict.

Suggested branch: feature/markdown-renderings-2a-leftovers
Worktree root: ~/Code/PyAutoLabs-wt/markdown-renderings-2a-leftovers/

Implementation Steps

Phase A — PyAutoHands (library)

1. autohands/generate_markdown.py: add --optimize-only to the argparse block (:439). When set, skip load_examples / render / write_index; walk Path.cwd()/markdown for *_files/ dirs, call the existing optimize_pngs(files_dir) on each, print a per-repo before/after byte total. ~20 lines, no new dependency (Pillow already imported inside optimize_pngs).
2. Do not add a dpi knob.
3. Update the module docstring and docs/internals.md (if it documents this tool) to note the retro mode.

Phase B — PNG retro pass (workspaces, mechanical)

4. Per repo in autolens_workspace, autogalaxy_workspace, autofit_workspace, HowToGalaxy, HowToFit, HowToLens, from the workspace root: python ../PyAutoHands/autohands/generate_markdown.py <project> --optimize-only.
5. Verify: re-run the PNG mode census (expect all P), spot-check 3 rendered pages per repo, and confirm git status shows only markdown/**/*.png modified — nothing outside markdown/.
6. HowToLens is expected to be a near-no-op; if it produces zero changes, no commit there.

Phase C — ellipse/modeling (measure, then decide)

7. From autogalaxy_workspace, time a single fit in isolation (fit_0 of the Multiple-Ellipses loop, n_live=50, 2 free params) and extrapolate x22. Note this machine is WSL with a throttled nproc, so the script's number_of_cores=4 may itself be the cost driver — check that before blaming the sampler. Ceiling: 2 hours for the full 22-fit pre-run.
8. If under the ceiling: run python scripts/ellipse/modeling.py directly to populate output/, then git status and restore any tracked file the script mutated — running outside generate_markdown.py loses its tracked-file protection, and dataset/imaging/ellipse does not currently exist so the auto-simulator will fire. Then add the entry to config/build/markdown_examples.yaml with max_minutes: 360, run generate_markdown.py autogalaxy --only ellipse/modeling (resume path), and replace the exclusion comment.
9. If over the ceiling: leave the page out, and rewrite the yaml comment from "Tracked as a follow-up" to a closed decision recording the measured cost — so this does not come back a third time.

Phase D — ship

10. ship_library for PyAutoHands, then ship_workspace for the workspace commits behind the library-first merge gate. ship_workspace has a known binary-leak trap — this task is all binaries by design, so state explicitly that markdown/**/*.png is the intended payload and check nothing else rides along.

Key Files

• PyAutoHands/autohands/generate_markdown.py — optimize_pngs() at :240, called at :379; main()/argparse at :439
• autogalaxy_workspace/config/build/markdown_examples.yaml — curated list; ellipse/modeling exclusion comment
• autogalaxy_workspace/scripts/ellipse/modeling.py — 22-fit Multiple Ellipses + Masking sections, lines 445-583
• <workspace>/markdown/**/*_files/*.png — the 405 RGBA images to retro-optimize

Testing approach

No unit tests exist for generate_markdown.py. Verification is empirical: the PNG mode census before/after, byte totals per repo, git status scoped to markdown/, and visual spot-checks of rendered pages on the PR.

Original Prompt

Click to expand starting prompt

# Markdown renderings batch 2a — leftovers (ellipse/modeling + PNG size)

Type: docs
Target: workspaces
Difficulty: small
Autonomy: safe
Priority: low
Status: formalised

Two deferred items from batch 2a ([[markdown-example-renderings]] rollout,
autolens_workspace#264):

1. **autogalaxy `ellipse/modeling.py`** — excluded from the curated set because
   its "Multiple Ellipses" section runs many sequential DynestyStatic/Drawer
   fits that exceed nbconvert's per-cell timeout (CellTimeoutError at 7200s;
   confirmed a timeout, NOT a bug — corner works, imaging/modeling renders
   fine). To add it: bump its `max_minutes` in
   `autogalaxy_workspace/config/build/markdown_examples.yaml` to something
   generous (try 360+; the multi-fit cell is genuinely multi-hour) and run
   `generate_markdown.py autogalaxy --only ellipse/modeling`. If it's still
   impractical, leave it out — ellipse/simulator + ellipse/fit already showcase
   the dataset and a fit.

2. **PNG size** — batch 2a committed large image galleries (autolens markdown/
   ~61M total incl. phase-1's 19M; autogalaxy ~22M) at generate_markdown's
   default dpi. If repo size becomes a concern, do a single optimization pass
   across ALL markdown/*_files/*.png in every workspace (pngquant lossy ~60-70%
   reduction, visually fine for plots) — or add a dpi/optimize step to
   generate_markdown.py so it's consistent going forward (would also cover the
   phase-1 autolens pages). Do it repo-wide for consistency, not piecemeal.

[Jammy2211]

Correction to the detailed plan — testing approach.

The plan's "Testing approach" says "No unit tests exist for generate_markdown.py". That is wrong: tests/test_generate_markdown.py exists (317 lines) and already contains a TestOptimizePngs class covering optimize_pngs() in-place shrink and the missing-dir no-op.

Phase A therefore gains a step: extend that file with a TestOptimizeOnly class covering the new --optimize-only walk — a tmp_path workspace with a couple of markdown/<page>_files/ dirs, asserting every PNG shrinks, that nothing outside markdown/ is touched, and that an absent markdown/ dir is a clean no-op. Run with pytest tests/test_generate_markdown.py.

The empirical checks in the original plan (PNG mode census, per-repo byte totals, git status scoped to markdown/, visual spot-checks) still stand for Phase B — they verify the corpus, the unit test verifies the tool.

[Jammy2211]

Library PR Created

PyAutoHands#248 — feat(generate_markdown): --optimize-only retro pass over committed PNGs (pending-release, MERGEABLE).

API Changes

None. --optimize-only is a new opt-in flag; default behaviour is byte-for-byte unchanged. optimize_pngs() now returns (before, after) instead of None — internal helper, one in-tree caller, no downstream consumer.

Workspace impact — option (i), new content, workspace PRs follow

The workspace repos are not affected by an API change; they consume the new flag's output. Six workspaces were run through it, and the results are committed on the same branch in each repo:

Repo	Before	After	Reclaimed	PNGs changed
autolens_workspace	61.8 MB	14.9 MB	46.9 MB	196
autogalaxy_workspace	21.5 MB	5.6 MB	15.9 MB	115
HowToGalaxy	3.8 MB	1.0 MB	2.7 MB	57
HowToFit	1.1 MB	0.4 MB	0.8 MB	31
autofit_workspace	0.3 MB	0.1 MB	0.2 MB	6
HowToLens	2.7 MB	2.7 MB	0.0 MB	0
Total	91.2 MB	24.7 MB	66.5 MB	405

HowToLens reclaimed exactly zero — it was rendered post-#200 and is already fully optimized. No commit there; its branch carries nothing.

Phase C — the exclusion premise did not reproduce

The prompt recorded ellipse/modeling.py as "genuinely multi-hour" after a 7200s CellTimeoutError. Measured before acting on it: one DynestyStatic fit takes ~32s, flat in major_axis (0.3" → 32.1s, 3.6" → 32.7s), so there is no blow-up at the large end that diagnosis implied.

Post-#246 the notebook splits the work into two heavy cells (10 and 11 fits), so the per-cell timeout has to cover ~11 fits ≈ 6 min, not all 23. The full page rendered in 591s (9.9 min) end to end, covering Multiple Ellipses, Final Fit and Masking. Added at max_minutes: 120 (~20× headroom), with the measurement recorded in the yaml comment so the "multi-hour" claim is not re-derived from the old note.

Incidental fix: regenerating the index repaired three pre-existing broken links — the committed markdown/README.md pointed at markdown/multi/{start_here,simulator,modeling}.md, but those pages live under markdown/multi_dataset/. Verified the old targets do not exist and the new ones do.

Gate

• pytest tests/ in PyAutoHands: 354 passed (5 new).
• Heart at PR-open: stale (85), sole reason release validation stale: source moved since rehearsal (PyAutoGalaxy) — unrelated to this work. No RED, no YELLOW. Nothing merged; merge is held for a human.

Workspace PRs next via /ship_workspace.

[Jammy2211]

Workspace PRs Created — all six PRs open, nothing merged

Repo	PR	Contents
PyAutoHands	#248	--optimize-only flag + 5 tests
autolens_workspace	#489	196 PNGs, 61.8 → 14.9 MB
autogalaxy_workspace	#217	115 PNGs + ellipse/modeling page + index fix, 21.5 → 5.6 MB
HowToGalaxy	#69	57 PNGs, 3.8 → 1.0 MB
HowToFit	#46	31 PNGs, 1.1 → 0.4 MB
autofit_workspace	#144	6 PNGs, 0.3 → 0.1 MB

All five workspace PRs carry pending-release; all six are MERGEABLE.

HowToLens has an empty branch and no PR — it was rendered post-#200 and reclaimed exactly 0 bytes, as predicted.

Merge order (library-first gate)

PyAutoHands#248 must merge first — it adds the flag the workspace contents were produced with. The five workspace PRs follow in any order.

Not done

• Nothing is merged. The gate is satisfied for PR-open only; merging is a human act and is held.
• No completion record written and the active.md entry is not pruned — both wait on merge.
• Heart is stale (85), sole reason release validation stale: source moved since rehearsal (PyAutoGalaxy). Not GREEN, but the reason is PyAutoGalaxy release-rehearsal drift and unrelated to this work; no RED, no YELLOW. Flagging it rather than treating it as a pass.
• .git history is unchanged — the 88 MB of original blobs remain in every pack. The saving is to the working tree and future clone-of-tip only.

[Jammy2211]

Shipped — all six PRs merged 2026-08-20

Repo	PR	Merge commit
PyAutoHands	#248	cdea28c
autolens_workspace	#489	d45cdca5
autogalaxy_workspace	#217	c91a0e5
autofit_workspace	#144	1c3df68
HowToGalaxy	#69	4e1e6b4
HowToFit	#46	8e2ad02

Merged library-first (#248 before the workspace PRs). All 35 CI legs were green beforehand.

Verified on main after merge

Repo	markdown/	PNG modes
autolens_workspace	14.9 MB	207 × P
autogalaxy_workspace	5.8 MB	129 × P (122 + 7 new ellipse)
HowToLens	2.7 MB	94 × P
HowToGalaxy	1.0 MB	57 × P
HowToFit	0.4 MB	31 × P
autofit_workspace	0.1 MB	6 × P
Total	24.9 MB (was 91.2 MB)	66.3 MB reclaimed

Both prompt premises were wrong

• "Add a dpi/optimize step" — optimize_pngs() already existed (feat: markdown generator per-cell stdout truncation + PNG optimization #200); being forward-only was the actual gap. Fixed with a retro pass reusing the same function, not the suggested pngquant sweep, so retro bytes match what a future re-render emits.
• "Genuinely multi-hour" — one fit is ~32 s, flat in major_axis (0.3″ → 32.1 s, 3.6″ → 32.7 s); the whole page rendered in 591 s. Added at max_minutes: 120 with the measurement recorded in the yaml comment so the false claim can't be re-derived.

Incidental: three pre-existing broken index links repaired (markdown/multi/… → markdown/multi_dataset/…).

Standing caveat

.git history is unchanged — the ~88 MB of original blobs remain in every pack. The saving is to the working tree and future clone-of-tip only.

Completion record: PyAutoMind/complete/2026/08/markdown-renderings-2a-leftovers.md.