diff --git a/complete/2026/08/howto-setup-notebook-audit.md b/complete/2026/08/howto-setup-notebook-audit.md new file mode 100644 index 00000000..d64b39bf --- /dev/null +++ b/complete/2026/08/howto-setup-notebook-audit.md @@ -0,0 +1,134 @@ +- issue: (none — run end-to-end from a cloud session; the six PRs are the record) +- completed: 2026-08-18 +- prs: HowToFit#45, HowToGalaxy#66, HowToLens#70, autofit_workspace#138, + autogalaxy_workspace#211, autolens_workspace#485 — all squash-merged 2026-08-18, + every check green on each. +- merge-commits: HowToFit `1a4e7132`, HowToGalaxy `d973bc57`, HowToLens `e0fbdf61`, + autofit_workspace `fa55c438`, autogalaxy_workspace `47227c79`, autolens_workspace `c5960f69`. +- summary: Full audit of the `# from auto* import setup_notebook; setup_notebook()` boilerplate + across every notebook-generating repo. 39 scripts were missing it — HowToFit 3, HowToGalaxy 2, + HowToLens 6, autofit_workspace 1, autogalaxy_workspace 5, autolens_workspace 22 — and all 39 + were fixed with their generated notebooks updated to the uncommented form PyAutoHands emits. + autocti_workspace was already clean (79 scripts, 0 missing). The line chdir's to the workspace + root and enables inline plotting; without it a notebook that loads data by a relative path + fails under nbconvert, which runs with CWD set to the notebook's own directory. +- the only live breakage: `HowToLens/scripts/simulator/lens_x{2,3}.py`, which resolve + `dataset_path = Path("dataset", ...)` against CWD and write FITS output there. Everything else + in the sweep is prose-only or import-only and passed today for exactly the reason the original + report predicted — they never load data by relative path. +- the omissions were not policy: they sit alongside siblings in the SAME directory that do carry + the line (`autolens_workspace/scripts/cluster/` has 6 without and 5 with). Anyone re-auditing + should not read a missing line as a deliberate per-subtree exemption. +- placement rule (two forms, both in the tree already): after the module docstring, OR after the + `from auto* import jax_wrapper` line where a script has one — jax_wrapper must set the JAX + environment before anything else. One outlier, + `autolens_workspace/scripts/guides/units/mass_to_light_ratio_units.py`, opens on imports rather + than a docstring; the line went at the top of the file, still ahead of every import. +- SCOPE BOUNDARY worth keeping: `autoreduce_workspace` has 30 scripts and NONE carry the line, + which looks alarming until you notice it has no `notebooks/` directory — nothing generates + notebooks there, so the convention does not apply yet. Same for every `*_workspace_test` and + `*_workspace_developer` repo (verified by tree listing, not assumption). If autoreduce_workspace + ever starts generating notebooks, those 30 scripts need this sweep. +- notebooks were NOT regenerated by PyAutoHands: it is not available in a cloud session, so the + `.ipynb` files were patched directly to the exact shape the generator emits. Verified three ways + before pushing — the edit reproduces already-correct sibling pairs byte-for-byte, + `json.dumps(nb, indent=1)` round-trips every touched file unchanged, and every `.py` still + parses. A real `generate.py` run should be a no-op on these; if it is not, that is the thing to + look at first. +- CRLF trap: `HowToFit/scripts/chapter_1_introduction/tutorial_1_models.py` is committed with CRLF + line endings while everything around it is LF. A naive read/split/write rewrote all 547 lines as + a whitespace-only diff. The fix preserves the file's existing terminator. Anything doing a + mechanical sweep across these repos will hit this file. +- pre-existing drift found, deliberately NOT touched: nine `start_here.ipynb` files in + autolens_workspace and six in autogalaxy_workspace are stored with a JSON formatting that + differs from the generator's `indent=1` output. Flagged in both PR bodies. Unrelated to this + task, real, and unclaimed. +- no pending-release gate applied to the three workspace legs: `setup_notebook` is long-shipped in + the released libraries and is already called by every other example script in those repos, so + there is no unreleased API for the library-first merge gate to guard. +- adjacent but distinct: `draft/maintenance/workspaces/notebook_setup_notebook_drift_siblings.md` + covers notebooks that carry the COMMENTED form and need regenerating. Different failure, still + open, untouched here. +- `HowToGalaxy/chapter_1_introduction/tutorial_4_methods.py`, named in the original report as + missing, already carried the line — fixed before this audit ran. + +## Original prompt + +# Audit HowTo tutorials for missing setup_notebook() line + +Type: bug +Target: HowToFit +Repos: +- @HowToFit +- @HowToGalaxy +- @HowToLens +- @autofit_workspace +- @autogalaxy_workspace +- @autolens_workspace +Difficulty: small +Autonomy: safe +Priority: normal +Status: formalised + +Discovered during the batch-2b markdown rollout ([[markdown-example-renderings]]). +Several HowTo tutorials are missing the standard +`# from autoconf import setup_notebook; setup_notebook()` line that every other +workspace/tutorial script carries (it chdir's to the workspace root + enables +inline plotting). Without it, a tutorial that loads data via a relative path +(or runs a simulator subprocess) FAILS when executed by nbconvert (which runs +with CWD = the notebook's own directory), and only works interactively if the +user happens to launch jupyter from the repo root. + +Confirmed missing in chapter_1_introduction alone: HowToFit start_here.py + +tutorial_1_models.py; HowToGalaxy tutorial_4_methods.py; (HowToGalaxy +tutorial_3_fitting.py + HowToLens tutorial_7_fitting.py were fixed in batch 2b's +PRs since they blocked rendering). The 3 still-missing pass today only because +they don't load data by relative path. + +Fix: audit ALL chapters of all three HowTo repos (and re-check the workspaces) +for scripts lacking the setup_notebook line; add it right after the opening +docstring, matching the sibling convention; regenerate the affected notebooks. +Low-risk boilerplate consistency fix. + +## Audit result (2026-08-18) + +Every `.py` under `scripts/` — plus each HowTo repo's root `start_here.py` — was checked in the +three HowTo repos and all five user-facing workspaces. 39 scripts were missing the line: + +| Repo | Missing | Notes | +|---|---|---| +| HowToFit | 3 | `chapter_1_introduction/{start_here,tutorial_1_models}.py`, root `start_here.py` | +| HowToGalaxy | 2 | `chapter_2_modeling/tutorial_8_need_for_speed.py`, root `start_here.py` | +| HowToLens | 6 | `chapter_2_lens_modeling/{tutorial_8_need_for_speed,tutorial_11_slam}.py`, `chapter_3_pixelizations/tutorial_9_model_fit.py`, `simulator/{lens_x2,lens_x3}.py`, root `start_here.py` | +| autofit_workspace | 1 | `overview/overview_3_statistical_methods.py` | +| autogalaxy_workspace | 5 | under `guides/`, `imaging/data_preparation/`, `interferometer/` | +| autolens_workspace | 22 | mostly `cluster/*`, `guides/*` and feature `simulator.py` scripts | +| autocti_workspace | 0 | clean | + +`tutorial_4_methods.py` in HowToGalaxy, named in the original report, already carries the line — +it was fixed before this audit ran. + +The omissions are not a deliberate per-subtree policy: they sit alongside siblings in the same +directory that do carry the line (e.g. `autolens_workspace/scripts/cluster/` has 6 without and 5 +with). + +The two genuinely load-bearing cases are `HowToLens/scripts/simulator/lens_x{2,3}.py`, which write +to a relative `dataset/` path; the rest are prose-only or import-only scripts that pass today for +the reason the report gives. + +**Out of scope, deliberately:** `autoreduce_workspace` has 30 scripts and none carry the line, but +it has no `notebooks/` directory — the convention does not apply there yet. Same for every +`*_workspace_test` and `*_workspace_developer` repo (checked by tree listing; none generates +notebooks). + +**Placement:** after the module docstring, or after the `from auto* import jax_wrapper` line where +a script has one. One outlier — `autolens_workspace/scripts/guides/units/mass_to_light_ratio_units.py` +opens on imports rather than a docstring, so the line went at the top of the file. + +**Notebooks:** patched by hand to the exact shape PyAutoHands emits (uncommented, same cell, +`json.dumps(nb, indent=1)` round-trips byte-identically), because the generator is not available in +a cloud session. A real `generate.py` run should be a no-op on these — confirm before merging. + +Branch `claude/howto-setup-notebook-audit-dm2j9e` in all six repos, one commit each: +HowToFit#45, HowToGalaxy#66, HowToLens#70, autofit_workspace#138, autogalaxy_workspace#211, +autolens_workspace#485. diff --git a/complete/index.md b/complete/index.md index 3c51183d..a0a54eb9 100644 --- a/complete/index.md +++ b/complete/index.md @@ -6,7 +6,7 @@ Token-light navigation over the finished-work records (schema: only then grep a dated bucket. Curators: edit the band between the CURATED markers; everything below GENERATED is rebuilt. -1005 records across 7 buckets. +1006 records across 7 buckets. ## Highlights @@ -59,6 +59,7 @@ _(curate hard-won records here — survives regeneration.)_ - [hazard-profiling-likelihood-tier](2026/08/hazard-profiling-likelihood-tier.md) — Tier-2 likelihood profiling landed with five persistent findings and corrected NNLS continuity semantics. - [health-conductor-stale-verdict](2026/08/health-conductor-stale-verdict.md) - [heart-red-guarded-sample-escape](2026/08/heart-red-guarded-sample-escape.md) — Closed the escape path that kept release validation RED after +- [howto-setup-notebook-audit](2026/08/howto-setup-notebook-audit.md) — none — run end-to-end from a cloud session; the six PRs are the record - [howto-smoke-all-tutorials](2026/08/howto-smoke-all-tutorials.md) - [howtofit-stub-fork-cleanup](2026/08/howtofit-stub-fork-cleanup.md) - [hpc-sync-first-push-race](2026/08/hpc-sync-first-push-race.md) — `hpc/sync push()` fired the `CODE_DIRS` rsyncs in parallel before diff --git a/dashboard.md b/dashboard.md index 882b4d9f..5a67e707 100644 --- a/dashboard.md +++ b/dashboard.md @@ -11,7 +11,7 @@ Tasks only — the organism's health lives with the Heart (`/health`), not here. | [In flight](#in-flight) (`active/`) | 4 | | [Parked](#parked) (`parked.md`) | 1 | | [Planned](#planned) (`planned.md`) | 7 | -| [Backlog](#backlog) (`draft/`) | 139 | +| [Backlog](#backlog) (`draft/`) | 138 | Live on GitHub: [open issues](https://github.com/search?q=org%3APyAutoLabs+is%3Aissue+is%3Aopen&type=issues) · [open pull requests](https://github.com/search?q=org%3APyAutoLabs+is%3Apr+is%3Aopen&type=prs) @@ -34,7 +34,6 @@ Live on GitHub: [open issues](https://github.com/search?q=org%3APyAutoLabs+is%3A **Quick wins** (small enough, and safe enough to run unattended) -- [Audit HowTo tutorials for missing setup_notebook() line](draft/bug/howto/missing_setup_notebook_audit.md) — howto · small · safe · normal - [PyAutoFit CLI-noise batch: unclosed search.log handler + four small warning](draft/maintenance/pyautofit/cli_noise_pyautofit_batch.md) — pyautofit · small · safe · normal - [Tenant firewall: release_run.py carries an unlisted 'PyAutoLabs' instance fact](draft/bug/pyautoheart/tenant_firewall_release_run_instance_fact.md) — pyautoheart · small · safe · normal - [Silence the three autonerves-rooted CLI-noise sources (fits leak, pytest collection,](draft/maintenance/pyautonerves/cli_noise_autonerves_batch.md) — pyautonerves · small · safe · normal @@ -83,10 +82,10 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned. ## Backlog -**139** filed prompts, not started. Each section is sorted most-pickable first (priority, then size). +**138** filed prompts, not started. Each section is sorted most-pickable first (priority, then size).
-bug — 40 +bug — 39 - [pre_build stages untracked files, publishing uncommitted human work](draft/bug/pyautohands/pre_build_stages_untracked_wip.md) — pyautohands · small · supervised · high - [UniformPrior bounds are not enforced in the objective on the](draft/bug/autofit/uniform_prior_bounds_unenforced_on_numpy_path.md) — autofit · medium · human-required · high @@ -109,7 +108,6 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned. - [TEST_MODE bypass crashes on ordered-parameter assertion ties](draft/bug/autofit/test_mode_bypass_ordered_assertion_ties.md) — autofit · small · supervised · normal - [point.py JAX-vmap parity assert is non-deterministic under the smoke env](draft/bug/autolens/point_jax_vmap_parity_nondeterministic.md) — autolens · small · supervised · normal - [Scripts derive geometry from a hardcoded pixel_scale while the dataset](draft/bug/autolens_workspace/script_local_pixel_scale_vs_dataset_pixel_scales.md) — autolens_workspace · small · supervised · normal -- [Audit HowTo tutorials for missing setup_notebook() line](draft/bug/howto/missing_setup_notebook_audit.md) — howto · small · safe · normal - [HowToGalaxy small API drifts: ellipse kwargs + plot_grid_lines (parked NEEDS_FIX)](draft/bug/howtogalaxy/small_api_drift_ellipse_and_plot_grid_lines.md) — howtogalaxy · small · supervised · normal - [`generate.py` deletes notebooks/ before rejecting an unknown project](draft/bug/pyautohands/generate_rejects_autocti_after_deleting_notebooks.md) — pyautohands · small · supervised · normal - [Tenant firewall: release_run.py carries an unlisted 'PyAutoLabs' instance fact](draft/bug/pyautoheart/tenant_firewall_release_run_instance_fact.md) — pyautoheart · small · safe · normal diff --git a/draft/bug/howto/missing_setup_notebook_audit.md b/draft/bug/howto/missing_setup_notebook_audit.md deleted file mode 100644 index d16b5165..00000000 --- a/draft/bug/howto/missing_setup_notebook_audit.md +++ /dev/null @@ -1,28 +0,0 @@ -# Audit HowTo tutorials for missing setup_notebook() line - -Type: bug -Target: HowToFit -Difficulty: small -Autonomy: safe -Priority: normal -Status: formalised - -Discovered during the batch-2b markdown rollout ([[markdown-example-renderings]]). -Several HowTo tutorials are missing the standard -`# from autoconf import setup_notebook; setup_notebook()` line that every other -workspace/tutorial script carries (it chdir's to the workspace root + enables -inline plotting). Without it, a tutorial that loads data via a relative path -(or runs a simulator subprocess) FAILS when executed by nbconvert (which runs -with CWD = the notebook's own directory), and only works interactively if the -user happens to launch jupyter from the repo root. - -Confirmed missing in chapter_1_introduction alone: HowToFit start_here.py + -tutorial_1_models.py; HowToGalaxy tutorial_4_methods.py; (HowToGalaxy -tutorial_3_fitting.py + HowToLens tutorial_7_fitting.py were fixed in batch 2b's -PRs since they blocked rendering). The 3 still-missing pass today only because -they don't load data by relative path. - -Fix: audit ALL chapters of all three HowTo repos (and re-check the workspaces) -for scripts lacking the setup_notebook line; add it right after the opening -docstring, matching the sibling convention; regenerate the affected notebooks. -Low-risk boilerplate consistency fix.