Skip to content

chore: clear the Nerves board's 79 workspace orphan keys (library-first) #441

Closed

Description

@Jammy2211

Overview

The Nerves board (PyAutoNerves#175) marks 14 workspace config files YELLOW for 79 "orphan" keys. An orphan is a key a workspace sets that no library's default YAML defines, which is not the same as a key nothing reads. A per-key classification, with the reading code checked, splits them into DELETE 51 / KEEP-WS 7 / PROMOTE 5 / FIX 16. The approved approach is library-first: fix the board (directory/file equivalence plus an autonerves allow-list), promote the 5 keys into PyAutoGalaxy, then delete the dead workspace keys. Target: 0 orphans.

Plan

  • PyAutoNerves scripts/board.py: when a library ships X.yaml and the workspace uses an X/ directory (or the reverse), compare section by section, the same way autonerves merges them. This clears the 13 CTI false positives. Add a NERVES_OWNED allow-list for the version.* keys that autonerves itself reads, which clears the 7 KEEP-WS keys.
  • PyAutoGalaxy config: add the dPIE ra/rs notation label and label_format entries, and add areas_factor: Constant 0.5 to priors/mesh/delaunay.yaml. The second change fixes "areas_factor Prior Missing" for af.Model(al.mesh.Delaunay). Unit tests are included.
  • Ship the library half, then the workspaces, one PR per repo:
    • Delete the 51 dead keys and the top-level subplot_shape_to_figsize_factor.
    • Fix fit_imaging {}: in autogalaxy_workspace.
    • Delete the 5 now-promoted keys from autolens_workspace.
  • Witness: a local python3 scripts/board.py --json reports 0 orphan files, and each workspace's start_here/smoke scripts still load their config.
Detailed implementation plan

Affected Repositories

  • PyAutoNerves (library: board)
  • PyAutoGalaxy (library: config promotion)
  • autofit_workspace, autogalaxy_workspace, autolens_workspace, autocti_workspace (workspace deletes)

Branch Survey

Repository Current Branch Dirty?
./organs/PyAutoNerves main clean
./galaxy/PyAutoGalaxy main clean
./fit/autofit_workspace main clean
./galaxy/autogalaxy_workspace main clean
./lens/autolens_workspace main clean
./cti/autocti_workspace main clean

Suggested branch: feature/workspace-config-cleanup, worktree ~/Code/PyAutoLabs-wt/workspace-config-cleanup/.

Parallel claim: PyAutoNerves is also claimed by eyes-organ-order (PyAutoMind#439). Its Nerves diff is only AGENTS.md, so it does not overlap with scripts/board.py. The human approved a parallel worktree on 2026-09-27.

Implementation Steps

  1. PyAutoNerves scripts/board.py:
    • In build_snapshot/diff_against_stack, handle file↔directory equivalence: a library X.yaml section <f> corresponds to the workspace file X/<f>.yaml, and the reverse. The semantics mirror autonerves RecursiveConfig.
    • Add the NERVES_OWNED allow-list for general.yaml version.{workspace_version, minimum_library_version, workspace_version_check, python_version_check}. These are excluded from orphans and noted "owned by autonerves".
    • Add tests: the equivalence clears a defined key, an owned key is not flagged, and an unknown key is still flagged.
  2. PyAutoGalaxy:
    • autogalaxy/config/notation.yaml: add label.ra/rs and label_format.ra/rs, using the autolens_workspace values.
    • autogalaxy/config/priors/mesh/delaunay.yaml: add areas_factor as a Constant 0.5, which matches the Delaunay(areas_factor=0.5) default.
    • Add unit tests: a Delaunay model has no missing prior, and the dPIE ra label is not the first-letter fallback.
  3. Ship the library half (/ship_library).
  4. Workspace edits:
    • Delete the DELETE keys, the top-level subplot_shape_to_figsize_factor (it equals the library default of mat_plot.figure.*) and the PROMOTE keys in autolens_workspace.
    • Fix fit_imaging: {}.
    • Leave the CTI and version keys alone.
    • Before each delete, re-grep the workspace's scripts and notebooks. Mirror a change into notebooks or markdown only where they carry config snippets. Remove any YAML comment that describes a deleted key.
  5. Ship the workspaces (/ship_workspace), then /prm, then dispatch nerves_board.yml and re-check the published board.

Per-key verdicts (79)

File Key(s) Verdict Reason
autofit_workspace/config/general.yaml hpc.live_visual_update, hpc.quick_update_background DELETE PyAutoFit reads updates.* (abstract_search.py:240-251), not hpc.*
autofit_workspace/config/general.yaml test.check_preloads, test.preloads_check_threshold DELETE preloads check removed; nothing reads them
autofit_workspace/config/general.yaml version.minimum_library_version, version.workspace_version_check KEEP-WS read by autonerves (workspace.py / init.py); board allow-list
autofit_workspace/config/visualize/plots_search.yaml mle.corner_cornerpy DELETE not read
autogalaxy_workspace/config/general.yaml hpc.live_visual_update, hpc.quick_update_background DELETE read from updates.*
autogalaxy_workspace/config/general.yaml test.disable_positions_lh_inversion_check DELETE only PyAutoLens reads it
autogalaxy_workspace/config/general.yaml version.minimum_library_version, version.workspace_version_check KEEP-WS autonerves; allow-list
autogalaxy_workspace/config/notation.yaml label.contribution_factor, label_format.contribution_factor DELETE param no longer exists
autogalaxy_workspace/config/visualize/general.yaml subplot_shape.{1,2,4,6,9,12,16,20,36,49,64,81,100} (13) DELETE table not read
autogalaxy_workspace/config/visualize/general.yaml subplot_shape_to_figsize_factor FIX (delete) top level is never read; PyAutoArray reads mat_plot.figure.*, which equals the library default
autogalaxy_workspace/config/visualize/plots.yaml:30 fit_imaging {}: FIX malformed key → fit_imaging: {}
autolens_workspace/config/general.yaml hpc.live_visual_update, hpc.quick_update_background DELETE read from updates.*
autolens_workspace/config/general.yaml version.minimum_library_version, version.python_version_check, version.workspace_version_check KEEP-WS autonerves; allow-list
autolens_workspace/config/notation.yaml label.contribution_factor, label_format.contribution_factor, label.sigma_scale DELETE params no longer exist
autolens_workspace/config/notation.yaml superscript.hyperbackgroundnoise, hypergalaxy, hyperimagesky DELETE hyper classes removed
autolens_workspace/config/notation.yaml label.ra, label.rs, label_format.ra, label_format.rs PROMOTE dPIE params read by the library; add to PyAutoGalaxy notation.yaml, then delete here
autolens_workspace/config/priors/mesh/delaunay.yaml Delaunay.areas_factor PROMOTE the PyAutoGalaxy prior file is a bare Delaunay:, so "areas_factor Prior Missing"
autolens_workspace/config/visualize/general.yaml subplot_shape.* (13) DELETE not read
autolens_workspace/config/visualize/general.yaml subplot_shape_to_figsize_factor FIX (delete) as above
autocti_workspace/config/general.yaml analysis.n_cores, analysis.preload_attempts, profiling.should_profile, test.check_preloads, test.disable_positions_lh_inversion_check, test.preloads_check_threshold DELETE not read by PyAutoCTI
autocti_workspace/config/visualize/general.yaml general.subplot_ascending_fpr, general.symmetric_cmap_value FIX (board) false positive: PyAutoCTI ships a single visualize.yaml and autonerves merges the directory form
autocti_workspace/config/visualize/plots.yaml combined_only, dataset.{data,data_binned,data_logy,fpr_non_uniformity,subplot_dataset_regions}, fit.{data_logy,fits_fit,residual_map_logy,subplot_fit_regions}, subplot_format (11) FIX (board) same false positive
autocti_workspace/config/visualize/plots_search.yaml mle.corner_cornerpy DELETE not read

Totals: DELETE 51, KEEP-WS 7, PROMOTE 5, FIX 16 (13 CTI board false positives, 2 × subplot_shape_to_figsize_factor, 1 fit_imaging typo).

Out of scope: library-side unused keys (human-review-gated, NOT removed here)

  • PyAutoLens:
    • config/non_linear.yaml: the dead 57-key nest.DynestyStatic / nest.DynestyDynamic block.
    • general.yaml: output.fit_dill.
  • PyAutoFit:
    • general.yaml: output.log_level, output.log_to_file, output.log_file, output.search_internal and profiling.repeats. These may be read dynamically.
    • non_linear/GridSearch.yaml: parallel, parallel.number_of_cores and parallel.step_size.
  • PyAutoArray general.yaml: numba.use_numba.
  • PyAutoGalaxy general.yaml: adapt.adapt_noise_limit.
  • PyAutoCTI general.yaml: fits.flip_for_ds9, hpc.iterations_per_update, model.ignore_prior_limits, output.log_file, output.log_level and output.log_to_file.

Remove any of these only on explicit human confirmation, as a separate library prompt.

Key Files

  • PyAutoNerves/scripts/board.py: board snapshot, stack diff, orphans
  • PyAutoNerves/autonerves/directory_config.py, autonerves/conf.py: the merge semantics to mirror
  • PyAutoGalaxy/autogalaxy/config/notation.yaml, autogalaxy/config/priors/mesh/delaunay.yaml
  • The workspace config/ trees listed in the table

Original Prompt

Click to expand starting prompt

Workspace config cleanup: remove orphan config keys the Nerves board flags

Type: maintenance
Target: workspaces
Repos:

  • autofit_workspace
  • autogalaxy_workspace
  • autolens_workspace
  • autocti_workspace
    Difficulty: small
    Autonomy: supervised
    Priority: normal
    Status: formalised
    Consequence: glance
    Witness: after the change, a local Nerves board render (python3 scripts/board.py --json in PyAutoNerves) reports 0 workspace files with orphan keys (down from 14 files / 79 keys), except any key this task keeps with a recorded reason; each touched workspace's start_here/smoke scripts still load their config with no config-key error.
    Review-minutes: 3
    Unattended: ready
    Filed: 2026-09-27
    Epic: organ-cockpit
    Lane: local-dev

Why

The Nerves board (PyAutoNerves#175, shipped 2026-09-27 in the cockpit-followups bundle) now lists
workspace config keys that no library in the stack defines. On the 2026-09-27 render
(https://pyautolabs.github.io/PyAutoNerves/) the board is YELLOW with 14 workspace config files
carrying 79 orphan keys
. They are dead settings that mislead users who edit them expecting an effect.

The orphan keys (workspace side — in scope)

  • autofit_workspace/config/general.yaml (L14, 6): hpc.live_visual_update, hpc.quick_update_background, test.check_preloads, test.preloads_check_threshold, version.minimum_library_version, version.workspace_version_check
  • autofit_workspace/config/visualize/plots_search.yaml (L6, 1): mle.corner_cornerpy
  • autogalaxy_workspace/config/general.yaml (L26, 5): hpc.live_visual_update, hpc.quick_update_background, test.disable_positions_lh_inversion_check, version.minimum_library_version, version.workspace_version_check
  • autogalaxy_workspace/config/notation.yaml (L24, 2): label.label.contribution_factor, label_format.format.contribution_factor
  • autogalaxy_workspace/config/visualize/general.yaml (L14, 14): subplot_shape.1, subplot_shape.100, subplot_shape.12, subplot_shape.16, subplot_shape.2, subplot_shape.20, subplot_shape.36, subplot_shape.4, subplot_shape.49, subplot_shape.6, subplot_shape.64, subplot_shape.81, subplot_shape.9, subplot_shape_to_figsize_factor
  • autogalaxy_workspace/config/visualize/plots.yaml (L30, 1): fit_imaging {}
  • autolens_workspace/config/general.yaml (L25, 5): hpc.live_visual_update, hpc.quick_update_background, version.minimum_library_version, version.python_version_check, version.workspace_version_check
  • autolens_workspace/config/notation.yaml (L24, 10): label.label.contribution_factor, label.label.ra, label.label.rs, label.label.sigma_scale, label.superscript.hyperbackgroundnoise, label.superscript.hypergalaxy, label.superscript.hyperimagesky, label_format.format.contribution_factor, label_format.format.ra, label_format.format.rs
  • autolens_workspace/config/priors/mesh/delaunay.yaml (L2, 1): delaunay.areas_factor
  • autolens_workspace/config/visualize/general.yaml (L14, 14): subplot_shape.1, subplot_shape.100, subplot_shape.12, subplot_shape.16, subplot_shape.2, subplot_shape.20, subplot_shape.36, subplot_shape.4, subplot_shape.49, subplot_shape.6, subplot_shape.64, subplot_shape.81, subplot_shape.9, subplot_shape_to_figsize_factor
  • autocti_workspace/config/general.yaml (L2, 6): analysis.n_cores, analysis.preload_attempts, profiling.should_profile, test.check_preloads, test.disable_positions_lh_inversion_check, test.preloads_check_threshold
  • autocti_workspace/config/visualize/general.yaml (L5, 2): general.subplot_ascending_fpr, general.symmetric_cmap_value
  • autocti_workspace/config/visualize/plots.yaml (L1, 11): combined_only, dataset.data, dataset.data_binned, dataset.data_logy, dataset.fpr_non_uniformity, dataset.subplot_dataset_regions, fit.data_logy, fit.fits_fit, fit.residual_map_logy, fit.subplot_fit_regions, subplot_format
  • autocti_workspace/config/visualize/plots_search.yaml (L6, 1): mle.corner_cornerpy

Before deleting each key, grep the workspace itself (scripts, notebooks, start_here.py, any
workspace-local helper) for a reader: the board only checks library code, so a key read by the
workspace's own code (possibly the version.* keys via the version handshake in autonerves, or
subplot_shape.* read by an integer lookup) is not an orphan. Keep any such key and record why.
Mirror the removals into the matching notebooks/markdown only if they carry config snippets.

The typo

autogalaxy_workspace/config/visualize/plots.yaml:30 has fit_imaging {}: — a malformed key (the
{} looks like a leftover flow-mapping), so the fit_imaging section it was meant to open is not
read. Fix it to the intended fit_imaging: section (compare with autolens_workspace's plots.yaml),
not just delete it.

Separate, human-review-gated: library-side unused keys (NOT in scope to remove without review)

The same scan flags library config keys no library code reads. These need a human to confirm
before removal, because a key may be read dynamically (string-built lookups the scan cannot see):

  • PyAutoLens config/non_linear.yaml: a dead 57-key nest.DynestyStatic / nest.DynestyDynamic
    block (Dynesty searches live in PyAutoFit's config; PyAutoLens's copy is not read).
  • PyAutoLens general.yaml: output.fit_dill.
  • PyAutoFit general.yaml: output.log_level, output.log_to_file, output.log_file,
    output.search_internal, profiling.repeats (may be read dynamically — needs review);
    non_linear/GridSearch.yaml: parallel, parallel.number_of_cores, parallel.step_size.
  • PyAutoArray general.yaml: numba.use_numba; PyAutoGalaxy general.yaml: adapt.adapt_noise_limit.
  • PyAutoCTI general.yaml: fits.flip_for_ds9, hpc.iterations_per_update, model.ignore_prior_limits,
    output.log_file, output.log_level, output.log_to_file.

Present this list to the human; remove only what they confirm, as a library change shipped before
the workspace edits (library-first), or re-file it as its own library prompt.

Related prompts

  • draft/maintenance/workspaces/config_key_mirror_drift.md — the opposite direction (library keys
    missing from workspace configs); do both in one pass if picked up together, and do not re-add a
    key this task removes.
  • draft/maintenance/workspaces/sync_remaining_workspace_config_priors_copies.md.

Activity

  1. Jammy2211 commented on Sep 27, 2026

    @Jammy2211
    CollaboratorAuthor

    Parallel claim: PyAutoNerves is also claimed by eyes-organ-order (PyAutoMind#439). That task's Nerves diff is only AGENTS.md. This task touches only scripts/board.py and its tests, so the file sets don't overlap. The human approved a parallel worktree (~/Code/PyAutoLabs-wt/workspace-config-cleanup) on 2026-09-27.

  2. Jammy2211 commented on Sep 27, 2026

    @Jammy2211
    CollaboratorAuthor

    Scope change (human-approved 2026-09-27):

    1. PyAutoLens added as a 7th repo (branch feature/workspace-config-cleanup). Configuring Delaunay.areas_factor in PyAutoGalaxy turns two model-figure acceptance tests red: test_autolens/model_figure/test_acceptance_lens.py test__mge__areas_factor_is_missing_on_the_mesh and test__mge__counts. They used that parameter as the real-class witness of the "missing" state. The witness moves to a class that is still unconfigured.
    2. Workspace step: also delete autogalaxy_workspace/config/priors/mesh/delaunay.yaml, a bare Delaunay: file. After the promotion it is the board's only new orphan. It has no runtime effect, because areas_factor already resolves to 0.5 from the library.
  3. Jammy2211 commented on Sep 27, 2026

    @Jammy2211
    CollaboratorAuthor

    Library PR Created

    API changes: no Python API change. Config-default change in PyAutoGalaxy: Delaunay models now get areas_factor=0.5 Constant from library config (previously needed a workspace prior or reported Missing). Lens/Nerves: internal/test-only.

    Heart YELLOW (85) acknowledged by human: manifest drift (hub organism blurb 7, organism-map blocks 1, workspace checkouts 1); release validation stale (PyAutoFit, PyAutoArray, PyAutoGalaxy, PyAutoLens).

    Next: workspace PRs (autofit_workspace, autogalaxy_workspace, autolens_workspace, autocti_workspace) — /ship_workspace in the same task.

  4. Jammy2211 commented on Sep 27, 2026

    @Jammy2211
    CollaboratorAuthor

    Workspace PRs Created — 2026-09-27

    Library PRs

    Workspace PRs

    Merge order

    1. PyAutoGalaxy#630 → PyAutoLens#751; PyAutoNerves#176 any time (independent).
    2. Workspace PRs only after the library PRs are merged (library-first gate).
    3. autolens_workspace#578 only after PyAutoGalaxy#630 is merged AND released — it deletes priors/mesh/delaunay.yaml and the dPIE ra/rs notation, relying on the new PyAutoGalaxy defaults. Recorded as release-gate: PyAutoGalaxy in active.md.

    Evidence

    • Heart YELLOW (85), human-acknowledged: manifest drift (hub organism blurb 7, organism-map blocks 1, workspace checkouts 1); release validation stale (source moved since rehearsal: PyAutoFit, PyAutoArray, PyAutoGalaxy, PyAutoLens).
    • Nerves board against the worktree: 0 orphans (canonical control 12 files / 59 keys).
    • Smoke (pyauto-heart smoke --root <wt>): autofit 10/10, autogalaxy 16/16, autolens 43/43. autocti: smoke env not buildable locally (no GSL headers for arcticpy); fallback .github/scripts/run_smoke.py base env 3/3 — relying on PR CI.
    • Notebooks for the 3 changed guide scripts regenerated through the PyAutoHands generator's per-script path (build_util.py_to_notebook + inject_colab_setup); output byte-identical to the committed notebooks, no local paths; no markdown mirror exists for these guides.

    Task stays open; merge + close-out via /prm.

  5. Jammy2211 commented on Sep 27, 2026

    @Jammy2211
    CollaboratorAuthor

    /prm — 6 of 7 merged (2026-09-27)

    All legs green, mergeable CLEAN, Heart freeze clear. Merged in library-first order:

    • PyAutoGalaxy#630 → cbd89ced
    • PyAutoLens#751 → 1e6372fd
    • PyAutoNerves#176 → 0b6e7c78
    • autofit_workspace#164 → aa7361df
    • autogalaxy_workspace#250 → 7fd1953d
    • autocti_workspace#34 → 7aa79ac6

    Held: autolens_workspace#578 — release-gate PyAutoGalaxy (it drops the notation ra/rs and Delaunay areas_factor entries that only the new PyAutoGalaxy defaults supply). Merge with /prm once PyAutoGalaxy with #630 is on PyPI; full close-out (records, this issue, worktree) follows then.

    nerves_board.yml re-dispatched. Until #578 merges, the published board will still list autolens_workspace's dead keys (the 21 DELETE keys + top-level subplot_shape_to_figsize_factor); the 5 PROMOTE keys stop counting as orphans now that PyAutoGalaxy defines them. Every other workspace should read 0.

  6. Jammy2211 commented on Sep 30, 2026

    @Jammy2211
    CollaboratorAuthor

    Shipped. All 7 PRs merged: PyAutoGalaxy#630 (cbd89ced), PyAutoLens#751 (1e6372fd), PyAutoNerves#176 (0b6e7c78), autofit_workspace#164 (aa7361df), autogalaxy_workspace#250 (7fd1953d), autocti_workspace#34 (7aa79ac6) on 2026-09-27, and autolens_workspace#578 (bf77575d) on 2026-09-30. Library PRs remain pending release (PyAutoGalaxy, PyAutoLens, PyAutoNerves).

  7. added 2 commits that reference this issue on Oct 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions