Repository navigation
refactor: repoint hard-coded RAL paths to the category layout #67
Description
Overview
Science/ and RAL /mnt/ral/jnightin now share a category layout (active/, inference/, profiling/, archive/, complete/). The folder moves are done; 8 temporary old-path symlinks remain on RAL because tracked files hard-code the old top-level paths. This task repoints that live config so the links can be removed.
Heart RED development override granted by the human 2026-10-15 for this task (config/docs only, no library code); never extends to a release.
Plan
- Make
activate.shderive its own root; SLURM templates keep one overridable root variable; old submit scripts get a mechanical old→new rewrite (no history rewritten). - Phase 1 (now): autolens_inference, autofit_profiling, Mind
policy/hpc_ral.md, PyAutoPulse mentions, the euclid project's 26 submit scripts. - Phase 2 (blocked on claims): autolens_profiling after #418 merges; autofit_inference + Cortex
ral_rootsweep after search-ext-b5b2-remainder releases them; PyAutoMemory. - After each merge:
git pullon RAL, then remove that repo's link (human-run).
Tier: judge — merge mode: human /prm
Detailed implementation plan
Affected Repositories
- autolens_inference, autofit_profiling, PyAutoMind, PyAutoPulse (phase 1)
- autolens_profiling, autofit_inference, PyAutoCortex, PyAutoMemory (phase 2, claimed)
- euclid project (personal repo, laptop
Science/active/euclid)
Branch Survey
| Repository | Claim |
|---|---|
| autolens_inference | free |
| autofit_profiling | free |
| PyAutoMind | free |
| PyAutoPulse | free |
| autolens_profiling | hst-mge-pilot-qualification (awaiting merge #418) |
| autofit_inference, PyAutoCortex | search-ext-b5b2-remainder (jobs running) |
Suggested branch: feature/ral-category-paths
Design
Do not swap one hard-coded path for another: activate.sh derives its own root from BASH_SOURCE; SLURM templates keep one overridable root variable at the top (AP_ROOT=${AP_ROOT:-/mnt/ral/jnightin/profiling/autolens_profiling}) because sbatch spools the script; historical per-experiment submit scripts get a mechanical old→new rewrite. results/, provenance JSON, Cortex archive/, rulings, complete/ records are never rewritten.
Phase 1 — now (repos free)
- autolens_inference: activate.sh, hpc/README.md, hpc/batch_{cpu,gpu}/template + submit scripts (14 files) →
/mnt/ral/jnightin/inference/autolens_inference. - autofit_profiling: activate.sh, hpc/README.md, hpc/batch_cpu/template (3 files) →
/mnt/ral/jnightin/profiling/autofit_profiling. - PyAutoMind
policy/hpc_ral.md"Projects live under …" → category layout; PyAutoPulse live path mentions (4 files). - euclid (personal repo Jammy2211/euclid-dr1-modeling, laptop
Science/active/euclid): 26 submit scriptsPROJECT_PATH=/mnt/ral/jnightin/euclid/…→/mnt/ral/jnightin/active/euclid/…; commit ONLY those files (the repo has unrelated staged renames catalogue_swg/ → catalogue/); no GitHub push unless the human asks;hpc/sync push --no-data. - After merge: human runs
git pullon RAL for each repo, then removes the autolens_inference, autofit_profiling and euclid links.
Phase 2 — blocked (claims)
- autolens_profiling (+_wt, _corpus; ~77 files incl. hpc/sync line 62 LOCAL_PULL_ROOT default →
Science/profiling/autolens_profiling) — after autolens_profiling#418 (hst-mge-pilot-qualification) merges. - autofit_inference (+ b5a_dryrun; 15 files) and PyAutoCortex
ral_rootfor every remaining row + registersubhalo(active) +euclid→ active atactive/euclid+ schema comment — after search-ext-b5b2-remainder releases them (its wave-2 jobs run on RAL now). - PyAutoMemory live path mention (1 file) — with phase 2.
- Remove the remaining RAL links (human-run).
Original Prompt
Click to expand starting prompt
Science + RAL folder restructure — point live path config at the new category layout
Type: refactor
Target: cortex
Repos:
- PyAutoCortex
- PyAutoMind
- PyAutoPulse
- PyAutoMemory
- autolens_profiling
- autolens_inference
- autofit_profiling
- autofit_inference
Difficulty: medium
Autonomy: supervised
Priority: medium
Status: formalised
Consequence: judge
Witness:cortex.py checkpasses on the edited projects.yaml; every active row'slocal_pathandral_rootexist (laptoptest -d, RALssh test -donce the RAL moves ran);grep -rnover the edited repos' live config (projects.yaml, policy/hpc_ral.md, activate.sh, hpc/ submit defaults) finds no top-level/mnt/ral/jnightin/<project>orScience/<project>path for a moved project; history (results/, provenance, archive/, complete/, rulings, Cortex archive/) untouched.
Review-minutes: 10
Issued: 2026-10-15
Original request (verbatim, 2026-10-14)
Ok I removed a load of old folders on RAL, so now do the folder updates but I want to sync this with updates in the "Science" Folder which have active, complete (instead of archive?), inference, profiling, scratch. I also want PyAutoCortex to be updated to use this structure and any other organs
I want to get rid of these weird z_ folders, I guess complete does that for z_projects_Complete, maybe archieve is still warranted as a way to dump unifished projects and we probably need a generic dataset folder like we do now.
Human decisions (same session): euclid_dr1 is NOT moved now (sorted separately later); euclid → active; subhalo → active and registered in the Cortex; collaborator/ stays a top-level Science folder; admin_jammy to be deleted (held in archive/ — it has uncommitted edits, human to confirm); old-path symlinks kept one month, then a Gut sweep.
Layout (laptop /mnt/c/Users/Jammy/Science ⇄ RAL /mnt/ral/jnightin) — DONE 2026-10-15
active/— analytic_gaussian, ep_toy_gaussian, euclid, ic50_workspace, slope_hierarchy_scale, subhalo, subhalo_validationinference/— RAL: autolens_inference, autofit_inference, autofit_inference_b5a_dryrun (laptop: pull roots only)profiling/— RAL: autolens_profiling (+_wt, _corpus, _delaunay), autofit_profiling, tune_profiling (laptop: pull roots only)archive/,complete/,scratch/,dataset/,collaborator/(laptop only)- RAL
PyAuto/and (for now)euclid_dr1/stay top-level.
Status (2026-10-15)
- Laptop: moved; old-path symlinks REMOVED (PyCharm clutter); Cortex
local_path/mirrorrepointed (PyAutoCortex#64, projects.yaml: analytic_gaussian, ep_toy_gaussian → active/ #66 merged).~/.bashrcProjectEuclid/ProjectSubhalo repointed. - RAL: moved via
scratch/ral_reorg_20261015.sh(human-run) + toys to active/; worktrees repaired; 11 unused old-path links removed. 8 links REMAIN because tracked files hard-code the old paths: autolens_profiling, autolens_profiling_wt, autolens_profiling_corpus, autolens_inference, autofit_inference, autofit_inference_b5a_dryrun, autofit_profiling, euclid. - Local untracked
hpc/sync.confHPC_BASE repointed for all 9 projects/repos and witnessed withhpc/sync logs. - Human decisions 2026-10-15: keep profiling/ and inference/ split (mirrors Pulse/Insight); RAL links are temporary — removed as soon as their repo edits merge (no Gut month sweep);
admin_jammyuntouched; Heart RED development override granted for this task.
Design
Do not swap one hard-coded path for another: activate.sh derives its own root from BASH_SOURCE; SLURM templates keep one overridable root variable at the top (AP_ROOT=${AP_ROOT:-/mnt/ral/jnightin/profiling/autolens_profiling}) because sbatch spools the script; historical per-experiment submit scripts get a mechanical old→new rewrite. results/, provenance JSON, Cortex archive/, rulings, complete/ records are never rewritten.
Phase 1 — now (repos free)
- autolens_inference: activate.sh, hpc/README.md, hpc/batch_{cpu,gpu}/template + submit scripts (14 files) →
/mnt/ral/jnightin/inference/autolens_inference. - autofit_profiling: activate.sh, hpc/README.md, hpc/batch_cpu/template (3 files) →
/mnt/ral/jnightin/profiling/autofit_profiling. - PyAutoMind
policy/hpc_ral.md"Projects live under …" → category layout; PyAutoPulse live path mentions (4 files). - euclid (personal repo Jammy2211/euclid-dr1-modeling, laptop
Science/active/euclid): 26 submit scriptsPROJECT_PATH=/mnt/ral/jnightin/euclid/…→/mnt/ral/jnightin/active/euclid/…; commit ONLY those files (the repo has unrelated staged renames catalogue_swg/ → catalogue/); no GitHub push unless the human asks;hpc/sync push --no-data. - After merge: human runs
git pullon RAL for each repo, then removes the autolens_inference, autofit_profiling and euclid links.
Phase 2 — blocked (claims)
- autolens_profiling (+_wt, _corpus; ~77 files incl. hpc/sync line 62 LOCAL_PULL_ROOT default →
Science/profiling/autolens_profiling) — after autolens_profiling#418 (hst-mge-pilot-qualification) merges. - autofit_inference (+ b5a_dryrun; 15 files) and PyAutoCortex
ral_rootfor every remaining row + registersubhalo(active) +euclid→ active atactive/euclid+ schema comment — after search-ext-b5b2-remainder releases them (its wave-2 jobs run on RAL now). - PyAutoMemory live path mention (1 file) — with phase 2.
- Remove the remaining RAL links (human-run).
Activity
Jammy2211 commentedon Oct 15, 2026
Phase 1 shipped (2026-10-15)
AP_ROOT, submit scripts/README/AGENTS.md/sync.conf.example→/mnt/ral/jnightin/inference/autolens_inference/mnt/ral/jnightin/profiling/autofit_profilingpolicy/hpc_ral.mddescribes the category layoutactive/euclidand pushed to RAL; 0 old-path hits on RALRAL follow-up (human-run):
git pullininference/autolens_inference+profiling/autofit_profiling, then remove theautolens_inference,autofit_profilingandeuclidlinks.Left open for phase 2 (blocked on claims): autolens_profiling (+_wt, _corpus) after #418; autofit_inference (+b5a_dryrun) and the Cortex
ral_rootsweep after search-ext-b5b2-remainder; PyAutoMemory. Re-filed asdraft/refactor/cortex/ral_category_paths_phase2.md.Jammy2211 commentedon Oct 15, 2026
Phase 2 started 2026-10-15 (Mind
active/ral_category_paths_phase2.md): autofit_inference, PyAutoCortex, PyAutoMemory now; autolens_profiling after #418 merges. Heart still RED with the same reasons; the human's development override for this issue carries to phase 2 ("go", session 4b6b15e2).Jammy2211 commentedon Oct 15, 2026
Phase 2 shipped (2026-10-15)
AP_ROOT; wave-2 submit scripts +plan_wave2.pygenerator, README, AGENTS.md, wiki,sync.conf.example→/mnt/ral/jnightin/inference/autofit_inference;dashboard/summary.jsonregenerated (_wave2.pyis a producer path)ral_rootmirrors its local category;subhaloregistered (active, decision 61);euclidstays dormant, superseded by euclid_dr1 (euclid_dr1_prelim defunct) per the humanRAL follow-up (human-run):
git pullininference/autofit_inference, then remove theautofit_inferenceandautofit_inference_b5a_dryrunlinks.Left: autolens_profiling (+_wt, _corpus) once autolens_profiling#418 merges — re-filed as
draft/refactor/cortex/ral_category_paths_autolens_profiling.md.Jammy2211 commentedon Oct 18, 2026
Last repo: autolens_profiling — plan (approved by human 2026-10-18)
Heart RED override (human, 2026-10-18, chat): Heart was RED at start_dev (PyAutoFit/PyAutoArray Tests, workspace_test smokes, CI wall-clock, worktree drift; "autolens_profiling: red" while its main lint + Pages are green on 3f12474). The human authorized this non-corrective development branch past the RED gate per AUTONOMY.md "Human override for Heart RED (development only)". This is not a Heart acknowledgement.
Claim: autolens_profiling is also claimed by
cluster-pointsolver-robust-settings(#436). The file sets are disjoint (29 target files vs that branch's 9), so this task uses its own parallel worktree, as approved.High level
/mnt/ral/jnightin/autolens_profiling{,_wt,_corpus}to/mnt/ral/jnightin/profiling/…(29 files, ~80 occurrences).AP_ROOT=${AP_ROOT:-/mnt/ral/jnightin/profiling/autolens_profiling}. Same design as autolens_inference#22 and autofit_inference#15.hpc/sync.conf.exampleHPC_BASE=/mnt/ral/jnightin/profiling. Thehpc/sync:62LOCAL_PULL_ROOTdefault changes fromScience/inference_programme, which is missing, to/mnt/c/Users/Jammy/Science/profiling/autolens_profiling.PyAuto_wt/…andp2d_checkfolders stay as they are.git pullin RALprofiling/autolens_profiling, thenrmthe links autolens_profiling, autolens_profiling_wt, autolens_profiling_corpus. Live jobs (412567 and the 41253x/412565 inference arrays) do not use the links (checked 2026-10-18).Tier: judge — merge mode: human /prm
Detail
hpc/batch_gpu/*17,hpc/batch_cpu/*5):AP_ROOT=becomes the overridable form.*_BRANCH_SRCdefaults underautolens_profiling_wt/becomeprofiling/autolens_profiling_wt/, keeping their${VAR:-…}form.hpc/README.md:182,activate.sh:14(comment),.gitignore:114(comment),scripts/lens/solver/README.md,scripts/lens/solver/_corpus.py,wiki/campaigns/linear_solver_accuracy.md,wiki/research/jacobi_a100_batched_divergence.md,hpc/sync.conf.example,hpc/sync:62.bash -non every edited submit script; the repo's lint.yml steps run locally.feature/ral-category-paths-autolens-profiling, worktree~/Code/PyAutoLabs-wt/ral-category-paths-autolens-profiling/. Shipped with ship_workspace; the PR references this issue.Jammy2211 commentedon Oct 18, 2026
PR opened: PyAutoLabs/autolens_profiling#437 (Heart RED override, development only)
Human authorization (live, 2026-10-18, this task): "I authorize override". Covers commit, push and PR-open only. Merge needs a separate human
/prmwith all checks green; no release.Passed branch gates: Witness grep empty; bash -n + py_compile; lint.yml steps run locally (ruff, build_* --check, check_wiki, check_results_layout, check_submits, list_timing_assertions); pytest scripts/misc/test 1457 passed, 6 skipped; author diff review path-only; smoke n/a.
Exact Heart RED reasons at ship time:
After merge (human, RAL):
git pull --ff-onlyin profiling/autolens_profiling, thenrmthe 3 root links.Jammy2211 commentedon Oct 18, 2026
Shipped — RAL category layout complete
/prmcommand; lint CI green. The Heart RED override covered shipping only; no release.profiling/autolens_profilingfast-forwarded from 9ad12abd (#305) to 173ab272 (326 commits). Sixteen untrackedresults/breakdown/imaging/fixed_light_trace_*files blocked the ff. All 16 were byte-identical to main and were moved to/mnt/ral/jnightin/scratch/ap_aside_20261018/; nothing was lost, so that folder is redundant. The final git process sat in an NFS D-state wait after writing; HEAD and the index were already updated, with no lock left.autolens_profiling,autolens_profiling_wt,autolens_profiling_corpus. The only remaining root link iseuclid_dr1 → active/euclid_dr1, which is intended. Running jobs (412567 and the autolens_inference arrays) use category paths, not the links.inference/autofit_inferencefast-forward 0587a44 → main started (logscratch/afi_ff_20261018.log).Every repo is now on the category layout, so this issue closes.