Schema extension (Cortex script, small)

projects.yaml needs two things the phase-1 grammar lacks: status: planned (for
euclid_dr1_prelim) and an optional free-text note: (the prompt's dormant rows).
scripts/cortex.py: PROJECT_STATUSES += ("planned",); note joins PROJECT_FIELDS
as the one optional field (_finish_row no longer lists it as missing); remote
accepts any owner/repo (already does) and the header comment says "the remote —
PyAutoLabs for active projects; a personal remote is recorded as a fact with a note:".
Tests + REFERENCE.md subset text + docs/schema_decisions.md 50.

The pull manifest (decision 40 → amended as decision 51)

<mirror or local_path>/.cortex/pull.json, written by each sync CLI's pull after the
last rsync (dry-run guarded — status calls pull --dry-run):

{"schema": 1, "pulled_at": "<ISO UTC>",
 "checkpoints": {"<run dir relative to the pull root>": {"bytes": N, "mtime": "<ISO>"}},
 "runs": {"<jobid or jobid_task>": {"checkpoint_bytes": N, "checkpoint_mtime": "<ISO>"}}}

Gathered by one ssh "$HPC_HOST" "find '<ral_root>/output' -name checkpoint.hdf5 -printf '%s %T@ %p\n'"
over the existing ControlMaster mux, before the manifest write. checkpoints is always
filled (keyed by run-dir path — the only key the RAL tree can name); runs is filled
only where the CLI can link a job id to a run dir (subhalo's .out names the sample;
profiling cannot — leave runs empty there). Brain collect gains the third lookup
(runs[ident] → runs[stem] → checkpoints[<resolved run dir rel>]) in a two-line
change + test. .cortex/ is gitignored in every project (it is pull state, like
output/).

Files

/mnt/c/Users/Jammy/Science/subhalo_validation (direct commits on main; human creates the remote)

1. .gitignore: add hpc/batch_gpu/output/*, hpc/batch_gpu/error/* (+ .gitkeep
   negations like the cpu pair), .cortex/, *.hdf5, *.dill. output/, dataset/raw|reduced|external,
   sync.conf already ignored; the simulated dataset/ stays tracked (15 MB, not Euclid —
   say so in the README).
2. git add results/delaunay_adapt_split/pl_eff_0_no_subhalo.json pl_eff_1_outer_no_subhalo.json
   (the witnesses the README promises).
3. hpc/sync.conf.example (three keys, values as aliases) so a fresh clone has a template.
4. hpc/sync pull() (:194-230): after the for dir loop, [[ -z "$dryrun" ]] && write_pull_manifest → ${PROJECT_ROOT}/.cortex/pull.json (bash + python3 - heredoc
   for the JSON; keys per above; runs linked via hpc/batch_cpu/output/output.<jobid>_<task>.out
   → its Sample: / output dir line when present).
5. README: the witness sentence made true; one paragraph "Rulings of record live in
   PyAutoCortex (projects.yaml row subhalo_validation); wiki/project/state.md and the
   journal are commentary and cite ruling ids."
6. Human step: gh repo create PyAutoLabs/subhalo_validation --private (empty), then
   git remote add origin … && git push -u origin main. Witness: gh repo view … --json visibility → PRIVATE; git ls-files results/ lists the pl_eff pair; no tracked file

   5 MB, no .fits outside dataset/ (git ls-files | grep -c '\.fits$' = the 15
   simulated ones, all under dataset/).

/home/jammy/Code/PyAutoLabs/autolens_profiling (PR, branch feature/cortex-registration)

7. CORTEX.md (one paragraph: rulings of record live in PyAutoCortex; DECISIONS.md is
   commentary that cites ruling ids; the backfill is phase 4); link it from README.
8. hpc/sync pull() (:183-203): after pull_indices …, [[ -z "$dryrun" ]] && write_pull_manifest → ${LOCAL_PULL_ROOT}/.cortex/pull.json (checkpoints only);
   .cortex/ added to the mirror's .gitignore (the mirror is a local repo — commit that
   one-liner there; it stays remote-less).
9. hpc/sync.conf.example gains nothing; hpc/README mentions the manifest.

/mnt/c/Users/Jammy/Science/euclid — no changes (ruling: remote: none).

PyAutoCortex (PR, branch feature/cortex-registration)

10. scripts/cortex.py schema extension (above) + tests/test_cortex.py cases
    (planned legal; note optional; unknown field still an error).
11. projects.yaml rows (in-grammar, PyYAML-parity):
    • inference_programme: remote PyAutoLabs/autolens_profiling, local_path
      /home/jammy/Code/PyAutoLabs/autolens_profiling, ral_root
      /mnt/ral/jnightin/autolens_profiling, mirror /mnt/c/Users/Jammy/Science/inference_programme,
      sync_cli hpc/sync, sync_verbs [pull, logs, status, submit, jobs, sacct, cancel, tail, du, check],
      ledger results/notes/inference/PROGRAMME.md, witness_file results/searches/**/*.json,
      partition both, status active, note "DECISIONS.md is commentary; pull-only CLI — RAL is a git checkout; gpu-1 excluded".
    • subhalo_validation: remote PyAutoLabs/subhalo_validation, local_path
      /mnt/c/Users/Jammy/Science/subhalo_validation, ral_root /mnt/ral/jnightin/subhalo_validation,
      mirror none, sync_cli hpc/sync, sync_verbs [push, pull, logs, sync, push-data-init, pull-full, status, submit, push-submit, jobs, sacct, cancel, wait-and-pull, tail, du, check, clear-logs],
      ledger wiki/project/state.md, witness_file results/**/*.json, partition ral,
      status active, note "pulls land in the checkout (PULL_DIRS output results); run_chain.sh chains A→B afterok; witness evidence_increase < 5, subhalo_stage, test_mode".
    • euclid: remote none, local_path /mnt/c/Users/Jammy/Science/euclid, ral_root
      /mnt/ral/jnightin/euclid_strong_lens_modeling_pipeline, mirror none, sync_cli
      hpc/sync, sync_verbs [push, pull, sync, status], ledger CLAUDE.md, witness_file
      catalogue/**/*.csv, partition both, status active, note "NO PyAutoLabs remote by ruling 2026-09-01: tracks DR1 tile-keyed catalogue CSV, personal remote Jammy2211/euclid-dr1-modeling, nested Overleaf paper repo, 34 GB; code half = PyAutoLabs/euclid_strong_lens_modeling_pipeline; witness undefined until a science phase pre-registers one".
    • euclid_dr1_prelim: remote none, local_path /mnt/c/Users/Jammy/Science/euclid_dr1_prelim,
      ral_root /mnt/ral/jnightin/euclid_dr1_prelim, mirror none, sync_cli hpc/sync,
      sync_verbs [push, pull, sync, status], ledger wiki/project/state.md, witness_file
      results/**/*.json, partition both, status planned, note "created by euclid-dr1-prep phase 4 (a Cortex phase after migration); every path is the planned one".
    • dormant rows (git repos only; non-git trees subhalo, euclid_group, z_* are storage,
      skipped; aris_PJ011646 is an assistant clone, skipped): profiling (remote none, note
      "2026-05 checkpointed sweep harness, PyAutoNSS venv; name collides with autolens_profiling"),
      cowls_diana (none), subhalo_simulations (none; "git init, zero commits, 21 GB"),
      slope_hierarchy (Jammy2211/slope_hierarchy), pj011646 (Jammy2211/PJ011646),
      concr (Jammy2211/cosmology_and_cancer), ic50_workspace (Jammy2211/ic50_workspace)
      — each with ral_root = /mnt/ral/jnightin/<name> (planned/unknown, noted), mirror
      none, sync_cli hpc/sync, sync_verbs [pull] (unverified, noted), ledger README.md,
      witness_file results/**/*.json, partition both. Keys must match ^[a-z][a-z0-9_]*$
      (so pj011646, aris not needed).
12. projects.yaml header: the boundary sentence "science repos are NOT added to
    PyAutoMind/repos.yaml — that map is the workspace; this one is the science" and the
    remote wording above. REFERENCE.md subset + fields; docs/schema_decisions.md 50–52
    (schema extension; manifest v1 schema + gather method + .cortex/ ignored; the euclid
    ruling and the dormant-row rules: git repos only, personal remotes recorded as facts).
13. python3 scripts/cortex.py check OK with the rows; pytest tests -q green;
    dashboard --cortex . --apply so the Projects table renders the rows (commit the render;
    the Cortex dashboard_refresh.yml PR check must be clean).

PyAutoBrain (PR, branch feature/cortex-registration) — tiny

14. _cortex.py collect: manifest schema 1 reader with the third lookup
    (checkpoints[<run dir rel>]), pull.json written to the tmp mirror in the existing
    collect tests; docs line in agents/conductors/cortex/AGENTS.md.

Execution (Opus)

• /create_issue → PyAutoMind#…; prompt → active/; active.md status: workspace-dev,
  worktree ~/Code/PyAutoLabs-wt/cortex-registration for **PyAutoCortex + autolens_profiling
  • PyAutoBrain** (the Science-folder repo is edited in place on its own main — it is not a
    workspace repo and has no worktree helper; the subagent works there directly with explicit
    pathspecs; the mirror likewise).
• Slice A (Cortex + Brain): schema extension, rows, decisions, render, collect lookup, tests.
• Slice B (projects, laptop lane): subhalo .gitignore/witnesses/README/sync.conf.example/
  manifest hook + commit on main; profiling CORTEX.md + manifest hook + mirror ignore; a real
  hpc/sync pull on both projects to prove the manifest lands (read-only on RAL; rsync
  --update so only new files transfer); then, after the human creates the private remote,
  push subhalo main and verify the witness.
• Ship: Cortex PR → Brain PR (either order; Cortex first keeps the render honest) →
  autolens_profiling PR; /prm per the standing authorisation; the subhalo push is not a PR.

Human steps (please do when the plan is approved — the rest proceeds around them)

! gh repo create PyAutoLabs/subhalo_validation --private --description "Validation of DM-subhalo SLaM detection pipelines on simulated no-subhalo lenses (science project; witness JSONs tracked, outputs local)"

and confirm the euclid ruling (remote: none, reasons above) — approving this plan is that
confirmation unless you say otherwise.

Verification

• Prompt witness: projects.yaml rows for inference_programme, subhalo_validation, euclid,
  euclid_dr1_prelim (planned) + dormant rows; git -C …/subhalo_validation ls-files results/
  lists the pl_eff JSONs; gh repo view PyAutoLabs/subhalo_validation --json visibility →
  PRIVATE.
• cortex.py check OK; Cortex pytest green; yaml.safe_load(projects.yaml) parity test
  green; dashboard Projects table shows every row; Brain pytest green.
• Both manifests exist after a real pull, with pulled_at and ≥1 checkpoints entry
  (subhalo has live checkpoints on RAL; profiling should too under output/searches);
  cortex collect on a synthetic member reports the checkpoint leg from the manifest.
• Pushed subhalo main: no file > 5 MB, no .fits outside dataset/, no sync.conf.

Branch Survey

Suggested branch: feature/cortex-registration (worktree root ~/Code/PyAutoLabs-wt/cortex-registration/)

Cortex phase 3 — private remotes for the Science-folder projects, and the body-map rows

Type: feature
Target: pyautocortex
Repos:

• PyAutoCortex
• autolens_profiling
  Themes:
• hpc-gpu
• mind-workflow
  Difficulty: medium
  Autonomy: supervised
  Priority: high
  Status: draft
  Consequence: judge
  Witness: projects.yaml has rows for inference-programme, subhalo_validation, euclid, euclid_dr1_prelim (planned) and every other Science-folder tree as dormant; git -C /mnt/c/Users/Jammy/Science/subhalo_validation ls-files results/ lists the pl_eff witness JSONs; gh repo view PyAutoLabs/subhalo_validation --json visibility says PRIVATE
  Review-minutes: 15
  Unattended: never
  Lane: local-dev
  Epic: cortex-birth
  Phase: 3
  Parent: draft/feature/pyautocortex/cortex_birth_epic.md
  Filed: 2026-09-01

Phase 3 of 7 in the PyAutoCortex birth epic. Gate: phase 1 (the
projects.yaml schema). May overlap phase 2. Gates phase 4. Laptop lane
throughout — the projects live under /mnt/c/Users/Jammy/Science/ and the
remotes are created by the human.

Context (ruled 2026-09-01)

Each Science-folder project gets a private PyAutoLabs remote holding code,
config, wiki and witness result files only. Outputs, checkpoints and laptop
mirrors stay local and gitignored. No Euclid data files of any kind —
consortium-controlled, private repo or not. The inference_programme mirror
stays remote-less ("holds data, not science"). PyAutoLabs rather than a
personal account so repo_settings.yml's live org enumeration applies policy
to them; the personal-repo convention applies to papers, not projects.

State of the trees today (audit 2026-09-01): subhalo_validation is a local git
repo, no remote, not in any body map; its README says the witness JSONs are
tracked but the pl_eff pair is untracked. Science/euclid is the private DR1
source-of-truth tree with no remote. euclid_dr1_prelim does not exist yet
(euclid-dr1-prep phase 4 creates it). inference_programme is the profiling
mirror (LOCAL_PULL_ROOT), local git, no remote by design. Other trees:
PJ011646, aris_PJ011646, concr, cowls_diana, euclid_group,
ic50_workspace, profiling (2026-05 sweep harness — name collides with
autolens_profiling), slope_hierarchy, subhalo, subhalo_simulations,
z_dataset, z_projects_complete/, z_vault.

Task

1. subhalo_validation: write a .gitignore that excludes output/,
   hpc/batch_*/output/, hpc/batch_*/error/, *.zip, *.hdf5, *.dill,
   dataset/ FITS, and keeps scripts/, simulators/, config/, hpc/
   (scripts and sync, not sync.conf), wiki/, results/**/*.json,
   results/figures/*.jpg. Track the untracked pl_eff witness JSONs. Human
   creates PyAutoLabs/subhalo_validation private; push main. Fix the README
   line that claims the witnesses are tracked so it is true.
2. Science/euclid: same pattern, with a data audit first: list every
   file over 1 MB and every .fits/.h5/catalogue CSV, and confirm with the
   human which are consortium data before any remote exists. Push only after
   that list is ruled on. If the ruling is "too entangled", record the reason in
   projects.yaml (remote: none — data-entangled, see ruling R-…) and move on
   — the epic does not block on it.
3. autolens_profiling: already has a remote. Add a CORTEX.md pointer
   (one paragraph: "rulings of record live in PyAutoCortex; DECISIONS.md is
   commentary and cites ruling ids") — the DECISIONS backfill itself is phase 4.
4. projects.yaml rows in the Cortex, from the audit's where-to-look table:
   • inference-programme: repo autolens_profiling, RAL
     /mnt/ral/jnightin/autolens_profiling (git checkout), mirror
     /mnt/c/Users/Jammy/Science/inference_programme (no remote), sync
     hpc/sync (pull-only: pull logs status submit jobs sacct cancel tail du check — no push by design), ledger
     results/notes/inference/PROGRAMME.md + DECISIONS.md, witness result
     JSON (results/searches/**/hpc_hpc_a100_fp64_*.json: stamp + target_id +
     provenance triple), partition gpu (gpu-1 excluded) + ral.
   • subhalo_validation: local /mnt/c/Users/Jammy/Science/subhalo_validation,
     RAL /mnt/ral/jnightin/subhalo_validation (rsync), mirror = project
     itself (PULL_DIRS=(output results)), sync hpc/sync (push/pull/
     push-submit/wait-and-pull/…) + hpc/run_chain.sh, ledger
     wiki/project/state.md + journal + results_summary.md, witness
     results/<pipeline>/<lens>_no_subhalo.json → evidence_increase (< 5),
     subhalo_stage, test_mode, partition ral.
   • euclid: local /mnt/c/Users/Jammy/Science/euclid, pipeline repo
     euclid_strong_lens_modeling_pipeline, RAL
     /mnt/ral/jnightin/euclid_strong_lens_modeling_pipeline, sync hpc/sync
     (push pull sync status; submit from the cluster), ledger: none (the
     euclid-dr1-prep epic file until phase 4 migrates it), witness: undefined
     until the science phase pre-registers one.
   • euclid_dr1_prelim: status: planned, everything else tbd — created by
     euclid-dr1-prep phase 4.
   • Every other tree above: status: dormant, local_path only, one-line
     note: (e.g. profiling: "2026-05 checkpointed sweep harness, PyAutoNSS
     venv; name collides with autolens_profiling").
5. Add the three private remotes to PyAutoMind/repos.yaml? No. The Mind's
   body map is the workspace; science projects are the Cortex's body map. Record
   that boundary in projects.yaml's header comment.

Acceptance

• The witness above; cortex.py check passes with the seeded rows; each
  pushed remote's main contains no file over the size the human ruled and no
  .fits.
• Commits on the project repos' main (they are not PR-shaped repos; the
  Cortex PR carries projects.yaml).

Out of scope

• Unifying the three hpc/sync CLIs (deferred; epic ledger).
• Moving any tree out of /mnt/c/Users/Jammy/Science/.
• Creating euclid_dr1_prelim (euclid-dr1-prep phase 4, which will by then be a
  Cortex phase).