Skip to content

refactor: move job logs and JSON sidecars out of results/notes; provenance block on every result JSON #341

Closed

Description

@Jammy2211

Overview

results/notes/ is the ledger tree, but 16 .out job logs (5,311 lines) and 10 .json sidecars (5,406 lines) sit beside the 46 markdown ledgers and are 38 % of it, and recent measurement PRs are 16k–26k lines of per-repeat JSON. This task writes the artefact policy into results/README.md, moves the 26 files to where they belong with every ledger and wiki link rewritten (history preserved by git mv, nothing deleted), adds a lint-gate check that results/notes/ holds only markdown, and makes every result JSON carry a provenance block (host, loadavg at start and at write, SLURM job id, library revisions, dependency versions). Epic profiling-research-wiki follow-on; Mind prompt draft/refactor/autolens_profiling/notes_logs_and_sidecars_out_of_results_notes.md. Nothing is re-run and no script changes what it measures.

Plan

  • Write the artefact policy into results/README.md: logs under results/logs/<campaign>/ (or cited by job id and left on RAL), JSON sidecars beside the result JSONs they describe, committed result JSON summarised (per-row medians + CI + provenance) rather than per-repeat dumps, the notes-only rule, the two frozen evidence-pack folders as the named exception.
  • git mv the 10 sidecars to results/breakdown/imaging/ (every one describes a fixed-light trace or numba result there) and the 16 logs to results/logs/point_source_image/ (12) and results/logs/point_source_source/ (4); rewrite the links in the 12 ledgers and 5 wiki pages that cite them.
  • Add scripts/misc/tooling/check_results_layout.py --check (stdlib) to lint.yml: results/notes/ top level is .md only plus the two allowlisted folders, and every result JSON with a device block carries device.provenance with the required keys unless it is in the grandfather manifest of pre-policy files.
  • Provenance block in _profile_cli.py: provenance_dict() attached inside device_info_dict(), so the 60 cells that already record the device get it without per-script edits; reuses likelihood_breakdown/provenance.py for the library revisions.
  • Tests for the check and the block; check_wiki.py --check, ruff, build_readme.py --check, pytest and lychee stay green.
Detailed implementation plan

Affected Repositories

  • autolens_profiling (primary; workspace-type repo)
  • PyAutoMind (ledger only: active.md registration, prompt move)

Branch Survey

Repository Current Branch Dirty?
./autolens_profiling claude/profiling-wiki-phase-2-b8vtjm (restarted from main 5deb704) clean
./PyAutoMind claude/profiling-wiki-phase-2-b8vtjm (= origin/main) clean

Suggested branch: claude/profiling-wiki-phase-2-b8vtjm (the session's designated branch, restarted from main after #340 merged).

Work Classification: Workspace (profiling repo; tooling + docs + file moves, no library change).
Worktree root: none — remote container clone.

Implementation Steps

  1. Moves (git mv, filenames unchanged):
    • results/notes/{certified_solver_phase_c1_job350768,certified_solver_policy_phase_b_job350588,fixed_light_numba_s4b_source_job343397,fixed_light_numba_s5_source_job343398,fixed_light_numba_s5b_source_job343413,fixed_light_numba_s6_source_jobs,hst_gpu_residue_phase2_historical_job343376,hst_gpu_residue_phase2_job344635,hst_gpu_residue_phase3_job350573,hst_gpu_residue_phase4_job350651}.json → results/breakdown/imaging/.
    • results/notes/point_source_cpu_*.out (12) → results/logs/point_source_image/; results/notes/point_source_source_plane_*.out (4) → results/logs/point_source_source/. Add results/logs/README.md (what a log is, how it is cited, the RAL-only alternative).
    • Rewrite links: ledgers certified_solver_policy_phase_b_2026_09.md, certified_solver_phase_c1_lane_rate_2026_09.md, fixed_lens_light_s4_2026_09.md, fixed_lens_light_numba_memo_2026_09.md, fixed_lens_light_numba_memo_policy_2026_09.md, fixed_lens_light_numba_scaling_2026_09.md, hst_gpu_residue_phase{2_vmap,3_psf,4_logdet}_2026_09.md, point_source_cpu_campaign.md, point_source_source_plane_campaign.md; wiki pages certified_positive_solver.md, fixed_light_numba_cpu.md, hst_gpu_residue.md, point_source_image_plane_cpu.md, point_source_source_plane.md (whichever cite a moved file). clipper_campaign/ and point_source_cpu_2026_09_17_reported/ stay where they are (frozen evidence packs with their own READMEs).
  2. Policy: new "Artefact policy" section in results/README.md (+ a logs/ row in the sections table): where logs, sidecars and summaries live; the summarised-JSON rule for new measurements; the provenance contract (field list, where the block sits, start/at-write loadavg semantics); the lint that enforces the notes-only rule and the provenance ratchet.
  3. Provenance block: in _profile_cli.py add _LOADAVG_AT_IMPORT and provenance_dict() → {captured_at, host, slurm: {job_id, array_job_id, array_task_id}, loadavg_at_import, loadavg_at_write, profiling_revision, library_revisions (from likelihood_breakdown/provenance.source_revisions), library_versions, dependency_versions (jax, jaxlib, numpy, scipy, numba, nufftax, …), provenance_schema: 1}; device_info_dict() gains "provenance": provenance_dict(). Every failure inside it degrades to None fields, never raises.
  4. Check: scripts/misc/tooling/check_results_layout.py [--check] [--write-grandfather] with the two rules above; results/provenance_grandfathered.txt generated once (pre-policy result JSONs, one path per line, header explains the ratchet: entries may only be removed). New step in lint.yml beside check_wiki.py.
  5. Tests: scripts/misc/test/test_check_results_layout.py (tmp trees: clean, stray .out, stray dir, missing provenance, grandfathered) and scripts/misc/test/test_provenance_dict.py (keys, SLURM env pickup, no raise without git).
  6. Validate: check_results_layout.py --check, check_wiki.py --check, build_readme.py --check, ruff check / format --check, pytest scripts/misc/test -q; open the PR; merge stays human (/prm).

Key Files

  • _profile_cli.py — provenance_dict(), device_info_dict()
  • scripts/misc/likelihood_breakdown/provenance.py — reused for source_revisions
  • scripts/misc/tooling/check_results_layout.py — new gate; results/provenance_grandfathered.txt — manifest
  • results/README.md, results/logs/README.md — policy
  • .github/workflows/lint.yml — new step
  • results/notes/*.md, wiki/campaigns/*.md — link rewrites only

Original Prompt

Click to expand starting prompt

Move committed job logs and JSON sidecars out of results/notes/

  • Work type: refactor
  • Target: @autolens_profiling
  • Epic: profiling-research-wiki
  • Autonomy: human-required
  • Filed: 2026-09-27

Original prompt

From the 2026-09-27 profiling review: results/notes/ holds 74 files / 28,474 lines, of which
16 .out job logs (5,311 lines) and 10 .json sidecars (5,406 lines) are 38 % of the "notes"
tree; recent measurement PRs are 16k–26k lines each, mostly result JSON (e.g. #327 = 24,100 lines
of JSON, #335 = 15.5k). The maintainer agreed this is a problem.

Scope

  1. Decide and document the artefact policy in results/README.md: .out logs go to
    results/logs/<campaign>/ (or stay on RAL and are cited by job id); JSON sidecars go beside the
    result JSONs they describe, not under notes/; committed result JSON is summarised (per-row
    medians + CI + provenance block), not per-repeat dumps.
  2. git mv the existing 26 files with the ledger links rewritten (and check_wiki.py, lychee
    green). History is preserved by the move; nothing is deleted.
  3. Add a lint-gate check that results/notes/ contains only .md.
  4. Mandatory provenance block for every result JSON: host, loadavg at start/end, SLURM job id,
    library revisions, dependency floors (nufftax etc.) — the fields the wiki index and the future
    dashboard read.

Out of scope

Re-running anything; changing what any script measures.

Activity

  1. Jammy2211 commented on Sep 27, 2026

    @Jammy2211
    ContributorAuthor

    Shipped

    • PR refactor: job logs and JSON sidecars out of results/notes; artefact policy, layout gate, provenance block #342 merged to main at 282076cf (head ab1e4fd, 45 files, +1150/−25) on the human's /prm after lint.yml went green.
    • 16 .out logs → results/logs/{point_source_image,point_source_source}/, 10 JSON sidecars → results/breakdown/imaging/ (git mv; ledger and wiki links rewritten). results/notes/ is markdown-only again.
    • Artefact policy in results/README.md (+ results/logs/README.md); check_results_layout.py --check in lint.yml with the 530-file grandfather manifest; _profile_cli.provenance_dict() attached as device.provenance by device_info_dict().
    • Mind: record complete/2026/09/profiling-notes-sidecars-out.md.

    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions