Skip to content

feat: profiling-summary@3 class/status, walltime-validation@1, Class-first board #69

Closed

Description

@Jammy2211

Overview

Phase 0, sub-issue 0A of the likelihood-lifecycle epic (PyAutoMind draft/research/autolens_profiling/likelihood_lifecycle_epic.md, report §3.1–§3.2, §3.6, §3.8, §4). Pulse is the contract owner, so it goes first: this issue adds profiling-summary@3 with producer-declared class (optimal/secondary/experimental) and status (current/superseded/archived), the walltime-validation@1 evidence axis, the designation decision kind, and a Class-first, current-only setup browser. 0B (autolens_profiling builder), 0C (autolens_inference exporter), 0D (Insight) and 0E (Brain template) follow as their own issues once this ships. Pulse still validates and renders only: it infers no optimal, selects no fastest row and computes no ratio.

Rulings assumed (approved in chat 2026-10-21): R1 vocabulary/setup_id grammar (§3.1); R2 label-only history cut; R3 provisional designations as phase-0 labels; R7 host_class is provenance and laptop/local-GPU rows are never current; R4/R5 enter as validator enumerations only.

Plan

  • Add profiling-summary@3 to the reader: v2 shape plus lifecycle fields with the one-current-per-key, optimal-needs-decision_id and laptop-never-current rules; v2 instances (autofit_profiling) validate unchanged.
  • Add the walltime-validation@1 schema as a second producer-declared evidence axis, registrable as its own instance row (the autolens_inference row itself lands in 0C).
  • Add the designation decision kind to the decision index and template; no designation record is written here (phase 3).
  • Rework the setup browser for @3 snapshots: Class is the first selector with Optimal default, current-only view, History toggle for superseded rows, per-class Coverage panel, archived records counted never rendered. v2 snapshots keep the current browser.
  • Fix REFERENCE/README drift that still calls v1 live; add the @3, walltime and designation sections; one CHECKIN.md line that research lookups read class: optimal + status: current unless asked otherwise.
  • Fixtures and tests for every rule and for the board selector order.

Tier: judge — merge mode: human /prm

Detailed implementation plan

Affected Repositories

  • PyAutoPulse (primary, only)

Branch Survey

Repository Current Branch Dirty?
./organs/PyAutoPulse main (e3d411e) clean

Warning (not a conflict): unregistered worktree .worktrees/pulse-checkin-2026-10-17 on checkin/2026-10-17. Pulse PR #68 (docs/receipts) is open and unmerged: branch from main, rebase after it merges.

Suggested branch: feature/likelihood-lifecycle-p0-pulse (worktree ~/Code/PyAutoLabs-wt/likelihood-lifecycle-p0-pulse/)

Implementation Steps

  1. pulse/summary.py: SUPPORTED adds ("profiling-summary", 3) and ("walltime-validation", 1); validate() dispatches v3 to catalogue.validate(doc, version=3) and walltime to a new pulse/walltime.py; hardware()/group_label() learn host_class.
  2. pulse/catalogue.py: _record gains a v3 branch validating likelihood {dataset_type, variant}, instrument, regime, setup {backend, device, operator, precision, transformer, threads} + setup_id, host_class, model_variant, measurement, class, status, superseded_by, day_zero, figures[] {kind, path, sha256}, decision_id, release_pin, measured_at; enumerations: ten dataset_types (imaging, interferometer, datacube, point_source_image, point_source_source, multi_galaxy, group, cluster, multi_dataset, weak), host_class ∈ {hpc_ral_cpu, hpc_a100, local_laptop, local_gpu}, measurement ∈ {runtime, breakdown, gradient, compile, memory_host, memory_device, parity, walltime_validation}, class ∈ {optimal, secondary, experimental}, status ∈ {current, superseded, archived}. validate(version=3): key = (likelihood, instrument, setup_id, host_class, model_variant, measurement); refuse a second current per key; refuse class: optimal without decision_id; refuse status: current on local_laptop/local_gpu; refuse any archived record in records[] (they belong in archive_record_count); superseded requires superseded_by resolving in-document; planned_cells[] must carry class.
  3. pulse/ingest.py + check CLI: resolve every decision_id against decisions.yaml at ingest/check time (doc-only validator checks presence); receipt records schema and current/superseded/archived counts.
  4. pulse/walltime.py (new): records keyed by (likelihood, instrument, setup_id, host_class, model_variant, sampler, seed) with wall_s, likelihood_evals, implied_ms_per_eval, reference_runtime_record_id, predicted_ms_per_eval, ratio_measured_over_predicted, cores, parallel_efficiency, release_pin, slurm_job_id, max_log_likelihood, parity_status ∈ {match, diverged, not_assessed}; all producer-declared; duplicate keys refused.
  5. pulse/registry.py: supported_schema accepts walltime-validation@1; a registry fixture row exercises it (no live row here).
  6. pulse/decisions.py + decisions/README.md: optional kind: designation with likelihood, regime, hardware_class ∈ {cpu_many_core, gpu_a100, laptop}, optimal, secondary[], evidence[], rule, supersedes; template appended; validation of the designation fields when kind is set.
  7. pulse/setup_browser.js: for v3 snapshots the selector block (before today's Instrument selector, ~line 909) becomes Class (Optimal default | Secondary | Experimental | All) → Likelihood (dataset type → variant) → Instrument → Hardware class → Measurement; default filter class == optimal && status == current; History checkbox includes superseded; Coverage panel lists planned_cells with class: optimal lacking a current record; v2 eligibility/curated() heuristics (lines 150–198) bypassed for v3. v2 path unchanged.
  8. pulse/board.py + dashboard.md: per-instance status lines show current/superseded/archived counts; walltime axis renders as its own table under Profiling Results.
  9. Tests: fixtures tests/fixtures/setup_summary_v3.json, walltime_validation_v1.json, refusal fixtures (duplicate current, optimal without decision, laptop current, archived in records); new cases in test_catalogue.py, test_summary.py, test_registry.py, test_decisions.py, test_ingest.py, test_board.py, test_setup_browser.py (first <select> is Class with Optimal selected; superseded hidden until History).
  10. Docs: REFERENCE.md lines 9, 36, 42, 99–101 corrected; new @3, walltime and designation sections; README mirrors; CHECKIN.md lookup sentence.

Trade-off: one versioned reader with a v3 branch, not a parallel module, keeps the v2 instance untouched and lets 0B migrate lens by changing one registry pin.

Key Files

  • pulse/summary.py — SUPPORTED set and version dispatch
  • pulse/catalogue.py — v2/v3 record and catalogue validation
  • pulse/walltime.py — new walltime-validation@1 validator
  • pulse/decisions.py, decisions/README.md, decisions.yaml — designation kind
  • pulse/setup_browser.js, pulse/board.py — Class-first browser and board
  • REFERENCE.md, README.md, CHECKIN.md — contract docs
  • tests/ — fixtures and cases

Original Prompt

Click to expand starting prompt

Likelihood lifecycle phase 0A: profiling-summary@3 (class/status), walltime-validation@1, designation decisions and Class-first Pulse board

Type: feature
Target: PyAutoPulse
Repos:

  • PyAutoPulse
    Themes:
  • profiling
  • dashboard
    Difficulty: large
    Autonomy: supervised
    Priority: high
    Status: formalised
    Consequence: judge
    Witness: pyauto-pulse check accepts a fixture profiling-summary@3 catalogue in which every record carries class and status, refuses a second current record on one (likelihood, instrument, setup_id, host_class, model_variant, measurement) key, refuses class: optimal without a resolvable decision_id, accepts a walltime-validation@1 document, and the rendered board's first <select> is Class with Optimal selected and superseded rows hidden until the History toggle
    Review-minutes: 30
    Unattended: never
    Epic: likelihood-lifecycle
    Phase: 0
    Parent: draft/research/autolens_profiling/likelihood_lifecycle_epic.md
    Filed: 2026-10-21
    Note: Pulse PR docs: link /prm close-outs and refresh receipts after 2026-10-17 merges #68 (docs/receipts) is open and unmerged on 2026-10-21; branch from main and rebase after it merges — not a gate.

Phase 0 of the likelihood-lifecycle epic (ledger: draft/research/autolens_profiling/likelihood_lifecycle_epic.md; bodies and architecture: likelihood_lifecycle_epic_report.md §3.1, §3.2, §3.6, §3.8, §4 phase 0; evidence: likelihood_lifecycle_epic_surveys/02_pulse_insight_contracts.md). Phase 0 is the spine and has five sub-issues, one per repo, issued in order: 0A PyAutoPulse (this prompt) → 0B autolens_profiling → 0C autolens_inference → 0D PyAutoInsight → 0E PyAutoBrain. Each later sub-issue is filed as its own prompt (likelihood_lifecycle_p0_<target>.md, Phase: 0) once 0A is shipped, because 0B's catalogue builder and 0C's exporter emit the contract this issue defines. Library first: Pulse is the contract owner, so it goes first.

Rulings assumed (report §5; the human confirms or amends at plan approval)

  • R1 vocabulary and setup_id grammar as report §3.1 (<backend>_<device>_<operator>_<precision>[_<transformer>]; site/tier moves to host_class).
  • R2 day-zero history cut is label-only; nothing is deleted or re-run in phase 0.
  • R3 the provisional designations in report §3.3 are the phase-0 labels; phases 2–3 confirm.
  • R7 host_class is provenance, never identity; local_laptop / local_gpu rows can never be current.
  • R4/R5 enter as validator enumerations only (mixed precision allowed with class: experimental; compile is a measurement, not a class axis).

What (sub-issue 0A, PyAutoPulse only)

  1. profiling-summary@3 contract (pulse/summary.py SUPPORTED, pulse/catalogue.py validate, REFERENCE.md): additive on v2 with a version bump. Record fields: likelihood {dataset_type, variant}, instrument, regime, setup {backend, device, operator, precision, transformer, threads} + setup_id, host_class, model_variant, measurement, class ∈ {optimal, secondary, experimental}, status ∈ {current, superseded, archived}, superseded_by, day_zero, figures[] {kind, path, sha256}, decision_id, release_pin, measured_at. Enumerations for the ten dataset_types, host_class, measurement (adds gradient, parity, walltime_validation) and class/status live in REFERENCE.md and the validator. Rules: exactly one current per (likelihood, instrument, setup_id, host_class, model_variant, measurement); class: optimal ⇒ decision_id present and resolvable in decisions.yaml; host_class ∈ {local_laptop, local_gpu} ⇒ status != current; archived records are excluded from records[] and counted in archive_record_count; planned_cells carry class so coverage is reported per class. v2 instances keep validating unchanged (autofit_profiling stays @2).
  2. walltime-validation@1 evidence axis (new schema, producer autolens_inference): record keyed by (likelihood, instrument, setup_id, host_class, model_variant, sampler, seed) with wall_s, likelihood_evals, implied_ms_per_eval, reference_runtime_record_id, predicted_ms_per_eval, ratio_measured_over_predicted, cores, parallel_efficiency, release_pin, slurm_job_id, max_log_likelihood, parity_status ∈ {match, diverged, not_assessed}. All derived numbers producer-declared; Pulse validates and renders only. Registry accepts a second summary per instance (or a second instance kind) for this axis; REFERENCE.md states these are full-fit measurements.
  3. designation decision kind (decisions/README.md template, decisions.yaml, pulse/decisions.py): fields likelihood, regime, hardware_class ∈ {cpu_many_core, gpu_a100, laptop}, optimal: <setup_id>, secondary: [<setup_id>…], evidence: [record ids], rule (one sentence in dataset-property terms), supersedes. The validator's optimal⇒decision_id rule resolves against these rows. No designation records are written in 0A (phase 3 writes them); the kind, template and test fixture are.
  4. Board (pulse/setup_browser.js, pulse/board.py, dashboard.md): for @3 instances the selector order becomes Class (Optimal default | Secondary | Experimental | All) → Likelihood (dataset type → variant) → Instrument → Hardware class → Measurement; a History toggle reveals superseded rows; a Coverage panel lists Optimal cells that are not_measured; archived records are counted, never rendered. @2 instances keep today's browser. The check-in prose (CHECKIN.md) says research lookups read class: optimal + status: current unless asked otherwise.
  5. Doc drift: REFERENCE.md :9, :36, :42, :99-101 still describe v1 as live; fix alongside the @3 section. README mirrors.
  6. Tests: @3 fixture catalogue under tests/fixtures/ exercising every rule above (accept, duplicate-current refusal, optimal-without-decision refusal, laptop-current refusal, archived exclusion); walltime-validation@1 fixture; browser test asserting the first <select> is Class with Optimal default and that superseded rows are hidden until History.

Out of scope for 0A

The autolens_profiling catalogue builder and move-only PRs (0B), the autolens_inference exporter (0C), Insight's additive fields (0D), the Brain lifecycle template and conductor grid (0E), any designation decision record (phase 3), any re-measurement (phase 3) and the setup-baseline campaign closure (phase 3).

Boundaries (epic invariants)

Optimal is always a producer-declared class backed by a Pulse decision record; Pulse infers nothing, selects no fastest row and computes no ratio. Exactly one current per key, chosen by the producer under the day-zero rule, never by date. Sessions end at their deliverable; the human runs /prm.

🤖 Generated with Claude Code

Activity

  1. Jammy2211 commented on Oct 21, 2026

    @Jammy2211
    ContributorAuthor

    Library PR Created: #70 (branch feature/likelihood-lifecycle-p0-pulse, 5 commits; ruff clean, pytest 296 passed, node browser checks pass, pyauto-pulse check --offline OK).

    API change summary: profiling-summary@3 (class/status lifecycle records, one current per key, optimal ⇒ decision_id, laptop never current), walltime-validation@1, designation decision kind, Class-first current-only board with History and Coverage. v2 instances unchanged. Two shape decisions versus the plan (records carry metric/unit/values; v3 has no setups[]/selections[], planned_cells instead) are flagged in the PR body for /prm.

    No workspace impact (organ repo). Tier judge: the human runs /prm. Next phase step after merge: file the 0B autolens_profiling prompt.

  2. added a commit that references this issue on Oct 21, 2026
  3. Jammy2211 commented on Oct 21, 2026

    @Jammy2211
    ContributorAuthor

    Shipped: #70 merged (91870ad). Pulse now reads profiling-summary@3 (producer-declared class/status, one current per key, optimal ⇒ decision_id, laptop never current), walltime-validation@1, designation decisions, and renders a Class-first current-only browser with History and Coverage. v2 instances unchanged; the lens pin moves to @3 in phase 0B (autolens_profiling builder emits the v3 shape per REFERENCE.md). Closing; Mind record follows.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions