diff --git a/CHECKIN.md b/CHECKIN.md index 8dd34bb..5c049c6 100644 --- a/CHECKIN.md +++ b/CHECKIN.md @@ -13,7 +13,8 @@ campaigns and tasks. This static page does not run jobs or monitor them itself. 2. Read each campaign's current project wiki/ledger, results and issue/PR state. Evidence URLs pin the last reviewed source; inspect the current default branch for subsequent changes too. Consult Brain's profiling conductor for - scientific interpretation. With HPC access, use project pull/status commands; + scientific interpretation. Research lookups in Pulse catalogues read + `class: optimal` + `status: current` records unless asked otherwise. With HPC access, use project pull/status commands; otherwise report which jobs/results remain unverified. Never infer completion from an old job id or absent data. 3. Compare with the last check-in and report one table: changes, current state, diff --git a/README.md b/README.md index 0427199..ab17531 100644 --- a/README.md +++ b/README.md @@ -18,17 +18,25 @@ It owns: - the **instance registry** (`registry.yaml`) — which projects publish a profiling summary, where, under which contract version; each project is named by its PyAutoMind body-map identity, never a second repository catalogue; -- the versioned **`profiling-summary` read contract** — v2 is live at - `autolens_profiling/dashboard/catalogue.json`, documented in that repo's - [dashboard/README.md](https://github.com/PyAutoLabs/autolens_profiling/blob/main/dashboard/README.md); +- the versioned **`profiling-summary` read contract** — v2 is the live pin for + `autolens_profiling` and `autofit_profiling` (`dashboard/catalogue.json`, + documented in autolens_profiling's + [dashboard/README.md](https://github.com/PyAutoLabs/autolens_profiling/blob/main/dashboard/README.md)); + v3, the lifecycle catalogue in which the producer declares each record's + `class` (optimal/secondary/experimental) and `status` (current/superseded), + is supported for the next migration; v1 snapshots still validate; +- the **`walltime-validation` v1 contract** — full-fit sampler runs, registered + as their own instance row; +- the **decision index** (`decisions.yaml`), including `designation` entries + that `class: optimal` records must cite; - the **ingest receipts** (`receipts/`) — the one commit each project was read at, per render — and the last-good **snapshots** (`snapshots/`), shown as *cached* when a later fetch fails; - the cross-project **Pages board** (`dashboard.md`, `dashboard.html`, plus `state.json` for the Brain cockpit and `badge.json`). -It validates the exchange contract only. It never moves pins, combines unmatched -timings, applies the compile threshold to runtime, computes an ecosystem-wide +It validates the exchange contract only. It never moves pins, infers an optimal +setup, selects the fastest row, combines unmatched timings, applies the compile threshold to runtime, computes an ecosystem-wide speed score or issues a release-health verdict, and it never judges a timing — the Brain's profiling conductor does (`/profiling triage `). The project repos keep their producers, results, drift policy and their own Pages page. @@ -70,7 +78,10 @@ Pulse embeds the validated captured v2 index and loads a chosen evidence shard from that **same captured commit**, verifying its SHA-256 and setup identity. A failed or cached fetch remains visible; a missing deep link never substitutes a different run. Local reads cannot load remote shards as if they were published. -Inline v2 records and v1 diagnostics remain supported. No rendering action accepts +Inline v2 records and v1 diagnostics remain supported. A v3 catalogue instead +opens on Class (Optimal by default) → Likelihood → Instrument → Hardware class → +Measurement, shows current records unless History is ticked, counts archived +records without showing them and lists optimal cells not yet measured. No rendering action accepts a baseline, changes campaign state, runs profiling, or selects the fastest run. Temporal diagnostics remain collapsed while the front page focuses on setup choices. Existing archive evidence still requires scientific review. diff --git a/REFERENCE.md b/REFERENCE.md index 100572d..06abeab 100644 --- a/REFERENCE.md +++ b/REFERENCE.md @@ -6,8 +6,9 @@ active campaigns and Pulse-owned tasks above the detailed profiling evidence. prompts and their source commit. Board refreshes never change task status or stamp a check-in. Run `bin/pyauto-pulse board --offline` after ledger edits. -The registry schema, the `profiling-summary` v1 contract as this organ reads -it, and the formats of the receipts, snapshots, board markers and cockpit feed. +The registry schema, the `profiling-summary` contracts as this organ reads +them (v1 historical, v2 the live registry pin, v3 the lifecycle catalogue now +supported), the `walltime-validation` v1 evidence axis, and the formats of the receipts, snapshots, board markers and cockpit feed. Prose boundaries live in [AGENTS.md](AGENTS.md); the design is [profiling_inference_organs.md](https://github.com/PyAutoLabs/PyAutoBrain/blob/main/docs/research/profiling_inference_organs.md). @@ -20,28 +21,33 @@ One row per instance under `instances:`. Unknown fields are refused. | `instance` | yes | Stable key within the organ, `^[a-z][a-z0-9_-]*$`, unique. | | `repo` | yes | A PyAutoMind body-map identity (`repos.yaml` key). `path` and `github` are read from the body map at load time and **may not** be written in the row. | | `summary_path` | yes | Repository-relative path of the producer's summary (no `/`-prefix, `..`, URL). | -| `supported_schema` | yes | `@` the reader expects, e.g. `profiling-summary@1`. | +| `supported_schema` | yes | `@` the reader expects and implements: `profiling-summary@1`, `@2`, `@3` or `walltime-validation@1`. Any other pin is refused. | | `dashboard_url` | yes | The project's own human-facing page (`https://`). | | `library_refs` | no | Body-map identities the project exercises (each must be in the body map). | | `cortex_project` | no | An existing Cortex key, or `null`. | Further rules: the same `(repo, summary_path)` may be registered once (disjoint -scopes; no inflated coverage). Every ingest reads branch `main`. +scopes; no inflated coverage). Every ingest reads branch `main`. A project that +publishes wall-time validation registers it as a second instance row with its +own `summary_path` and `supported_schema: walltime-validation@1`. Body map resolution order: `--mind PATH` (must hold `repos.yaml`, never a silent fall-through), `$PYAUTO_MIND`, `$PYAUTO_ROOT/organs/PyAutoMind`, `$PYAUTO_ROOT/PyAutoMind`, then beside this organ (grouped `organs/` or flat task-bundle layout). No body map → an error, never a guess. -## `profiling-summary` v1, as read here +## `profiling-summary` v1, as read here (historical) The producer documents the contract ([autolens_profiling `dashboard/README.md`](https://github.com/PyAutoLabs/autolens_profiling/blob/main/dashboard/README.md)); -this is what `pulse/summary.py` enforces. Additive fields are ignored. +this is what `pulse/summary.py` enforces. Additive fields are ignored. No live +registry row pins v1 today; saved v1 snapshots and fixtures still validate. -**Support.** `(schema, version)` must be `("profiling-summary", 1)` *and* match the +**Support.** `(schema, version)` must be one this reader implements +(`profiling-summary` 1, 2 or 3, `walltime-validation` 1) *and* match the registry row's `supported_schema`; otherwise the outcome is **unsupported** and -nothing else is validated. +nothing else is validated. The envelope, record and comparison rules below are +the v1 shape; v2 and v3 reuse the envelope. **Envelope** — required: `schema, version, project, scope, generated_at, evidence_updated_at, producer_revision, comparison_policy, coverage, records, @@ -96,9 +102,9 @@ precision`, in producer order (never sorted by ratio). ## `profiling-summary` v2: setup catalogue -The v2 reader lands before the project producer migrates. Registry entries -still pin an exact version; supporting v2 does **not** silently upgrade the live -`profiling-summary@1` registry or reinterpret a saved v1 snapshot. The full +V2 is the live registry pin for both instances (`lens`, `fit`). Registry entries +pin an exact version; supporting a newer version never silently upgrades a row +or reinterprets a saved snapshot of an older one. The full [synthetic fixture](tests/fixtures/setup_summary_v2.json) is an executable producer example. Validation is stdlib-only in `pulse/catalogue.py`, through `pulse.summary.classify`; it never opens evidence files or judges science. @@ -199,13 +205,126 @@ a recommended solver is correct. Consumers must inspect the cited records' hardware/method and applicability, and must not extrapolate to a different setup or interpret a per-likelihood measurement as a total-fit prediction. +## `profiling-summary` v3: lifecycle catalogue + +V3 is v2's envelope with every record declaring where it sits in the +likelihood lifecycle. The [synthetic fixture](tests/fixtures/setup_summary_v3.json) +is an executable producer example; validation is stdlib-only in +`pulse/catalogue.py`. The producer declares `class` and `status`; Pulse checks +that the declarations are consistent and renders them. It infers no optimal +setup, selects no fastest row and computes no ratio. + +**Envelope.** The common v1 envelope (as v2), `comparisons: []`, plus: + +- `records`: list (empty is valid; an empty catalogue is no evidence). +- `planned_cells` (optional list): the cells the producer intends to measure, + each with the key fields below and `class`. Duplicate cells are refused. +- `archive_record_count`: non-negative integer. Archived records are moved out + of `records[]` by the producer and only counted here. +- `hazards`, `recommendations` (optional): the v2 shape; `applies_to.setup_ids` + name canonical v3 `setup_id`s present in `records`. +- `coverage.expected.cells` equals the `planned_cells` count; + `coverage.observed.records` equals the record count, and `current` / + `superseded`, when present, match the declared statuses. + +**Vocabulary** (the validator's enumerations): + +| Field | Values | +|---|---| +| `likelihood.dataset_type` | `imaging`, `interferometer`, `datacube`, `point_source_image`, `point_source_source`, `multi_galaxy`, `group`, `cluster`, `multi_dataset`, `weak` | +| `likelihood.variant` | non-empty string, e.g. `delaunay`, `mge`, `source_plane` | +| `setup.backend` | `numba`, `jax` | +| `setup.device` | `cpu`, `gpu` | +| `setup.operator` | `dense`, `sparse`, `na` | +| `setup.precision` | `fp64`, `mixed` (mixed is expected to be `class: experimental`) | +| `setup.transformer` | `dft`, `nufft`, `wtilde`, `streaming`, `na` | +| `setup.threads` | positive integer or `null` | +| `setup_id` | `___`, plus `_` when the transformer is not `na`, e.g. `jax_gpu_sparse_fp64_wtilde` | +| `host_class` | `hpc_ral_cpu`, `hpc_a100`, `local_laptop`, `local_gpu` (provenance, never identity) | +| `measurement` | `runtime`, `breakdown`, `gradient`, `compile`, `memory_host`, `memory_device`, `parity`, `walltime_validation` | +| `class` | `optimal`, `secondary`, `experimental` | +| `status` | `current`, `superseded`, `archived` | + +**Records.** Each record has `id` (unique), `run_id`, `likelihood +{dataset_type, variant}`, `instrument`, `regime` (the named size bin, e.g. +`standard`), `setup {backend, device, operator, precision, transformer, +threads}` and the matching `setup_id`, `host_class`, `model_variant` (e.g. +`default`, `light_fixed`), `measurement`, `metric`, `unit`, `values`, `class`, +`status`, `superseded_by`, `day_zero`, `figures`, `decision_id`, +`release_pin`, `measured_at`, and v2's `identity`, `method`, `provenance`, +`validation` and `evidence` with the same rules. + +- `unit` belongs to the measurement: seconds for the time measurements, a + memory unit for `memory_host`/`memory_device`, `dimensionless` for `parity`. +- `values` is a non-empty object of finite, non-negative numbers that contains + the headline `metric` (a breakdown carries its components beside its total). + Missing cells are `planned_cells`, never zero-valued records. +- `superseded_by` is `null` unless `status: superseded`, when it names another + record in the same document. +- `day_zero`, `release_pin`, `decision_id`: string or `null`. `measured_at`: + ISO-8601 UTC. +- `figures`: list of `{kind, path, sha256}` with safe repo-relative paths and + 64-hex digests (may be empty). + +**Rules.** + +- Exactly one `current` record per key (`likelihood`, `instrument`, + `setup_id`, `host_class`, `model_variant`, `measurement`). The producer + chooses which one under the day-zero rule; a second `current` is refused. +- `class: optimal` requires a `decision_id`. `pyauto-pulse check` and ingest + resolve every `decision_id` against this organ's `decisions.yaml` + (the `designation` decisions below); an unresolved id makes the capture + **invalid**. +- `host_class` `local_laptop` or `local_gpu` is never `current`; such rows may + be kept as `superseded` provenance. +- An `archived` record inside `records[]` is refused. + +**Board.** The setup browser's first selector is Class (Optimal by default, +then Secondary, Experimental, All), then Likelihood (dataset type, variant), +Instrument, Hardware class and Measurement. Only `current` rows show until +History is ticked, which adds `superseded` rows. Archived records appear only +as a count. A Coverage panel lists optimal `planned_cells` with no current +record. Research lookups read `class: optimal` + `status: current` unless asked +otherwise. The v2 eligibility heuristics never apply to a v3 catalogue. + +## `walltime-validation` v1: full-fit evidence axis + +A second evidence axis, produced by autolens_inference and registered as its +own instance row. These are **full-fit** measurements: one sampler run per +record. A per-likelihood `runtime` record is never read as a total-fit +prediction, and a full-fit wall time is never read as a per-likelihood cost. +The [fixture](tests/fixtures/walltime_validation_v1.json) is an executable +example; validation is stdlib-only in `pulse/walltime.py`. + +**Envelope** — required: `schema` (`walltime-validation`), `version` (1), +`project` (non-empty), `generated_at` (ISO-8601 UTC), `producer_revision` +(40-hex or `null`), `records`; optional `limitations` (non-empty strings). + +**Records** — each with a unique `id` and one record per key (`likelihood +{dataset_type, variant}`, `instrument`, `setup_id`, `host_class`, +`model_variant`, `sampler`, `seed`); a duplicate key is refused. Fields: + +- `wall_s`, `implied_ms_per_eval`, `predicted_ms_per_eval`, + `ratio_measured_over_predicted`, `parallel_efficiency` (relative to the + 1-core row of the same key), `max_log_likelihood`: finite number or `null`. +- `likelihood_evals`, `cores`: non-negative integer or `null`; `seed`: + non-negative integer. +- `reference_runtime_record_id` (the day-zero `runtime` record compared with), + `release_pin`, `slurm_job_id`: string or `null`. +- `parity_status` ∈ `match | diverged | not_assessed`. + +Every derived number is computed and declared by the producer. Pulse renders +the records as their own table under Profiling Results and computes nothing; +the instance has no qualification flag, so it raises no "unqualified" item. + ## Ingest 1. `GET https://api.github.com/repos///commits/main` → one 40-hex SHA (stdlib `urllib`, 20 s timeout, `User-Agent`, `Authorization: Bearer $GH_TOKEN` on API calls only, when set). 2. `GET https://raw.githubusercontent.com////`. -3. Validate; write the receipt; on `ok` also write the snapshot. +3. Validate; for a `profiling-summary@3` document resolve every `decision_id` + against `decisions.yaml`; write the receipt; on `ok` also write the snapshot. `--offline` reads the committed receipt + snapshot and writes nothing. `--from [INSTANCE=]PATH` reads a local checkout (or the file), reports its `HEAD` and @@ -224,6 +343,10 @@ writes **no** receipt or snapshot. } ``` +A `profiling-summary@3` receipt also carries `"lifecycle": {"current": N, +"superseded": N, "archived": N}` (the producer's declared counts); receipts of +other schemas keep the shape above. + ### Snapshot — `snapshots/.json` `{instance, repo, github, summary_path, commit, fetched_at, summary}` — the last @@ -247,6 +370,9 @@ nothing; a new project commit moves both. `/profiling triage `. Freshness without a policy is shown as "freshness policy unspecified · age N d at fetch " (the HTML adds a live "N d today"); a passed `valid_until` reads "stale". +- A v3 instance's status lines add **Lifecycle** (`N current · N superseded · + N archived`), and its census item carries the same counts. A + `walltime-validation` instance renders as its own table. - Instance marker (what `check` compares to the receipt): ``. - `badge.json`: shields endpoint, label `pulse`, message = the state headline. @@ -336,3 +462,16 @@ Results in HTML and Markdown. Both boards can link to one canonical record. The reader validates IDs, dates, GitHub destinations and owned record metadata; it never makes a scientific decision. Remote record existence is checked when publishing the cross-dashboard link, not by offline rendering. + +### Designation decisions + +An entry with `kind: designation` records the human's choice of setups for one +likelihood, regime and hardware class. Its extra fields are validated only when +the kind is set: `likelihood` (`/`), `regime`, +`hardware_class` ∈ `cpu_many_core | gpu_a100 | laptop`, `optimal` (one +canonical `setup_id`), `secondary` (list of distinct canonical `setup_id`s), +`evidence` (non-empty list of record ids), `rule` (one sentence in +dataset-property terms) and `supersedes` (a decision id in the index, or +`null`). `profiling-summary@3` records with `class: optimal` cite these entries +in `decision_id`. The template is in [decisions/README.md](decisions/README.md); +no designation is recorded yet. diff --git a/dashboard.html b/dashboard.html index 0dce4f9..cbeefb2 100644 --- a/dashboard.html +++ b/dashboard.html @@ -626,6 +626,31 @@ .timing-diagnostics { margin-top: 0.8rem; } +/* profiling-summary@3: Class-first lifecycle browser. */ +.lifecycle-selectors { + grid-template-columns: repeat(auto-fit, minmax(9rem, 1fr)); +} +.lifecycle-selectors .history-toggle { + align-self: end; + font-weight: 500; +} +.lifecycle-table tr[hidden], +.lifecycle-browser [hidden] { + display: none; +} +.lifecycle-table td, +.lifecycle-table th { + font-size: 0.8rem; + padding: 0.35rem 0.45rem; + vertical-align: top; +} +.lifecycle-coverage { + margin: 1rem 0; +} +.lifecycle-coverage summary { + cursor: pointer; + font-weight: 600; +}

PyAutoPulseProfiling

Measure. Trace. Compare.

Check your Pulse

Read the prompt