Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 11 additions & 11 deletions campaigns.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -77,10 +77,10 @@ campaigns:
title: Measurement reliability and profiling tools
status: active
updated: '2026-10-08'
next: 'timing_noise_audit phase 1 merged (autolens_profiling#402): inventory of 31 timing assertions/gates, 10 UNSAFE-SILENT
including the dashboard qualify() flag Pulse relies on (pulse/catalogue.py requires qualified: true). Fix phase (2)
will change the profiling-summary contract - Pulse must coordinate. New campaign page wiki/campaigns/measurement_tools.md.
Other tasks unchanged and unissued.'
next: 'timing_noise_audit fix phases (1) and (2) done (autolens_profiling#404 merged; #405 open, awaiting /prm). Correction
2026-10-08: the live Pulse lens feed is dashboard/catalogue.json (profiling-summary@2, registry switched 2026-10-05
in 3c7bed2), whose producer hard-codes qualified: false, so the v1 qualify() gap never reached Pulse live and fix (2)
needed no Pulse change. Next: fix phase (3) INCONCLUSIVE for go/lever rules. Other tasks unchanged and unissued.'
evidence: https://github.com/PyAutoLabs/autolens_profiling/blob/main/wiki/campaigns/measurement_tools.md
- id: autofit
title: PyAutoFit profiling bootstrap
Expand Down Expand Up @@ -139,14 +139,14 @@ tasks:
priority: high
path: tasks/timing_noise_audit.md
updated: '2026-10-08'
next: 'Phase 2 MERGED 2026-10-08 as autolens_profiling#404: one abba_overhead_verdict (excess ms vs the 12 ms budget,
t-interval, PASS/FAIL/FAIL_GROSS/INCONCLUSIVE) serves the CI test and the fixed-light numba cell; promotion requires
PASS; 6/17 published RAL rows re-judge INCONCLUSIVE (recorded, no JSON rewritten). Human decision: the ~16 ms CI
fixture cannot resolve 12 ms, so the CI test is the coverage + gross-breakage guard. Inventory now 13 SOUND / 9 FRAGILE /
9 UNSAFE-SILENT. Next: fix phase (2) dashboard qualify()/drift wording - a profiling-summary contract change that needs
a paired PyAutoPulse reader change (plan as a two-repo task); then (3) INCONCLUSIVE for go/lever rules.'
next: 'Fix phase (2) PR OPEN 2026-10-08 as autolens_profiling#405 (awaiting /prm; Heart YELLOW reasons human-acknowledged):
dashboard qualify() leaves laptop rows, rows without a load average and hpc rows without a host unqualified with a
reason; a within-2x comparison is flat only when both endpoints carry a repeat summary (none do yet), else insufficient;
drifted/improved keep status with a single-sample caveat (human decision 2026-10-08). Effect on summary.json: qualified
2 -> 2, flat 4 -> 0, insufficient 135 -> 139. Not a Pulse contract change (live feed is v2 catalogue.json, qualified
hard-false). Next after merge: fix phase (3) INCONCLUSIVE for go/lever rules, tie sets instead of argmax.'
issue: https://github.com/PyAutoLabs/autolens_profiling/issues/362
evidence: https://github.com/PyAutoLabs/autolens_profiling/pull/404
evidence: https://github.com/PyAutoLabs/autolens_profiling/pull/405
- id: mge_likelihood_breakdown_steps_are_cumulative_an
campaign: measurement-tools
title: MGE likelihood_breakdown steps are cumulative and `linear_gaussians` is reported as 0
Expand Down
2 changes: 1 addition & 1 deletion dashboard.html

Large diffs are not rendered by default.

6 changes: 3 additions & 3 deletions dashboard.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# PyAutoPulse — profiling dashboard

<!-- pulse-campaigns:af6f0d180a436637f5db5be8ad42e33eaec6af7c87278d4294ed46e61fe379d4 -->
<!-- pulse-campaigns:652e6582f3ead66f68ae1120a79f68f1c18219dbc4ffee221291672133f5fc79 -->
## Profiling Check In

<details><summary>Full check-in prompt</summary>
Expand Down Expand Up @@ -133,13 +133,13 @@ Reviewed 2026-10-04. No new results since the pin. Over-sampling phase 1: add --

<details><summary>Measurement reliability and profiling tools — active</summary>

Reviewed 2026-10-08. timing_noise_audit phase 1 merged (autolens_profiling#402): inventory of 31 timing assertions/gates, 10 UNSAFE-SILENT including the dashboard qualify() flag Pulse relies on (pulse/catalogue.py requires qualified: true). Fix phase (2) will change the profiling-summary contract - Pulse must coordinate. New campaign page wiki/campaigns/measurement_tools.md. Other tasks unchanged and unissued.
Reviewed 2026-10-08. timing_noise_audit fix phases (1) and (2) done (autolens_profiling#404 merged; #405 open, awaiting /prm). Correction 2026-10-08: the live Pulse lens feed is dashboard/catalogue.json (profiling-summary@2, registry switched 2026-10-05 in 3c7bed2), whose producer hard-codes qualified: false, so the v1 qualify() gap never reached Pulse live and fix (2) needed no Pulse change. Next: fix phase (3) INCONCLUSIVE for go/lever rules. Other tasks unchanged and unissued.

[Campaign evidence](https://github.com/PyAutoLabs/autolens_profiling/blob/main/wiki/campaigns/measurement_tools.md)

### Active tasks

- [Audit timing tests and profiling gates for measurement noise](https://github.com/PyAutoLabs/PyAutoPulse/blob/main/tasks/timing_noise_audit.md) — active: Phase 2 MERGED 2026-10-08 as autolens_profiling#404: one abba_overhead_verdict (excess ms vs the 12 ms budget, t-interval, PASS/FAIL/FAIL_GROSS/INCONCLUSIVE) serves the CI test and the fixed-light numba cell; promotion requires PASS; 6/17 published RAL rows re-judge INCONCLUSIVE (recorded, no JSON rewritten). Human decision: the ~16 ms CI fixture cannot resolve 12 ms, so the CI test is the coverage + gross-breakage guard. Inventory now 13 SOUND / 9 FRAGILE / 9 UNSAFE-SILENT. Next: fix phase (2) dashboard qualify()/drift wording - a profiling-summary contract change that needs a paired PyAutoPulse reader change (plan as a two-repo task); then (3) INCONCLUSIVE for go/lever rules.
- [Audit timing tests and profiling gates for measurement noise](https://github.com/PyAutoLabs/PyAutoPulse/blob/main/tasks/timing_noise_audit.md) — active: Fix phase (2) PR OPEN 2026-10-08 as autolens_profiling#405 (awaiting /prm; Heart YELLOW reasons human-acknowledged): dashboard qualify() leaves laptop rows, rows without a load average and hpc rows without a host unqualified with a reason; a within-2x comparison is flat only when both endpoints carry a repeat summary (none do yet), else insufficient; drifted/improved keep status with a single-sample caveat (human decision 2026-10-08). Effect on summary.json: qualified 2 -&gt; 2, flat 4 -&gt; 0, insufficient 135 -&gt; 139. Not a Pulse contract change (live feed is v2 catalogue.json, qualified hard-false). Next after merge: fix phase (3) INCONCLUSIVE for go/lever rules, tie sets instead of argmax.
- [MGE likelihood_breakdown steps are cumulative and `linear_gaussians` is reported as 0](https://github.com/PyAutoLabs/PyAutoPulse/blob/main/tasks/mge_likelihood_breakdown_steps_are_cumulative_an.md) — ready: Unchanged and unissued (no GitHub refs, 2026-10-04). Select one bounded step when measurement-tools is prioritised.
- [A gradient-cost probe: forward vs `value_and_grad` ms/eval and a strict FD check, on any registry cell](https://github.com/PyAutoLabs/PyAutoPulse/blob/main/tasks/gradient_cost_probe.md) — ready: Unissued. New context 2026-10-07: Heart unit-test timing flagged PyAutoLens forward-gradient tests +150%/+96% after PyAutoLens#768 rewrote that test file; not a controlled comparison, and this probe is the right instrument. Select one bounded step when prioritised.
- [Numba breakdown harness: perturb the instance so the operated-matrix memo cannot hide a step](https://github.com/PyAutoLabs/PyAutoPulse/blob/main/tasks/numba_breakdown_harness_memo_blind.md) — ready: Unchanged and unissued (PyAutoArray#496 closed 2026-08-27). Select one bounded step when prioritised.
Expand Down
4 changes: 4 additions & 0 deletions tasks/setup_baseline_campaign.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,3 +75,7 @@ the later collection and review deliverables.
- Specification merged: https://github.com/PyAutoLabs/autolens_profiling/pull/385 (2026-10-06; issue #384 closed 2026-10-06; PyAutoPulse#17 closed 2026-10-06). `baseline/campaign.json` is status draft, version 1, with `baseline/README.md` readiness contract (https://github.com/PyAutoLabs/autolens_profiling/blob/719459fefb40c7e5be157f652818dbf923830557/baseline/campaign.json, https://github.com/PyAutoLabs/autolens_profiling/blob/719459fefb40c7e5be157f652818dbf923830557/wiki/campaigns/setup_baseline.md).
- Six unknowns recorded verbatim: no frozen revisions/env lock; no collection authorization/window; hardware/affinity identity unselected; each cell needs complete settings, input hashes, witness + tolerances; compile per-cell builder unavailable; proposed fp64/mixed coverage is not evidence of support.
- Status stays `needs-decision`: no collection or acceptance authorized; the check-in authorizes none.

## Note 2026-10-08 (from timing_noise_audit fix phase 2)

- When the frozen specification names a reference host and the v2 catalogue starts writing `qualified: true`, reuse autolens_profiling's `build_dashboard.is_reference_host_class` rule (laptop rows never qualify; a missing load average or host is unqualified with a reason) so the v1 and v2 qualification cannot drift apart. Source: autolens_profiling#405.
11 changes: 11 additions & 0 deletions tasks/timing_noise_audit.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,17 @@ The original task below is retained verbatim. Current scheduling state lives in
- Human decision: on the ~15–17 ms CI fixture the shared 12 ms budget can only fail through the gross guard; the CI test is documented as the coverage + cached-site-count + gross-breakage guard, and the ms budget is judged on the 225–415 ms production rows.
- Next: fix phase (2) — `build_dashboard.qualify`/`drift` wording. This changes the `profiling-summary` contract Pulse ingests; plan it as a two-repo task (autolens_profiling producer + PyAutoPulse reader/fixtures) before any producer change lands.


## Execution 2026-10-08 — phase 3 (fix phase 2: dashboard qualification + drift wording)

- Human: "continue … --auto" after the plan was presented in the Pulse chat; issue #362 reused (plan comment https://github.com/PyAutoLabs/autolens_profiling/issues/362#issuecomment-6067418030). Mind `active/timing_noise_audit_phase3_qualify_drift.md` (f27988f1); branch `feature/timing-noise-audit-p3-qualify-drift`; Opus executes.
- **Correction to the phase 1/2 notes above:** Pulse's live `lens` registry reads `dashboard/catalogue.json` (`profiling-summary@2`) since 2026-10-05 (3c7bed2). That producer (`build_catalogue.py`) hard-codes `qualified: false` on every record, and its comparisons list is empty. `build_dashboard.qualify()`/`drift()` feed only the v1 `summary.json` (project dashboard and badge). The P6 gap therefore never reached Pulse live, and fix (2) is a single-repo change: no Pulse reader change was needed.
- Human decision 2026-10-08: `drifted` / `improved` keep their status and carry a "single-sample endpoint(s)" caveat; only the within-band case changes (`flat` only with repeat summaries on both endpoints, else `insufficient`).
- Heart readiness was YELLOW (8 organism-wide manifest-drift reasons + "no rehearsal for current source"); the human acknowledged that list for this PR on 2026-10-08.
- PR [autolens_profiling#405](https://github.com/PyAutoLabs/autolens_profiling/pull/405) opened 2026-10-08. Verified on the branch: 1169 passed / 6 skipped; independent review CLEAN; Pulse v1 `validate()` returns [] before and after. `summary.json` effect: 159 records, qualified 2 → 2 (no record changed), `flat` 4 → 0, `insufficient` 135 → 139, `improved` 6 → 6 with the caveat. Audit counts now 15 SOUND / 8 FRAGILE / 8 UNSAFE-SILENT.
- Carry-forward: when the v2 catalogue starts qualifying records (setup-baseline reference-host decision), it must reuse the producer's `is_reference_host_class` rule rather than a separate one.
- Next: /prm the PR (judge tier, human merge); then fix phase (3) — INCONCLUSIVE for go / lever rules, tie sets instead of argmax.

---

# Audit timing tests and profiling gates for measurement noise
Expand Down
Loading