Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 7 additions & 9 deletions campaigns.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -139,16 +139,14 @@ tasks:
priority: high
path: tasks/timing_noise_audit.md
updated: '2026-10-08'
next: 'Phase 1 (inventory) MERGED 2026-10-08 as autolens_profiling#402 (main un-reddened first by #403, a Brain theme
drift); Mind record complete/2026/10/timing-noise-audit-p1-inventory.md. 31 gates: 11 SOUND / 10 FRAGILE / 10 UNSAFE-SILENT;
worst is dashboard qualify() (laptop / no-loadavg / no-host rows qualify, and Pulse accepts evidence on that flag).
Phase 2 STARTED 2026-10-08 (human: "do the next step"): fix (1) shared ABBA overhead verdict for the #361 test and the
fixed-light numba cell, budget in ms of excess, t-interval vs budget, visible INCONCLUSIVE; Mind
active/timing_noise_audit_phase2_overhead_verdict.md, worktree timing-noise-audit-p2-overhead-verdict, Opus executing;
superseded "relax it a bit" draft retired. Next: /prm on its PR; then fix (2) dashboard qualification + drift wording
(profiling-summary contract change, Pulse coordination).'
next: 'Phase 2 MERGED 2026-10-08 as autolens_profiling#404: one abba_overhead_verdict (excess ms vs the 12 ms budget,
t-interval, PASS/FAIL/FAIL_GROSS/INCONCLUSIVE) serves the CI test and the fixed-light numba cell; promotion requires
PASS; 6/17 published RAL rows re-judge INCONCLUSIVE (recorded, no JSON rewritten). Human decision: the ~16 ms CI
fixture cannot resolve 12 ms, so the CI test is the coverage + gross-breakage guard. Inventory now 13 SOUND / 9 FRAGILE /
9 UNSAFE-SILENT. Next: fix phase (2) dashboard qualify()/drift wording - a profiling-summary contract change that needs
a paired PyAutoPulse reader change (plan as a two-repo task); then (3) INCONCLUSIVE for go/lever rules.'
issue: https://github.com/PyAutoLabs/autolens_profiling/issues/362
evidence: https://github.com/PyAutoLabs/autolens_profiling/blob/main/results/notes/timing_noise_audit_2026_10.md
evidence: https://github.com/PyAutoLabs/autolens_profiling/pull/404
- id: mge_likelihood_breakdown_steps_are_cumulative_an
campaign: measurement-tools
title: MGE likelihood_breakdown steps are cumulative and `linear_gaussians` is reported as 0
Expand Down
2 changes: 1 addition & 1 deletion dashboard.html

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions dashboard.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# PyAutoPulse — profiling dashboard

<!-- pulse-campaigns:9bb9ffb230c27eaee9126701594f90c5e513d7e288bcaaab839cdbce7921b822 -->
<!-- pulse-campaigns:af6f0d180a436637f5db5be8ad42e33eaec6af7c87278d4294ed46e61fe379d4 -->
## Profiling Check In

<details><summary>Full check-in prompt</summary>
Expand Down Expand Up @@ -139,7 +139,7 @@ Reviewed 2026-10-08. timing_noise_audit phase 1 merged (autolens_profiling#402):

### Active tasks

- [Audit timing tests and profiling gates for measurement noise](https://github.com/PyAutoLabs/PyAutoPulse/blob/main/tasks/timing_noise_audit.md) — active: Phase 1 (inventory) MERGED 2026-10-08 as autolens_profiling#402 (main un-reddened first by #403, a Brain theme drift); Mind record complete/2026/10/timing-noise-audit-p1-inventory.md. 31 gates: 11 SOUND / 10 FRAGILE / 10 UNSAFE-SILENT; worst is dashboard qualify() (laptop / no-loadavg / no-host rows qualify, and Pulse accepts evidence on that flag). Phase 2 STARTED 2026-10-08 (human: &quot;do the next step&quot;): fix (1) shared ABBA overhead verdict for the #361 test and the fixed-light numba cell, budget in ms of excess, t-interval vs budget, visible INCONCLUSIVE; Mind active/timing_noise_audit_phase2_overhead_verdict.md, worktree timing-noise-audit-p2-overhead-verdict, Opus executing; superseded &quot;relax it a bit&quot; draft retired. Next: /prm on its PR; then fix (2) dashboard qualification + drift wording (profiling-summary contract change, Pulse coordination).
- [Audit timing tests and profiling gates for measurement noise](https://github.com/PyAutoLabs/PyAutoPulse/blob/main/tasks/timing_noise_audit.md) — active: Phase 2 MERGED 2026-10-08 as autolens_profiling#404: one abba_overhead_verdict (excess ms vs the 12 ms budget, t-interval, PASS/FAIL/FAIL_GROSS/INCONCLUSIVE) serves the CI test and the fixed-light numba cell; promotion requires PASS; 6/17 published RAL rows re-judge INCONCLUSIVE (recorded, no JSON rewritten). Human decision: the ~16 ms CI fixture cannot resolve 12 ms, so the CI test is the coverage + gross-breakage guard. Inventory now 13 SOUND / 9 FRAGILE / 9 UNSAFE-SILENT. Next: fix phase (2) dashboard qualify()/drift wording - a profiling-summary contract change that needs a paired PyAutoPulse reader change (plan as a two-repo task); then (3) INCONCLUSIVE for go/lever rules.
- [MGE likelihood_breakdown steps are cumulative and `linear_gaussians` is reported as 0](https://github.com/PyAutoLabs/PyAutoPulse/blob/main/tasks/mge_likelihood_breakdown_steps_are_cumulative_an.md) — ready: Unchanged and unissued (no GitHub refs, 2026-10-04). Select one bounded step when measurement-tools is prioritised.
- [A gradient-cost probe: forward vs `value_and_grad` ms/eval and a strict FD check, on any registry cell](https://github.com/PyAutoLabs/PyAutoPulse/blob/main/tasks/gradient_cost_probe.md) — ready: Unissued. New context 2026-10-07: Heart unit-test timing flagged PyAutoLens forward-gradient tests +150%/+96% after PyAutoLens#768 rewrote that test file; not a controlled comparison, and this probe is the right instrument. Select one bounded step when prioritised.
- [Numba breakdown harness: perturb the instance so the operated-matrix memo cannot hide a step](https://github.com/PyAutoLabs/PyAutoPulse/blob/main/tasks/numba_breakdown_harness_memo_blind.md) — ready: Unchanged and unissued (PyAutoArray#496 closed 2026-08-27). Select one bounded step when prioritised.
Expand Down
9 changes: 9 additions & 0 deletions tasks/timing_noise_audit.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,15 @@ The original task below is retained verbatim. Current scheduling state lives in
- Human: "do the next step" → fix phase (1) of the audit note, plan approved in chat; issue #362 reused (plan comment posted); Mind `active/timing_noise_audit_phase2_overhead_verdict.md` (3a3d4fc6); worktree `timing-noise-audit-p2-overhead-verdict`; Opus executes. The superseded Mind draft `call_accounting_ci_timing_threshold.md` ("increase it a bit so merge goes through") was retired on the human's approval — the guard is fixed, not relaxed.
- Scope: one `abba_overhead_verdict` used by both the CI test and `fixed_light_numba.py`; budget 12 ms of excess over the clean call; one-sided small-sample t-interval of the excess vs budget → PASS / FAIL / FAIL_GROSS / INCONCLUSIVE; n < 3 and below-1 ratios INCONCLUSIVE; cell raises only on FAIL/FAIL_GROSS and keeps the JSON otherwise; promotion requires PASS; the test warns visibly on INCONCLUSIVE; the 1.031 ratio constant goes. No budget raised.


### Phase 2 shipped 2026-10-08

- [autolens_profiling#404](https://github.com/PyAutoLabs/autolens_profiling/pull/404) merged 2026-10-08 (lint green after a catalogue/dashboard stamp refresh). Mind record `complete/2026/10/timing-noise-audit-p2-overhead-verdict.md`. Issue #362 stays open.
- Rule as built (`scripts/misc/likelihood_breakdown/overhead_verdict.py`): excess ms = (ratio − 1) × clean mean; one-sided 95 % t-bounds; invalid → ValueError; mean ratio > 1.5 → FAIL_GROSS; n < 3 → INCONCLUSIVE; upper bound < 0 → INCONCLUSIVE (host-noise signature); upper ≤ budget → PASS; lower > budget → FAIL; else INCONCLUSIVE. Cell raises only on FAIL/FAIL_GROSS; promotion requires PASS on both arms; CI INCONCLUSIVE is a visible warning; no budget raised.
- Exposed facts (no JSON rewritten): 6 of 17 published RAL rows, including the pinned 413 ms row ([−7.3, +19.4] ms), are INCONCLUSIVE on their own blocks rather than PASS.
- Human decision: on the ~15–17 ms CI fixture the shared 12 ms budget can only fail through the gross guard; the CI test is documented as the coverage + cached-site-count + gross-breakage guard, and the ms budget is judged on the 225–415 ms production rows.
- Next: fix phase (2) — `build_dashboard.qualify`/`drift` wording. This changes the `profiling-summary` contract Pulse ingests; plan it as a two-repo task (autolens_profiling producer + PyAutoPulse reader/fixtures) before any producer change lands.

---

# Audit timing tests and profiling gates for measurement noise
Expand Down
Loading