Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 11 additions & 11 deletions campaigns.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -77,10 +77,10 @@ campaigns:
title: Measurement reliability and profiling tools
status: active
updated: '2026-10-08'
next: 'timing_noise_audit fix phases (1) and (2) done (autolens_profiling#404 merged; #405 open, awaiting /prm). Correction
2026-10-08: the live Pulse lens feed is dashboard/catalogue.json (profiling-summary@2, registry switched 2026-10-05
in 3c7bed2), whose producer hard-codes qualified: false, so the v1 qualify() gap never reached Pulse live and fix (2)
needed no Pulse change. Next: fix phase (3) INCONCLUSIVE for go/lever rules. Other tasks unchanged and unissued.'
next: 'timing_noise_audit fix phases (1)-(4) MERGED 2026-10-08 (autolens_profiling#404, #405, #406, #407). The live Pulse
lens feed is the v2 catalogue.json (qualified hard-false), so none needed a Pulse change. Next: fix phase (5) median
headline + GPU-only marker, then (6) warm-up flag + witness band; leftovers phase 3b (C1/C3/C4/C5), Holm/Bonferroni
policy, gpu_bottleneck_map round bootstrap. Other tasks unchanged and unissued.'
evidence: https://github.com/PyAutoLabs/autolens_profiling/blob/main/wiki/campaigns/measurement_tools.md
- id: autofit
title: PyAutoFit profiling bootstrap
Expand Down Expand Up @@ -139,14 +139,14 @@ tasks:
priority: high
path: tasks/timing_noise_audit.md
updated: '2026-10-08'
next: 'Fix phase (2) PR OPEN 2026-10-08 as autolens_profiling#405 (awaiting /prm; Heart YELLOW reasons human-acknowledged):
dashboard qualify() leaves laptop rows, rows without a load average and hpc rows without a host unqualified with a
reason; a within-2x comparison is flat only when both endpoints carry a repeat summary (none do yet), else insufficient;
drifted/improved keep status with a single-sample caveat (human decision 2026-10-08). Effect on summary.json: qualified
2 -> 2, flat 4 -> 0, insufficient 135 -> 139. Not a Pulse contract change (live feed is v2 catalogue.json, qualified
hard-false). Next after merge: fix phase (3) INCONCLUSIVE for go/lever rules, tie sets instead of argmax.'
next: 'Fix phases (2)-(4) MERGED 2026-10-08 under human /prm: #405 dashboard qualification + drift wording; #406 A/B go/lever
rules gain INCONCLUSIVE + tie sets (shared ab_verdict.py; no published go/no-go/NO_LEVER changed; 7 of 9 committed C10
sweeps are tie sets); #407 paired whole-round bootstrap (shared round_bootstrap.py; RAL CPU/A100 calls unchanged; one
laptop C6 row GO -> INCONCLUSIVE). Issue #362 stays open. Next: fix phase (5) P8/P9 (median headline, GPU-only marker),
then (6) P3/P5 (authorized --auto; not started at session end); leftovers phase 3b (C1/C3/C4/C5 lack intervals),
multiple-comparison policy, gpu_bottleneck_map.'
issue: https://github.com/PyAutoLabs/autolens_profiling/issues/362
evidence: https://github.com/PyAutoLabs/autolens_profiling/pull/405
evidence: https://github.com/PyAutoLabs/autolens_profiling/pull/407
- id: mge_likelihood_breakdown_steps_are_cumulative_an
campaign: measurement-tools
title: MGE likelihood_breakdown steps are cumulative and `linear_gaussians` is reported as 0
Expand Down
2 changes: 1 addition & 1 deletion dashboard.html

Large diffs are not rendered by default.

6 changes: 3 additions & 3 deletions dashboard.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# PyAutoPulse — profiling dashboard

<!-- pulse-campaigns:652e6582f3ead66f68ae1120a79f68f1c18219dbc4ffee221291672133f5fc79 -->
<!-- pulse-campaigns:885c41acf471ee25bdaad4bb819d702e5d3e8e19f7b3662b9e9f46a04b538c6d -->
## Profiling Check In

<details><summary>Full check-in prompt</summary>
Expand Down Expand Up @@ -133,13 +133,13 @@ Reviewed 2026-10-04. No new results since the pin. Over-sampling phase 1: add --

<details><summary>Measurement reliability and profiling tools — active</summary>

Reviewed 2026-10-08. timing_noise_audit fix phases (1) and (2) done (autolens_profiling#404 merged; #405 open, awaiting /prm). Correction 2026-10-08: the live Pulse lens feed is dashboard/catalogue.json (profiling-summary@2, registry switched 2026-10-05 in 3c7bed2), whose producer hard-codes qualified: false, so the v1 qualify() gap never reached Pulse live and fix (2) needed no Pulse change. Next: fix phase (3) INCONCLUSIVE for go/lever rules. Other tasks unchanged and unissued.
Reviewed 2026-10-08. timing_noise_audit fix phases (1)-(4) MERGED 2026-10-08 (autolens_profiling#404, #405, #406, #407). The live Pulse lens feed is the v2 catalogue.json (qualified hard-false), so none needed a Pulse change. Next: fix phase (5) median headline + GPU-only marker, then (6) warm-up flag + witness band; leftovers phase 3b (C1/C3/C4/C5), Holm/Bonferroni policy, gpu_bottleneck_map round bootstrap. Other tasks unchanged and unissued.

[Campaign evidence](https://github.com/PyAutoLabs/autolens_profiling/blob/main/wiki/campaigns/measurement_tools.md)

### Active tasks

- [Audit timing tests and profiling gates for measurement noise](https://github.com/PyAutoLabs/PyAutoPulse/blob/main/tasks/timing_noise_audit.md) — active: Fix phase (2) PR OPEN 2026-10-08 as autolens_profiling#405 (awaiting /prm; Heart YELLOW reasons human-acknowledged): dashboard qualify() leaves laptop rows, rows without a load average and hpc rows without a host unqualified with a reason; a within-2x comparison is flat only when both endpoints carry a repeat summary (none do yet), else insufficient; drifted/improved keep status with a single-sample caveat (human decision 2026-10-08). Effect on summary.json: qualified 2 -&gt; 2, flat 4 -&gt; 0, insufficient 135 -&gt; 139. Not a Pulse contract change (live feed is v2 catalogue.json, qualified hard-false). Next after merge: fix phase (3) INCONCLUSIVE for go/lever rules, tie sets instead of argmax.
- [Audit timing tests and profiling gates for measurement noise](https://github.com/PyAutoLabs/PyAutoPulse/blob/main/tasks/timing_noise_audit.md) — active: Fix phases (2)-(4) MERGED 2026-10-08 under human /prm: #405 dashboard qualification + drift wording; #406 A/B go/lever rules gain INCONCLUSIVE + tie sets (shared ab_verdict.py; no published go/no-go/NO_LEVER changed; 7 of 9 committed C10 sweeps are tie sets); #407 paired whole-round bootstrap (shared round_bootstrap.py; RAL CPU/A100 calls unchanged; one laptop C6 row GO -&gt; INCONCLUSIVE). Issue #362 stays open. Next: fix phase (5) P8/P9 (median headline, GPU-only marker), then (6) P3/P5 (authorized --auto; not started at session end); leftovers phase 3b (C1/C3/C4/C5 lack intervals), multiple-comparison policy, gpu_bottleneck_map.
- [MGE likelihood_breakdown steps are cumulative and `linear_gaussians` is reported as 0](https://github.com/PyAutoLabs/PyAutoPulse/blob/main/tasks/mge_likelihood_breakdown_steps_are_cumulative_an.md) — ready: Unchanged and unissued (no GitHub refs, 2026-10-04). Select one bounded step when measurement-tools is prioritised.
- [A gradient-cost probe: forward vs `value_and_grad` ms/eval and a strict FD check, on any registry cell](https://github.com/PyAutoLabs/PyAutoPulse/blob/main/tasks/gradient_cost_probe.md) — ready: Unissued. New context 2026-10-07: Heart unit-test timing flagged PyAutoLens forward-gradient tests +150%/+96% after PyAutoLens#768 rewrote that test file; not a controlled comparison, and this probe is the right instrument. Select one bounded step when prioritised.
- [Numba breakdown harness: perturb the instance so the operated-matrix memo cannot hide a step](https://github.com/PyAutoLabs/PyAutoPulse/blob/main/tasks/numba_breakdown_harness_memo_blind.md) — ready: Unchanged and unissued (PyAutoArray#496 closed 2026-08-27). Select one bounded step when prioritised.
Expand Down
10 changes: 10 additions & 0 deletions tasks/timing_noise_audit.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,16 @@ The original task below is retained verbatim. Current scheduling state lives in
- Carry-forward: when the v2 catalogue starts qualifying records (setup-baseline reference-host decision), it must reuse the producer's `is_reference_host_class` rule rather than a separate one.
- Next: /prm the PR (judge tier, human merge); then fix phase (3) — INCONCLUSIVE for go / lever rules, tie sets instead of argmax.


### Phases 3–5 merged 2026-10-08 (fix phases 2–4)

- [autolens_profiling#405](https://github.com/PyAutoLabs/autolens_profiling/pull/405) merged 19:46Z (f6e6982); Mind record `complete/2026/10/timing-noise-audit-p3-qualify-drift.md` (d774cbb1).
- [autolens_profiling#406](https://github.com/PyAutoLabs/autolens_profiling/pull/406) merged 20:19Z (94861700): shared `ab_verdict.py` (`ab_rule_verdict`, `tie_set`, `paired_block_ratio_interval`) wired into C6, C7, P2, C10. Re-judged committed rows: no go / no-go / NO_LEVER call changed; 7 of 9 committed C10 "best" picks are tie sets (IP-4a's 2.37x leader ties with four others; the human decision not to ship it stands). Flagged decision: C10 vmap rows measured for the tie set's `point_leader` only. Mind record f55f220e.
- [autolens_profiling#407](https://github.com/PyAutoLabs/autolens_profiling/pull/407) merged 20:43Z (8be806cc): shared `round_bootstrap.py` (paired whole-round resampling, `effective_n = n_rounds`) in seven cells. Witness: iid CI 0.14–0.39x as wide as the round CI; 40-seed coverage 0.85 vs 0.33 at nominal 0.90. RAL CPU / A100 calls unchanged; laptop C6 solved `rev_analytic` GO -> INCONCLUSIVE ([0.805, 0.854] vs 0.85). The review returned FINDINGS (label strings); the fix commit 63642d52 was re-reviewed by the main session before merge.
- Heart was YELLOW throughout with an organism-wide manifest-drift + no-rehearsal set; the human acknowledged that set for these PRs on 2026-10-08.
- #407 close-out: Mind record `complete/2026/10/timing-noise-audit-p5-round-bootstrap.md` (a9eb5864). Fix phases (5) and (6) NOT started at session end (no branch, Mind entry or plan comment); resume at start_dev with slug `timing-noise-audit-p6-median-headline-gpu-marker` from main 8be806c.
- Leftovers recorded in the audit note: phase 3b (C1/C3/C4/C5 have no interval), multiple-comparison policy (Holm/Bonferroni), `gpu_bottleneck_map` still per-call. Next: fix phase (5) P8/P9, then (6) P3/P5 (human authorized both under --auto 2026-10-08).

---

# Audit timing tests and profiling gates for measurement noise
Expand Down
Loading