Repository navigation
CI performance dashboard #1182
Description
Maintained by the CI profiler routine, which runs every 4h. Each run replaces this body and carries the history rows over.
Headline (run 2026-10-20 16:16 UTC)
Window: 2026-10-19 16:16 → 2026-10-20 16:16, 83 runs (21 CI: 19 merge_group, 1 pull_request finished + 1 running, 1 nightly). The previous profile was 10-18 00:15; the 10-18 → 10-19 runs were not profiled. Quiet until 15:47, then a 9-PR queue burst.
🔴 Main is red and the merge queue is draining by eviction
- Since uv 0.13.2 shipped (PyPI, 10-19 20:59),
e2e (ubuntu-latest, e2e_vex_build, pipenv:: --ignored, 2026.8.0)fails on main:uv pip install -r pylock.toml→ "The target environment is not supported by the lockfile". The cause is the unpinnedastral-sh/setup-uv. First red: nightly 39376690526 (10-20 05:46; the macOS pipenv leg failed too). - Merge queue: 15:54 batch (Fix lockless nested npm projects in root scans (#1554) #1591, Fix PyPI vendored wheels lost to gitignore/LFS (#1572) #1593, Fix bare requirements wheel paths not contesting VEX (#1587) #1599, Fix yarn copies with invalid manifest names (#1598) #1605, Fix hosted uv missing uv<0.5.6 advisories (#1569) #1609) all failed at that leg (39452719874 etc.) → Fix lockless nested npm projects in root scans (#1554) #1591 evicted 16:01. 16:01 batch (Fix PyPI vendored wheels lost to gitignore/LFS (#1572) #1593…Fix mixed line endings lost in vendored requirements (#1567) #1612) all failed again → Fix PyPI vendored wheels lost to gitignore/LFS (#1572) #1593 evicted 16:15. The 16:15 batch (Fix bare requirements wheel paths not contesting VEX (#1587) #1599, Fix yarn copies with invalid manifest names (#1598) #1605, Fix hosted uv missing uv<0.5.6 advisories (#1569) #1609, Fix mixed line endings lost in vendored requirements (#1567) #1612, Fix uv restore ignoring lock file filters (#1604, #1613) #1614) will hit the same failure. Each eviction burns ~5 rebuilds and 6–13 min.
- Cost so far: 549 Linux + 37 Windows job-min in merge_group runs in 22 min, all wasted. 0 merges in 24h.
- Fix: #1639 (ci-janitor) pins
setup-uv. Its PR CI is green (39451691962, 9.6 min toci-ok). It is not in the queue. Needs a human to queue it first (or jump it), then re-enqueue Fix lockless nested npm projects in root scans (#1554) #1591 and Fix PyPI vendored wheels lost to gitignore/LFS (#1572) #1593. - No new
ci-perfissue: the fix is already open as ci-janitor. Updated #1512 (slow Depot hosts are back: 92% slow-start in the 16:01 batch, which doubled its time toci-ok). - The 10-18 nightly 39101574188 also failed, and the 10-19 nightly was green. I didn't dig into 10-18 because it falls outside the window.
Sample: all 83 runs. Jobs (filter=all) for all 21 CI runs (the nightly's first page only, 100/246 jobs). 1 job log read. About 35 REST requests. GraphQL is blocked, so merge-queue state comes from run branches and PR timelines.
| metric | 24h |
|---|---|
| PRs merged | 0 (last: #1429, #1592 on 10-19 13:3x) |
| Merge queue enqueue → merge | — (n=0); 2 evictions, both from the main-wide pipenv failure |
merge_group created → ci-ok |
0 successes; failing runs 5.7–6.3 (fast hosts) and 11.4–13.4 (slow hosts) |
PR run created → ci-ok |
9.6 (n=1 full run, 69% slow-start) |
| Nightly | 10-20 05:46 failure (pipenv pylock, uv 0.13.2) |
| Linux queue wait p50 / max | Depot 0.02 / 0.05 |
| macOS queue wait max | 0.22 (nightly only) |
| Job-min/day (Linux / Windows / macOS) | ~0.70k / 0.15k / 0.06k: merge_group 549 L / 37 W (wasted), PR 65 L, nightly ~306 / 91 / 57 carried |
Critical path (unchanged when green): clippy → coverage → ci-ok, in parallel with e2e-build → Gradle e2e leg → ci-ok.
Merge-queue settings (ruleset 24668460, read 16:16): unchanged: build 5 / merge 5, min wait 5 min, ALLGREEN, timeout 90, SQUASH.
History
| run (UTC) | merges/24h | MQ enq→merge p50/p90 (min) | mg CI success p50 | PR push→ci-ok p50/p90 | mac queue p50/p90 | linux queue p50/p90 | job-min/day L/W/M |
|---|---|---|---|---|---|---|---|
| 2026-10-20 16:16 | 0 PRs merged (last merges #1429/#1592 on 10-19 13:3x; 13 merge_group CI runs failed or were cancelled from 15:54, 2 heads evicted (#1591, #1593): main is red from uv 0.13.2, fix #1639 not yet queued) | — (n=0) | — (0 mg successes; failing runs reached ci-ok at 5.7–13.4) |
9.6 (n=1 full run, slow hosts) | 0.22 max (nightly only) | 0.02 / 0.03 (Depot) | ~0.70k / 0.15k / 0.06k (mg 549 L / 37 W in 22 min, all wasted; PR 65 L; nightly 10-20 failed (pipenv leg), 306 / 91 / 57 carried; Depot slow-start 57% (235/409), the 16:01 mg batch 92%, median 50 s) |
| 2026-10-18 00:15 | 1 PR merged (#1590 at 14:47, none since; 0 mg failures, 0 evictions) | 7.93 (n=1, #1590) | 7.48 (n=1) | 7.43 / 7.83 (n=14 full runs, range 7.27–7.88) | 0.13 / 0.13 (nightly only, carried) | 0.02 / 0.02 (Depot, max 0.07) | ~0.84k / 0.09k / 0.06k (PR ~0.47k L in 39 runs (14 full at ~31.4 L, 1 cancelled, 24 stubs); mg 32.2 L / 1.8 W + watcher 8.0 L; push 7.6 L; nightly 306 / 91 / 57 carried; Depot slow-start 10-17 day ~2% (8/~420), 21h 0/29) |
| 2026-10-17 20:16 | 1 PR merged (#1590 at 14:47, none since; 0 mg failures, 0 evictions; 17 PRs open and not draft) | 7.93 (n=1, #1590) | 7.47 (n=1) | 7.47 / 7.88 (n=15 full runs, range 7.27–8.32) | 0.13 / 0.13 (nightly only, carried) | 0.02 / 0.02 (Depot, max 0.07) | ~0.85k / 0.09k / 0.06k (PR ~0.48k L in 41 runs (15 full at ~31.5 L, 25 stubs); mg 32.2 L / 1.8 W + watcher 8.0 L; push 7.6 L; nightly 306 / 91 / 57 carried; Depot slow-start 17h 28% (8/29, 4-core, +0.4 min), 19h 0% (0/58)) |
| 2026-10-17 16:16 | 1 PR merged (#1590 at 14:47; 1 mg CI success, 0 failures, 0 evictions; first merge after 42.8 h quiet; 15 PRs open and not draft) | 7.93 (n=1, #1590) | 7.47 (n=1) | 7.48 / 7.88 (n=12 full runs, range 7.27–8.32) | 0.13 / 0.13 (nightly only, carried) | 0.02 / 0.02 (Depot, max 0.07) | ~0.75k / 0.09k / 0.06k (PR ~0.38k L in 37 runs (12 full at ~31 L, 25 stubs); mg 32.2 L / 1.8 W + fail-fast watcher 8.0 L; push 7.6 L; nightly 306 / 91 / 57 carried; Depot slow-start 0/69 in the 5 new runs (13h–14h)) |
| 2026-10-17 12:16 | 0 PRs merged (0 mg CI runs, 0 push runs; none since 10-15 19:59, 40.3 h quiet; 14 PRs Ready for review) | 9.22 / 11.52 (carried, n=5; no merges in window) | — (no mg runs) | 7.53 / 8.31 (n=13 full runs, range 7.25–8.55) | 0.13 / 0.13 (nightly only) | 0.02 / 0.02 (Depot, n≈450) | ~0.75k / 0.09k / 0.06k (PR ~0.45k L in 40 runs; nightly 306 / 91 / 57 carried; Depot slow-start 15h 71% (20/28), 10h 0% (0/29); 1 new full run since 08:16, 7.25 min) |
| 2026-10-17 08:16 | 0 PRs merged (0 mg CI runs, 0 push runs; none since 10-15 19:59, 36.3 h quiet; 13 PRs Ready for review) | 9.22 / 11.52 (carried, n=5; no merges in window) | — (no mg runs) | 7.53 / 8.30 (n=12 full runs, range 7.32–8.48) | 0.13 / 0.13 (nightly only) | 0.02 / 0.02 (Depot, n=495) | ~0.76k / 0.09k / 0.06k (PR 457 L in 37 runs; nightly 306 / 91 / 57 carried; Depot slow-start 15h 71% (20/28), 21h 7% (4/58), 05h nightly 0% (0/92); 10-17 nightly green, 24.15 min) |
| 2026-10-17 04:16 | 0 PRs merged (0 mg CI runs, 0 push runs; none since 10-15 19:59, 32.3 h quiet; 12 PRs Ready for review) | 9.22 / 11.52 (carried, n=5; no merges in window) | — (no mg runs) | 7.80 / 8.48 (n=11 full runs, range 7.32–9.18) | 0.14 / 0.16 (nightly only) | 0.02 / 0.02 (Depot, n=433) | ~0.69k / 0.09k / 0.06k (PR 383 L in 34 runs; nightly 306 / 91 / 57 carried; Depot slow-start 05h 67% (nightly), 15h 69% (20/29), 01h–02h 0% (0/87)) |
| 2026-10-17 00:16 | 0 PRs merged (0 mg CI runs, 0 push runs; none since 10-15 19:59, 28.3 h quiet; 10 PRs Ready for review with green CI) | 9.22 / 11.52 (carried, n=5; no merges in window) | — (no mg runs) | 7.82 / 8.62 (n=9 full runs, range 7.32–9.18) | 0.14 / 0.16 (nightly only) | 0.02 / 0.02 (Depot, n=374) | ~0.63k / 0.09k / 0.06k (PR 321 L in 31 runs; nightly 306 / 91 / 57 carried; Depot slow-start 05h 74% (67/91, nightly), 15h 71% (20/28), 21h 7% (4/59)) |
| 2026-10-16 20:16 | 0 PRs merged (0 mg CI runs, 0 push runs; none since 10-15 19:59, 24.3 h quiet; 9 PRs Ready for review with green CI) | 9.22 / 11.52 (carried, n=5; no merges in window) | — (no mg runs) | 8.05 / 9.28 (n=15 full runs, range 7.32–10.60; 17:14 run 7.82) | 0.15 / 0.17 (nightly only, n=4 sampled) | 0.02 / 0.02 (Depot, n=550) | ~0.87k / 0.09k / 0.06k (PR 565 L in 38 runs, 14.9 per run incl. stubs; nightly 306 / 91 / 57 carried; Depot slow-start 15h 71% (20/28), 17h 10% (3/31)) |
| 2026-10-16 16:16 | 5 PRs merged (5 mg CI successes 16:49–19:48; 0 failures, 0 evictions; none since 19:59, 20.3 h quiet; 8 PRs Ready for review, green CI) | 9.22 / 11.52 (n=5, range 8.77–11.52) | 8.67 (n=5, p90 11.12; fast 8.17–8.88; slow 11.12) | 9.13 / 11.52 (n=21 full runs, range 7.32–12.15; today 7.32–8.48) | 0.13 / 0.18 (nightly + 1 CI dispatch only) | 0.02 / 0.02 (Depot, n=1,222) | ~1.75k / 0.21k / 0.11k (incl. 1 CI dispatch ≈ 0.30k / 0.09k / 0.06k; mg 39.6 L / 4.4 W per run; Depot slow-start 14h 2% (1/58), 15h 71% (20/28): #1512 relapse again, but that run lost only ~0.5 min (8.48 to ci-ok)) |
| 2026-10-16 12:20 | 14 PRs merged (14 mg CI successes since 14:13; 0 failures, 0 evictions; none since 19:59, 16 h quiet; 7 PRs Ready for review with green CI) | 9.10 / 11.83 (n=14, carried: no new merges) | 8.67 (n=14, p90 10.78; fast 7.43–8.98; slow 9.75–11.12) | 9.17 / 11.52 (n=23 full runs, range 7.53–12.15; 09:22 run 8.02) | 0.13 / 0.18 (nightly + 1 CI dispatch only) | 0.02 / 0.02 (Depot, n=1,601) | ~2.2k / 0.24k / 0.11k (incl. 1 CI dispatch ≈ 0.30k / 0.09k / 0.06k; mg 39.5 L / 3.9 W per run; Depot slow-start 09h 3% (1/29), too thin to call; last busy hour 05h 77%) |
| 2026-10-16 08:16 | 14 PRs merged (14 mg CI successes since 14:13; 0 failures, 0 evictions; none since 19:59, 12 h quiet) | 9.10 / 11.83 (n=14, range 8.18–15.22; slow-host entries 10.95–11.83) | 8.52 (n=14, p90 10.63; fast 7.43–8.98; slow 9.75–11.12) | 9.13 / 11.49 (n=25 full runs, range 7.53–12.15) | 0.13 / 0.18 (nightly + 1 CI dispatch only) | 0.02 / 0.02 (Depot, n=1,663) | ~2.3k / 0.24k / 0.11k (incl. 1 CI dispatch ≈ 0.30k / 0.09k / 0.06k; mg 39.5 L / 3.9 W per run; Depot slow-start 00h 0% (0/31), 04h 16% (5/32), 05h 77% (149/194, the nightly): #1512 not over) |
| 2026-10-16 04:16 | 16 PRs merged (16 mg CI successes since 05:49; 0 failures, 0 evictions; none since 19:59, 8 h quiet) | 9.05 / 11.81 (n=15, range 8.18–15.22; slow-host entries 10.95–11.83) | 8.35 (n=16, p90 10.53; fast 7.43–8.98; slow 9.75–11.12) | 7.93 / 9.81 (n=17 full runs; fast 7.30–8.23, slow 9.28–12.15) | 0.13 / 0.17 (nightly + 3 CI dispatches only) | 0.02 / 0.02 (Depot, n=1,884) | ~3.1k / 0.39k / 0.22k (incl. 3 CI dispatches ≈ 0.80k / 0.25k / 0.17k; mg 38.7 L / 3.9 W per run; Depot slow-start 15h 80%, 19h 67%, 20h 63%, 23h 80%; 00h 0% (0/29): #1512 relapse ended or traffic too thin to tell) |
| 2026-10-16 00:16 | 21 PRs merged (19 mg CI successes since 01:14; 0 failures, 0 evictions; none since 19:59) | 9.05 / 11.83 (n=15, carried: no new merges since the last run) | 8.33 (n=19, p90 10.37; fast 7.43–9.03; slow 10.27–11.12) | 8.12 / 11.34 (n=12 full runs; fast 7.50–8.18, slow 9.28–11.45) | none ran (no macOS on PR/mg/push) | 0.02 / 0.02 (Depot, n=1,055) | ~3.9k / 0.55k / 0.36k (incl. 5 CI dispatches ≈ 1.25k / 0.40k / 0.30k; mg 37.9 L / 3.7 W per run; Depot slow-start 66–83% at 15h, 17h, 19h, 20h, 23h: #1512 relapse runs 10 h) |
| 2026-10-15 20:16 | 24 PRs merged (21 mg CI successes since 23:48; 0 failures, 0 evictions) | 9.05 / 11.83 (n=15, range 8.18–15.22; slow-host merges 11.5–11.8) | 8.17 (n=21, p90 10.27; fast 7.43–8.88; slow 10.27–11.12) | 8.05 / 11.45 (n=9 full runs) | 0.15 / 0.18 (CI dispatch only) | 0.02 / 0.02 (Depot, n=1,240) | ~4.1k / 0.65k / 0.42k (incl. 6 CI dispatches ≈ 1.5k / 0.48k / 0.36k; mg 37.4 L / 3.6 W per run; Depot slow-start 76% / 32% / 71% / 25% / 71% / 83% at 15h–20h: #1512 relapse sustained) |
| 2026-10-15 16:16 | 24 PRs merged (20 mg CI successes since 19:49; 0 failures, 0 evictions) | 8.82 / 11.83 (n=15, range 8.13–15.22; #1566 waited 5.8 min for a build slot behind a 6-entry burst) | 8.17 (n=20, p90 10.62; 14:13–14:27 runs 7.43–9.75; 15:49 runs 10.27–10.78 on slow hosts) | 7.93 / 9.93 (n=8 full runs) | 0.13 / 0.17 (nightly + 6 CI dispatches only) | 0.02 / 0.02 (Depot, n=1,633) | ~3.8k / 0.62k / 0.43k (incl. 6 CI dispatches ≈ 1.44k / 0.47k / 0.37k; mg 36.8 L / 3.2 W per run; Depot slow-start 11% at 14h, 81% at 15h, 26% at 16h: #1512 relapse on the first busy afternoon) |
| 2026-10-15 12:16 | 19 PRs merged (17 mg CI successes since 13:21; 0 failures, 0 evictions; none since 05:58) | 9.05 / 12.22 (n=15; since 23:48 8.07–9.75; 16:48–20:48 11.37–12.42) | 8.33 (n=17, p90 11.33; the 10 runs since 23:48 7.52–9.03) | 7.93 / 10.18 (n=7 full runs) | 0.13 / 0.17 (nightly + 6 CI dispatches only) | 0.02 / 0.02 (Depot, n=1,470) | ~3.35k / 0.60k / 0.43k (incl. 6 CI dispatches ≈ 1.44k / 0.47k / 0.37k; mg 38.9 L / 2.7 W per run; Depot slow-start 1% at 06h, 0% at 11h; quiet morning, no busy-day confirmation for #1512 yet) |
| 2026-10-15 08:16 | 23 PRs merged (19 mg CI successes since 12:24; 0 failures, 0 evictions) | 9.05 / 12.2 (n=15; the 9 entries since 23:48 took 8.07–9.75; 16:48–20:48 entries 11.37–12.42) | 8.55 (n=19, p90 11.38; the 9 runs since 23:48 7.52–9.03) | 9.68 / 12.13 (n=11 full runs) | 0.13 / 0.17 (nightly + 6 CI dispatches only) | 0.02 / 0.02 (Depot, n=1,647) | ~4.45k / 0.61k / 0.43k (incl. 6 full-matrix CI dispatches ≈ 1.44k / 0.47k / 0.37k; mg 38.6 L / 2.8 W per run; Depot slow-start 0% in 06h–08h, 05h hits were the nightly; nightly green → #1525 delivered) |
Open ci-perf issues by ROI
ROI on one scale: (thousands of job-min/day weighted Linux ×1, Windows ×2, macOS ×3, plus 1 per critical-path or PR-latency minute) × confidence ÷ effort (S=1, M=2, L=3).
| ROI | issue | saving | status |
|---|---|---|---|
| ~2.5 | #1512 (settings) slow Depot hosts | ~2–3 min on each merge that hits a slow host (1 of 5 merge_group runs in 24h, 11.12 vs 8.17–8.88), plus ~0.5–3.5 min on PR feedback | Slow hosts at 17h–23h on 10-15 (24–75% slow-start), 05h in the nightly (77%), and 15h on 10-16 (71%). 00h–09h on 10-17 were fast, including the nightly (0/92). 10-17 as a whole: ~2% slow (8/~420, all in one 17h run, +0.4 min). Relapse 10-20 16h: 92% slow-start in the 16:01 mg batch (median 50 s, max 189 s), making the batch about 2× slower to ci-ok. Last update: comment (10-20 16:16). Needs a human to raise it with Depot. |
| ~1.0 | #1248 (settings) merge-queue build concurrency | ~6 min for each entry past the 5th in a burst | Needs a human decision. Not hit: the 10-20 queue (9 entries) is draining through evictions, not through build slots. |
Open ci-perf / ci-janitor PRs: #1639 pins setup-uv and fixes the pipenv pylock leg broken by uv 0.13.2 (blocking every merge; PR CI green, 9.6 min; not in the queue at 16:16); #1620 ci-watch.sh without GraphQL (in the queue); #1634 reuse self-tests (correctness).
Merged ci-perf / ci-janitor PRs (last 7 days) and measured effect
| PR | merged | before → after |
|---|---|---|
| #1590 move action pins off deprecated Node 20 (#1588, ci-janitor) | 10-17 14:47 | Maintenance; claimed no saving. Its own merge_group run: 7.47 min to ci-ok (fast-host range before: 7.43–8.98); next push run 3.15 (3.1–4.85 before). No measurable change, as expected. |
| #1585 core covgap tests join their suites (#1089) | 10-15 18:58 | Drops 4 core test binaries. 1 merge_group run since (19:48, slow hosts, 11.12). PR full runs after it on fast hosts: 7.32–8.48 to ci-ok (n=4 today), coverage 5.53–6.22. No measurable change, and it claimed no saving. |
| #1568 fold last top-level covgap test binaries (#1089) | 10-15 14:21 | coverage on fast hosts (start-up gap ≤ 8 s): 5.82–6.03 min (n=8 mg runs before) → 5.87–6.25, p50 6.0 (n=5 after). No measurable change. It was a test refactor and claimed no saving, so no follow-up on #1089. |
| #1563, #1565 vex / cli suite tests assert codes, not sentences (#1089) | 10-15 14:36 / 14:22 | Test refactors. No CI-time effect expected or measurable. |
| #1560 scan mod tests assert codes, not sentences (#1089) | 10-15 05:57 | Test refactor. No CI-time effect expected or measurable. |
| #1556 scan hosted tests assert codes, not sentences (#1089) | 10-15 02:57 | Test refactor. No CI-time effect expected or measurable. |
| #1546 apply suite asserts codes, not sentences (#1089) | 10-14 21:00 | Test refactor. No CI-time effect expected or measurable. |
| #1544 vendor suite asserts codes, not sentences (#1089) | 10-14 20:00 | Test refactor. No CI-time effect expected or measurable yet. |
| #1525 fix nightly old-uv orphan-unwind failure in uv e2e (ci-janitor) | 10-14 09:01 | The 10-14 nightly failed on old-uv legs → the 10-15 nightly 38688589166 was green (0 failed jobs). Delivered. |
Method notes
- GraphQL is blocked in this environment, so merge-queue state comes from REST timelines of merged PRs. The timelines use the last enqueue before the merge.
- Jobs are fetched with
filter=all. Queue wait = jobstarted_at−created_at; skipped jobs are excluded. "Start-up gap" = jobstarted_at→ first stepstarted_at; over 15 s counts as a slow host. - Critical path = merge_group run
created_at→ci-okcompleted_at. - Daily job-min = per-run mean × 24h run count per event. This run one page covered the whole 24 h (100 runs), so no scaling was needed. Merges per day come from the closed-PR list.
Generated by Claude Code
Activity
mikolalysenko commentedon Oct 8, 2026
[agent] There are now two open dashboard issues: this one and #1175, both titled "CI performance dashboard" and labeled
ci-perf-dashboard. The body says it is maintained in place, so the profiler routine should update one issue rather than open a new one each run. I'm not closing either one, because I can't tell which issue the routine will update next. Whoever owns the profiler should close the copy it no longer updates.Generated by Claude Code