[rubenvdlinde]

Package invariants is a required check on main and hits its 30-minute timeout on every pull request (#888, #896, #897, #898, and others' PRs), so it reads CANCELLED. This PR makes it fit without raising the cap and without dropping a suite.

Why it got slow

The job runs all 141 scripts/lib/test_* suites one after another. Per-suite times from CI run 41600349698 (deltas between PASS lines), which reached 133 suites at 29 minutes:

suite	seconds
test_gate_acceptance_matrix.sh	268
test_gate_45_to_55_acceptance.sh	166
test_gate_crashed_checker_is_not_a_finding.sh	165
test_gate_45_selector_parity.sh	103
test_gate_a11y_helper_wiring.sh	100
test_gate_1_11_honest_verdicts.sh	59
next 10	29 to 50 each

Most of the time is full run-hydra-gates.sh runs over fixtures (about 3 s each in CI, 100 gates at a fraction of a second each; no single gate dominates, gate 122 is the largest at about 13 percent). The suite count keeps growing with every gate, which is why a 10-minute job in September is a 30+ minute job now. The 8 suites after test_gate_scope_matrix.sh had not run in CI at all while it timed out.

What changes

• Sharding. run-helper-suites.sh takes HELPER_SHARDS / HELPER_SHARD and runs only its shard. tests/partition_helper_suites.py splits longest-first by the measured times in tests/helper-suite-timings.tsv. A suite missing from the table gets a default weight, so a stale table can only unbalance the shards, never drop a suite. Discovery, the quarantine and its rot check are unchanged; each quarantine entry is checked on the shard that owns it.
• Workflow. invariants-shard runs 6 legs (Package invariants (shard N of 6)), each with the old 30-minute cap. The bin invariant suite and the no-Ajv manifest check run on shard 0. Each leg uploads its per-suite verdicts.
• Package invariants stays the required name. It is now the aggregate. tests/check-helper-shard-reports.py re-discovers the suites and fails unless every one has exactly one PASS or QUARANTINED verdict from exactly one shard, every shard left a report, and the matrix succeeded. A cancelled, crashed or lossy shard cannot read green.
• Timing per suite. Every PASS/FAIL line now ends with the suite's seconds, so the next slowdown is visible in the log.
• Slowest suite sped up. test_gate_acceptance_matrix.sh runs its 82 read-only fixture arms up front, nproc at a time (HYDRA_ACCEPT_JOBS, 1 keeps the old sequential path), then grades in the same order from the stored output. The other heavy suites mutate one fixture between runner calls, so they are left sequential.
• Caching. Nothing worth caching: the only install is ajv ajv-formats (under a second).

Verified locally

• tests/test-partition-helper-suites.py: 12 cases, including every real suite on exactly one shard, a suite missing from the table still running, and the aggregate refusing a lost suite, a duplicate, a FAIL, a missing shard report, an unknown suite and a cancelled matrix.
• The measured table splits the suites into 6 shards of about 5.6 minutes each.
• Sharded runner: single-suite shards report and pass, the quarantined suite's shard prints QUARANTINED and "still failing as documented", and an out-of-range shard is refused.
• Acceptance matrix: two prefetched arms compared with a sequential runner run over the same fixture give identical verdict lines (105 each, 0 diff lines). The fixtures stay untouched (git status clean).
• shellcheck at warning level clean; assert-run-steps-resolvable.py OK.
• Not run locally: the whole suite, because this box was at load 40 to 45 and one runner run took 26 s here against about 3 s in CI. CI is the proof; see below.

CI result

Run 41646864752 on 6bb8df8: Package invariants (the aggregate) passed, and so did all 6 legs.

leg	wall time	suite seconds	suites
shard 0 of 6	3m49s	146 (plus the bin suite and the no-Ajv check)	19
shard 1 of 6	4m39s	272	23
shard 2 of 6	6m33s	386	23
shard 3 of 6	4m04s	237	25
shard 4 of 6	6m26s	378	25
shard 5 of 6	6m42s	393	26
aggregate	6s		141

The slowest leg took 6m42s against the 30-minute cap that the single job used to hit. The 6 shard reports hold 141 suites, each exactly once: 140 PASS and 1 QUARANTINED (test_gate_orphaned_capability_fixtures.sh, as documented).

The suites that never ran before now pass. Run 41600349698 timed out before reaching them. All 11 suites that are absent from its log pass here: test_gates_23_33_never_green_over_nothing.sh, test_generate_mock_register.py, test_js_scope.js, test_lib_commonjs_scope.sh, test_manifest_diff_scope.py, test_manifest_scope_filter.js, test_or_abstraction_exceptions.sh, test_or_abstraction_pdok.sh, test_php_template_scope.py, test_source_scope.py and test_vendored_schema_freshness.js. This includes the 8 after test_gate_scope_matrix.sh.

Rebalanced. The legs were uneven (146 to 393 suite seconds), mainly because the table still held the acceptance matrix at 268 s, while the prefetch brought it down to 124 s. ebaafd5 replaces the table with the per-suite seconds measured in this run. The quarantined suite keeps its old weight. Fed the measured times, the old table reproduces the observed loads exactly, and the new table predicts 302 to 303 s on every shard. tests/test-partition-helper-suites.py passes (12 tests). That commit gets a new CI run, and I have not read it.

AI disclosure: prepared with Claude Code (Opus 5.5).

🤖 Generated with Claude Code