[finding] `Test Core` is red on three unrelated PRs across three different shards, and `main` has not advanced in ~2.5h — the merge queue looks stalled repo-wide

[os-elon]

Observation-class finding, filed unassigned, from the domain:engine seat at end of shift. No shipped behaviour is implicated. This is a CI/throughput observation that no single PR's author is positioned to see, because it is only visible across several PRs at once.

The measurement

Three open PRs, three unrelated diffs, checked at 2026-08-20 ~05:40Z:

PR	card	diff touches	merged with current main?	red shard
#10093	#9974	packages/objectql, plugin-audit, service-storage	yes	Test Core (**1**/3)
#10094	#9864	packages/core, packages/cli	yes	Test Core (**2**/3)
#10060	#9960	metadata-protocol, rest, runtime	no — base 18 commits behind	Test Core (**3**/3)

Three different shards. On every one of the three, Lint & Repo Gates and TypeScript Type Check are green (except #10060's Lint, which is the separate #10121 eslint stack overflow), as are Build Core, Temporal Conformance, every Dogfood shard, and every claim guard.

check-test-completeness reported OK on the shards inspected ("18 package(s), 6847 test(s) declared and all 6847 accounted for"), which per that step's own documentation means real assertion failures rather than dead workers.

What this rules out, and what it does not

⛔ It is not any one of these three diffs. Three unrelated file surfaces producing failures on three different shards is not a shared code cause in the PRs.

⛔ "Something landed in main" does not explain it either — that hypothesis is falsified by #10060. #10060 was deliberately left unmerged, 18 commits behind, and is red anyway. Two of the three are on current main and one is not; all three are red. Recording this explicitly because it was my first hypothesis and it looked strong until #10060 was checked.

⚠️ What remains open, and this card does not choose between them:

1. a broad environmental/load condition on the runners right now;
2. several independent non-deterministic tests that happen to be distributed one per shard;
3. something else.

The exact failing assertion was not recovered for any of the three: GitHub truncates these job logs to a tail that lands inside the post-job cleanup block, past the failing task's own output. That is itself worth noting — it makes per-PR root-causing disproportionately expensive.

The corroborating signal, and why it is the load-bearing half

main's recent landing cadence, from git log origin/main:

00:19  01:01  01:07  01:23  01:30  01:43  02:15  02:56   ← then nothing

Roughly one merge every 20–40 minutes, then ~2.5 hours with zero landings, with refs/heads/gh-readonly-queue/* empty (nothing being tested) the whole time. In a repo that runs ~18 merges in a working day, that is a break in cadence, not a quiet patch.

⇒ The single-PR reading ("my card is flaky") and the repo reading ("nothing is landing for anyone") are very different problems, and only the second one is consistent with all the evidence above.

Suggested first step

⛔ Do not start by re-running the three PRs — that treats it as three flakes and would confirm nothing whichever way it went.

Start with the base: check whether Test Core is red on main itself. ⚠️ Note that main's real CI history lives in merge_group events, not push — a branch=main run listing returns stale push runs and will mislead (measured; recorded on the domain:engine seat post as a platform fact). If the base is red, this whole card collapses into that one fact and the three PRs are innocent bystanders.

Related, but distinct — do not merge these

• [finding] eslint . intermittently dies with "Maximum call stack size exceeded" parsing migrations/registry.ts — a monotonically-growing file now at the parser's stack cliff #10121 — eslint . intermittently blowing its parse stack on migrations/registry.ts. That is Lint & Repo Gates, a different job, with a measured non-deterministic signature of its own. It is also currently red on refactor(metadata-protocol,rest,runtime): one declared shape for the protocol.deletePackage seam #10060, which is why that PR shows two independent reds.
• [finding] deliveredInbox returns SHORT on timeout, so a slow run is asserted as a wrong value — and in one call site that makes the test PASS for the wrong reason #10106 — deliveredInbox in examples/app-showcase returning short on a 5-second wall-clock timeout. A named candidate mechanism for one shard's flakiness, but ⛔ it was explicitly not reproduced (including at 3× CPU oversubscription), and it cannot account for three different shards.
• queue-only failure ×2 in one day: sql-driver-datetime-mysql-storage.test.ts "os migrate plan lists the MySQL widening (#3954)" — root-cause flake vs #9274 probe interference #9350 — non-deterministic driver-sql live-DB failures. Different job (Temporal Conformance), and that job is green on all three PRs here.

Priority

Medium, and the reason it is not low: if the second reading is right, every seat's throughput is currently zero and each PM is independently diagnosing it as their own card's flake. That misattribution is the expensive part, exactly as with #10121.

Evidence: #10093 run 32331045284 job 96311618594 · #10094 run 32331060241 job 96311657328 · #10060 run 32333753299 job 96319225484.

[os-elon]

RESOLVED — cause confirmed by the maintainer, and every prediction this card made has now been checked

The maintainer states directly: #10120 fixed a merge-queue problem. e61ee6832 — "pay the plugin-security transform at module load, out of every clocked window" — landed 06:31Z and the stall broke immediately.

The card's own falsification criterion, discharged

This card said ⛔ do not start by re-running the three PRs, because that treats it as three flakes and confirms nothing whichever way it goes. That still holds as method. What actually settled it is that all three were re-run against a changed base, which is a different experiment: same three diffs, same three shards, one variable moved.

PR	shard that was red	after e61ee6832
#10093	Test Core (1/3)	✅ success — and all 25 checks green
#10094	Test Core (2/3)	✅ success (all three shards)
#10060	Test Core (3/3)	✅ success (all three shards)

Three unrelated diffs, three different shards, all clearing on the same base change. Combined with the maintainer's confirmation, the mechanism is settled: a clocked window that a queue-side cost was being charged into, so the failure surfaced wherever the timing happened to land — which is exactly why it presented as one red shard per PR and did not follow the diffs.

⭐ Worth keeping: the reason this card exists is that no single PR's author could see it. Each of the three looked like "my card has a flaky test", and each would have been re-run individually and misdiagnosed. The signal only appeared by looking across PRs — three different shards is not a shape any one author observes.

⭐ And the hypothesis this card explicitly killed was the obvious one: "something landed in main" — falsified by #10060, which was deliberately 18 commits behind and red anyway. Had that not been checked, the fix would have been sought in the wrong place. Recording it because "the obvious hypothesis was wrong and cheap to disprove" is the transferable part.

Cadence, closed out

… 01:43  02:15  02:56   ← stall begins
                          ~3.5h, zero landings, queue empty
      06:31  06:46  06:54  07:00   ← recovered

⚠️ One thing that is NOT closed by this — see #10121

eb2bebe53 (#10124) landed at 07:00: "give the eslint parser stack headroom for migrations/registry.ts". That unblocked Lint & Repo Gates for everyone and was the right call under a stall.

⛔ But #10121 says in as many words: do not treat raising the stack as the answer. It buys back a few months of appends to a file that is append-only by process (6,795 lines / 528 KB, one entry per ADR-0087 retirement) and hides the next occurrence — and once CI has headroom, the next one surfaces on a contributor's machine instead, where nobody is watching. The open question there is unchanged and unanswered: which construct in that file the parser recurses on. #10121 should not be closed on the strength of #10124.

Closing this card. Lint & Repo Gates reds still showing on #10060 and #10094 are from runs at 06:48–06:50, i.e. before the 07:00 fix; both branches are being updated to pick it up.

---

Generated by Claude Code