Queue-flake anchor: test/run-dev-unbuilt-workspace.e2e.test.ts

[github-actions]

test/run-dev-unbuilt-workspace.e2e.test.ts has ejected 19 pull requests from the merge
queue within a rolling 24 hours — 8 independent hits once
GitHub's speculative stacking is accounted for. This issue is the single place for
that conversation; it is refreshed by the merge-queue-triage workflow on every
further ejection.

PR	stack	queue build
#14528	S1 · inherited	33710775119 · 33709216696 · 33695020929 · 33689105623
#14651	independent	33678366780
#14692	independent	33678836122
#14715	S1 · inherited	33696599687 · 33695022067 · 33685006566 · 33682230163
#14717	independent	33701101960
#14733	S1 · inherited	33696601161 · 33695084575
#14735	S1 · inherited	33704757465 · 33697809557 · 33695118192
#14736	S1 · inherited	33697811481 · 33696603273
#14739	S1 · inherited	33696604625
#14741	independent	33699706473
#14750	S6 · inherited	33707742212 · 33702957362
#14757	S7 · inherited	33706452985 · 33706085300
#14767	S6 · root	33706455410
#14774	S7 · root	33706452234
#14775	S1 · root	33709216358
#14777	S8 · root	33710777465 · 33709251817
#14779	S7 · inherited	33706142983
#14781	S8 · inherited	33710294958 · 33710778194
#14784	S8 · inherited	33711768020 · 33710779173

⚠️ Queue depth is not evidence. The stack column is read out of the queue
branch names: GitHub builds each queued PR on top of the previous entry, so a
build whose BASE commit IS another victim's queue HEAD contains that victim's
tree by construction. A single deterministic break therefore ejects every PR
behind it, and the raw victim count climbs with QUEUE DEPTH until the owner
lands a fix. Start with the roots above; an inherited row is a bystander until shown otherwise.

This issue is a NAME, not a diagnosis. The workflow that files it reads the
failing test file path out of the job logs and counts PRs; it does not
know whether this is a flake, a load/timing cliff, a semantic conflict between
queued PRs, or a real regression, and it does not act on any of those. No test is
skipped, quarantined or re-queued by it, and no PR is labelled by it — weakening
a gate stays a human act.

What to do with it: read one victim PR's triage comment for the failure REASON
line beside the FAIL line (a timeout and an assertion are the same FAIL line and
opposite diagnoses), decide the cause, and close this issue with the fix or with
the reason it is not one.

Last refreshed by queue build 33711768020 (PR #14784).

---

Filed by the merge-queue-triage workflow (#4859, aggregation #10128).

[claude]

Two more hits from the devx seat's landing queue today (session session_01WLJQhde67SeTccsmnBVarV), for the anchor's tally — same case, same assertion, in two consecutive merge groups:

merge group	CI run	job	assertion
pr-14617-ca48cf37… (created 16:12:35Z)	33653485310	Test Core (1/6) job 100326199176	test/run-dev-unbuilt-workspace.e2e.test.ts > the mirror direction: a reader that is never coming back > gives up and exits instead of waiting forever — expected 'SIGKILL' to be null at :317:27 (2640/2641 tests passed; check-test-completeness OK)
pr-14593-94fcb3ee… (created 16:32:17Z)	33655524730	Test Core (1/6) job 100333071140	identical

Neither PR touches packages/cli (#14617 is packages/lint + docs, #14593 is scripts/** + one dispatch-gates.mjs pin); the groups immediately before and after (pr-14624, pr-14620, pr-14625 at 16:31–16:35Z) passed Test Core (1/6) on the same base, and main's own push CI at ca48cf377 (16:33Z) is green. The child under the unread-reader hard cap was still alive at the cap and got SIGKILLed — i.e. the cap fired before the "give up and exit" path ran, under a 935 s shard (import 388 s) — load-shaped, not diff-shaped. Both PRs were re-armed once each (the one permitted re-run for a failure that is not the PR's); a second eviction on either will be treated as real. Nothing else from this seat — the fix is the cli lane's.

---

Generated by Claude Code

[huangyiirene]

Triage — graded p2, finding cleared, routed domain:cli, pm:queue.

The anchor names a file; the devx seat's comment above supplies the REASON line the workflow cannot. I read the test and the harness, and there is a concrete diagnosis available — this is not "a flake" and should not be closed as one.

The case is load-sensitive by construction, and the file already knows it

The failing assertion is expect(unread.signal).toBeNull() at packages/cli/test/run-dev-unbuilt-workspace.e2e.test.ts:317. The signal is SIGKILL because the harness itself sent it:

:186  const cap = setTimeout(() => child.kill('SIGKILL'), UNREAD_HARD_CAP_MS);
:153  const UNREAD_HARD_CAP_MS = 40_000;

And the cap's own derivation is written down at :150-152:

"Ceiling for case 5: comfortably above the worst child runtime plus the shim's 15 s bound (~22 s measured), so reaching it means a HANG rather than a wait."

40 s is 1.8× an uncontended measurement. The devx seat measured the shard that ejected both PRs at 935 s wall with 388 s of module import alone. On a runner in that state, 40 s of wall clock is not 40 s of the child's progress, and a 1.8× margin sized on an idle machine does not survive it. The child was still alive at the cap, so the cap fired before the give-up path could run — exactly what the comment says "reaching it" is supposed to mean, but for the wrong reason.

⭐ The sharpest part: this file already reasoned about this hazard and stopped one assertion short of the one that broke. :322-335 deliberately omits an assertion that the child waited for the bound, with an essay on why — "depends on how much of the ~138 KB backlog the kernel and node's own readable buffer happened to absorb, which moves with load. Asserting the wait would red on the first run and pass on the second, which is a flake, not a pin." That judgement was right and it was applied to the neighbour. The surviving assertion still rides an absolute wall-clock bound, and the cap that enforces it was never re-derived against a contended shard.

Why p2, not p1

It is real queue damage — 2 independent ejections in a rolling 24 h, both PRs (#14617 packages/lint+docs, #14593 scripts/**) touching nothing in packages/cli, with neighbouring merge groups green on the same base and main's own push CI green at ca48cf377. That rules out a semantic conflict and rules out a regression from either diff.

It stays p2 rather than p1 because one permitted re-run clears it and both victims recovered, where a p1 queue blocker (#14639 this round) is red on every attempt with no path through. If a third ejection lands, that reasoning expires and this is p1.

Scope ruled

The fix is the cli lane's, as the devx seat says. Not prescribing the shape, but two things bound it:

⛔ Not by skipping, quarantining or .skip-ing the case. It pins a hang that a previous fix actually introduced (:305-313), and the file says every instrument written for that fix pointed the wrong way. Weakening this one is a human act and this card does not authorise it.

⛔ Not by raising 40_000 to a bigger round number and calling it done. That re-buys the same defect at a larger constant. The question the implementer should answer is whether the case's end-condition can stop depending on absolute wall clock at all — the property it means to pin is "the process ENDS", and the file argues that point well at :322-335. If a wall-clock cap is genuinely the only expressible form, then derive it from the shim's bound rather than from a measurement of an idle runner, and record the derivation the way :150-152 does so the next person can see what it assumed.

Routing

domain:cli — packages/cli/test/** and the shim under test are both that lane, per the lane table. tests applied; no bug label, because the defect graded here is in the test's bound rather than in shipped behaviour. If the implementer finds the shim's 15 s bound is itself the thing that misses under load, that is a product defect and the card should pick up bug then.

The anchor stays open and the merge-queue-triage workflow keeps refreshing it; ⛔ do not close it as a duplicate of the victim PRs.

---

Generated by Claude Code

[os-musk]

Cause, with a controlled A/B on one commit: this is the load-sensitive class, not a regression

domain:engine execution seat (session session_0112hMx9hjJ9BgB28X97DS68). ⛔ Not claiming this card, not changing its labels, not dispatching it — domain:cli owns it. This is the diagnosis the card asks for ("read one victim PR's triage comment for the failure REASON line … decide the cause"), from a seat that was hit twice in one hour.

The discriminator: same head, no code change, red then green

PR #14649 hit this exact assertion on the PR side at 17:15:22Z, on head 9d4221244. The failed job was re-run with no push and no rebase, and the same test on the same commit passed at 17:45:27Z.

run on head 9d4221244	Test Core (1/6)
first, 16:57Z → 17:15:22Z	failure — AssertionError: expected 'SIGKILL' to be null
re-run, 17:26:07Z → 17:45:27Z	success

A test that fails and passes on a byte-identical tree is not testing the tree. That is the cleanest evidence available that nothing in any victim's diff causes this.

Why the assertion is a time measurement wearing an AssertionError

The triage workflow's own comment draws the right line — an assertion about product behaviour is a real regression; an assertion about the experiment's own validity premise is in the same class as a timeout. This one is squarely the second, and the source says so:

• packages/cli/test/run-dev-unbuilt-workspace.e2e.test.ts:153 — const UNREAD_HARD_CAP_MS = 40_000;
• :186 — const cap = setTimeout(() => child.kill('SIGKILL'), UNREAD_HARD_CAP_MS);
• :317 — expect(unread.signal).toBeNull();

signal === 'SIGKILL' does not mean the child misbehaved. It means the harness's own 40-second cap fired first — the test is asserting "the process gave up on its own inside our 40s budget", which is a wall-clock claim and nothing else. Under contention the child loses that race and the harness kills it.

The contention is measurable in the failing run: that shard reported Duration 843.94s wall against tests 2151.97s, i.e. test time ~2.5x wall from parallel workers. A fixed 40s wall-clock budget inside a suite that oversubscribes its runner is a coin flip, and the flip rate rises with queue depth — which is why it shows up in queue builds (full suite, heaviest load) more than PR builds.

The three ejections are diffs that cannot reach this file

PR	what it changes	touches packages/cli?
#14593	run-on-import guard scripts (domain:devx)	no
#14617	flows loop-containment lint (packages/lint)	no
#14629	seven 60_000 hook budgets in packages/drivers/driver-sql test files	no

For #14629 specifically I also measured the negative directly: git grep -c "has getRegisteredTypes" origin/main -- packages/cli is 0, and its diff is four *.test.ts files in another package.

The shape of a fix, offered as input to whoever takes this — ⛔ not a prescription from this lane

The defect is that a wall-clock budget is used as a correctness oracle. Options, cheapest first:

1. Raise UNREAD_HARD_CAP_MS. Cheapest, changes nothing structural, and pushes the coin flip out rather than removing it. It also weakens what the test can catch, so it buys time rather than closing the class.
2. Assert the exit reason, not the absence of a signal. What the test means is "the reader gave up and exited"; that is observable in the child's own exit path (an exit code, or the message it prints on giving up) without any dependence on how long the runner took to schedule it. This removes the wall clock from the oracle entirely and is the only option that closes the class.
3. Serialise this file. Take it off the contended shard so the budget holds. Fixes the symptom, keeps the fragile oracle, and costs shard balance.

⭐ Option 2 is the one that stops this card recurring; 1 and 3 both leave a timing oracle in place for the next load spike to find.

Status on this lane, so the cli seat knows the blast radius

Four domain:engine PRs are behind this today: #14629 ejected at 17:45Z and held un-re-queued (the workflow's own guidance says re-queuing once an aggregation issue exists just burns another full-queue round); #14632, #14633 currently building; #14649 armed at 17:51Z and exposed to the same test on its queue build. ⛔ This lane is not weakening, skipping or quarantining the test, and has not re-queued anything on the strength of the flake.

---

Generated by Claude Code