Repository navigation
CI Coverage job dies silently on about a third of PR runs since the Blacksmith migration #4247
Description
Purpose
The Coverage job in CI dies without reporting a cause on about one PR run in three. Since the Blacksmith runner migration (#4185, 2026-11-03), 11 of the last 38 CI runs that ran Coverage failed it, and the PR diff isn't the cause. For example, #4231 changes only help text and failed it twice, while #4243 passed on its first run.
Current behavior
Example: job 127467430367 on #4231 at 5b9d769.
runs-on: blacksmith-4vcpu-ubuntu-2404,timeout-minutes: 30. The job ran 10 minutes, so it did not time out.- The
Run coveragestep printsagent-device: building macOS helper (first run or helper update)four times at 19:12:57. It then prints nothing for about 3.5 minutes, and the step fails at 19:16:18. - No test is named as failing, and no
Test FilesorTestssummary is printed. There is noELIFECYCLEline and noProcess completed with exit codeline. - The error lines earlier in the log (
--iterations must be a positive integer,Unknown option '--not-a-flag', Android helper diagnostics) are expected output from tests of failure paths.
A step that ends with no exit line at all suggests the test process was killed, for example by the out-of-memory killer, not that a test failed.
Required behavior
- Find out what ends the step, for example from the runner's memory usage or kernel OOM messages, or by running the coverage step with
--reporter=verboseand a heap limit. - If it is memory, give the Coverage job enough of it: a larger Blacksmith runner, a GitHub-hosted runner, or fewer coverage workers. Choose by measurement.
- If the process can be killed, the job must print a cause, such as the signal or the last test file, so the next failure names it.
Completion conditions
- 10 consecutive Coverage runs on PRs pass, or each failure names a failing test.
- The fix is a workflow or test-runner configuration change. No test is skipped and no coverage threshold is lowered.
Dependencies
Related to #4185 (the Blacksmith migration).
Activity
apex-sentinel-staging commentedon Nov 5, 2026
Apex Sentinel · Issue review
Suggested label:
bugThe issue reports an existing CI Coverage job intermittently failing silently after the Blacksmith migration, with concrete failing run examples, timing, missing exit/error output, and a plausible process-kill/OOM hypothesis. It provides enough detail for a maintainer to investigate the runner/workflow or coverage runner configuration.
Required sections
thymikee commentedon Nov 6, 2026
A lead to check first. A rebase comment on #4231 (#4231) attributes a Coverage failure there to a
launch-confirmation.test.tsturn-bound flake, fixed on main by #4225 (merged 2026-11-06 04:41Z,a9e286e30b). The run analyzed above, job 127467430367, never mentionslaunch-confirmation, so that explanation may not cover every failure. Before changing runners, check whether Coverage still dies without naming a test on PR runs based ona9e286e30bor later. If it stops, close this issue as fixed by #4225.thymikee commentedon Nov 6, 2026
Still happening after #4225: Coverage on #4245 at
cf83272, which contains #4225, died the same way (job 127909416075). The step goes quiet afterbuilding macOS helperat 13:11:59, then fails at 13:14:23 with no test named, no test summary and no exit line. So #4225 is not the whole fix, and the original analysis above still applies.