Skip to content

Depot runners lost mid-job evicted 2 merge-queue entries in 27h #1660

Open
Open
Depot runners lost mid-job evicted 2 merge-queue entries in 27h#1660
Labels
agent:needs-humanagent:triagedci-janitorOpened by the CI janitor routine (flakes, redundant tests, CI perf)priority:p3

Description

@mikolalysenko
Mikola Lysenko (mikolalysenko)
opened on Oct 21, 2026

Two merge-queue entries were evicted in about 27 hours because a Depot runner (depot-ubuntu-24.04-4) stopped reporting partway through a job. GitHub then failed the job with "The self-hosted runner lost communication with the server". Both jobs normally finish in 1–5 minutes on the same label. The failed ones were declared lost about 10 minutes after the runner went quiet.

When (UTC) merge_group run Job Runner Normal duration Failed after
2026-10-20 16:48 39460266425 (PR #1617) e2e (ubuntu-latest, e2e_vex_build, pipenv) depot-wkpbh4r80h ~1 min 10m00s
2026-10-21 19:02 39657631762 (PR #1593) e2e (ubuntu-latest, e2e_gradle_discovery_build …) depot-v42fm5z0wd ~5 min 15m35s

Both PRs merged after a rebuild, so each loss cost one full queue cycle (about 10 minutes plus the rebuild of every entry behind them). Neither job log could be downloaded, so I can't tell whether the runner died on its own or was starved for memory. The Gradle leg is the heaviest Linux e2e leg. The pipenv leg is light, which points away from memory pressure in that case.

This repo can't fix the cause; it's runner infrastructure. Next steps for a maintainer:

  • Check the Depot dashboard for instance terminations or OOM kills at those times.
  • If this keeps happening, consider larger runners for the e2e legs or a Depot support ticket.

The CI janitor will add new occurrences here.

Generated by Claude Code

Activity

  1. Mikola Lysenko (mikolalysenko)
    added
    ci-janitorOpened by the CI janitor routine (flakes, redundant tests, CI perf)
    agent:needs-human
    agent:triaged
    priority:p3
    on Oct 21, 2026

  2. mikolalysenko commented on Oct 21, 2026

    @mikolalysenko
    Mikola Lysenko (mikolalysenko)
    on Oct 21, 2026
    CollaboratorAuthor

    [agent] Triage (CI lane): this is runner infrastructure, so no change in this repo fixes it. scripts/merge-queue-fail-fast.py already ignores runner-loss failures, which keeps a lost runner from cancelling sibling jobs. It cannot save the evicted entry, though. Labelled p3 and needs-human for the Depot dashboard and support-ticket checks above. The CI janitor keeps logging new occurrences here.

    Generated by Claude Code

  3. mikolalysenko commented on Oct 22, 2026

    @mikolalysenko
    Mikola Lysenko (mikolalysenko)
    on Oct 22, 2026
    CollaboratorAuthor

    New occurrence (3rd eviction in ~49h), a different failure mode from the first two:

    When (UTC) merge_group run Job Runner What happened
    2026-10-22 17:14 39820682471 (PR #1664) yarn-berry 4.18.0 (ubuntu-latest) depot-btxmz0j7s6 Job created 17:14:32, the runner only picked it up at 17:21:24 (~7 min wait), then failed at "Getting action download info" with ##[error]Step canceled by GitHub (Depot doc). Every other job in the run passed.

    The cancelled job failed ci-ok, the queue removed #1664 at 17:24:11, and the five entries behind it (#1652, #1653, #1640, #1642, #1663) were rebuilt on new refs. That cost roughly one ~12 minute queue cycle for all of them. #1664 itself is still open and needs re-queueing once it's green.

    There's still no in-repo fix: the job never ran a step. Same next steps as above, plus checking the Depot dashboard for slow job assignment around 17:14–17:21 UTC on 2026-10-22.

    Generated by Claude Code

  4. mikolalysenko commented on Oct 22, 2026

    @mikolalysenko
    Mikola Lysenko (mikolalysenko)
    on Oct 22, 2026
    CollaboratorAuthor

    New occurrence, and a fourth failure mode: a required clippy check stayed in_progress after its job had finished. This stalled the whole merge queue for about 77 minutes.

    When (UTC) merge_group run Job Runner What happened
    2026-10-22 17:24 39822372512 (PR #1653, queue head) clippy (attempt 1, job 119976695886) depot-0gnjv73jr6 Every step, Complete job included, finished with success by 17:27. ci-ok in the same run saw clippy: success and passed, and the run completed success at 17:38. The clippy check run on b12ecc1 still read in_progress with no completed_at at 18:38.
    2026-10-22 18:40 same run, attempt 2 (re-run by the CI janitor) clippy (job 120015236768) depot-vwjzg530l6 Same thing again: all steps succeeded by 18:41:50, ci-ok passed at 18:48, the run completed success at 18:49, and the check stayed in_progress. The attempt-1 check flipped to completed/success (completed_at 17:27:37) only after the re-run.

    Impact: clippy is a required check, so the queue head waited from 17:38 until the 90-minute check_response_timeout evicted #1653 at 18:55. Nothing merged to main in that time (last merge bc307ce at 17:38). Seven entries behind it (#1640, #1642, #1663, #1667, #1644, #1658, #1664) had already passed their own merge-group CI and waited too. After the eviction the queue rebuilt all of them. #1653 was re-added at 18:55.

    What this shows:

    • Re-running the job does not clear the stuck check. The new attempt got stuck the same way. So a janitor that auto-reruns stuck checks would not help, and I'm not opening a PR for one.
    • Both stuck jobs ran on Depot runners, and nothing in the job touches the Checks API. That points at the runner's job-completion report to GitHub not landing, not at anything in this repo.
    • The 90-minute queue timeout is the only backstop. Recent merge-group CI runs take 11 to 14 minutes. See the settings issue filed alongside this comment for lowering the timeout.

    For the maintainer, in addition to the Depot dashboard check above: report the two job IDs (119976695886, 120015236768) to Depot support as "job finished, GitHub check never completed".

    Generated by Claude Code

  5. mikolalysenko commented on Oct 22, 2026

    @mikolalysenko
    Mikola Lysenko (mikolalysenko)
    on Oct 22, 2026
    CollaboratorAuthor

    CI janitor (2026-10-22 19:45 UTC): two more occurrences since the last update. Both evicted an entry.

    When (UTC) merge_group run Job Runner What happened
    2026-10-22 18:55 39835618551 (PR #1642) e2e (ubuntu-latest, e2e_vex_build, pip:: --ignored, 22 23 24 25 26) depot-b01s6vn6kr "The self-hosted runner lost communication with the server" (the only annotation). Same mode as the first two.
    2026-10-22 19:17 39838789738 (PR #1664) lint-ecosystems depot-5l9kdtstdq Started 19:17:33, cancelled in Set up job at 19:24:12. No step ran. Same mode as the 17:14 yarn-berry cancel. Every other job passed, and ci-ok failed only on this one.

    #1664 has now been evicted twice by the Set-up-job cancel mode (17:24 and 19:31). Its head da3c353 passed full merge-group CI twice in between. It is approved, clean and green, but it is out of the queue now with auto-merge off.

    Tally: 5 Depot-side evictions in ~51h, 3 of them today. Every one cost a ~12 min rebuild of each entry behind it. Two modes: runner lost mid-job (3) and a ~7 min pickup wait ending in a GitHub cancel during Set up job (2). Nothing in the repo causes either: merge-queue-fail-fast.py ignores runner loss, and the janitor workflow cancels whole runs whose ref is gone, never single jobs. A maintainer still needs to look at the Depot dashboard or open a support ticket.

    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent:needs-humanagent:triagedci-janitorOpened by the CI janitor routine (flakes, redundant tests, CI perf)priority:p3

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions