Skip to content

fix: keep scheduled builds running through a busy store and planner skips - #1413

Merged
kvlonge merged 14 commits into
mainfrom
fix/sqlite-busy-scheduler
Nov 11, 2026
Merged

kvlonge merged 14 commits into
mainfrom
fix/sqlite-busy-scheduler

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Why

This combines three fixes so they share one CI run. Each was reviewed and passed checks on its own.

  • With a SQLite control store, a write lock held past the 30 s busy timeout stopped the scheduler. It terminated healthy workers and failed the build.
  • Workers reported a start for nodes the planner had already skipped. The API rejected the start, the worker stopped, and independent branches ended skipped.
  • The fresh-process cache guards failed on runner speed alone. The same-runner cold limits had been sized before native analysis.

Changes

  • Busy store: ControlError::transient covers SQLite busy and locked. The scheduler and platform maintenance retry the next cycle, the API answers 503, and workers retry a 503 report with the same event ID. A lost worker after a restart now says so. DuckShop: busy_control_store.json, lost_worker_restart.json.
  • Skipped nodes: a worker never reports a start for a node whose outcome it already reported. DuckShop: worker_freshness_skip.yml.
  • Perf guards: the cache guard scales its wall budgets by the runner calibration factor. The dense and fresh base/head cold comparisons and the model analysis CPU gate go back to 1.10x. These are test-only changes.

Verification

  • Combined branch: the worker and local recovery integration files pass (35 tests). make check-ci passes.
  • Each part's own checks passed before combining: orchestration scheduler, API and platform tests on SQLite and PostgreSQL. The new busy-lock and restart tests fail on main and pass here, as does the freshness-skip worker test. The same-machine perf guard check failed an injected 20% slowdown at 1.234x.

…the 1.10x cold ratio

The fresh-process cache guards failed on runner speed alone: cold wall and
CPU spread 1.6x to 1.9x between hosted runner classes, while cold over the
in-job cache-disabled control stayed within 1.05 on CPU. Their wall budgets
now scale by the fixed calibration workload the dense guards already use.

A slower compiler is caught by the same-runner base/head comparisons. Their
cold and model analysis allowances were sized against main before native
analysis; main now has it on both sides, and recent readings sit at 0.95 to
1.03, so they return to 1.10x. An injected slowdown of 20% of cold wall
fails the restored limit at 1.234.
When another connection held the SQLite control store's write lock past
the busy timeout, the scheduler's lease renewal failed with "database is
locked" and the scheduler stopped, terminating healthy workers and exiting
the control service. A build that would have succeeded ended failed.

The scheduler loop and platform maintenance now retry their next cycle
after a transient store failure (SQLite busy or locked, or the existing
transient PostgreSQL errors). Every write still rechecks the scheduler
lease, and a retry reruns claim recovery so a claim with an uncertain
commit is still adopted. The API answers a transient store failure with
503, and workers retry such a run report with the same event ID.

A worker finalized before its child started publishes its launch failure
first. A run whose adopted worker stopped on its own after a restart now
fails with a reason that names the restart.
@kvlonge
kvlonge enabled auto-merge (squash) November 11, 2026 11:02
@kvlonge kvlonge added the full-verify Run the full Verify matrix on this pull request label Nov 11, 2026 — with Amp for GitHub
@kvlonge kvlonge changed the title fix: keep the scheduler running through a busy control store fix: keep scheduled builds running through a busy store and planner skips Nov 11, 2026
@kvlonge
kvlonge added this pull request to the merge queue Nov 11, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 11, 2026
@kvlonge
kvlonge added this pull request to the merge queue Nov 11, 2026
@kvlonge
kvlonge removed this pull request from the merge queue due to a manual request Nov 11, 2026
@kvlonge
kvlonge added this pull request to the merge queue Nov 11, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 11, 2026
@kvlonge
kvlonge added this pull request to the merge queue Nov 11, 2026
Merged via the queue into main with commit 16747e6 Nov 11, 2026
51 checks passed
@kvlonge
kvlonge deleted the fix/sqlite-busy-scheduler branch November 11, 2026 23:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-verify Run the full Verify matrix on this pull request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant