Repository navigation
fix: keep scheduled builds running through a busy store and planner skips - #1413
Merged
Merged
Conversation
…the 1.10x cold ratio The fresh-process cache guards failed on runner speed alone: cold wall and CPU spread 1.6x to 1.9x between hosted runner classes, while cold over the in-job cache-disabled control stayed within 1.05 on CPU. Their wall budgets now scale by the fixed calibration workload the dense guards already use. A slower compiler is caught by the same-runner base/head comparisons. Their cold and model analysis allowances were sized against main before native analysis; main now has it on both sides, and recent readings sit at 0.95 to 1.03, so they return to 1.10x. An injected slowdown of 20% of cold wall fails the restored limit at 1.234.
When another connection held the SQLite control store's write lock past the busy timeout, the scheduler's lease renewal failed with "database is locked" and the scheduler stopped, terminating healthy workers and exiting the control service. A build that would have succeeded ended failed. The scheduler loop and platform maintenance now retry their next cycle after a transient store failure (SQLite busy or locked, or the existing transient PostgreSQL errors). Every write still rechecks the scheduler lease, and a retry reruns claim recovery so a claim with an uncertain commit is still adopted. The API answers a transient store failure with 503, and workers retry such a run report with the same event ID. A worker finalized before its child started publishes its launch failure first. A run whose adopted worker stopped on its own after a restart now fails with a reason that names the restart.
kvlonge
enabled auto-merge (squash)
November 11, 2026 11:02
A scheduled build reported a source-freshness skip as finished and then reported the same node as started when the executor reached it. The control API rejects that change, so the worker stopped and every node it had not reached yet ended skipped, including branches independent of the stale source. Workers now never report a start for a node whose outcome is already final.
…o consolidate/scheduler
…nto consolidate/scheduler
This was referenced Nov 11, 2026
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Nov 11, 2026
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Nov 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
This combines three fixes so they share one CI run. Each was reviewed and passed checks on its own.
Changes
ControlError::transientcovers SQLite busy and locked. The scheduler and platform maintenance retry the next cycle, the API answers 503, and workers retry a 503 report with the same event ID. A lost worker after a restart now says so. DuckShop:busy_control_store.json,lost_worker_restart.json.worker_freshness_skip.yml.Verification
make check-cipasses.