Repository navigation
fix(stovepipe): fail validations that exceed max_validation_ms - #967
Open
sbalabanov wants to merge 1 commit into
Open
sbalabanov wants to merge 1 commit into
sbalabanov wants to merge 1 commit into
Conversation
sbalabanov
force-pushed
the
sergeyb/sp-oct-002-validation-deadline
branch
from
October 27, 2026 23:43
655b844 to
cfb56bd
Compare
buildsignal held a nonterminal build unconditionally, and holds do not consume the retry budget, so a runner that kept answering accepted or running was re-polled forever. The request never dead-lettered and, at MaxConcurrent 1, kept the queue's only build slot. Record Build.CreatedAt and add QueueConfig.MaxValidationMs (two hours in the reference config; zero disables it). When a poll leaves the build nonterminal past the deadline, buildsignal runs the normal terminal path and marks the request failed with reason validation_timeout. The timed-out build established nothing: it keeps its polled status, is not recorded as the terminal build, and gets no build_finished entry or terminal analysis. A retry re-derives the same timeout from the poll, and finishRequest is a no-op once the outcome is stored. Existing MySQL deployments must add created_at to build before deploying. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
sbalabanov
force-pushed
the
sergeyb/sp-oct-002-validation-deadline
branch
from
October 27, 2026 23:46
cfb56bd to
37e2795
Compare
sbalabanov
marked this pull request as ready for review
October 27, 2026 23:47
Contributor
Author
|
Two things for the reviewer, neither blocks CI: Default value needs a human call. Merge order. This PR is stacked on #965. It also touches |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why?
buildsignal holds a nonterminal build on every poll, and holds do not consume the retry budget, so a runner that keeps answering
acceptedorrunningis re-polled forever. Nothing dead-letters and nothing releases the build slot. Repro:MaxConcurrent1.runningon every poll, so the buildsignal message holds and redelivers indefinitely.What?
BuildgainsCreatedAt, which build sets at creation and MySQL persists in a newbuild.created_atcolumn.QueueConfiggainsMaxValidationMs(zero disables the deadline). The reference default store sets it to two hours. When a poll leaves the build nonterminal past the deadline, buildsignal runs the same terminal sequence as a polled failure and marks the requestfailedwith reasonvalidation_timeout(stored on the request by #965). The timed-out build established nothing: it keeps its last polled status, the request'sTerminalBuildIDstays empty, and there is nobuild_finishedentry or terminal analysis publish. Existing MySQL deployments must addcreated_at BIGINT NOT NULL DEFAULT 0tobuildfirst.Convergence: a retry after any failure in the timeout sequence polls again, finds the build still nonterminal and still past the deadline, and takes the same path with the same reason;
finishRequestis a no-op once the outcome landed. If the build reached a real terminal status in the meantime and the request has no outcome yet, that real verdict is recorded instead. Nothing lists nonterminal builds and the request is terminal, so nothing reopens.Test Plan
make fmt && make lint && make check-tidy && make check-gazelle && make check-mocks./tool/bazel test //stovepipe/... //service/stovepipe/... --test_output=errors && ./tool/bazel test //test/integration/stovepipe/extension/storage/mysql:go_default_test --test_output=errorsStack
Audit: SP-OCT-002 in doc/audit/stovepipe-fable-v2.md (Sol v2 #12, Opus v2 #4).
🤖 Generated with Claude Code