Repository navigation
🤖 fix: aggregated writes treat a proxy 3xx and a build answer without the requested build as uncertain - #427
Conversation
Change-Id: I0de1665234fe62303560942880cf94d6c4f1fd1b Signed-off-by: Thomas Kosiewski <tk@coder.com>
…he requested build as uncertain (#421) Change-Id: I7c8ce7baed2ed0fe35a881dc224b6337414acd74 Signed-off-by: Thomas Kosiewski <tk@coder.com>
…rs, HTTP/2 retries (#421) Change-Id: I49bd466932f1dafc15ffb785dd24b3a4a9e9b60f Signed-off-by: Thomas Kosiewski <tk@coder.com>
… answer without the stop build (#421) Change-Id: I098d050e62e27a5ace15fdc12dd98925acce5129 Signed-off-by: Thomas Kosiewski <tk@coder.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
@codex security review |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
…must be uncertain (#421) Change-Id: I3a19b029a165970d31de9287a82caee4fa6b0e0e Signed-off-by: Thomas Kosiewski <tk@coder.com>
… answer nor the re-read confirms the build (#421) Change-Id: I63e691d570029dbda529d3e5cbab94532075940a Signed-off-by: Thomas Kosiewski <tk@coder.com>
…tain (#421) Change-Id: I5c9bbcffc37d24d500c9a7586713a9661933651c Signed-off-by: Thomas Kosiewski <tk@coder.com>
…s the uncertain 504 (#421) Change-Id: Iafd41fcefa94c277e520dcf4f8f332305e8cb4a1 Signed-off-by: Thomas Kosiewski <tk@coder.com>
|
@codex review |
|
@codex security review |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b03e004c80
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
|
@codex security review |
|
Codex Review: Didn't find any major issues. Breezy! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
…irming re-reads, are uncertain (#421) Change-Id: I6e51f82724d1369ee23c78401a9dbaf9c040a70a Signed-off-by: Thomas Kosiewski <tk@coder.com>
…irming re-read needs a build ID and status (#421) Change-Id: I1a7327a2046c2221500c876dac88a4d7755eabde Signed-off-by: Thomas Kosiewski <tk@coder.com>
…ion (#421) Change-Id: Iaf022227c99c14bc0c191b221d4276f4eec9b487 Signed-off-by: Thomas Kosiewski <tk@coder.com>
… take Change-Id: I3abf08046eca2466bb13706a6e4e7d74a27ed814 Signed-off-by: Thomas Kosiewski <tk@coder.com>
…umption is explicit (#421) Change-Id: I86b3da046a268efb0b006fe46315283b45fa084c Signed-off-by: Thomas Kosiewski <tk@coder.com>
|
@codex review |
|
@codex security review |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 22b1a82785
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
|
@codex security review |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
|
Codex Review: Didn't find any major issues. You're on a roll. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Summary
Closes the six follow-ups that review and verification of #400 left open for aggregated writes. Three are fixed in code: a stopped workspace create no longer reports an empty build, a proxy
3xxwithoutLocationon a write returns the uncertain504, and thecoder-errorslog keeps the error of a failed post-activation re-read. Build answers must now also carry the requested transition. The remaining limits (answers that a proxy forges, partly written HTTP/1.1 headers, Go's HTTP/2 retry) are documented as limits of the checks.Fixes #421
Background
The goal is to land the follow-up issues of the last run on
main. The #242 contract stays: a write that can have reached Coder never returns the non-uncertain500. A write counts as "not sent" only if no request headers were written on any attempt.Item by item:
postWorkspaceBuildsreturns the new build). This rejects201 {"id":"<uuid>"}, the example from the review thread. I did not add a workspace-ID check: a body that echoes one request field can echo another, so a second field adds no real protection. Also, theapiserverappintegration mock answers builds withoutworkspace_id. A full forgery, or a template metadata answer with the right ID and stale content, cannot be told apart from Coder's answer. I documented these cases as a trust assumption about the network path to Coder.3xxwithoutLocation. Fixed.IsUncertainStatusnow counts every3xx. The SDK sees a3xxonly when net/http did not follow it, and the server that answered it got the write. Writes (and the start/stop confirming re-read) treat it as uncertain. Reads keepMapCoderError. TheTestMapCoderWriteErrorrow that pinned300as a500for writes now expects the uncertain504.coder.UncertainWriteErrorFromlogs the original error, with its status or0for a transport error, before it returns the uncertain504.spec.running=false. Fixed. When the stop POST fails, or its answer is not the stop build (201 {},null, an ID only), the server re-reads the workspace and returns Coder's latest build. Before this change, the API reported an empty build ID. The server uses the re-read only if its latest build has an ID, a status, and the start or stop transition. If the re-read fails or has no such build, the result depends on the stop:504. The message says that the stop build was not confirmed after the workspace was created. The server never returns the start build as a success when it can be stale.504still holds.RST_STREAM(PROTOCOL_ERROR). Documented. This is Go behavior (x/net/http2: Server rejects Transport with ErrCodeProtocol stream error golang/go#47635). It applies only tohttpsCoder URLs that negotiate HTTP/2.Found in verification of this PR: a start/stop whose POST answer left the result open (
307,502) and whose confirming re-read returned the new build in a workspace without its name returned a non-uncertain500. Thequeuedconversion failed after the write. It now returns the uncertain504. In the stopped create, the final conversion uses only the identity of the create answer, which the server checked before the stop. A re-read failure there is covered above, so no post-write error path returns a500.Found in the second verification round (last code round):
MapCoderWriteErrortreated every non-timeout status as definite. A proxy can apply the write and then answer with an HTML400or500. The newcoder.IsCoderErrorAnsweraccepts only a JSON body with a message. It rejects the SDK's own messages for a non-JSON, empty or message-less body. Any other error answer to a write is now uncertain. This fixes the stopped create (HTML400/500stop answer, then a failed re-read used to return the stale start build) and every other write.409. A409goes to the conflict re-read only if it is Coder's own answer. Otherwise it goes to the confirming re-read. When the re-read after a409fails, the result is now the uncertain504, because a proxy can also send a409in Coder's format. Before this, the result was a500.Queuedwith the zero UUID, and a build without a status returnedQueued.Implementation
internal/aggregated/coder/errors.go:IsUncertainStatuscovers200..399plus502/503/504. NewUncertainWriteErrorFrom(err, ...).internal/aggregated/storage/errors.go: newisRequestedBuild(build, transition), used by workspace update, delete, the start/stop subresource and the stopped create.internal/aggregated/storage/workspace.go: the stopped create re-reads by workspace ID when the stop answer is not the stop build. If the stop can have reached Coder and the re-read fails, it returnsUncertainWriteErrorFromwith the re-read error.internal/aggregated/storage/workspace_transition.go: oneuncertain()504 for the unconfirmed re-read and for aqueuedworkspace that does not convert. The message drops "in time", because a malformed re-read is not a timeout.internal/aggregated/storage/template.go: the activation re-read failure goes throughUncertainWriteErrorFrom.docs/reference/aggregated-api-behavior.md: updated the uncertain list and the start/stop retries. Added the stopped-create behavior and a "Limits of the checks" section for items 1, 5 and 6.Validation
The first commit adds only the tests. They fail on
main, and the fix commits make them pass:TestMapCoderWriteError:300and307give the uncertain504(onmain: a500"with status 300").TestWriteOutcomes: a new "proxy 307 without Location" outcome for every write (onmain: a500"with status 307"). A new "201 with only a build ID" outcome for workspace update and delete (onmain: success).TestWorkspaceTransitionUncertainPost:201with only a build ID gives the uncertain504after one re-read (onmain:Queuedwith the forged ID).307withoutLocationafter the build was queued givesQueuedthrough the re-read (onmain: a500).TestWorkspaceCreateStoppedStopBuildAnswer:{},null, an ID only and a proxy502all return Coder's real latest build, both when the stop reached Coder and when it did not (onmain: an empty build ID).TestWorkspaceCreateStoppedFailedReRead: a stop that Coder applied, answered201 {}or a proxy502, followed by a re-read that fails with502or404, gives the uncertain504and no object (on239676e4: success with the stale start build andrunning=true). A re-read whose latest build has only an ID also gives the504(onb03e004c: success withrunning=false, from the Codex finding). A control case, a Coder400for a stop that never reached Coder, still succeeds with the start build.TestWorkspaceTransitionMalformedConfirmingReRead: POST307or502after the build was queued, then a re-read without the workspace name, gives the uncertain504(on239676e4:500"without a valid name").05aa1abd):TestWriteOutcomesgets "proxy 400 page", "proxy 500 page" and "409 with an empty JSON object" for every write. 31 subtests fail without theIsCoderErrorAnswercheck.TestWorkspaceCreateStoppedFailedReReadgets HTML400/500stop answers combined with re-read502/404.TestWorkspaceTransitionMalformedConfirmingReReadcovers re-reads without the build ID or the status, a proxy409page and a Coder-format409, each followed by a re-read502.TestMapCoderWriteErrorgets a500without a Coder message, which is uncertain. The definite rows now carry a Coder message.TestTemplateFilesUpdateReReadAfterActivation: thecoder-errorslog keeps a Coder503with its status and a refused connection with status0(onmain: no line).Round 3 overlays: the verifier's
proxy_refusalandproxy_conflict_read_failureprobes pass on the new head. Themissing_build_idandmissing_build_statusprobes now get the uncertain504they assert. They still stop at their preconditionoperatorReads == 1, because the server now refuses the re-read before it calls/users/me.The verifier's probe tests for the last two cases (
stopped_reread_probe_test.go,transition_reread_probe_test.go) also pass on the new head as overlays.Negative controls on the fix commits: reverting only the
IsUncertainStatusbound makes the307start/stop row fail. Restoring the previous stopped-create code makes the proxy502row fail.I ran no full-route (generic apiserver) tests. Each new outcome comes from storage after the Coder call, and generic apiserver validation runs before storage, so it cannot intercept them.
Local gates passed:
make verify-vendor,make test,make build,make lint,make docs-check(nix dev shell), pluscspellandmarkdownlint-cli2on the changed page. An independent pre-PR reviewer (gpt-6.1-sol) reported no material findings.Round 4 (two Codex findings on
22b1a827):504. The start build is still a result after a stop that Coder refused (an error answer in Coder's format) or that was never sent. NewTestWorkspaceCreateStoppedStopBuildAnswerrows cover201 {}, proxy502and proxy504for a stop that Coder has not queued yet (on22b1a827: success withrunning=true), and a Coder400for the stop (start build, success).409. When the re-read after that409fails, the server now logs the error through the newcoder.LogHiddenError.TestWorkspaceTransitionConflictReReadFailureIsLoggedchecks the line (fails on22b1a827).Docs guarantee to test map
Every guarantee sentence in "Coder timeouts" and "Time limit and retries" of
docs/reference/aggregated-api-behavior.mdmaps to a test. I scoped the audit to these two sections, the ones this change and #400 own. Other sections of the page are unchanged.502/503TestWriteOutcomes(proxy 502/503),TestMapCoderWriteErrorTestWriteOutcomes(proxy 400 page, proxy 500 page, 500 with an empty body, 409 with an empty JSON object),TestWorkspaceTransitionMalformedConfirmingReRead(proxy 409 page)2xxthat Coder does not sendTestCreateWithUnreadable2xxAnswerIsUncertain(201/200 HTML, 200 with Coder JSON),TestWriteOutcomes(200 and 201 with malformed JSON, 201 with an empty body, 201 with a JSON array)TestWriteOutcomes(201{},null, answer without the ID, 201 with only a build ID)3xxnot followedTestWriteOutcomes(proxy 307),TestMapCoderWriteError(300, 307)TestMapCoderWriteError,TestMapCoderWriteErrorRealTransport,config_redirect_test.goTestTemplateFilesUpdateReReadAfterActivation,TestTemplateUpdateReadAfterChangeTestMapCoderWriteErrorRealTransport,TestWriteOutcomes(closed, body breaks off)TestMapCoderWriteErrorRealTransport. DNS failure uses a synthetic error inTestMapCoderWriteError.TestMapCoderWriteError(reset after the send)TestWorkspaceCreateStoppedStopBuildAnswer,TestWorkspaceCreateStoppedFailedReRead,TestWorkspaceCreateStoppedStopNotSent(new)504TestMapCoderWriteErrorPartlyWrittenHeaders(new, ported from the verifier's probe)coder-errorsTestMapCoderErrorHidesCoderURL,TestMappedErrorsLogTheFullError(new),TestTemplateFilesUpdateReReadAfterActivation409re-read, uncertain POST and confirming re-readTestWorkspaceTransitionCoderConflictRereads,TestWorkspaceTransitionUncertainPost,TestWorkspaceTransitionMalformedConfirmingReReadTwo sentences are limits, not guarantees, and the page says so. The HTTP/2 duplicate is Go behavior that the server does not detect. The trust assumption is now explicit: nothing between the server and Coder forges a well-formed Coder JSON answer. A forged success answer can make the server report success, and a forged Coder-format error can make it report a refusal. I removed one claim that no test covers ("a retry of the create gets
409 AlreadyExists"). The page now says not to retry the create.Test fix:
closedAddressreturned a port that it had just closed. With more parallel subtests, another test's server took that port, and themake testrun flaked once (a refused write reached another test's proxy). It now returns127.0.0.1:1, which is outside the ephemeral range.go test -count=4 ./internal/aggregated/storage/passes.Risks
500for a3xxnow returns the uncertain504. Clients that retried on that500now get the "re-read before you retry" message instead.403or404HTML page) now returns the uncertain504instead of403/404/500. The server cannot tell whether the write reached Coder. A re-read shows the real state, and the read keeps the proxy's status.504although the workspace exists. A retry of the create gets409 AlreadyExists. Re-read the workspace and updatespec.runninginstead.transitionin build answers would now see uncertain results.Generated with
xum• Model:anthropic:claude-opus-5-5• Thinking:high