Skip to content

🤖 fix: aggregated writes treat a proxy 3xx and a build answer without the requested build as uncertain - #427

Merged
ThomasK33 merged 17 commits into
mainfrom
aggregated-write-followups
Oct 31, 2026
Merged

ThomasK33 merged 17 commits into
mainfrom
aggregated-write-followups

Conversation

@ThomasK33

@ThomasK33 ThomasK33 commented Oct 30, 2026 •

Copy link
Copy Markdown
Member

Summary

Closes the six follow-ups that review and verification of #400 left open for aggregated writes. Three are fixed in code: a stopped workspace create no longer reports an empty build, a proxy 3xx without Location on a write returns the uncertain 504, and the coder-errors log keeps the error of a failed post-activation re-read. Build answers must now also carry the requested transition. The remaining limits (answers that a proxy forges, partly written HTTP/1.1 headers, Go's HTTP/2 retry) are documented as limits of the checks.

Fixes #421

Background

The goal is to land the follow-up issues of the last run on main. The #242 contract stays: a write that can have reached Coder never returns the non-uncertain 500. A write counts as "not sent" only if no request headers were written on any attempt.

Item by item:

  1. Forged success answers. Partly fixed, the rest documented. A build answer must have a nonzero ID and the requested transition. Real Coder always returns both (postWorkspaceBuilds returns the new build). This rejects 201 {"id":"<uuid>"}, the example from the review thread. I did not add a workspace-ID check: a body that echoes one request field can echo another, so a second field adds no real protection. Also, the apiserverapp integration mock answers builds without workspace_id. A full forgery, or a template metadata answer with the right ID and stale content, cannot be told apart from Coder's answer. I documented these cases as a trust assumption about the network path to Coder.
  2. Proxy 3xx without Location. Fixed. IsUncertainStatus now counts every 3xx. The SDK sees a 3xx only when net/http did not follow it, and the server that answered it got the write. Writes (and the start/stop confirming re-read) treat it as uncertain. Reads keep MapCoderError. The TestMapCoderWriteError row that pinned 300 as a 500 for writes now expects the uncertain 504.
  3. Lost log detail. Fixed. The new coder.UncertainWriteErrorFrom logs the original error, with its status or 0 for a transport error, before it returns the uncertain 504.
  4. Create with spec.running=false. Fixed. When the stop POST fails, or its answer is not the stop build (201 {}, null, an ID only), the server re-reads the workspace and returns Coder's latest build. Before this change, the API reported an empty build ID. The server uses the re-read only if its latest build has an ID, a status, and the start or stop transition. If the re-read fails or has no such build, the result depends on the stop:
    • A stop that can have reached Coder (an uncertain answer or a timeout) returns the uncertain 504. The message says that the stop build was not confirmed after the workspace was created. The server never returns the start build as a success when it can be stale.
    • A stop that Coder refused, or that never reached Coder, keeps the start build, and the create succeeds.
  5. HTTP/1.1 partial headers. Documented. The write error is not one of the not-sent errors, so the 504 still holds.
  6. HTTP/2 retry after RST_STREAM(PROTOCOL_ERROR). Documented. This is Go behavior (x/net/http2: Server rejects Transport with ErrCodeProtocol stream error golang/go#47635). It applies only to https Coder URLs that negotiate HTTP/2.

Found in verification of this PR: a start/stop whose POST answer left the result open (307, 502) and whose confirming re-read returned the new build in a workspace without its name returned a non-uncertain 500. The queued conversion failed after the write. It now returns the uncertain 504. In the stopped create, the final conversion uses only the identity of the create answer, which the server checked before the stop. A re-read failure there is covered above, so no post-write error path returns a 500.

Found in the second verification round (last code round):

  • Only Coder's own error answer proves a refusal. MapCoderWriteError treated every non-timeout status as definite. A proxy can apply the write and then answer with an HTML 400 or 500. The new coder.IsCoderErrorAnswer accepts only a JSON body with a message. It rejects the SDK's own messages for a non-JSON, empty or message-less body. Any other error answer to a write is now uncertain. This fixes the stopped create (HTML 400/500 stop answer, then a failed re-read used to return the stale start build) and every other write.
  • Start/stop 409. A 409 goes to the conflict re-read only if it is Coder's own answer. Otherwise it goes to the confirming re-read. When the re-read after a 409 fails, the result is now the uncertain 504, because a proxy can also send a 409 in Coder's format. Before this, the result was a 500.
  • Confirming re-read. A re-read confirms a start/stop only with a nonzero build ID, a status and the requested transition. Before this, a build without an ID returned Queued with the zero UUID, and a build without a status returned Queued.

Implementation

  • internal/aggregated/coder/errors.go: IsUncertainStatus covers 200..399 plus 502/503/504. New UncertainWriteErrorFrom(err, ...).
  • internal/aggregated/storage/errors.go: new isRequestedBuild(build, transition), used by workspace update, delete, the start/stop subresource and the stopped create.
  • internal/aggregated/storage/workspace.go: the stopped create re-reads by workspace ID when the stop answer is not the stop build. If the stop can have reached Coder and the re-read fails, it returns UncertainWriteErrorFrom with the re-read error.
  • internal/aggregated/storage/workspace_transition.go: one uncertain() 504 for the unconfirmed re-read and for a queued workspace that does not convert. The message drops "in time", because a malformed re-read is not a timeout.
  • internal/aggregated/storage/template.go: the activation re-read failure goes through UncertainWriteErrorFrom.
  • docs/reference/aggregated-api-behavior.md: updated the uncertain list and the start/stop retries. Added the stopped-create behavior and a "Limits of the checks" section for items 1, 5 and 6.

Validation

The first commit adds only the tests. They fail on main, and the fix commits make them pass:

  • TestMapCoderWriteError: 300 and 307 give the uncertain 504 (on main: a 500 "with status 300").
  • TestWriteOutcomes: a new "proxy 307 without Location" outcome for every write (on main: a 500 "with status 307"). A new "201 with only a build ID" outcome for workspace update and delete (on main: success).
  • TestWorkspaceTransitionUncertainPost: 201 with only a build ID gives the uncertain 504 after one re-read (on main: Queued with the forged ID). 307 without Location after the build was queued gives Queued through the re-read (on main: a 500).
  • TestWorkspaceCreateStoppedStopBuildAnswer: {}, null, an ID only and a proxy 502 all return Coder's real latest build, both when the stop reached Coder and when it did not (on main: an empty build ID).
  • TestWorkspaceCreateStoppedFailedReRead: a stop that Coder applied, answered 201 {} or a proxy 502, followed by a re-read that fails with 502 or 404, gives the uncertain 504 and no object (on 239676e4: success with the stale start build and running=true). A re-read whose latest build has only an ID also gives the 504 (on b03e004c: success with running=false, from the Codex finding). A control case, a Coder 400 for a stop that never reached Coder, still succeeds with the start build.
  • TestWorkspaceTransitionMalformedConfirmingReRead: POST 307 or 502 after the build was queued, then a re-read without the workspace name, gives the uncertain 504 (on 239676e4: 500 "without a valid name").
  • Round 3 (each fails on 05aa1abd):
    • TestWriteOutcomes gets "proxy 400 page", "proxy 500 page" and "409 with an empty JSON object" for every write. 31 subtests fail without the IsCoderErrorAnswer check.
    • TestWorkspaceCreateStoppedFailedReRead gets HTML 400/500 stop answers combined with re-read 502/404.
    • TestWorkspaceTransitionMalformedConfirmingReRead covers re-reads without the build ID or the status, a proxy 409 page and a Coder-format 409, each followed by a re-read 502.
    • TestMapCoderWriteError gets a 500 without a Coder message, which is uncertain. The definite rows now carry a Coder message.
  • TestTemplateFilesUpdateReReadAfterActivation: the coder-errors log keeps a Coder 503 with its status and a refused connection with status 0 (on main: no line).

Round 3 overlays: the verifier's proxy_refusal and proxy_conflict_read_failure probes pass on the new head. The missing_build_id and missing_build_status probes now get the uncertain 504 they assert. They still stop at their precondition operatorReads == 1, because the server now refuses the re-read before it calls /users/me.

The verifier's probe tests for the last two cases (stopped_reread_probe_test.go, transition_reread_probe_test.go) also pass on the new head as overlays.

Negative controls on the fix commits: reverting only the IsUncertainStatus bound makes the 307 start/stop row fail. Restoring the previous stopped-create code makes the proxy 502 row fail.

I ran no full-route (generic apiserver) tests. Each new outcome comes from storage after the Coder call, and generic apiserver validation runs before storage, so it cannot intercept them.

Local gates passed: make verify-vendor, make test, make build, make lint, make docs-check (nix dev shell), plus cspell and markdownlint-cli2 on the changed page. An independent pre-PR reviewer (gpt-6.1-sol) reported no material findings.

Round 4 (two Codex findings on 22b1a827):

  • Stale start build after an uncertain stop. This happens with real Coder: the stop answer times out while Coder is still queueing it, and the re-read still shows the start build. After a stop that can have reached Coder, only a stop build (with an ID and a status) in the re-read confirms it. Anything else is the uncertain 504. The start build is still a result after a stop that Coder refused (an error answer in Coder's format) or that was never sent. New TestWorkspaceCreateStoppedStopBuildAnswer rows cover 201 {}, proxy 502 and proxy 504 for a stop that Coder has not queued yet (on 22b1a827: success with running=true), and a Coder 400 for the stop (start build, success).
  • Lost log detail after a Coder-format 409. When the re-read after that 409 fails, the server now logs the error through the new coder.LogHiddenError. TestWorkspaceTransitionConflictReReadFailureIsLogged checks the line (fails on 22b1a827).

Docs guarantee to test map

Every guarantee sentence in "Coder timeouts" and "Time limit and retries" of docs/reference/aggregated-api-behavior.md maps to a test. I scoped the audit to these two sections, the ones this change and #400 own. Other sections of the page are unchanged.

Docs claim Test
502/503 TestWriteOutcomes (proxy 502/503), TestMapCoderWriteError
Error answer not in Coder's format (HTML 400/409/500, empty body, JSON without message) TestWriteOutcomes (proxy 400 page, proxy 500 page, 500 with an empty body, 409 with an empty JSON object), TestWorkspaceTransitionMalformedConfirmingReRead (proxy 409 page)
2xx that Coder does not send TestCreateWithUnreadable2xxAnswerIsUncertain (201/200 HTML, 200 with Coder JSON), TestWriteOutcomes (200 and 201 with malformed JSON, 201 with an empty body, 201 with a JSON array)
Answer without the needed object TestWriteOutcomes (201 {}, null, answer without the ID, 201 with only a build ID)
3xx not followed TestWriteOutcomes (proxy 307), TestMapCoderWriteError (300, 307)
Redirect to another origin, or a failed next request TestMapCoderWriteError, TestMapCoderWriteErrorRealTransport, config_redirect_test.go
Activation re-read, final read TestTemplateFilesUpdateReReadAfterActivation, TestTemplateUpdateReadAfterChange
Broken connection, TLS record/alert, HTTP/2 retry refused, TLS handshake refused TestMapCoderWriteErrorRealTransport, TestWriteOutcomes (closed, body breaks off)
Not-sent list TestMapCoderWriteErrorRealTransport. DNS failure uses a synthetic error in TestMapCoderWriteError.
Other transport errors are uncertain TestMapCoderWriteError (reset after the send)
Stopped create TestWorkspaceCreateStoppedStopBuildAnswer, TestWorkspaceCreateStoppedFailedReRead, TestWorkspaceCreateStoppedStopNotSent (new)
Partly written HTTP/1.1 headers give 504 TestMapCoderWriteErrorPartlyWrittenHeaders (new, ported from the verifier's probe)
Message without the Coder URL, full error in coder-errors TestMapCoderErrorHidesCoderURL, TestMappedErrorsLogTheFullError (new), TestTemplateFilesUpdateReReadAfterActivation
Start/stop 409 re-read, uncertain POST and confirming re-read TestWorkspaceTransitionCoderConflictRereads, TestWorkspaceTransitionUncertainPost, TestWorkspaceTransitionMalformedConfirmingReRead

Two sentences are limits, not guarantees, and the page says so. The HTTP/2 duplicate is Go behavior that the server does not detect. The trust assumption is now explicit: nothing between the server and Coder forges a well-formed Coder JSON answer. A forged success answer can make the server report success, and a forged Coder-format error can make it report a refusal. I removed one claim that no test covers ("a retry of the create gets 409 AlreadyExists"). The page now says not to retry the create.

Test fix: closedAddress returned a port that it had just closed. With more parallel subtests, another test's server took that port, and the make test run flaked once (a refused write reached another test's proxy). It now returns 127.0.0.1:1, which is outside the ephemeral range. go test -count=4 ./internal/aggregated/storage/ passes.

Risks

  • A write that returned the 500 for a 3xx now returns the uncertain 504. Clients that retried on that 500 now get the "re-read before you retry" message instead.
  • A write that gets an error page from a proxy (for example an ingress 403 or 404 HTML page) now returns the uncertain 504 instead of 403/404/500. The server cannot tell whether the write reached Coder. A re-read shows the real state, and the read keeps the proxy's status.
  • A stopped create can now return the uncertain 504 although the workspace exists. A retry of the create gets 409 AlreadyExists. Re-read the workspace and update spec.running instead.
  • A start/stop/update/delete answer with an ID but another transition is now uncertain. Real Coder does not send that answer. A fake Coder that omits transition in build answers would now see uncertain results.

Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: high

Change-Id: I0de1665234fe62303560942880cf94d6c4f1fd1b
Signed-off-by: Thomas Kosiewski <tk@coder.com>
…he requested build as uncertain (#421)

Change-Id: I7c8ce7baed2ed0fe35a881dc224b6337414acd74
Signed-off-by: Thomas Kosiewski <tk@coder.com>
…rs, HTTP/2 retries (#421)

Change-Id: I49bd466932f1dafc15ffb785dd24b3a4a9e9b60f
Signed-off-by: Thomas Kosiewski <tk@coder.com>
… answer without the stop build (#421)

Change-Id: I098d050e62e27a5ace15fdc12dd98925acce5129
Signed-off-by: Thomas Kosiewski <tk@coder.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-31T01:05:56.602614Z 8775be6 Manual request
🔒 Security Review ✅ Completed 2026-10-31T01:05:52.499878Z 8775be6 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 239676e4ca

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

…must be uncertain (#421)

Change-Id: I3a19b029a165970d31de9287a82caee4fa6b0e0e
Signed-off-by: Thomas Kosiewski <tk@coder.com>
… answer nor the re-read confirms the build (#421)

Change-Id: I63e691d570029dbda529d3e5cbab94532075940a
Signed-off-by: Thomas Kosiewski <tk@coder.com>
…tain (#421)

Change-Id: I5c9bbcffc37d24d500c9a7586713a9661933651c
Signed-off-by: Thomas Kosiewski <tk@coder.com>
…s the uncertain 504 (#421)

Change-Id: Iafd41fcefa94c277e520dcf4f8f332305e8cb4a1
Signed-off-by: Thomas Kosiewski <tk@coder.com>
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: b03e004c80

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b03e004c80

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…ly an ID (#421)

Change-Id: Ie53bfc3c17ce06ec5f61bbd6dc5e2e6ccdd3cfcd
Signed-off-by: Thomas Kosiewski <tk@coder.com>
…that has an ID and a status (#421)

Change-Id: I45d6350a7cbd631e0b75c47e241ee30b951ce91d
Signed-off-by: Thomas Kosiewski <tk@coder.com>
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Breezy!

Reviewed commit: 05aa1abd74

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 05aa1abd74

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

…irming re-reads, are uncertain (#421)

Change-Id: I6e51f82724d1369ee23c78401a9dbaf9c040a70a
Signed-off-by: Thomas Kosiewski <tk@coder.com>
…irming re-read needs a build ID and status (#421)

Change-Id: I1a7327a2046c2221500c876dac88a4d7755eabde
Signed-off-by: Thomas Kosiewski <tk@coder.com>
…ion (#421)

Change-Id: Iaf022227c99c14bc0c191b221d4276f4eec9b487
Signed-off-by: Thomas Kosiewski <tk@coder.com>
… take

Change-Id: I3abf08046eca2466bb13706a6e4e7d74a27ed814
Signed-off-by: Thomas Kosiewski <tk@coder.com>
…umption is explicit (#421)

Change-Id: I86b3da046a268efb0b006fe46315283b45fa084c
Signed-off-by: Thomas Kosiewski <tk@coder.com>
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 22b1a82785

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 22b1a82785

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…iled conflict re-read is logged (#421)

Change-Id: I3ffef0aa1e69517e6c6e63d5a29508e189bedcc3
Signed-off-by: Thomas Kosiewski <tk@coder.com>
…e re-read, and a failed conflict re-read is logged (#421)

Change-Id: I3dcca39489d5a99819e73d0c00fcc80af57de944
Signed-off-by: Thomas Kosiewski <tk@coder.com>
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 8775be6f56

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. You're on a roll.

Reviewed commit: 8775be6f56

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ThomasK33
ThomasK33 added this pull request to the merge queue Oct 31, 2026
Merged via the queue into main with commit 168bfcb Oct 31, 2026
12 of 13 checks passed
@ThomasK33
ThomasK33 deleted the aggregated-write-followups branch October 31, 2026 01:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Aggregated writes: follow-ups after #400 (forged answers, 3xx without Location, log detail, stop build 201 {})

1 participant