🤖 fix: map uncertain aggregated write answers to 504 and list unconfigured namespaces as empty

docs/how-to/troubleshooting.md

@@ -59,7 +59,7 @@ kubectl get deploy,svc -n <namespace>
 
 Coder answered an operator call with a redirect to another origin. The operator does not follow such redirects, so that the operator token stays on the control plane origin. The usual cause is `CODER_REDIRECT_TO_ACCESS_URL=true` in `spec.extraEnv` of the `CoderControlPlane`. Remove it, and redirect browsers from HTTP to HTTPS in your Ingress, Gateway or load balancer instead. See [Redirects to another origin](deploy-controller.md#redirects-to-another-origin).
 
-Most aggregated API calls then answer the caller with a `500` that says `the Coder API could not be reached`. The `coder-errors` line in the aggregated API server log shows the refusal. When Coder or a proxy redirects only the websocket request of the workspace build log stream (`coderworkspaces/log?follow=true`), that line shows `refusing request to a different origin than the coder URL` instead.
+Most aggregated API reads then answer the caller with a `500` that says `the Coder API could not be reached`. Writes answer with the uncertain `504`, because the server that sent the redirect got the write. Re-read the object before you retry. The `coder-errors` line in the aggregated API server log shows the refusal. When Coder or a proxy redirects only the websocket request of the workspace build log stream (`coderworkspaces/log?follow=true`), that line shows `refusing request to a different origin than the coder URL` instead.
 
 ## Operator calls to Coder are slow or time out under load
 
@@ -177,6 +177,7 @@ If the error does not name a field, the ServiceAccount does not have a permissio
 - `500` that mentions `subjectaccessreviews`: the ServiceAccount of the server cannot create SubjectAccessReviews. Apply `config/rbac/auth-delegator-binding.yaml`. In standalone mode, apply `config/apiserver-standalone/auth-delegator-binding.yaml` instead.
 - `403` that says `the Coder API denied the request`: kube-apiserver let the caller in, but Coder refused the call that the server made with its Coder token. The message leaves out the Coder URL and Coder's own text. The server log has the full Coder error, on a line from the `coder-errors` logger.
 - `500` that says `the Coder API failed the request ... with status <code>` or `the Coder API could not be reached`: Coder answered with a server error, or the server could not reach Coder. Read the `coder-errors` line in the server log for the Coder URL and the full error.
+- `504` that says `the result is uncertain`: Coder, or a proxy in front of it, gave an answer to a write that leaves it open whether Coder applied the write. Re-read the object with `kubectl get` before you retry. See [Coder timeouts](../reference/aggregated-api-behavior.md#coder-timeouts).
 
 ## Aggregated reads return `ServiceUnavailable`
 

docs/reference/aggregated-api-behavior.md

@@ -17,7 +17,7 @@ The aggregated API server serves `coderworkspaces`, `codertemplates` and `codert
 | [Template versions](#template-versions) | Read-only: `get` and `list`, no watch. Reads never download template source. |
 | [Promote a template version](#promote-a-template-version) | `codertemplates/<name>/promote` (`create`) makes a version active. A rollback promotes an older version. It needs its own RBAC grant. `dryRun=All` previews it. An unconfirmed result returns `503`, and a concurrent change returns `409`. |
 | [Template builds](#template-builds) | Create and update with `spec.files` wait until Coder completes the import. The full request must complete within 34 seconds. |
-| [Coder timeouts](#coder-timeouts) | A Coder call that runs out of time returns `504` on every resource. A create that gets a broken `2xx` answer returns `504` or `500`. After a `504` on a write, or a `500` on a create, re-read before you retry. |
+| [Coder timeouts](#coder-timeouts) | A Coder call that runs out of time returns `504` on every resource. A write whose answer leaves the result open, for example a `502` from a proxy, also returns `504`. After a `504` on a write, re-read before you retry. |
 | [Workspace build log](#workspace-build-log) | `coderworkspaces/log` returns the latest build log as text. It needs its own RBAC grant and has fixed size, time, and concurrency limits. |
 | [Start and stop](#start-and-stop) | `coderworkspaces/start` and `coderworkspaces/stop` start or stop a workspace. Each needs its own RBAC grant. A repeated request does not queue a second build while one is active. |
 
@@ -168,15 +168,16 @@ A namespace has no Coder backend in these cases:
 
 - `all` mode: the namespace contains no `CoderControlPlane` at all.
 - Standalone mode: the namespace is not the one that `--coder-namespace` names.
+- Standalone mode without `--coder-url` and `--coder-session-token`: every namespace.
 
 In such a namespace, the server answers requests as follows:
 
 | Request | Result |
 | --- | --- |
 | `list` of `coderworkspaces`, `codertemplates`, or `codertemplateversions` in that namespace | `200` with an empty list. The server does not call Coder. |
-| `get`, `create`, `update`, `delete`, and the subresources | An error: `503` in `all` mode, `400` in standalone mode |
+| `get`, `create`, `update`, `delete`, and the subresources | An error: `503` in `all` mode and in standalone mode without a Coder URL and token, `400` in standalone mode |
 | `watch` | Works as described in [Watch](#watch). It shows no events, because no write can succeed in that namespace. |
-| `list` in all namespaces (`-A`) | Unchanged. In `all` mode, it returns `503` when no `CoderControlPlane` is eligible in any namespace. |
+| `list` in all namespaces (`-A`) | Unchanged. In `all` mode, it returns `503` when no `CoderControlPlane` is eligible in any namespace. In standalone mode without a Coder URL and token, it returns `503`. |
 
 A namespace that contains a `CoderControlPlane` that is not eligible is not in this group. For example, operator access is not ready, or the name contains a `.` character. For the eligibility rules, see [Aggregated reads return `ServiceUnavailable`](../how-to/troubleshooting.md#aggregated-reads-return-serviceunavailable). In such a namespace, every request returns `503`, `list` included.
 
@@ -225,17 +226,34 @@ A Coder call that runs out of time returns `504 Gateway Timeout` on every aggreg
 - the request deadline ends first (see [The 34-second write budget](#the-34-second-write-budget), the [log limits](#workspace-build-log), and the [start and stop budget](#time-limit-and-retries)), or
 - Coder itself answers `504`.
 
-A create of a workspace or a template can also get a broken `2xx` answer, for example from a proxy in front of Coder. Coder can have created the object. The server returns `504` for some of these answers and `500` for others:
+A write (create, update, patch, or delete) of a workspace or a template can also get an answer that leaves its result open, for example from a proxy in front of Coder. Coder can have applied the write. The server returns `504`, with a message that says that the result is uncertain, for these answers:
 
-- `504`, with a message that says that the result is uncertain. Examples are a `201` with an HTML page, an empty body, invalid JSON, or a JSON array, and a `200` with an HTML page or with the JSON of Coder.
-- `500`. Examples are a `200` with the type `application/json` and invalid JSON or a JSON array, a `201` with `{}` or `null`, and a connection that breaks while the server reads the answer.
+- A `502` or a `503`.
+- A `2xx` that Coder does not send: a status other than the expected one, or a body that is not the expected JSON. Examples on a create are a `201` with an HTML page, an empty body, invalid JSON, or a JSON array, and a `200` with an HTML page, invalid JSON, or the JSON of Coder.
+- An answer without the object that the server needs: a create answer without the ID or name of the object, a build answer of a workspace update or delete without the ID of the build, a template version answer of a template create or files update without the ID of the version, or a template update answer without the template. An example is a `200` or `201` with `{}` or `null`.
+- A redirect to another origin, which the server refuses, or a redirect whose next request fails. The server that sent the redirect got the write.
+- A template files update whose activation Coder accepted, when the re-read of the template fails or shows another active version.
+- A template update that changed the files or the metadata in Coder, when the read of the template that ends the update fails.
+- A connection that breaks after the server sent the request, before or while the server reads the answer. This includes a malformed TLS record or a TLS alert from Coder or the proxy after the TLS handshake.
+- An error of a later attempt of a request that the server already sent. For example, the HTTP/2 client sends the request again on a new connection after Coder or the proxy resets the stream. If that connection is refused, the write is still uncertain.
+- A TLS handshake that Coder or the proxy refuses, for example because it does not support the TLS version or the cipher suites of the client. The server cannot tell this refusal apart from a TLS alert after the request.
 
-After a `504` or a `500` on a create, re-read the object before you retry.
+A write that the server knows did not reach Coder keeps the `500` that says that the server cannot reach Coder. Nothing changed in Coder, so you can retry it. A write counts as not sent only if the server wrote no request headers on any attempt of the request. Then the server knows it for these errors:
+
+- A refused connection to Coder or to the proxy, or a failed DNS lookup.
+- A proxy that refuses the `CONNECT` tunnel to Coder, for example with `407`.
+- A rejected TLS certificate.
+- An `https` Coder URL that a plain HTTP server answers.
+- A request to an origin other than the Coder URL, which the server refuses before it sends it.
+
+The server cannot tell other transport errors without a status apart from a connection that broke after the send. It treats them as uncertain and returns the `504`.
+
+A `get` or a `list` that gets a Coder object without its ID or the fields of its name, for example `{}`, returns `500` with a message that contains `the Coder API answered`.
 
 The message does not include the Coder URL. The same is true for a `403` that Coder answers, a `500` for a Coder server error, and a `500` when the server cannot reach Coder. These messages name only the resource, the object and the Coder status code. The server log keeps the full Coder error, on a line from the `coder-errors` logger.
 
 - After a `504` on a read, try again later.
-- After a `504` on a write (create, update, patch, or delete), the result is not certain. Coder can have applied the change before the call timed out. Re-read the object with `kubectl get` before you retry.
+- After a `504` on a write (create, update, patch, or delete), the result is not certain. Coder can have applied the change before the call timed out or the answer broke. Re-read the object with `kubectl get` before you retry.
 - A promotion handles this itself. It answers `504` only before it sends the activation, so a `504` on a promotion means that nothing changed. When the activation itself times out or fails without an answer, the server re-reads the template and answers `Promoted`, `409`, or `503`. See [Promote a template version](#promote-a-template-version).
 - After a `504` on start or stop, re-read the latest build of the workspace before you retry. See [Time limit and retries](#time-limit-and-retries).
 
@@ -451,7 +469,7 @@ Other status codes:
 One time budget of 25 seconds covers every Coder call of a request: the lookup, the build request, the re-read after a Coder `409`, and the confirming re-read after an uncertain build request. The calls before the confirming re-read must end within 20 seconds, so at least 5 seconds stay for it. Each call also ends after the Coder request timeout (`--coder-request-timeout`, 30 seconds by default).
 
 - If Coder answers `409` to the build request, another build started after the lookup. The server re-reads the workspace one time. If the latest build already does what you asked, the outcome is `InProgress` or `Unchanged`. Otherwise the server returns `409` with the message from Coder.
-- If the build request times out, fails on the network, or gets `502`, `503`, or `504` (for example from a proxy in front of Coder), the server cannot know whether Coder queued the build. It re-reads the workspace one time, in the rest of the budget. If the latest build is a new build of the requested kind, started by the operator user of the control plane, and of the template version that the request sent (if it sent one), the outcome is `Queued`. A build that another user started in the meantime, for example in the Coder UI, does not count. Otherwise, or when the server cannot read the operator user, it returns `504`, and the message says that the result is uncertain.
+- If the build request times out, fails on the network, gets `502`, `503`, or `504` (for example from a proxy in front of Coder), or gets an answer without the ID of the build (for example `201` with `{}`), the server cannot know whether Coder queued the build. It re-reads the workspace one time, in the rest of the budget. If the latest build is a new build of the requested kind, started by the operator user of the control plane, and of the template version that the request sent (if it sent one), the outcome is `Queued`. A build that another user started in the meantime, for example in the Coder UI, does not count. Otherwise, or when the server cannot read the operator user, it returns `504`, and the message says that the result is uncertain.
 - Known gap: if the operator user itself starts a matching build somewhere else in that window, for example with its token in the Coder CLI, the server cannot tell the two builds apart. It then answers `Queued` and names that build.
 - After a `504`, re-read the latest build of the workspace (`status.latestBuildID` and `status.latestBuildStatus` of the `CoderWorkspace`) before you retry.
 - The server never sends a second build request by itself.

go.mod

@@ -13,6 +13,7 @@ require (
 	github.com/prometheus/client_golang v1.24.1
 	github.com/stretchr/testify v1.12.0
 	github.com/yuin/gopher-lua v1.1.2
+	golang.org/x/net v0.60.0
 	golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da
 	k8s.io/api v0.35.0
 	k8s.io/apiextensions-apiserver v0.35.0
@@ -403,7 +404,6 @@ require (
 	golang.org/x/exp v0.0.0-20260410095643-746e56fc9e2f // indirect
 	golang.org/x/exp/typeparams v0.0.0-20251023183803-a4bb9ffd2546 // indirect
 	golang.org/x/mod v0.41.0 // indirect
-	golang.org/x/net v0.60.0 // indirect
 	golang.org/x/oauth2 v0.36.0 // indirect
 	golang.org/x/sync v0.23.0 // indirect
 	golang.org/x/sys v0.48.0 // indirect

internal/aggregated/coder/config.go

@@ -2,12 +2,16 @@
 package coder
 
 import (
+	"context"
+	"errors"
 	"fmt"
 	"io"
 	"net/http"
+	"net/http/httptrace"
 	"net/url"
 	"strings"
 	"sync"
+	"sync/atomic"
 	"time"
 
 	"github.com/coder/coder/v2/codersdk"
@@ -227,10 +231,75 @@ func newCoderTransport() (*http.Transport, error) {
 	transport.MaxConnsPerHost = maxConnsPerCoderHost
 	transport.MaxIdleConnsPerHost = maxConnsPerCoderHost
 	transport.MaxIdleConns = maxConnsPerCoderHost
+	transport.OnProxyConnectResponse = refuseFailedProxyConnect
 
 	return transport, nil
 }
 
+// errProxyConnectRefused marks a proxy answer other than 200 to the CONNECT for Coder. Go reports
+// that answer with an untyped error, and without this mark notSent cannot tell it apart from a
+// connection that broke after the request.
+var errProxyConnectRefused = errors.New("the proxy refused the CONNECT tunnel")
+
+func refuseFailedProxyConnect(_ context.Context, _ *url.URL, _ *http.Request, resp *http.Response) error {
+	if resp.StatusCode != http.StatusOK {
+		return fmt.Errorf("%w: %s", errProxyConnectRefused, resp.Status)
+	}
+	return nil
+}
+
+// newCoderHTTPClient returns the HTTP client of NewSDKClient for the Coder origin coderURL.
+func newCoderHTTPClient(coderURL *url.URL, transport http.RoundTripper, timeout time.Duration) *http.Client {
+	return &http.Client{
+		// codersdk dials websockets (WorkspaceBuildLogsAfter for ?follow=true)
+		// with a new http.Client that keeps only this Transport, not
+		// CheckRedirect, so the Transport also refuses other origins.
+		Transport: markMaybeSent{next: sameorigin.RoundTripper(coderURL, transport)},
+		Timeout:   timeout,
+		// net/http copies Coder-Session-Token to every redirect target, so refuse
+		// redirects that leave the Coder origin.
+		CheckRedirect: sameorigin.RedirectPolicy(coderURL),
+	}
+}
+
+// errAfterRedirect marks the error of a request that follows a redirect. The server that answered
+// with the redirect got the write, so a follow-up that fails before its own send does not make
+// the write certain.
+var errAfterRedirect = errors.New("the request after a redirect failed")
+
+// errAfterWrite marks the error of a request whose headers net/http wrote. A later error of a new
+// connection does not make the write certain: the HTTP/2 client sends a request again on a new
+// connection after the server reset its stream with PROTOCOL_ERROR, which it can do after it read
+// the request.
+var errAfterWrite = errors.New("the request failed after net/http wrote it")
+
+// markMaybeSent marks the errors of requests that can have reached Coder with errAfterRedirect or
+// errAfterWrite.
+type markMaybeSent struct{ next http.RoundTripper }
+
+func (m markMaybeSent) RoundTrip(req *http.Request) (*http.Response, error) {
+	var wrote atomic.Bool
+	traced := req.WithContext(httptrace.WithClientTrace(req.Context(), &httptrace.ClientTrace{
+		WroteHeaders: func() { wrote.Store(true) },
+	}))
+	resp, err := m.next.RoundTrip(traced)
+	switch {
+	case err == nil:
+		return resp, nil
+	case req.Response != nil:
+		return nil, fmt.Errorf("%w: %w", errAfterRedirect, err)
+	case wrote.Load():
+		return nil, fmt.Errorf("%w: %w", errAfterWrite, err)
+	}
+	return nil, err
+}
+
+func (m markMaybeSent) CloseIdleConnections() {
+	if closer, ok := m.next.(interface{ CloseIdleConnections() }); ok {
+		closer.CloseIdleConnections()
+	}
+}
+
 // Config describes how to construct a Coder SDK client.
 type Config struct {
 	CoderURL       *url.URL
@@ -261,16 +330,7 @@ func NewSDKClient(cfg Config) (*codersdk.Client, error) {
 		return nil, fmt.Errorf("get Coder transport: %w", err)
 	}
 
-	client := codersdk.New(&coderURL, codersdk.WithHTTPClient(&http.Client{
-		// codersdk dials websockets (WorkspaceBuildLogsAfter for ?follow=true)
-		// with a new http.Client that keeps only this Transport, not
-		// CheckRedirect, so the Transport also refuses other origins.
-		Transport: sameorigin.RoundTripper(&coderURL, connectionWaitRoundTripper{base: transport}),
-		Timeout:   requestTimeout,
-		// net/http copies Coder-Session-Token to every redirect target, so refuse
-		// redirects that leave the Coder origin.
-		CheckRedirect: sameorigin.RedirectPolicy(&coderURL),
-	}))
+	client := codersdk.New(&coderURL, codersdk.WithHTTPClient(newCoderHTTPClient(&coderURL, connectionWaitRoundTripper{base: transport}, requestTimeout)))
 	if client == nil {
 		return nil, fmt.Errorf("assertion failed: coder SDK client is nil after successful construction")
 	}

internal/aggregated/coder/controlplane_provider.go

@@ -88,7 +88,7 @@ func (p *ControlPlaneClientProvider) ClientForNamespace(ctx context.Context, nam
 		if namespace == "" || listed > 0 {
 			return nil, unavailable
 		}
-		return nil, newNamespaceNotServedError(unavailable)
+		return nil, NewNamespaceNotServedError(unavailable)
 	case 1:
 		// handled below
 	default: