[fylorn]

What this changes

When every candidate has been tried and the last attempt failed temporarily before any answer content reached the client, the gateway backs off and tries again within the request's wait deadline instead of passing the error to the client. With a single upstream that means resending to it.

Example: a Responses/Codex upstream answers 200, then reports {"type":"error","code":"server_is_overloaded"} in the stream before any content. Before this change, if it was the last (or only) candidate, that error went straight to the client.

Rules

• Temporary failures only: statuses 500, 502, 503, 504 and 529 (not when the body names a request error such as invalid_request_error); in-stream errors whose code or type is on an allowlist (server_error, api_error, overloaded_error, server_is_overloaded, slow_down, Gemini UNAVAILABLE / INTERNAL or a 5xx numeric code, Bedrock serviceUnavailableException / internalServerException), in response.failed and in the top-level Responses error event; a connection that cannot be made or breaks before the response head; a stream that breaks, or ends cleanly after only opening events, before content (new code gw.upstream.stream_ended_early). Not retried: other 5xx (501, 505, Cloudflare 52x …), unlisted in-stream codes (invalid_image, content_filter, invalid_encrypted_content …), rate limits (they keep using the throttle queue), used-up quota, no balance, rejected credentials, missing model, idle timeouts, and failures after content. Failover to another candidate is unchanged for all of these.
• Backoff 2, 4, 8 s, then 16 s each time, ±20% jitter. retry-after-ms / Retry-After (or the body's retry delay) can lengthen a pause but never shorten it.
• Deadline: the request's existing wait deadline (failover.wait_secs, default 120 s, shared with queueing). Retrying stops when the next pause would end after it; the last error is handed over. wait_secs: 0 disables retrying. An attempt already in flight is not cut by the deadline.
• Which upstreams: each round retries the candidates whose failure in the previous round was temporary, in candidate order. Cooling candidates are skipped unless all of them are (same rule as the initial breaker filter). A conversation that stays on an upstream (session affinity, sealed history) retries only that upstream.
• Each retry is a normal hop (request hooks, content filter, redaction, parameter rewrites run per hop). Breaker accounting is unchanged. Token-counting requests are not retried.
• Abort during a backoff ends the request without another hop, like an abort while queued.

Opening hold

The last candidate's streamed answer is now also held when the client receives an SSE stream and the request can still wait (deadline not passed); before, it was held only when the upstream may be rate limited (Throttles::wary, which still applies). A held error before content can then be retried, and an in-stream 429 on such a stream reaches the throttle queue. The existing 15 s rule is unchanged: after 15 s the client gets 200 and : keep-alive comments. Non-streamed answers from the last candidate are not held (their non-2xx statuses are still classified and retried). Before the last candidate's own in-stream error is discarded for a retry, the same check as the retry decision runs (time left for the next pause, pinned upstream); when a retry cannot happen, the upstream's response is passed on unchanged.

Hits in plugin-written content (SecretsFound, ContentMatched) are reported once per client request, not once per attempt.

Control protocol 53

• New Event::RequestRetrying { id, upstreams, attempt, retry_at_ms, until_ms, at_ms } (tag request_retrying), emitted when a backoff starts; RequestDequeued is emitted when it ends. Both are kept in InFlightRequest::events; the recorder ignores them.
• AttemptView gains retry (1-based retry number on every hop of a retry round) and backoff_ms (pause before the first hop of that round, not part of ms), both optional.

No config schema change and no request-DB change. Docs for failover.wait_secs and the failover section are updated.

How it was verified

• New crates/tw-gateway/tests/retry.rs (23 tests, fake upstreams, backoff scaled down via AppState::retry_backoff with jitter off): in-stream overload before content then success (Anthropic and Responses server_is_overloaded); immediate 503/502 retried twice; connect failure until the upstream comes up; stream broken before content; error after content not retried; 400/401/402/404/quota not retried; in-stream invalid_encrypted_content not retried; deadline reached hands over the last error; wait_secs: 0; a longer retry-after-ms honoured and a 5 ms one not shortening the backoff; Retry-After beyond the deadline gives up at once; 501, 524 and a 500 naming a request error not retried; an unlisted in-stream code not retried but still failing over; a stream ending after its opening retried; an in-stream error that cannot be retried in time handed over as the upstream sent it; the last whole answer not held; abort during backoff; two overloaded candidates retried in a second round; affinity retries only the pinned upstream; RequestRetrying/RequestDequeued pairing and AttemptView fields; held stream past 15 s gets keep-alives and either the retried answer or the final error in the client's format.
• Unit tests for the backoff schedule, jitter bounds, the hint floor, the status and kind allowlists, request-fault detection, and the in-stream classification per format.
• plugins_security.rs: a retry does not report the plugin-written key or content hit twice.
• Existing tests that check pass-through of the last upstream's error, or the unheld last stream (endings, first token, live, Claude Desktop pings, traffic, notices, a few passthrough/alias/bedrock/hotreload/control tests), now set wait_secs: 0. The sealed-history test now checks that retries stay on the sealing upstream.
• cargo fmt --all --check, cargo clippy --workspace --all-targets -- -D warnings, cargo test --workspace, cargo clippy -p tw-api --features ts, cargo test -p tw-api --features ts, and the TS export.

Notes for review

• A stream that ends after only opening events now also fails over on a non-last candidate (before, it was handed to the client empty).
• No changes to tw-guard, tw-dialect or tw-breaker.

🤖 Generated with Claude Code