[fylorn]

What

When every candidate route has been tried and the last failure is transient, and none of the answer has reached the client, the gateway now waits and sends the request again instead of returning the error.

• Transient: 500, 502, 503, 504, 529; a connection that could not be made or broke before the response head (NetworkError); a stream that broke off (transport) or reported an overload or server error (overloaded_error, api_error, server_error, server_is_overloaded, slow_down, Gemini UNAVAILABLE) before any content.
• Never retried: 429 (rate limit / quota), 401/403, other 4xx, timeouts, local limits/budgets/policies, sealed-history refusals, client disconnects, and anything after stream content went out.
• Backoff: 2, 4, 8, then 16 s, ±20% jitter; retry-after-ms / Retry-After on the failed response wins (now kept on ProviderHttpError::retry_after), except that a zero hint is ignored and shorter hints are held to 250 ms. A retry starts only if its wait ends before the deadline; an in-flight stream attempt is not cut.
• Deadline: new system setting gateway.retry_wait_secs (default 120, 0–600, 0 disables), counted from the request's arrival. Seeded, validated in the settings handler, read with a clamp, editable under Settings → Gateway → Routing strategy (en/zh).
• Hard deadline for whole answers: a new Arrived stamp is layered just inside the gateway port's TimeoutLayer (its 120 s is now the shared constant proxy::REQUEST_TIMEOUT). Non-streaming attempts are cut 2 s before that timeout and the last upstream error (or a 504 ProviderTimeout) goes through the normal error/log path instead of an empty 408; a retry starts only when its wait plus an attempt as long as the last one fit before it.

Non-streaming (select_route_with_failover)

After a round exhausts the candidates, if the last error is transient and the wait fits, the loop sleeps and starts another round restricted to the providers whose last failure in this request was transient. Each pick still goes through pick_admitted, so strategy, breaker, route caps, affinity and the sealed-history pin apply; each failed attempt is still recorded in health as before. A single-route model retries the same route.

Streaming (build_chat_pump)

Streaming has no cross-route failover; the response head (200) is returned before the upstream is opened, as before. OpenUpstream is now a factory so the pump can open the same route again. A Resend gate (routing.rs) checks the route's breaker (pinned-only conversations bypass it, as on the first pick) and its caps before each retry, and records the failed attempt in health.

While a retry is still possible, the upstream's opening (message_start/ping, response.created/in_progress, role-only Chat chunks, Bedrock messageStart) is held back on the raw upstream bytes (Opening, read with tw_dialect::convert::Reader) until the first content, so a retry never duplicates it. An error is classified before release: transient → retry; anything else → the held opening and the error go out as before. Content ahead of an error in the same chunk commits the stream. While holding or waiting, non-Gemini clients get : keep-alive SSE comments after 15 s of silence (dropped by the WebSocket relay); Gemini clients never get them, since the google-genai SDK parses every SSE line as JSON. Deciding a retry clears what the failed attempt left for the stream's tail, so a client that leaves during the wait is logged as cancelled, unbilled, and the failed attempt is not counted twice against the route. With retry_wait_secs = 0 nothing is held and the pump behaves as before.

Observability

• gateway_provider_retry_total{provider} next to gateway_provider_fallback_total.
• The gateway_logs detail carries "retries": n on success and failure rows when a request was sent again.
• In-stream server_error now maps to 500 (ProviderHttpError) like api_error, instead of 502 (ProviderError).
• A request reqwest cannot build (invalid base URL) is a non-transient ProviderError instead of NetworkError.

Tests

• Unit: classification, backoff schedule and jitter, Retry-After precedence, deadline gating, disabled budget, hint floor, hard-deadline gating (proxy::retry); opening hold per dialect, error-before/after-content, hold cap, in-stream transient classification, keep-alive cadence with paused time, no keep-alives for Gemini, retry decision clears the failed attempt (lifecycle); Retry-After/retry-after-ms parsing on 5xx and builder errors not transient (transport); setting validation.
• Integration (gateway_retry.rs, wiremock): buffered retry after 503s, backoff without a hint, 400/404/429 not retried, 0 disables, deadline stops retries, two routes both retried, stream retried when it cannot open, a zero Retry-After still waits out the backoff, keep-alives reach a Chat client but not a Gemini SSE client, stream retried after an in-stream overload before content (one message_start reaches the client), not retried after content, not retried on an in-stream refusal. analytics_clickhouse.rs: "retries" on the log row for a buffered and a streamed request.
• The test harness sets gateway.retry_wait_secs = 0 so existing tests with failing upstreams keep their timing; the retry tests turn it on.

🤖 Generated with Claude Code