From 3675989f26bf7377653113ffd9b7d914a1dc0892 Mon Sep 17 00:00:00 2001 From: fylorn <249551762+fylorn@users.noreply.github.com> Date: Sat, 24 Oct 2026 23:00:09 +0800 Subject: [PATCH 1/3] Retry the last upstream after a temporary failure before any content When every candidate has been tried and the last attempt failed temporarily before any answer content reached the client (5xx or overload, an overload or server error reported in the stream, a connection that could not be made or broke before the response head, a stream that broke before content), the gateway now backs off and runs another round over the candidates whose failure was temporary, instead of passing the error on. - Backoff 2, 4, 8 s, then 16 s each time, +-20% jitter; retry-after-ms / Retry-After from the failed response replace it. - Bounded by the request's wait deadline (failover.wait_secs, shared with queueing): retrying stops when the next pause would end after it, and the last error is handed over. wait_secs: 0 disables retrying. - Rate limits, used-up quotas, no balance, rejected credentials, missing models, request errors (including in-stream invalid_request_error) and idle timeouts are not retried. Token counting requests are not retried. - A conversation that stays on an upstream (affinity, sealed history) retries only that upstream; cooling candidates are skipped unless all of them are. - Each retry is a normal hop on the attempt chain. Aborting during the backoff ends the request without another hop. - The last candidate's opening is now held whenever the request can still wait, so an error before content can be retried (and an in-stream 429 on the last candidate always reaches the throttle queue). The 15 s early head and keep-alives are unchanged; a final failure after the head is written in the client's format. Throttles::wary is removed. Control protocol 53: new Event::RequestRetrying (paired with RequestDequeued, kept in InFlightRequest::events); AttemptView gains retry and backoff_ms. Tests that exercise pass-through of the last upstream's error, or the unheld last stream, now set wait_secs: 0. Co-Authored-By: Claude Opus 5.5 --- README.md | 4 +- README.zh-CN.md | 2 +- crates/tw-api/src/lib.rs | 38 +- crates/tw-config/src/failover.rs | 5 +- crates/tw-config/tests/manual/schema.rs | 4 +- crates/tw-control/tests/error_answers.rs | 4 +- crates/tw-control/tests/handover.rs | 2 +- crates/tw-control/tests/in_flight.rs | 2 + crates/tw-control/tests/key_limits.rs | 4 + crates/tw-control/tests/live_state.rs | 4 +- crates/tw-control/tests/replay.rs | 2 + crates/tw-control/tests/upstream_limits.rs | 2 + crates/tw-gateway/src/lib.rs | 1 + crates/tw-gateway/src/retry.rs | 103 +++ crates/tw-gateway/src/server.rs | 10 + crates/tw-gateway/src/server/pipeline/hop.rs | 181 +++- crates/tw-gateway/src/server/pipeline/idle.rs | 6 +- .../tw-gateway/src/server/pipeline/opening.rs | 30 +- .../tw-gateway/src/server/pipeline/relay.rs | 2 +- crates/tw-gateway/src/state.rs | 3 + crates/tw-gateway/src/throttle.rs | 28 - crates/tw-gateway/tests/alias_routing.rs | 4 + crates/tw-gateway/tests/bedrock.rs | 31 +- crates/tw-gateway/tests/claude_desktop.rs | 5 + crates/tw-gateway/tests/endings.rs | 12 +- crates/tw-gateway/tests/first_token.rs | 6 + crates/tw-gateway/tests/hotreload.rs | 2 + crates/tw-gateway/tests/live.rs | 5 + crates/tw-gateway/tests/notices.rs | 5 + crates/tw-gateway/tests/passthrough.rs | 34 +- crates/tw-gateway/tests/retry.rs | 860 ++++++++++++++++++ crates/tw-gateway/tests/sealed.rs | 24 +- crates/tw-gateway/tests/traffic.rs | 5 + crates/tw-observe/src/bus.rs | 8 +- crates/tw-store/src/db.rs | 4 + crates/tw-store/src/recorder.rs | 30 + docs/config.md | 38 +- docs/config.zh-CN.md | 20 +- 38 files changed, 1431 insertions(+), 99 deletions(-) create mode 100644 crates/tw-gateway/src/retry.rs create mode 100644 crates/tw-gateway/tests/retry.rs diff --git a/README.md b/README.md index 53af4952..6e50b1c9 100644 --- a/README.md +++ b/README.md @@ -62,7 +62,9 @@ Documentation: [configuration reference](docs/config.md) · images, thinking and more, and send requests to an upstream or a group (in order, manual, rotation, lowest latency, lowest cost). Until the first byte of the response reaches the client, a failing upstream is replaced by the - next candidate and set aside for a time that depends on the reason it gives. + next candidate and set aside for a time that depends on the reason it gives; + when the last one is only overloaded or briefly unreachable, it is retried + after a pause. - **Many kinds of upstream.** Provider API keys, any compatible endpoint, relays such as OpenRouter, local models, Amazon Bedrock, and ChatGPT or Z.ai accounts. Health checks and warm-ups are answered locally by default, at no cost. diff --git a/README.zh-CN.md b/README.zh-CN.md index 1da1df98..d62c13f9 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -28,7 +28,7 @@ ThinkWatch Core 是 ThinkWatch 的网关引擎，由一组 Rust crate 及其构 - **切断恶意工具调用**。中转站可以改写回答，塞入让客户端执行的工具调用。回答中的工具调用若是下载即执行、外发环境变量或凭据文件、读取私钥、读写 ThinkWatch 自己的配置、写入开机启动项或定时任务，工具调用审查可以在客户端收到完整调用之前切断回答。 - **清除隐藏指令**。屏幕上看不见的字符可以夹带模型会读取的指令，内容过滤可以在请求发出前把它们从用户消息和工具结果中删除，也可以拒绝要求模型忽略自身指令的请求。出站脱敏、工具调用审查和内容过滤出厂均为观察档，只记录检出的内容，不改变任何请求。 - **每个请求都可追溯**。每个请求连同决定其去向的规则、每次尝试、格式转换、用量、费用及价格来源、首 token 时间和生成速度一并保存。试算可以在不发出请求的情况下说明请求会被送往何处；已保存的请求可以对另一个上游重放，以便对比。 -- **路由与故障转移**。规则可按模型、密钥、格式、请求大小、工具、图片、思考等条件匹配，把请求交给一个上游或策略组（按顺序、手动指定、轮流、最低延迟、最低价格）。响应的首字节到达客户端之前，失败的上游由下一个候选替换，并按其给出的失败原因暂停相应的时间。 +- **路由与故障转移**。规则可按模型、密钥、格式、请求大小、工具、图片、思考等条件匹配，把请求交给一个上游或策略组（按顺序、手动指定、轮流、最低延迟、最低价格）。响应的首字节到达客户端之前，失败的上游由下一个候选替换，并按其给出的失败原因暂停相应的时间；最后一个候选只是过载或暂时连不上的，等待片刻后重试。 - **多种上游**。服务商的 API 密钥、任意兼容接口、OpenRouter 等中转站、本地模型、Amazon Bedrock，以及 ChatGPT 和 Z.ai 账号。连通性检查和预热请求默认在本地应答，不产生费用。 - **如实计费**。用量（含缓存读写）按每日刷新的公开价目表或自定义价目表计价。估算的费用会标明，无法计价的用量单独统计，不计作零。 - **单一配置，即改即生效**。全部设置保存在 `config.yaml` 中；无论改动来自编辑器、命令行还是控制面，通过校验后一秒内生效，最近五十个版本均可恢复。 diff --git a/crates/tw-api/src/lib.rs b/crates/tw-api/src/lib.rs index a148f41f..5760632c 100644 --- a/crates/tw-api/src/lib.rs +++ b/crates/tw-api/src/lib.rs @@ -1066,7 +1066,13 @@ pub const MSG_CODES: &str = include_str!("../msg-codes.txt"); /// 父进程、开交接端点、等接替者；[`Status`] 多了 `handover`（[`HandoverState`]）和 `draining` /// （排空中的旧 core 手上还有几个请求，只有接管了它的新 core 报，平时是 0）。 /// 新码：`control.handover.*` -pub const CONTROL_API_VERSION: u32 = 52; +/// +/// **53 起最后一家暂时出错时退避重试**：候选都试过了、最后那一跳在第一段内容之前过载、5xx、 +/// 连不上、流断了，请求在等待期限（`wait_secs`，等上游和重试共用）之内退避一阵，把暂时出错的 +/// 那几家再试一轮。新事件 [`Event::RequestRetrying`]（开始退避：再试哪几家、第几次、什么时候 +/// 发出去），退避结束发 [`Event::RequestDequeued`]，两条都在 `InFlightRequest::events` 里。 +/// [`AttemptView`] 多了 `retry`（这一跳是第几次重试）和 `backoff_ms`（发出之前退避了多久） +pub const CONTROL_API_VERSION: u32 = 53; /// `/live` 的速度看最近多久（见 [`LiveView::tokens_per_sec`]），毫秒 pub const LIVE_RATE_WINDOW_MS: u64 = 3_000; @@ -1423,10 +1429,28 @@ pub enum Event { }, /// 这个请求不排队了：等到了一家、要发出去了，被手动中止了，或者等不到了。和它之前的那条 /// [`Event::RequestQueued`] 成对；路由事件要等整条尝试链走完才发，界面靠这一条知道它不在 - /// 排队了。也在 [`InFlightRequest::events`] 里。 + /// 排队了。也在 [`InFlightRequest::events`] 里。退避重试结束时也发，和之前的那条 + /// [`Event::RequestRetrying`] 成对。 /// /// 客户端在排队时走了的不发：紧跟着的结局（取消）就是它的结束 RequestDequeued { id: u64, at_ms: u64 }, + /// 这个请求在退避，之后重试：候选都试过了，最后那一跳在第一段内容之前出了暂时的错（过载、 + /// 5xx、连不上、流断了），暂时出错的那几家在等待期限之内再试一轮。 + /// + /// **一次退避发一条**，退避结束发 [`Event::RequestDequeued`]（重试发出去了、被手动中止了、 + /// 等不到了）。在 [`InFlightRequest::events`] 里 + RequestRetrying { + id: u64, + /// 要再试哪几家，按候选的顺序 + upstreams: Vec, + /// 接下来是第几次重试，从 1 数 + attempt: u32, + /// 下一次什么时候发出去（Unix 毫秒） + retry_at_ms: u64, + /// 这个请求的等待期限（Unix 毫秒），同 [`Event::RequestQueued`] 的 `until_ms` + until_ms: u64, + at_ms: u64, + }, /// 路由决定完了，尝试链也走完了。 /// /// **单独一个事件，因为成功和失败两条路都要发它。**挂在 @@ -1975,6 +1999,13 @@ pub struct AttemptView { /// 不算进这家的用量上限（[`Self::counts_for_upstream`]）。别的结果没有 #[serde(default, skip_serializing_if = "Option::is_none")] pub pause_ms: Option, + /// 这一跳是第几次重试，从 1 数（见 [`Event::RequestRetrying`]）：一轮重试里的每一跳都带着。 + /// 第一轮的没有 + #[serde(default, skip_serializing_if = "Option::is_none")] + pub retry: Option, + /// 这一跳发出去之前退避了多少毫秒：一轮重试的头一跳带着。不算在 `ms` 里。别的没有 + #[serde(default, skip_serializing_if = "Option::is_none")] + pub backoff_ms: Option, /// 这一跳走的代理：这家上游配置的代理名（跟随系统的是 `system`），直连是 None。没发出去 /// 的一跳（满着、被拒、转换不了……）是它要走的那个 —— 连不上的那一跳，问题可能就出在 /// 这个代理上。WebSocket 只连直连的上游，一律是 None @@ -2215,6 +2246,7 @@ impl Event { | Event::ThrottleChanged { id, .. } | Event::RequestQueued { id, .. } | Event::RequestDequeued { id, .. } + | Event::RequestRetrying { id, .. } | Event::ModelsChanged { id, .. } | Event::ProxyChanged { id, .. } | Event::AuthChanged { id, .. } @@ -6892,6 +6924,8 @@ mod tests { queued_ms: None, skipped: None, pause_ms: None, + retry: None, + backoff_ms: None, proxy: None, } } diff --git a/crates/tw-config/src/failover.rs b/crates/tw-config/src/failover.rs index 9175bfd9..76fa89cc 100644 --- a/crates/tw-config/src/failover.rs +++ b/crates/tw-config/src/failover.rs @@ -45,8 +45,9 @@ pub struct Failover { pub idle_timeout_secs: u64, /// 一个请求最多等多少秒，**整个请求合起来算**：准入时等密钥的分钟、小时上限空出名额， /// 之后等上游 —— 并发数满了（`providers[].max_concurrent`）、被限流暂停着、分钟或小时的 - /// 用量上限满了 —— 共用这一段。留在那一家的对话等它，候选都等不上时等先能发的那一家； - /// 等不到的换下一家，或者回 429。0 是不等 + /// 用量上限满了 —— 和最后一家在内容之前暂时出错（过载、5xx、连不上）之后退避重试，共用这 + /// 一段。留在那一家的对话等它，候选都等不上时等先能发的那一家；等不到的换下一家，或者回 + /// 429；重试到时还没成的交出最后那个错误。0 是不等，也不重试 #[serde(default = "d_wait_secs")] pub wait_secs: u64, } diff --git a/crates/tw-config/tests/manual/schema.rs b/crates/tw-config/tests/manual/schema.rs index b0ecadfe..06a23081 100644 --- a/crates/tw-config/tests/manual/schema.rs +++ b/crates/tw-config/tests/manual/schema.rs @@ -1579,8 +1579,8 @@ pub fn sections() -> Vec
{ Kind::Int, Def::Is("120"), t( - "Seconds a request waits in all, counted once the key's own `max_concurrent` lets it in: for a key's `minute` or `hour` limit to free up, and for an upstream that cannot take it yet — at its `max_concurrent`, paused after answering 429, or at one of its `minute` or `hour` usage limits. A key limit that does not free up in time refuses the request. An upstream that cannot take it is skipped for the next one, and when none is left the request waits for the first to become available; a conversation that stays on an upstream waits for it. When nothing became available in time, the client gets 429 with `Retry-After`. `0`: never wait. From 0 to 600.", - "一个请求合计最多等的秒数，从过了密钥自己的 `max_concurrent` 时算起：等密钥的 `minute`、`hour` 上限空出名额，和等暂时不能发的上游 —— 并发数满了（`max_concurrent`）、回了 429 正在暂停、到了这家 `minute` 或 `hour` 的用量上限 —— 都算在里面。密钥的上限到时空不出来就拒绝。上游暂时不能发的先换下一家，没有下一家了就等最先恢复的那一家；留在某一家的对话等这一家。到时都没有恢复，客户端收到带 `Retry-After` 的 429。`0`：不等。取值 0 到 600。", + "Seconds a request waits in all, counted once the key's own `max_concurrent` lets it in: for a key's `minute` or `hour` limit to free up, for an upstream that cannot take it yet — at its `max_concurrent`, paused after answering 429, or at one of its `minute` or `hour` usage limits — and before retrying upstreams that failed temporarily before any content (overloaded, another 5xx, unreachable). A key limit that does not free up in time refuses the request. An upstream that cannot take it is skipped for the next one, and when none is left the request waits for the first to become available; a conversation that stays on an upstream waits for it. When nothing became available in time, the client gets 429 with `Retry-After`; when retries did not succeed in time, it gets the last upstream error. `0`: never wait or retry. From 0 to 600.", + "一个请求合计最多等的秒数，从过了密钥自己的 `max_concurrent` 时算起：等密钥的 `minute`、`hour` 上限空出名额，等暂时不能发的上游 —— 并发数满了（`max_concurrent`）、回了 429 正在暂停、到了这家 `minute` 或 `hour` 的用量上限 —— 和上游在内容之前暂时出错（过载、其他 5xx、连不上）后重试之前的等待，都算在里面。密钥的上限到时空不出来就拒绝。上游暂时不能发的先换下一家，没有下一家了就等最先恢复的那一家；留在某一家的对话等这一家。到时都没有恢复，客户端收到带 `Retry-After` 的 429；到时重试仍未成功，客户端收到最后一次的上游错误。`0`：不等，也不重试。取值 0 到 600。", ), ), ], diff --git a/crates/tw-control/tests/error_answers.rs b/crates/tw-control/tests/error_answers.rs index b965c04b..c8502859 100644 --- a/crates/tw-control/tests/error_answers.rs +++ b/crates/tw-control/tests/error_answers.rs @@ -89,10 +89,12 @@ struct World { async fn world() -> World { let up = upstream().await; let d = tempfile::tempdir().unwrap(); + // 不等、不重试（`wait_secs: 0`）：唯一的上游不压开头，第三轮吐的那一帧当场交给客户端 let yaml = format!( "version: 1\nlisten:\n control:\n key: c0ffee00c0ffee00c0ffee00c0ffee00c0ffee00c0ffee00c0ffee00c0ffee00\n\ clients:\n - name: 我\n key: tw-k\n\ - providers:\n - name: 中转\n base_url: http://{up}\n key: sk-upstream\n protocol: anthropic\n" + providers:\n - name: 中转\n base_url: http://{up}\n key: sk-upstream\n protocol: anthropic\n\ + failover:\n wait_secs: 0\n" ); let p = d.path().join("config.yaml"); std::fs::write(&p, &yaml).unwrap(); diff --git a/crates/tw-control/tests/handover.rs b/crates/tw-control/tests/handover.rs index 0763a0df..e6f2ceab 100644 --- a/crates/tw-control/tests/handover.rs +++ b/crates/tw-control/tests/handover.rs @@ -146,7 +146,7 @@ async fn status_reports_the_handover_phase_and_the_draining_count() { let st = status(&s).await; assert_eq!(st.handover, tw_api::HandoverState::Draining); assert_eq!(st.draining, 3); - assert_eq!(st.api_version, 52); + assert_eq!(st.api_version, 53); } /// arm 之后（以及接替者正在接管时）不开始新的账号登录：回调端口不交接。（平常、排空中的 diff --git a/crates/tw-control/tests/in_flight.rs b/crates/tw-control/tests/in_flight.rs index 97c60be6..46de16b3 100644 --- a/crates/tw-control/tests/in_flight.rs +++ b/crates/tw-control/tests/in_flight.rs @@ -102,6 +102,8 @@ async fn it_gives_every_running_request_with_what_has_happened_to_it_so_far() { queued_ms: None, skipped: None, pause_ms: None, + retry: None, + backoff_ms: None, proxy: None, }], billing: tw_api::Billing::PerToken, diff --git a/crates/tw-control/tests/key_limits.rs b/crates/tw-control/tests/key_limits.rs index 8adb609f..7ff289b7 100644 --- a/crates/tw-control/tests/key_limits.rs +++ b/crates/tw-control/tests/key_limits.rs @@ -145,6 +145,8 @@ fn request(id: u64, key: &str, at: i64, input: u64) -> Vec { queued_ms: None, skipped: None, pause_ms: None, + retry: None, + backoff_ms: None, proxy: None, }], billing: tw_api::Billing::PerToken, @@ -456,6 +458,8 @@ fn hop(outcome: tw_api::AttemptOutcome, code: &str, busy: bool) -> tw_api::Attem queued_ms: None, skipped: busy.then_some(tw_api::ServeSkip::Busy), pause_ms: None, + retry: None, + backoff_ms: None, proxy: None, } } diff --git a/crates/tw-control/tests/live_state.rs b/crates/tw-control/tests/live_state.rs index 0cb6f4b0..98e64db2 100644 --- a/crates/tw-control/tests/live_state.rs +++ b/crates/tw-control/tests/live_state.rs @@ -159,6 +159,8 @@ async fn a_request_still_running_can_be_opened_and_becomes_whole_when_it_ends() queued_ms: None, skipped: None, pause_ms: None, + retry: None, + backoff_ms: None, proxy: None, }], billing: tw_api::Billing::PerToken, @@ -249,7 +251,7 @@ async fn the_overview_says_which_proxy_is_down_and_why() { let dead = dead_addr(); let d = tempfile::tempdir().unwrap(); let yaml = format!( - "version: 1\nlisten:\n control:\n key: c0ffee00c0ffee00c0ffee00c0ffee00c0ffee00c0ffee00c0ffee00c0ffee00\nclients:\n - name: 我\n key: tw-k\nproxies:\n - name: 代理一\n type: http\n addr: {dead}\nproviders:\n - name: 中转\n base_url: http://127.0.0.1:9\n protocol: anthropic\n key: sk-x\n proxy: 代理一\n" + "version: 1\nlisten:\n control:\n key: c0ffee00c0ffee00c0ffee00c0ffee00c0ffee00c0ffee00c0ffee00c0ffee00\nclients:\n - name: 我\n key: tw-k\nproxies:\n - name: 代理一\n type: http\n addr: {dead}\nproviders:\n - name: 中转\n base_url: http://127.0.0.1:9\n protocol: anthropic\n key: sk-x\n proxy: 代理一\nfailover:\n wait_secs: 0\n" ); let (gw, app) = control(&d, &yaml, None); let mut rx = gw.bus.subscribe(); diff --git a/crates/tw-control/tests/replay.rs b/crates/tw-control/tests/replay.rs index 34030445..e439bae4 100644 --- a/crates/tw-control/tests/replay.rs +++ b/crates/tw-control/tests/replay.rs @@ -392,6 +392,8 @@ fn routed(hops: &[(&str, Option<&str>)]) -> Option { queued_ms: None, skipped: None, pause_ms: None, + retry: None, + backoff_ms: None, proxy: None, }) .collect(); diff --git a/crates/tw-control/tests/upstream_limits.rs b/crates/tw-control/tests/upstream_limits.rs index 8e7526ed..6aa965f7 100644 --- a/crates/tw-control/tests/upstream_limits.rs +++ b/crates/tw-control/tests/upstream_limits.rs @@ -108,6 +108,8 @@ fn attempt(provider: &str, status: u16) -> tw_api::AttemptView { queued_ms: None, skipped: None, pause_ms: None, + retry: None, + backoff_ms: None, proxy: None, } } diff --git a/crates/tw-gateway/src/lib.rs b/crates/tw-gateway/src/lib.rs index 9f3a85b6..3ddbb9d4 100644 --- a/crates/tw-gateway/src/lib.rs +++ b/crates/tw-gateway/src/lib.rs @@ -50,6 +50,7 @@ pub mod probe; mod pulse; pub mod quota; pub mod quote; +pub mod retry; pub mod seal; pub mod sent; pub mod server; diff --git a/crates/tw-gateway/src/retry.rs b/crates/tw-gateway/src/retry.rs new file mode 100644 index 00000000..2af3dca0 --- /dev/null +++ b/crates/tw-gateway/src/retry.rs @@ -0,0 +1,103 @@ +//! 最后一家暂时出错时退避重试。 +//! +//! 候选都试过了，最后那一跳在**第一段内容之前**出了暂时的错 —— 过载（503、529、流里的 +//! `overloaded_error`、`server_is_overloaded`）、别的 5xx、连不上、流在内容之前断了。这时客户端 +//! 什么都还没收到，等一会儿再发一遍它察觉不到；只有一家上游的请求更是这样，没有下一家可换， +//! 上游过载几秒钟的事，不该把错误交给客户端。 +//! +//! 几条规矩（做法见 `server::pipeline::hop`）： +//! +//! - **只重试暂时的错**（[`transient`]），也就是说不出原因的失败。限流有自己的队（见 +//! [`crate::throttle`]）；额度用完、没钱了、凭据被拒、没有这个模型、请求本身的问题，再发一遍 +//! 也一样；没有内容超时的已经等过一整个无响应超时，不再等。 +//! - **在这个请求的等待期限之内**（`failover.wait_secs`，和排队共用一段）：剩下的时间不够下一次 +//! 退避就不等了，交出最后那个错误。`0` 是不重试。已经发出去的一跳不受期限限制。 +//! - **把暂时出错的那几家再试一轮**，按候选的顺序，在冷却的跳过（都在冷却的照样试，和开头的 +//! 熔断过滤一样）；留在一家的对话（会话粘性、封存的历史）只再试那一家。每一跳和故障转移的 +//! 一跳一样：插件、内容过滤、脱敏、参数改写都照常跑，熔断的账照常记。 +//! - **退避 2、4、8 秒，之后每次 16 秒**，上下浮动两成，免得一起失败的请求一起再来；上游说了 +//! 多久之后再来的（`retry-after-ms`、`Retry-After`）按它说的。 + +use std::time::Duration; + +use crate::failure::Cause; + +/// 第 1、2、3 次重试之前退避几秒，之后每次都是最后那个数 +const STEPS: [u32; 4] = [2, 4, 8, 16]; + +/// 退避多久。 +#[derive(Debug, Clone, Copy)] +pub struct Backoff { + /// 退避的一秒有多长。**测试会把它调短** + pub tick: Duration, + /// 上下浮动多少（`0.2` 是两成）。**测试设成 0**：退避多久是定的 + pub jitter: f64, +} + +impl Default for Backoff { + fn default() -> Self { + Self { + tick: Duration::from_secs(1), + jitter: 0.2, + } + } +} + +impl Backoff { + /// 第 `n` 次重试（从 1 数）之前退避多久 + pub fn before(&self, n: u32) -> Duration { + let i = (n.max(1) - 1).min(STEPS.len() as u32 - 1) as usize; + let base = self.tick.saturating_mul(STEPS[i]); + let jitter = self.jitter.clamp(0.0, 0.5); + if jitter == 0.0 { + return base; + } + base.mul_f64(1.0 + rand::random_range(-jitter..=jitter)) + } +} + +/// 这个原因是不是暂时的错：说不出原因的失败（5xx、过载）。别的见模块说明 +pub fn transient(cause: Cause) -> bool { + cause == Cause::Unexplained +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn the_backoff_doubles_up_to_sixteen_seconds() { + let b = Backoff { + tick: Duration::from_secs(1), + jitter: 0.0, + }; + let secs: Vec = (1..=6).map(|n| b.before(n).as_secs()).collect(); + assert_eq!(secs, [2, 4, 8, 16, 16, 16]); + } + + #[test] + fn the_jitter_stays_within_a_fifth() { + let b = Backoff::default(); + for _ in 0..200 { + let d = b.before(1); + assert!( + (1600..=2400).contains(&d.as_millis()), + "{d:?} 不在 2 秒上下两成之内" + ); + } + } + + #[test] + fn only_unexplained_failures_are_retried() { + assert!(transient(Cause::Unexplained)); + for c in [ + Cause::NoBalance, + Cause::QuotaUsedUp { resets_at_ms: None }, + Cause::RateLimited { retry_after: None }, + Cause::ModelUnavailable, + Cause::AuthRejected, + ] { + assert!(!transient(c), "{c:?}"); + } + } +} diff --git a/crates/tw-gateway/src/server.rs b/crates/tw-gateway/src/server.rs index 640050fa..b376472a 100644 --- a/crates/tw-gateway/src/server.rs +++ b/crates/tw-gateway/src/server.rs @@ -387,6 +387,8 @@ pub(crate) fn hop( queued_ms: None, skipped: None, pause_ms: None, + retry: None, + backoff_ms: None, proxy: None, } } @@ -415,6 +417,8 @@ pub(crate) fn hop_busy( queued_ms, skipped: Some(tw_api::ServeSkip::Busy), pause_ms: None, + retry: None, + backoff_ms: None, proxy: None, } } @@ -465,6 +469,8 @@ pub(crate) fn hop_waiting( queued_ms, skipped: Some(skipped), pause_ms: None, + retry: None, + backoff_ms: None, proxy: None, } } @@ -494,6 +500,8 @@ pub(crate) fn hop_used_up( queued_ms: None, skipped: Some(tw_api::ServeSkip::PeriodUsedUp), pause_ms: None, + retry: None, + backoff_ms: None, proxy: None, } } @@ -516,6 +524,8 @@ pub(crate) fn hop_failed( queued_ms: None, skipped: None, pause_ms: None, + retry: None, + backoff_ms: None, proxy: None, } } diff --git a/crates/tw-gateway/src/server/pipeline/hop.rs b/crates/tw-gateway/src/server/pipeline/hop.rs index 36d6d9b0..de11fb8e 100644 --- a/crates/tw-gateway/src/server/pipeline/hop.rs +++ b/crates/tw-gateway/src/server/pipeline/hop.rs @@ -12,6 +12,9 @@ //! //! 发出去之后 `failover.idle_timeout_secs` 还没有内容的一家也换掉（见 [`super::idle`]）；在等 //! 上游的时候被手动中止的，丢掉这一跳、不再换（见 [`crate::abort`]）。 +//! +//! 候选都试过了，最后一跳在内容之前出了暂时的错（过载、5xx、连不上、流断了）：在这个请求的 +//! 等待期限之内退避一阵，把暂时出错的那几家再试一轮（见 [`crate::retry`]）。 use bytes::Bytes; @@ -49,7 +52,7 @@ pub(super) struct Served<'a> { pub(super) slot: crate::slots::Slot, /// 这一跳发出去的那一刻。这一家的快慢样本从这里算起（见 [`crate::latency`]） pub(super) sent_at: std::time::Instant, - /// 这一家答没答上还不知道：最后一家的开头没压（整包的、等不起的），或者压着时到点了还没有 + /// 这一家答没答上还不知道：最后一家的开头没压（这个请求等不起了），或者压着时到点了还没有 /// 内容、在开头报了限流以外的错，原样交了出去。成败的账等回程看到第一段内容才记（见 /// `relay`）。没有内容就超时了记失败，和压着开头时一样 pub(super) unconfirmed: bool, @@ -187,9 +190,38 @@ pub(super) async fn try_upstreams<'a>( // 等发不出去的那几家等到最后的话，交出去的是这一家的错，不是「都发不出去」（见下面的 // `stalled`）。**回 429 的不算**：那是让等一会儿，不是失败 let mut reached = false; + // 退避重试的账（见 `crate::retry`）：最后一个失败是不是暂时的错、这一轮里暂时出错的是哪几家、 + // 重试过的那几轮 + let mut retries = Retries::default(); loop { stamp_queued(&mut chain, queued.take()); + // 候选都试过了，最后一跳在内容之前暂时出错：等待期限之内退避一阵，把暂时出错的那几家再 + // 试一轮。不重试的，交出去的是最后那个错误 + if queue.is_empty() && waiting.is_empty() { + let Some((round, wait)) = retry_round(state, started, &mut retries, counting, until) + else { + break; + }; + let n = retries.rounds.len() as u32 + 1; + retrying(state, id, &round, n, wait, deadline_ms); + let t = std::time::Instant::now(); + let woke = tokio::select! { + biased; + _ = tokio::time::sleep(wait) => true, + // 退避的时候被手动中止：一个字节都还没发出去，尝试链上不多一跳 + _ = req.abort.wait() => false, + }; + dequeued(state, id); + if !woke { + halt = Some(GatewayError::aborted()); + break; + } + retries + .rounds + .push((chain.len(), n, t.elapsed().as_millis() as u64)); + queue.extend(round); + } // 下一个候选。都看过了，就在等得到的那几家里等先能发的：等到的那一家带着占到的位置 let (name, mut granted) = match queue.pop_front() { Some(name) => (name, None), @@ -246,6 +278,8 @@ pub(super) async fn try_upstreams<'a>( }; // 后面没有别的候选了。暂时发不出去、还等得到的那几家也算在后面 let last = queue.is_empty() && waiting.is_empty(); + // 最后一个失败是不是暂时的错，看的是这一跳：暂时出错的那几处再记上 + retries.fault = None; let Some(provider) = rt.config.providers.iter().find(|p| &p.name == name) else { // 校验时挡过一次，能到这儿说明配置在运行中被换过。 last_err = Some(GatewayError::config(msg!( @@ -889,6 +923,10 @@ pub(super) async fn try_upstreams<'a>( &provider.name, state.health.record_cause(&provider.name, cause), ); + if crate::retry::transient(cause) { + let hint = crate::throttle::pause_hint(&headers, &head, now_ms()); + retries.shaky(name, hint); + } chain.push(hop( &provider.name, model.clone(), @@ -957,18 +995,12 @@ pub(super) async fn try_upstreams<'a>( // 回答的开头：**第一段内容到之前**上游在流里报的错照样换下一家（报的是限流就排队 // 等它，和回 429 一样），一直没有内容也换（见 `opening`）。 // - // 最后一家只在这一家可能在限流时压（探路的、最近被限流过的，见 - // `throttle::Throttles::wary`），而且只压流式的、这个请求又还等得起的：流的开头报的 - // 限流要暂停这一家、排队等它恢复，客户端察觉不到（不压的话那个错误直接交给了客户端）。 - // 别的时候不压：没有下一家可换，客户端该当场拿到响应头和开头的事件；整包的回答不在 - // 正文里报限流，等不起的也排不了队。不压的由回程按无响应超时收尾（见 `relay`） + // 最后一家在这个请求还等得起时也压：开头报的暂时的错（过载、5xx）、流在内容之前断了， + // 退避一阵再试一遍（见 `crate::retry`）；报的限流暂停这一家、排队等它恢复。客户端都 + // 察觉不到 —— 不压的话那个错误直接交给了客户端。等不起的不压：没有下一家可换、也不 + // 能再试，客户端该当场拿到响应头和开头的事件，由回程按无响应超时收尾（见 `relay`） let hold = hold_of(req, reading, provider, &out, &r); - let held = hold.as_ref().is_some_and(|h| { - !last - || (matches!(h, Hold::Stream(..)) - && tokio::time::Instant::now() < until - && state.throttles.wary(&provider.name)) - }); + let held = hold.is_some() && (!last || tokio::time::Instant::now() < until); // 不压开头的最后一家：它答没答上（成败的账）要等回程看到第一段内容才知道 let mut unconfirmed = last && hold.is_some() && !held; let r = match hold.filter(|_| held) { @@ -1061,9 +1093,16 @@ pub(super) async fn try_upstreams<'a>( if !limited { to_throttle(state, &provider.name, sent_at, verdict); } - if verdict == Verdict::ClientError || (last && !limited) { - // 请求本身的问题，或者最后一家报了限流以外的错：原样交出去，和 - // 不压开头时一样（后者的成败由回程记） + // 最后一家报的暂时的错，这个请求又还等得起：和换下一家时一样记下， + // 候选都试过了就退避重试（见 `crate::retry`）。说的是请求本身的问题的 + // 不算（见 `opening::faults_the_request`） + let shaky = matches!(verdict, Verdict::Failed(c) if crate::retry::transient(c)) + && !super::opening::faults_the_request(&body); + let retried = shaky && tokio::time::Instant::now() < until; + if verdict == Verdict::ClientError || (last && !limited && !retried) + { + // 请求本身的问题，或者最后一家报了限流以外、不再重试的错：原样 + // 交出去，和不压开头时一样（后者的成败由回程记） unconfirmed |= verdict != Verdict::ClientError; response } else { @@ -1114,6 +1153,14 @@ pub(super) async fn try_upstreams<'a>( &provider.name, state.health.record_cause(&provider.name, cause), ); + if shaky { + let hint = crate::throttle::pause_hint( + &headers, + &body, + now_ms(), + ); + retries.shaky(name, hint); + } chain.push(a); last_err = Some(if status == 429 { GatewayError::rate_limited(said) @@ -1129,6 +1176,7 @@ pub(super) async fn try_upstreams<'a>( } Some(super::opening::Opening::Broken(err)) => { reached = true; + retries.shaky(name, None); state.throttles.released(&provider.name, sent_at); note_health( &state.bus, @@ -1190,6 +1238,7 @@ pub(super) async fn try_upstreams<'a>( SendError::Http(e) if e.is_connect() => { usage_hold.release(); reached = true; + retries.shaky(name, None); // 没有结论：探路的不再探，交给熔断（见 `crate::throttle`） state.throttles.released(&provider.name, sent_at); state.check_proxy(&provider.proxy); @@ -1197,6 +1246,8 @@ pub(super) async fn try_upstreams<'a>( } SendError::Http(e) => { reached = true; + // 响应头之前断了：和连不上一样是暂时的错 + retries.shaky(name, None); state.throttles.released(&provider.name, sent_at); // 连不上的可能是代理而不是上游 —— 检一次那个代理，说清是哪一件事 state.check_proxy(&provider.proxy); @@ -1229,6 +1280,7 @@ pub(super) async fn try_upstreams<'a>( } stamp_queued(&mut chain, queued.take()); + stamp_retries(&mut chain, &retries.rounds); // 尝试链走完了，两条路都要发 —— 挂在 RequestFinished 上的话， // 失败那条路就没有尝试链，而那恰恰是最需要看它的时候。 // 最终服务的那家怎么收钱。**跟着请求走，不能事后查配置** —— @@ -1399,6 +1451,79 @@ impl Drop for Spent { } } +/// 退避重试的账（见 [`crate::retry`]）。 +#[derive(Default)] +struct Retries<'n> { + /// 最后一个失败是暂时的错：外面一层是有没有，里面是那一跳上游说了多久之后再来（没说是 + /// None，按退避算） + fault: Option>, + /// 这一轮里暂时出错的那几家。重试时再试它们 + shaky: Vec<&'n String>, + /// 重试过的每一轮：在尝试链上从哪一跳起、是第几次、发出去之前退避了多少毫秒 + rounds: Vec<(usize, u32, u64)>, +} + +impl<'n> Retries<'n> { + /// 这一跳暂时出错了。`hint` 是上游说的多久之后再来 + fn shaky(&mut self, name: &'n String, hint: Option) { + self.fault = Some(hint); + if !self.shaky.contains(&name) { + self.shaky.push(name); + } + } +} + +/// 候选都试过了，要不要退避重试（见 [`crate::retry`]）：要的话，交回这一轮再试哪几家、先退避 +/// 多久。最后一个失败不是暂时的错的、数 token 的（它不排队，也不该为它等）、剩下的时间不够 +/// 这一次退避的不重试。留在一家的对话只再试那一家；在冷却的跳过，都在冷却的照样试（同 +/// [`crate::health::Health::filter`]） +fn retry_round<'n>( + state: &AppState, + started: &'n Started, + retries: &mut Retries<'n>, + counting: bool, + until: tokio::time::Instant, +) -> Option<(Vec<&'n String>, std::time::Duration)> { + let hint = retries.fault.take()?; + let shaky = std::mem::take(&mut retries.shaky); + if counting { + return None; + } + let n = retries.rounds.len() as u32 + 1; + let wait = hint.unwrap_or_else(|| state.retry_backoff.before(n)); + if until.saturating_duration_since(tokio::time::Instant::now()) < wait { + return None; + } + let pinned = started.choice.stayed_on.as_ref(); + let mut round: Vec<&String> = started + .alive + .iter() + .filter(|a| shaky.contains(a) && pinned.is_none_or(|p| p == *a)) + .collect(); + let open: Vec<&String> = round + .iter() + .copied() + .filter(|a| state.health.is_available(a)) + .collect(); + if round.len() > 1 && !open.is_empty() { + round = open; + } + (!round.is_empty()).then_some((round, wait)) +} + +/// 重试的那几轮：在尝试链上补上每一跳是第几次重试，每一轮的头一跳发出去之前退避了多久 +fn stamp_retries(chain: &mut [tw_api::AttemptView], rounds: &[(usize, u32, u64)]) { + for (i, &(from, n, ms)) in rounds.iter().enumerate() { + let to = rounds.get(i + 1).map_or(chain.len(), |r| r.0); + for a in chain.get_mut(from..to).into_iter().flatten() { + a.retry = Some(n); + } + if from < to { + chain[from].backoff_ms = Some(ms); + } + } +} + /// 一家暂时发不出去、等得到：跳过它时它是哪一样。 #[derive(Debug, Clone, Copy, PartialEq, Eq)] enum Wait { @@ -1522,7 +1647,29 @@ fn queued_up( }); } -/// 这个请求不排队了（见 [`tw_api::Event::RequestDequeued`]）：等到了要发出去、被中止了、等不到了 +/// 这个请求开始退避、之后重试（见 [`tw_api::Event::RequestRetrying`]）：再试哪几家、第几次、 +/// 退避多久，最多等到什么时候 +fn retrying( + state: &AppState, + id: u64, + round: &[&String], + attempt: u32, + wait: std::time::Duration, + deadline_ms: u64, +) { + let at_ms = now_ms(); + state.bus.emit(tw_api::Event::RequestRetrying { + id, + upstreams: round.iter().map(|n| n.to_string()).collect(), + attempt, + retry_at_ms: at_ms + wait.as_millis() as u64, + until_ms: deadline_ms, + at_ms, + }); +} + +/// 这个请求不排队、不退避了（见 [`tw_api::Event::RequestDequeued`]）：等到了要发出去、被中止了、 +/// 等不到了 fn dequeued(state: &AppState, id: u64) { state.bus.emit(tw_api::Event::RequestDequeued { id, @@ -1618,6 +1765,8 @@ fn estimated_hop( queued_ms: None, skipped: None, pause_ms: None, + retry: None, + backoff_ms: None, proxy: None, } } diff --git a/crates/tw-gateway/src/server/pipeline/idle.rs b/crates/tw-gateway/src/server/pipeline/idle.rs index 75a35386..4274ade6 100644 --- a/crates/tw-gateway/src/server/pipeline/idle.rs +++ b/crates/tw-gateway/src/server/pipeline/idle.rs @@ -13,7 +13,7 @@ //! 流式的回答压过 [`crate::OPENING_HOLD`] 之后响应头已经交出去了，它收到的只是保活注释（见 //! [`super::commit`]）。没有下一家了，回一个超时错误（504，见 //! [`crate::error::Source::Timeout`]），尝试链带着；响应头已经交出去的，在流里报这个错。 -//! - **最后一家的响应已经交出去了**（最后一家不压开头）、或者**已经有内容交给了客户端**： +//! - **最后一家的响应已经交出去了**（等不起了的最后一家不压开头）、或者**已经有内容交给了客户端**： //! 换不了 —— 换一家会把开头再发一遍。回答按客户端的格式以一条错误收尾，请求记成失败 //! （见 `relay`）。还没有内容的照样记这一家一次失败；内容出到一半才停的不记，和流在半路 //! 断了一样。 @@ -99,6 +99,8 @@ pub(super) fn abandoned( queued_ms: None, skipped: None, pause_ms: None, + retry: None, + backoff_ms: None, proxy: None, } } @@ -122,6 +124,8 @@ pub(super) fn aborted_hop( queued_ms: None, skipped: None, pause_ms: None, + retry: None, + backoff_ms: None, proxy: None, } } diff --git a/crates/tw-gateway/src/server/pipeline/opening.rs b/crates/tw-gateway/src/server/pipeline/opening.rs index f9ea1ee0..580c4622 100644 --- a/crates/tw-gateway/src/server/pipeline/opening.rs +++ b/crates/tw-gateway/src/server/pipeline/opening.rs @@ -11,10 +11,10 @@ //! 和无响应超时是同一个判据（见 [`crate::pulse`]）。 //! //! 等到无响应超时那一刻（`failover.idle_timeout_secs`，从请求发出去算起）还没有内容是 -//! [`Opening::Slow`]：调用方放弃这一家、换下一家。**最后一家只在它可能在限流时在这里等**（流式 -//! 的、请求还等得起，见 `hop`）：开头报的限流要暂停这一家、排队等它恢复；别的错、到点了还没有 -//! 内容，没有下一家可换，照原样交给回程。别的时候最后一家不等：压着它只会让客户端晚一点看到 -//! 同样的东西。 +//! [`Opening::Slow`]：调用方放弃这一家、换下一家。**最后一家在请求还等得起时也在这里等**（见 +//! `hop`）：开头报的暂时的错（过载、5xx）、流在内容之前断了，退避一阵再试（见 [`crate::retry`]）； +//! 报的限流暂停这一家、排队等它恢复；别的错、到点了还没有内容，没有下一家可换，照原样交给回程。 +//! 等不起的最后一家不等：压着它只会让客户端晚一点看到同样的东西。 //! 开头压得太多（[`HOLD_LIMIT`]）也不再压，交出去由回程照常计时。压着的这段时间里客户端等的是响应头：流式的回答过了 //! [`crate::OPENING_HOLD`] 先拿到 `200` 和保活注释，上游的事件照旧压在这里（见 //! [`super::commit`]）。 @@ -301,6 +301,19 @@ pub(super) fn stream_error(dialect: Dialect, data: &str) -> Option<(u16, Bytes)> } } +/// 流里报的这个错（[`Opening::Failed`] 的 `body`）说的是请求本身的问题：错误的类型是 +/// `invalid_request_error`。OpenAI 在流里报的码有的认不出（`invalid_encrypted_content`），按状态码 +/// 算是说不出原因的失败，照样换下一家；**可它不是暂时的错**，同一家再发一遍也一样（见 +/// [`crate::retry`]） +pub(super) fn faults_the_request(body: &[u8]) -> bool { + let Ok(v) = serde_json::from_slice::(body) else { + return false; + }; + ["/error/type", "/response/error/type"] + .iter() + .any(|p| v.pointer(p).and_then(|t| t.as_str()) == Some("invalid_request_error")) +} + fn error(status: u16, data: &str, kind: String, message: String) -> Judge { Judge::Error { status, @@ -501,6 +514,15 @@ mod tests { )); } + #[test] + fn an_invalid_request_in_the_stream_faults_the_request() { + let refused = r#"{"type":"error","error":{"type":"invalid_request_error","code":"invalid_encrypted_content","message":"x"}}"#; + assert!(faults_the_request(refused.as_bytes())); + let overloaded = r#"{"type":"error","error":{"type":"service_unavailable_error","code":"server_is_overloaded","message":"x"}}"#; + assert!(!faults_the_request(overloaded.as_bytes())); + assert!(!faults_the_request(b"not json")); + } + #[test] fn something_unrecognised_counts_as_content() { assert_eq!( diff --git a/crates/tw-gateway/src/server/pipeline/relay.rs b/crates/tw-gateway/src/server/pipeline/relay.rs index 926b2e42..c5068948 100644 --- a/crates/tw-gateway/src/server/pipeline/relay.rs +++ b/crates/tw-gateway/src/server/pipeline/relay.rs @@ -216,7 +216,7 @@ pub(super) fn respond( let mut said = false; // 被手动中止了（结局报手动中止，不说「流断了」） let mut aborted = false; - // 这一家的成败还没记（最后一家不压开头，见 `hop`）：第一段真内容到了记成功 + // 这一家的成败还没记（等不起了的最后一家不压开头，见 `hop`）：第一段真内容到了记成功 let mut unconfirmed = unconfirmed; loop { // 补心跳的那一刻。不补的格式给一个不会用到的时刻，那一支不参与 diff --git a/crates/tw-gateway/src/state.rs b/crates/tw-gateway/src/state.rs index 3586bbf1..4e2009b7 100644 --- a/crates/tw-gateway/src/state.rs +++ b/crates/tw-gateway/src/state.rs @@ -287,6 +287,8 @@ pub struct AppState { pub opening_hold: std::time::Duration, /// 响应头先交出去之后隔多久发一行保活（[`crate::KEEPALIVE_EVERY`]）。**测试会把它调短** pub keepalive_every: std::time::Duration, + /// 最后一家暂时出错时退避多久（见 [`crate::retry`]）。**测试会把它调短、去掉浮动** + pub retry_backoff: crate::retry::Backoff, /// 新旧 core 交接时网关这一边的东西：排空的标记、替旧 core 占着的名额（见 /// [`crate::handover`]）。**跨重载存活** pub(crate) handover: Arc, @@ -375,6 +377,7 @@ impl AppState { idle_tick: std::time::Duration::from_secs(1), opening_hold: crate::OPENING_HOLD, keepalive_every: crate::KEEPALIVE_EVERY, + retry_backoff: Default::default(), handover: Default::default(), }; // 手写的清单马上可用；向上游问是后台的事，不挡启动 diff --git a/crates/tw-gateway/src/throttle.rs b/crates/tw-gateway/src/throttle.rs index 9ee7d67a..0fe9144f 100644 --- a/crates/tw-gateway/src/throttle.rs +++ b/crates/tw-gateway/src/throttle.rs @@ -61,9 +61,6 @@ const MIN_HINT: Duration = Duration::from_millis(1); /// 每次 60 秒 const BACKOFF_SECS: [u64; 5] = [5, 10, 20, 40, 60]; -/// 一家最近一次被限流之后这么久之内，它还「可能在限流」（[`Throttles::wary`]） -const WARY_FOR: Duration = Duration::from_secs(300); - /// 两个时钟走的差，到了这么多才当是系统时间被拨过（见 [`Span::end_ms`]）；两个时钟换算出来的 /// 时刻差不到这么多的当是同一刻（见 [`Throttles::limit_state`]）。先后读两个时钟，差会抖一两毫秒 const SLACK_MS: u64 = 1_000; @@ -583,17 +580,6 @@ impl Throttles { (Peek::Open, None) } - /// 这一家此刻**可能在限流**：在探路、连续被限流还没有别的答复，或者最近 - /// （[`WARY_FOR`]）被限流过。最后一家的流式回答这时也压着开头（见 `hop`）：流的开头报的 - /// 限流要排队等它恢复，不交给客户端。一直好好的上游不压 —— 客户端照常当场拿到响应头和 - /// 开头的那几个事件 - pub fn wary(&self, name: &str) -> bool { - let now = self.moment(); - self.lock().get(name).is_some_and(|up| { - up.probing || up.strikes > 0 || up.limited_at.is_some_and(|t| t.since(now) < WARY_FOR) - }) - } - /// 留在这一家的对话还留不留（见 [`crate::affinity`]）：这一期用完了的、限流暂停要到 /// `deadline_ms`（这个请求的等待期限）之后才结束的，放开 —— 和熔断着的一样，留在一家发不出去 /// 的上游上，省下的缓存一分钱都兑现不了。等得到的照旧留，到时候等它 @@ -1955,20 +1941,6 @@ mod tests { assert!(matches!(t.gate("甲", &[], None), Gate::Go(Some(_)))); } - #[test] - fn an_upstream_is_wary_while_probing_and_for_a_while_after_a_429() { - let (t, c) = clocked(); - assert!(!t.wary("甲"), "没被限流过"); - t.limited("甲", c.at(T0), None); - assert!(t.wary("甲")); - c.advance(5_000); - t.answered("甲", c.at(T0 + 5_000)); - // 探路的过了、连续的次数归零：最近被限流过，还要当心 - assert!(t.wary("甲")); - c.advance(295_000); - assert!(!t.wary("甲")); - } - #[tokio::test] async fn a_renamed_upstream_keeps_its_pause_and_its_queue() { let (t, c) = clocked(); diff --git a/crates/tw-gateway/tests/alias_routing.rs b/crates/tw-gateway/tests/alias_routing.rs index 2c78be55..bad16d3d 100644 --- a/crates/tw-gateway/tests/alias_routing.rs +++ b/crates/tw-gateway/tests/alias_routing.rs @@ -346,6 +346,8 @@ async fn failover_never_reaches_a_pinned_model_the_key_may_not_use() { rules, ); cfg.clients[0].allow = Some(allow); + // 不重试：a 回的 500 当场交出去，这里只看 b 收没收到（重试见 `retry.rs`） + cfg.failover.wait_secs = 0; let (gw, rx) = serve(cfg).await; (gw, rx, a_seen, b_seen) } @@ -391,6 +393,8 @@ async fn failover_never_reaches_a_phase_two_rename_the_key_may_not_use() { ", ); cfg.clients[0].allow = Some(vec!["m-a".into()]); + // 不重试：a 回的 500 当场交出去，这里只看 b 收没收到（重试见 `retry.rs`） + cfg.failover.wait_secs = 0; let (gw, mut rx) = serve(cfg).await; let (status, body) = ask(gw, "m-a").await; diff --git a/crates/tw-gateway/tests/bedrock.rs b/crates/tw-gateway/tests/bedrock.rs index a1377cca..d34572f5 100644 --- a/crates/tw-gateway/tests/bedrock.rs +++ b/crates/tw-gateway/tests/bedrock.rs @@ -176,6 +176,30 @@ async fn gateway_with( SocketAddr, tw_gateway::AppState, tokio::sync::broadcast::Receiver, +) { + gateway_of(p, security, tw_config::Failover::default().wait_secs).await +} + +/// 起网关，一个请求最多等 `wait_secs` 秒（见 `failover.wait_secs`） +async fn gateway_waiting( + p: Provider, + wait_secs: u64, +) -> ( + SocketAddr, + tw_gateway::AppState, + tokio::sync::broadcast::Receiver, +) { + gateway_of(p, Security::default(), wait_secs).await +} + +async fn gateway_of( + p: Provider, + security: Security, + wait_secs: u64, +) -> ( + SocketAddr, + tw_gateway::AppState, + tokio::sync::broadcast::Receiver, ) { let cfg = Config { version: 1, @@ -187,6 +211,10 @@ async fn gateway_with( }], providers: vec![p], security, + failover: tw_config::Failover { + wait_secs, + ..Default::default() + }, ..Default::default() }; let state = tw_gateway::AppState::new(cfg).unwrap(); @@ -382,7 +410,8 @@ async fn an_exception_inside_the_stream_fails_the_request() { json!({"message": "Too many tokens, please wait before trying again."}), )); let (up, _) = bedrock(Arc::new(move |_| eventstream_answer(wire.clone()))).await; - let (gw, _, mut rx) = gateway(with_keys(up)).await; + // 不等：唯一的上游不压开头，它在流里报的限流原样交给客户端（等它恢复见 `throttle_queue.rs`） + let (gw, _, mut rx) = gateway_waiting(with_keys(up), 0).await; let (status, body) = post( gw, "/v1/messages", diff --git a/crates/tw-gateway/tests/claude_desktop.rs b/crates/tw-gateway/tests/claude_desktop.rs index 9a629d09..0d1eb55f 100644 --- a/crates/tw-gateway/tests/claude_desktop.rs +++ b/crates/tw-gateway/tests/claude_desktop.rs @@ -110,6 +110,11 @@ async fn gateway_pinging_for(p: Provider, ping_for: Duration) -> Gateway { ..Default::default() }], providers: vec![p], + // 不等、不重试：唯一的上游不压开头，内容之前的静默也由回程补 `ping` + failover: tw_config::Failover { + wait_secs: 0, + ..Default::default() + }, ..Default::default() }; let mut state = tw_gateway::AppState::new(cfg).unwrap(); diff --git a/crates/tw-gateway/tests/endings.rs b/crates/tw-gateway/tests/endings.rs index b49a4a59..b9f943b2 100644 --- a/crates/tw-gateway/tests/endings.rs +++ b/crates/tw-gateway/tests/endings.rs @@ -205,6 +205,8 @@ fn provider(upstream: SocketAddr) -> Provider { } } +/// 不等、不重试（`wait_secs: 0`）：唯一的上游不压开头，第一帧当场交给客户端 —— 这里看的是 +/// 之后怎么收场。最后一家压着开头、退避重试见 `retry.rs` fn cfg(provider: Provider) -> Config { Config { clients: vec![Client { @@ -213,6 +215,10 @@ fn cfg(provider: Provider) -> Config { ..Default::default() }], providers: vec![provider], + failover: tw_config::Failover { + wait_secs: 0, + ..Default::default() + }, ..Default::default() } } @@ -598,13 +604,11 @@ async fn a_phase_two_deny_after_the_request_started_is_reported_as_denied() { /// 每一家上游都没接住。**只报一次失败，`source` 是它真正的原因** —— 被限流 /// 就是 `rate_limited`，和客户端收到的 `x-thinkwatch-error` 是同一个词。 /// -/// 限流的上游请求会等它恢复（见 `tw_gateway::throttle`）：这里不让等（`wait_secs: 0`）， +/// 限流的上游请求会等它恢复（见 `tw_gateway::throttle`）：这里不让等（`cfg` 的 `wait_secs: 0`）， /// 被限流的那一次就是结局 #[tokio::test] async fn a_request_every_upstream_refused_is_failed_once_for_the_reason_it_was_refused() { - let mut c = cfg(provider(refusing_upstream(429).await)); - c.failover.wait_secs = 0; - let (gw, mut events) = serve(c).await; + let (gw, mut events) = serve(cfg(provider(refusing_upstream(429).await))).await; let r = post(gw).send().await.unwrap(); assert_eq!(r.status(), 429); let header = r.headers().get("x-thinkwatch-error").cloned(); diff --git a/crates/tw-gateway/tests/first_token.rs b/crates/tw-gateway/tests/first_token.rs index 711cc81c..5203c407 100644 --- a/crates/tw-gateway/tests/first_token.rs +++ b/crates/tw-gateway/tests/first_token.rs @@ -49,6 +49,8 @@ fn streaming(frames_after_prefill: Vec<&'static [u8]>, first: &'static [u8]) -> })) } +/// 不等、不重试（`wait_secs: 0`）：唯一的上游不压开头，响应头和流的开头当场交给客户端 —— +/// 这里量的正是它们各在什么时候到 fn cfg(upstream: SocketAddr, protocol: Protocol) -> Config { Config { clients: vec![Client { @@ -63,6 +65,10 @@ fn cfg(upstream: SocketAddr, protocol: Protocol) -> Config { protocol: Some(protocol), ..Default::default() }], + failover: tw_config::Failover { + wait_secs: 0, + ..Default::default() + }, ..Default::default() } } diff --git a/crates/tw-gateway/tests/hotreload.rs b/crates/tw-gateway/tests/hotreload.rs index 513297d2..8dd3fcc8 100644 --- a/crates/tw-gateway/tests/hotreload.rs +++ b/crates/tw-gateway/tests/hotreload.rs @@ -335,6 +335,8 @@ async fn changing_the_proxy_does_rebuild_that_client() { addr: "127.0.0.1:17890".into(), auth: None, }]; + // 不重试：连不上的那一次当场交出去 + next.failover.wait_secs = 0; state.reload(next).unwrap(); // 代理起不来，请求该失败 —— 这就是「真的换成新 Client 了」的证据 let gw = serve(state.clone()).await; diff --git a/crates/tw-gateway/tests/live.rs b/crates/tw-gateway/tests/live.rs index 1c5c46d0..94bfeaac 100644 --- a/crates/tw-gateway/tests/live.rs +++ b/crates/tw-gateway/tests/live.rs @@ -63,6 +63,11 @@ async fn serve(upstream: SocketAddr) -> (SocketAddr, tw_gateway::live::Live) { protocol: Some(tw_config::Protocol::Anthropic), ..Default::default() }], + // 不等、不重试：唯一的上游不压开头，流的第一帧当场交给客户端 + failover: tw_config::Failover { + wait_secs: 0, + ..Default::default() + }, ..Default::default() }; let state = tw_gateway::AppState::new(cfg).unwrap(); diff --git a/crates/tw-gateway/tests/notices.rs b/crates/tw-gateway/tests/notices.rs index a4ff92aa..21d409d3 100644 --- a/crates/tw-gateway/tests/notices.rs +++ b/crates/tw-gateway/tests/notices.rs @@ -113,6 +113,7 @@ async fn proxy(broken: Arc) -> SocketAddr { // ---------------------------------------------------------------- 网关 +/// 不重试（`wait_secs: 0`）：连不上的那一次当场报出来（重试见 `retry.rs`） fn config(p: Provider) -> Config { Config { version: 1, @@ -123,6 +124,10 @@ fn config(p: Provider) -> Config { ..Default::default() }], providers: vec![p], + failover: tw_config::Failover { + wait_secs: 0, + ..Default::default() + }, ..Default::default() } } diff --git a/crates/tw-gateway/tests/passthrough.rs b/crates/tw-gateway/tests/passthrough.rs index ac98fd5f..405b984d 100644 --- a/crates/tw-gateway/tests/passthrough.rs +++ b/crates/tw-gateway/tests/passthrough.rs @@ -353,7 +353,11 @@ async fn an_unreachable_upstream_emits_a_failure_event_and_a_502() { let dead_port = spare_port(); let cfg = Config { retention: Default::default(), - failover: Default::default(), + // 不重试：连不上的那一次当场交出去（退避着重试见 `retry.rs`） + failover: tw_config::Failover { + wait_secs: 0, + ..Default::default() + }, aliases: Default::default(), default_route: None, default_key: None, @@ -860,7 +864,7 @@ async fn the_only_provider_keeps_being_tried_no_matter_how_broken() { // 唯一的上游被自己熔断就把用户锁死了。没有别的家可切的时候， // 熔断纯粹是自伤。 let dead = start_broken_upstream(503).await; - let gw = serve_cfg(cfg_with( + let mut cfg = cfg_with( vec![Provider { name: "only".into(), base_url: format!("http://{dead}"), @@ -868,8 +872,10 @@ async fn the_only_provider_keeps_being_tried_no_matter_how_broken() { ..Default::default() }], vec![], - )) - .await; + ); + // 不重试：每个请求只打一次，这里数的是熔断（重试见 `retry.rs`） + cfg.failover.wait_secs = 0; + let gw = serve_cfg(cfg).await; // 打满熔断阈值之后再打一次，仍然应该是「上游的错误」而不是 // 「我们编的无可用上游」 for _ in 0..5 { @@ -885,7 +891,7 @@ async fn everything_broken_still_tries_rather_than_refusing() { // 返回一个我们自己编的「无可用上游」—— 后者会让用户以为是我们坏了。 let a = start_broken_upstream(503).await; let b = start_broken_upstream(503).await; - let gw = serve_cfg(cfg_with( + let mut cfg = cfg_with( vec![ Provider { name: "a".into(), @@ -901,8 +907,10 @@ async fn everything_broken_still_tries_rather_than_refusing() { }, ], vec![], - )) - .await; + ); + // 不重试：每个请求只走一遍，这里数的是熔断（重试见 `retry.rs`） + cfg.failover.wait_secs = 0; + let gw = serve_cfg(cfg).await; for _ in 0..6 { send_to(gw).await; } @@ -1788,7 +1796,7 @@ async fn a_stream_that_dies_midway_says_so_instead_of_just_stopping() { } }); - let cfg = cfg_with( + let mut cfg = cfg_with( vec![Provider { name: "up".into(), base_url: format!("http://{up}"), @@ -1797,6 +1805,8 @@ async fn a_stream_that_dies_midway_says_so_instead_of_just_stopping() { }], vec![], ); + // 不等、不重试：唯一的上游不压开头，第一帧当场交给客户端，断在交出去之后 + cfg.failover.wait_secs = 0; let state = tw_gateway::AppState::new(cfg).unwrap(); let mut rx = state.bus.subscribe(); let gw = tw_gateway::serve(state, ([127, 0, 0, 1], 0).into()) @@ -2200,7 +2210,7 @@ async fn a_request_that_fails_everywhere_still_reports_the_chain() { // **失败那条路才是最需要看尝试链的时候。**挂在 RequestFinished 上 // 的话，它恰好在那时缺席。 let dead = start_broken_upstream(503).await; - let state = tw_gateway::AppState::new(cfg_with( + let mut cfg = cfg_with( vec![Provider { name: "挂了的".into(), base_url: format!("http://{dead}"), @@ -2208,8 +2218,10 @@ async fn a_request_that_fails_everywhere_still_reports_the_chain() { ..Default::default() }], vec![], - )) - .unwrap(); + ); + // 不重试：链上只有那一跳（重试的一跳见 `retry.rs`） + cfg.failover.wait_secs = 0; + let state = tw_gateway::AppState::new(cfg).unwrap(); let mut rx = state.bus.subscribe(); let gw = tw_gateway::serve(state, ([127, 0, 0, 1], 0).into()) .await diff --git a/crates/tw-gateway/tests/retry.rs b/crates/tw-gateway/tests/retry.rs new file mode 100644 index 00000000..583b1fc1 --- /dev/null +++ b/crates/tw-gateway/tests/retry.rs @@ -0,0 +1,860 @@ +//! 最后一家暂时出错时退避重试（`tw_gateway::retry`），端到端。 +//! +//! 假上游按脚本一个个地答：先过载、5xx、断流几次，之后照常答。退避的一秒在测试里调成一百毫秒 +//! （`AppState::retry_backoff`），不浮动：退避 200、400、800 毫秒…… 等待期限 +//! （`failover.wait_secs`）是真的秒数。 + +mod common; + +use std::collections::VecDeque; +use std::net::SocketAddr; +use std::sync::{Arc, Mutex}; +use std::time::{Duration, Instant}; + +use axum::Router; +use futures::StreamExt; +use tw_api::{AttemptOutcome, AttemptView, Event}; + +/// 测试里退避的一秒 +const TICK: Duration = Duration::from_millis(100); + +/// 假上游的一次答复 +#[derive(Clone)] +struct Reply { + status: u16, + headers: Vec<(&'static str, String)>, + body: String, + /// 流式的回答：响应头之后一段一段地给。有的话 `body` 不用 + chunks: Vec, + /// 给完 `chunks` 之后把连接断掉，不好好收尾 + broken: bool, + content_type: &'static str, +} + +/// 整包地照常答 +fn ok() -> Reply { + Reply { + status: 200, + headers: Vec::new(), + body: r#"{"type":"message","role":"assistant","model":"claude-sonnet-5","content":[{"type":"text","text":"ok"}],"stop_reason":"end_turn","usage":{"input_tokens":3,"cache_read_input_tokens":5000,"output_tokens":1}}"#.into(), + chunks: Vec::new(), + broken: false, + content_type: "application/json", + } +} + +/// 回一个错误状态码，正文是 `body` +fn status(code: u16, body: &str) -> Reply { + Reply { + status: code, + body: body.into(), + ..ok() + } +} + +fn overloaded_503() -> Reply { + status( + 503, + r#"{"type":"error","error":{"type":"overloaded_error","message":"Overloaded"}}"#, + ) +} + +/// Anthropic 流式回答里的一个事件 +fn event(name: &str, data: &str) -> String { + format!("event: {name}\ndata: {data}\n\n") +} + +fn message_start() -> String { + event( + "message_start", + r#"{"type":"message_start","message":{"id":"msg_1","type":"message","role":"assistant","model":"claude-sonnet-5","content":[],"stop_reason":null,"usage":{"input_tokens":3,"output_tokens":0}}}"#, + ) +} + +fn text(word: &str) -> String { + event( + "content_block_delta", + &format!( + r#"{{"type":"content_block_delta","index":0,"delta":{{"type":"text_delta","text":"{word}"}}}}"# + ), + ) +} + +fn overloaded_event() -> String { + event( + "error", + r#"{"type":"error","error":{"type":"overloaded_error","message":"Overloaded"}}"#, + ) +} + +fn sse(chunks: Vec) -> Reply { + Reply { + chunks, + content_type: "text/event-stream", + ..ok() + } +} + +/// 流式地照常答：开头、一段内容「ok」、收尾 +fn streamed() -> Reply { + sse(vec![ + message_start(), + event( + "content_block_start", + r#"{"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}"#, + ), + text("ok"), + event( + "content_block_stop", + r#"{"type":"content_block_stop","index":0}"#, + ), + event( + "message_delta", + r#"{"type":"message_delta","delta":{"stop_reason":"end_turn"},"usage":{"output_tokens":1}}"#, + ), + event("message_stop", r#"{"type":"message_stop"}"#), + ]) +} + +/// 响应头是 200、流也开了，开头之后在流里报过载，一段内容都还没有 +fn overloaded_in_the_stream() -> Reply { + sse(vec![message_start(), overloaded_event()]) +} + +/// 一家假上游：先按 `script` 的顺序答，答完了一律答 `then` +struct Up { + addr: SocketAddr, + hits: Arc>>, +} + +impl Up { + fn hits(&self) -> usize { + self.hits.lock().unwrap().len() + } +} + +fn app(script: Vec, then: Reply, hits: Arc>>) -> Router { + let script = Arc::new(Mutex::new(VecDeque::from(script))); + Router::new().fallback(axum::routing::any(move || { + let (script, h, then) = (script.clone(), hits.clone(), then.clone()); + async move { + h.lock().unwrap().push(Instant::now()); + let r = script.lock().unwrap().pop_front().unwrap_or(then); + let mut b = axum::response::Response::builder() + .status(r.status) + .header("content-type", r.content_type); + for (k, v) in &r.headers { + b = b.header(*k, v.as_str()); + } + if r.chunks.is_empty() { + return b.body(axum::body::Body::from(r.body)).unwrap(); + } + let broken = r.broken; + let chunks = futures::stream::iter(r.chunks) + .map(|c| Ok::<_, std::io::Error>(bytes::Bytes::from(c))) + .chain(futures::stream::iter( + broken.then(|| Err(std::io::Error::other("cut"))), + )); + b.body(axum::body::Body::from_stream(chunks)).unwrap() + } + })) +} + +async fn upstream(script: Vec, then: Reply) -> Up { + let hits = Arc::new(Mutex::new(Vec::new())); + let l = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let addr = l.local_addr().unwrap(); + let app = app(script, then, hits.clone()); + tokio::spawn(async move { axum::serve(l, app).await.unwrap() }); + Up { addr, hits } +} + +/// 一家或者两家（按顺序的组「池」）上游，默认路由把请求都交给它们 +fn config(ups: &[(SocketAddr, &str)], wait_secs: u64) -> tw_config::Config { + let names = ["甲", "乙"]; + let mut yaml = + "version: 1\ndefault_route: 默认\nclients:\n - name: me\n key: tw-k\nproviders:\n" + .to_string(); + for (n, (addr, protocol)) in names.iter().zip(ups) { + yaml += &format!( + " - name: {n}\n base_url: http://{addr}\n key: sk-x\n protocol: {protocol}\n" + ); + } + let members = names[..ups.len()].join(", "); + yaml += &format!( + "groups:\n - name: 池\n type: fallback\n providers: [{members}]\nroutes:\n - name: 默认\n rules:\n - name: 全部\n to: 池\nfailover:\n wait_secs: {wait_secs}\n" + ); + serde_yaml_ng::from_str(&yaml).unwrap() +} + +fn anthropic(ups: &[&Up], wait_secs: u64) -> tw_config::Config { + let ups: Vec<_> = ups.iter().map(|u| (u.addr, "anthropic")).collect(); + config(&ups, wait_secs) +} + +/// 总线上发生过的事件，后台一直收着 +#[derive(Clone)] +struct Log(Arc>>); + +impl Log { + async fn until(&self, what: &str, f: impl Fn(&[Event]) -> Option) -> T { + for _ in 0..500 { + if let Some(t) = f(&self.0.lock().unwrap()) { + return t; + } + tokio::time::sleep(Duration::from_millis(20)).await; + } + panic!("10 秒内没等到{what}：{:#?}", self.0.lock().unwrap()); + } + + /// 客户端写的模型是 `model` 的那个请求的号 + async fn id(&self, model: &str) -> u64 { + self.until(&format!(" {model} 开始"), |evs| { + evs.iter().find_map(|e| match e { + Event::RequestStarted { id, model: m, .. } if m == model => Some(*id), + _ => None, + }) + }) + .await + } + + /// 那个请求的尝试链 + async fn attempts(&self, model: &str) -> Vec { + let id = self.id(model).await; + self.until(&format!(" {model} 的路由"), |evs| { + evs.iter().find_map(|e| match e { + Event::RequestRouted { + id: i, attempts, .. + } if *i == id => Some(attempts.clone()), + _ => None, + }) + }) + .await + } + + /// 那个请求的退避和排队事件，按先后 + async fn waits(&self, model: &str) -> Vec { + let id = self.id(model).await; + self.0 + .lock() + .unwrap() + .iter() + .filter(|e| match e { + Event::RequestRetrying { id: i, .. } + | Event::RequestQueued { id: i, .. } + | Event::RequestDequeued { id: i, .. } => *i == id, + _ => false, + }) + .cloned() + .collect() + } +} + +/// 起网关。退避的一秒是 `tick`，不浮动；流式回答最多压 `hold` 不给响应头 +async fn serve_with( + cfg: tw_config::Config, + tick: Duration, + hold: Duration, +) -> (SocketAddr, tw_gateway::AppState, Log) { + let mut state = tw_gateway::AppState::new(cfg).unwrap(); + state.retry_backoff = tw_gateway::retry::Backoff { tick, jitter: 0.0 }; + state.opening_hold = hold; + state.keepalive_every = Duration::from_millis(150); + let mut rx = state.bus.subscribe(); + let log = Log(Arc::default()); + let into = log.0.clone(); + tokio::spawn(async move { + while let Ok(ev) = rx.recv().await { + into.lock().unwrap().push(ev); + } + }); + let addr = tw_gateway::serve(state.clone(), ([127, 0, 0, 1], 0).into()) + .await + .unwrap(); + (addr, state, log) +} + +async fn serve(cfg: tw_config::Config) -> (SocketAddr, tw_gateway::AppState, Log) { + serve_with(cfg, TICK, Duration::from_secs(10)).await +} + +fn client() -> reqwest::Client { + reqwest::Client::builder().no_proxy().build().unwrap() +} + +/// 一个 Anthropic 的请求，客户端写的模型是 `model`（认请求用）。`conversation` 给了的话带着 +/// 会话头，`turn` 是这段对话的第几轮 +fn ask_in( + gw: SocketAddr, + model: &str, + stream: bool, + conversation: Option<&str>, + turn: usize, +) -> reqwest::RequestBuilder { + let mut messages = Vec::new(); + for i in 0..turn { + if i > 0 { + messages.push(r#"{"role":"assistant","content":"ok"}"#.to_string()); + } + let who = conversation.unwrap_or(model); + messages.push(format!(r#"{{"role":"user","content":"{who}：第 {i} 句"}}"#)); + } + let body = format!( + r#"{{"model":"{model}","max_tokens":16,"stream":{stream},"system":"你是一个助手","messages":[{}]}}"#, + messages.join(",") + ); + let mut r = client() + .post(format!("http://{gw}/v1/messages")) + .header("x-api-key", "tw-k") + .header("anthropic-version", "2023-06-01") + .body(body); + if let Some(c) = conversation { + r = r.header("x-claude-code-session-id", c); + } + r +} + +fn ask(gw: SocketAddr, model: &str) -> reqwest::RequestBuilder { + ask_in(gw, model, false, None, 1) +} + +fn ask_stream(gw: SocketAddr, model: &str) -> reqwest::RequestBuilder { + ask_in(gw, model, true, None, 1) +} + +/// 尝试链上每一跳的 (结果, 第几次重试, 退避过没有) +fn shape(chain: &[AttemptView]) -> Vec<(AttemptOutcome, Option, bool)> { + chain + .iter() + .map(|a| (a.outcome, a.retry, a.backoff_ms.is_some())) + .collect() +} + +#[tokio::test] +async fn an_overload_in_the_stream_before_content_is_retried_on_the_lone_upstream() { + let up = upstream(vec![overloaded_in_the_stream()], streamed()).await; + let (gw, state, log) = serve(anthropic(&[&up], 10)).await; + let r = tokio::spawn(ask_stream(gw, "单家流里过载").send()); + // 退避的时候半路才来看的一方也看得出它在重试 + let id = log.id("单家流里过载").await; + let at = log + .until("退避", |evs| { + evs.iter().find_map(|e| match e { + Event::RequestRetrying { + id: i, + upstreams, + attempt, + retry_at_ms, + until_ms, + at_ms, + } if *i == id => Some(( + upstreams.clone(), + *attempt, + *retry_at_ms - *at_ms, + *until_ms - *at_ms, + )), + _ => None, + }) + }) + .await; + let (upstreams, attempt, wait, left) = at; + assert_eq!((upstreams, attempt), (vec!["甲".to_string()], 1)); + assert_eq!(wait, 200, "第一次退避两秒（测试里的两百毫秒）"); + assert!(left > 9_000 && left <= 10_000, "{left}"); + let r = r.await.unwrap().unwrap(); + assert_eq!(r.status(), 200); + let body = r.text().await.unwrap(); + assert!(body.contains("\"ok\""), "{body}"); + assert!( + !body.contains("overloaded_error"), + "过载那一次客户端看不到：{body}" + ); + assert_eq!(up.hits(), 2); + + let chain = log.attempts("单家流里过载").await; + assert_eq!( + shape(&chain), + [ + (AttemptOutcome::Error, None, false), + (AttemptOutcome::Served, Some(1), true) + ] + ); + assert_eq!( + chain[0].error.as_ref().unwrap().code, + "gw.upstream.stream_opening_error" + ); + assert!(chain[1].backoff_ms.unwrap() >= 190, "{:?}", chain[1]); + // 退避和退避结束成对 + let waits = log.waits("单家流里过载").await; + assert!( + matches!( + waits.as_slice(), + [Event::RequestRetrying { .. }, Event::RequestDequeued { .. }] + ), + "{waits:#?}" + ); + // 成功了就是成功：这一家照样记成功 + assert!(state.health.is_available("甲")); +} + +/// Codex（Responses）的上游：200 之后在流里报 `server_is_overloaded`，一段内容都没有 +#[tokio::test] +async fn a_responses_stream_that_reports_server_is_overloaded_is_retried() { + let created = event( + "response.created", + r#"{"type":"response.created","sequence_number":0,"response":{"id":"resp_1","status":"in_progress"}}"#, + ); + let overloaded = sse(vec![ + created.clone(), + event( + "response.in_progress", + r#"{"type":"response.in_progress","sequence_number":1,"response":{"id":"resp_1","status":"in_progress"}}"#, + ), + event( + "error", + r#"{"type":"error","sequence_number":3,"error":{"type":"service_unavailable_error","code":"server_is_overloaded","message":"Our servers are currently overloaded. Please try again later.","param":null}}"#, + ), + ]); + let answered = sse(vec![ + created, + event( + "response.output_text.delta", + r#"{"type":"response.output_text.delta","sequence_number":1,"item_id":"m1","output_index":0,"content_index":0,"delta":"ok"}"#, + ), + event( + "response.completed", + r#"{"type":"response.completed","sequence_number":2,"response":{"id":"resp_1","status":"completed","usage":{"input_tokens":3,"output_tokens":1,"total_tokens":4}}}"#, + ), + ]); + let up = upstream(vec![overloaded], answered).await; + let (gw, _state, log) = serve(config(&[(up.addr, "openai-responses")], 10)).await; + let r = client() + .post(format!("http://{gw}/v1/responses")) + .header("authorization", "Bearer tw-k") + .header("content-type", "application/json") + .body(r#"{"model":"codex-过载","stream":true,"input":"hi"}"#) + .send() + .await + .unwrap(); + assert_eq!(r.status(), 200); + let body = r.text().await.unwrap(); + assert!(body.contains("\"delta\":\"ok\""), "{body}"); + assert!(!body.contains("server_is_overloaded"), "{body}"); + assert_eq!(up.hits(), 2); + let chain = log.attempts("codex-过载").await; + assert_eq!(chain.len(), 2); + assert_eq!(chain[1].retry, Some(1)); +} + +#[tokio::test] +async fn an_immediate_503_is_retried_until_it_answers() { + let up = upstream(vec![overloaded_503(), status(502, "bad gateway")], ok()).await; + let (gw, _state, log) = serve(anthropic(&[&up], 10)).await; + let t = Instant::now(); + let r = ask(gw, "两次 5xx").send().await.unwrap(); + assert_eq!(r.status(), 200); + assert!(r.text().await.unwrap().contains("\"ok\"")); + assert_eq!(up.hits(), 3); + assert!( + t.elapsed() >= Duration::from_millis(600), + "退避 200 + 400 毫秒：{:?}", + t.elapsed() + ); + let chain = log.attempts("两次 5xx").await; + assert_eq!( + shape(&chain), + [ + (AttemptOutcome::Status, None, false), + (AttemptOutcome::Status, Some(1), true), + (AttemptOutcome::Served, Some(2), true) + ] + ); + assert_eq!(chain[0].status, Some(503)); + assert_eq!(chain[1].status, Some(502)); + assert!(chain[2].backoff_ms.unwrap() >= 390, "{:?}", chain[2]); + let waits = log.waits("两次 5xx").await; + let attempts: Vec = waits + .iter() + .filter_map(|e| match e { + Event::RequestRetrying { attempt, .. } => Some(*attempt), + _ => None, + }) + .collect(); + assert_eq!(attempts, [1, 2]); + assert_eq!(waits.len(), 4, "每次退避都有结束：{waits:#?}"); +} + +#[tokio::test] +async fn an_upstream_that_cannot_be_reached_yet_is_retried() { + let port = common::spare_port(); + let addr: SocketAddr = ([127, 0, 0, 1], port).into(); + let hits = Arc::new(Mutex::new(Vec::new())); + let later = hits.clone(); + // 一开始没人听这个端口，过一阵才起来 + tokio::spawn(async move { + tokio::time::sleep(Duration::from_millis(450)).await; + let l = tokio::net::TcpListener::bind(addr).await.unwrap(); + axum::serve(l, app(Vec::new(), ok(), later)).await.unwrap(); + }); + let (gw, _state, log) = serve(config(&[(addr, "anthropic")], 10)).await; + let r = ask(gw, "连不上").send().await.unwrap(); + assert_eq!(r.status(), 200); + assert_eq!(hits.lock().unwrap().len(), 1); + let chain = log.attempts("连不上").await; + assert!(chain.len() >= 3, "{chain:#?}"); + assert_eq!(chain.last().unwrap().outcome, AttemptOutcome::Served); + assert!( + chain[..chain.len() - 1] + .iter() + .all(|a| a.outcome == AttemptOutcome::Error), + "{chain:#?}" + ); +} + +#[tokio::test] +async fn a_stream_broken_before_content_is_retried() { + let cut = Reply { + broken: true, + ..sse(vec![message_start()]) + }; + let up = upstream(vec![cut], streamed()).await; + let (gw, _state, log) = serve(anthropic(&[&up], 10)).await; + let r = ask_stream(gw, "断流").send().await.unwrap(); + assert_eq!(r.status(), 200); + assert!(r.text().await.unwrap().contains("\"ok\"")); + assert_eq!(up.hits(), 2); + let chain = log.attempts("断流").await; + assert_eq!( + shape(&chain), + [ + (AttemptOutcome::Error, None, false), + (AttemptOutcome::Served, Some(1), true) + ] + ); +} + +#[tokio::test] +async fn an_error_after_content_is_not_retried() { + let half = sse(vec![message_start(), text("partial"), overloaded_event()]); + let up = upstream(vec![half], streamed()).await; + let (gw, _state, log) = serve(anthropic(&[&up], 10)).await; + let r = ask_stream(gw, "内容之后出错").send().await.unwrap(); + assert_eq!(r.status(), 200); + let body = r.text().await.unwrap(); + assert!(body.contains("partial"), "{body}"); + assert!(body.contains("overloaded_error"), "{body}"); + assert_eq!(up.hits(), 1, "已经有内容交给了客户端，再发一遍会重复开头"); + let chain = log.attempts("内容之后出错").await; + assert_eq!(shape(&chain), [(AttemptOutcome::Served, None, false)]); +} + +#[tokio::test] +async fn failures_that_are_not_temporary_are_handed_over_at_once() { + let cases = [ + ( + "请求本身错了", + status( + 400, + r#"{"type":"error","error":{"type":"invalid_request_error","message":"max_tokens: must be positive"}}"#, + ), + 400, + ), + ( + "凭据被拒", + status( + 401, + r#"{"type":"error","error":{"type":"authentication_error","message":"invalid x-api-key"}}"#, + ), + 401, + ), + ( + "没有这个模型", + status( + 404, + r#"{"type":"error","error":{"type":"not_found_error","message":"model: x"}}"#, + ), + 404, + ), + ( + "额度用完", + status( + 429, + r#"{"error":{"type":"usage_limit_reached","message":"The usage limit has been reached"}}"#, + ), + 429, + ), + ( + "没钱了", + status(402, r#"{"error":{"message":"Insufficient Balance"}}"#), + 402, + ), + ]; + for (model, reply, code) in cases { + let up = upstream(Vec::new(), reply).await; + let (gw, _state, log) = serve(anthropic(&[&up], 10)).await; + let r = ask(gw, model).send().await.unwrap(); + assert_eq!(r.status(), code, "{model}"); + assert_eq!(up.hits(), 1, "{model}"); + let waits = log.waits(model).await; + assert!(waits.is_empty(), "{model}：{waits:#?}"); + } +} + +/// 流里报的请求本身的问题（加密的历史解不开）：原样交出去，不重试 +#[tokio::test] +async fn an_invalid_request_reported_in_the_stream_is_not_retried() { + let refused = sse(vec![ + event( + "response.created", + r#"{"type":"response.created","sequence_number":0,"response":{"id":"resp_1"}}"#, + ), + event( + "error", + r#"{"type":"error","sequence_number":1,"error":{"type":"invalid_request_error","code":"invalid_encrypted_content","message":"The encrypted content could not be verified."}}"#, + ), + ]); + let up = upstream(Vec::new(), refused).await; + let (gw, _state, log) = serve(config(&[(up.addr, "openai-responses")], 10)).await; + let r = client() + .post(format!("http://{gw}/v1/responses")) + .header("authorization", "Bearer tw-k") + .header("content-type", "application/json") + .body(r#"{"model":"解不开","stream":true,"input":"hi"}"#) + .send() + .await + .unwrap(); + let body = r.text().await.unwrap(); + assert!(body.contains("invalid_encrypted_content"), "{body}"); + assert_eq!(up.hits(), 1); + assert!(log.waits("解不开").await.is_empty()); +} + +#[tokio::test] +async fn at_the_deadline_the_last_error_is_handed_over() { + let up = upstream(Vec::new(), overloaded_503()).await; + let (gw, _state, log) = serve(anthropic(&[&up], 1)).await; + let t = Instant::now(); + let r = ask(gw, "一直过载").send().await.unwrap(); + // 退避 200、400 毫秒之后剩下的不够 800 毫秒：交出最后那个错误 + assert_eq!(r.status(), 502); + let body = r.text().await.unwrap(); + assert!(body.contains("answered 503"), "{body}"); + assert_eq!(up.hits(), 3); + assert!( + t.elapsed() < Duration::from_millis(1500), + "{:?}", + t.elapsed() + ); + let chain = log.attempts("一直过载").await; + let retries: Vec> = chain.iter().map(|a| a.retry).collect(); + assert_eq!(retries, [None, Some(1), Some(2)]); +} + +#[tokio::test] +async fn wait_secs_zero_never_retries() { + let up = upstream(vec![overloaded_in_the_stream()], streamed()).await; + let (gw, _state, log) = serve(anthropic(&[&up], 0)).await; + let r = ask_stream(gw, "不等").send().await.unwrap(); + // 等不起的最后一家不压开头：它在流里报的错原样到了客户端 + assert_eq!(r.status(), 200); + assert!(r.text().await.unwrap().contains("overloaded_error")); + assert_eq!(up.hits(), 1); + assert!(log.waits("不等").await.is_empty()); +} + +#[tokio::test] +async fn retry_after_from_the_upstream_replaces_the_backoff() { + let said = Reply { + headers: vec![("retry-after-ms", "300".into())], + ..overloaded_503() + }; + let up = upstream(vec![said], ok()).await; + // 退避的一秒是十秒：按退避算要等二十秒，比等待期限还长 + let (gw, _state, log) = serve_with( + anthropic(&[&up], 10), + Duration::from_secs(10), + Duration::from_secs(10), + ) + .await; + let t = Instant::now(); + let r = ask(gw, "上游说了等多久").send().await.unwrap(); + assert_eq!(r.status(), 200); + assert!(t.elapsed() >= Duration::from_millis(300)); + assert!(t.elapsed() < Duration::from_secs(3), "{:?}", t.elapsed()); + let chain = log.attempts("上游说了等多久").await; + assert!(chain[1].backoff_ms.unwrap() >= 290, "{chain:#?}"); +} + +#[tokio::test] +async fn a_retry_after_beyond_the_deadline_gives_up_at_once() { + let said = Reply { + headers: vec![("retry-after", "5".into())], + ..overloaded_503() + }; + let up = upstream(Vec::new(), said).await; + let (gw, _state, log) = serve(anthropic(&[&up], 2)).await; + let t = Instant::now(); + let r = ask(gw, "等不到那时").send().await.unwrap(); + assert_eq!(r.status(), 502); + assert_eq!(up.hits(), 1); + assert!( + t.elapsed() < Duration::from_millis(500), + "{:?}", + t.elapsed() + ); + assert!(log.waits("等不到那时").await.is_empty()); +} + +#[tokio::test] +async fn aborting_while_backing_off_ends_the_request_without_another_attempt() { + let up = upstream(Vec::new(), overloaded_503()).await; + // 退避两秒 + let (gw, state, log) = serve_with( + anthropic(&[&up], 10), + Duration::from_secs(1), + Duration::from_secs(10), + ) + .await; + let r = tokio::spawn(ask(gw, "退避时中止").send()); + let id = log.id("退避时中止").await; + log.until("退避", |evs| { + evs.iter() + .any(|e| matches!(e, Event::RequestRetrying { id: i, .. } if *i == id)) + .then_some(()) + }) + .await; + let t = Instant::now(); + state.aborts.request(id).expect("还在跑"); + let r = r.await.unwrap().unwrap(); + assert_eq!(r.status(), 499); + assert!(t.elapsed() < Duration::from_secs(1), "不等退避完"); + assert_eq!(up.hits(), 1); + let chain = log.attempts("退避时中止").await; + assert_eq!(chain.len(), 1, "尝试链上不多一跳：{chain:#?}"); + let waits = log.waits("退避时中止").await; + assert!( + matches!( + waits.as_slice(), + [Event::RequestRetrying { .. }, Event::RequestDequeued { .. }] + ), + "{waits:#?}" + ); +} + +#[tokio::test] +async fn when_both_upstreams_are_overloaded_both_are_tried_again() { + let a = upstream(Vec::new(), overloaded_503()).await; + let b = upstream(vec![overloaded_503()], ok()).await; + let (gw, _state, log) = serve(anthropic(&[&a, &b], 10)).await; + let r = ask(gw, "两家都过载").send().await.unwrap(); + assert_eq!(r.status(), 200); + assert_eq!((a.hits(), b.hits()), (2, 2)); + let chain = log.attempts("两家都过载").await; + let seen: Vec<(&str, Option, bool)> = chain + .iter() + .map(|a| (a.provider.as_str(), a.retry, a.backoff_ms.is_some())) + .collect(); + // 退避只在一轮的头一跳上 + assert_eq!( + seen, + [ + ("甲", None, false), + ("乙", None, false), + ("甲", Some(1), true), + ("乙", Some(1), false) + ] + ); + let waits = log.waits("两家都过载").await; + assert!( + matches!( + &waits[0], + Event::RequestRetrying { upstreams, .. } if upstreams == &["甲".to_string(), "乙".to_string()] + ), + "{waits:#?}" + ); +} + +/// 这段对话留在乙那儿（缓存在那边）：重试只再试乙，甲出的错也是暂时的，照样不再试它 +#[tokio::test] +async fn a_conversation_that_stays_on_an_upstream_retries_only_that_one() { + let a = upstream(vec![overloaded_503()], overloaded_503()).await; + let b = upstream(vec![ok(), overloaded_503(), overloaded_503()], ok()).await; + let (gw, _state, log) = serve(anthropic(&[&a, &b], 10)).await; + // 第一轮：甲过载，乙答了、读了缓存 + let r = ask_in(gw, "粘住", false, Some("会话一"), 1) + .send() + .await + .unwrap(); + assert_eq!(r.status(), 200); + // 等第一轮的结局落定，第二轮才认得出它留在哪一家 + tokio::time::sleep(Duration::from_millis(100)).await; + // 第二轮：乙在头上。乙、甲都过载，之后只再试乙 + let r = ask_in(gw, "粘住", false, Some("会话一"), 2) + .send() + .await + .unwrap(); + assert_eq!(r.status(), 200); + assert_eq!(a.hits(), 2, "甲只在两轮里各试了一次"); + assert_eq!(b.hits(), 4); + let events = log.0.lock().unwrap().clone(); + let rounds: Vec> = events + .iter() + .filter_map(|e| match e { + Event::RequestRetrying { upstreams, .. } => Some(upstreams.clone()), + _ => None, + }) + .collect(); + assert_eq!(rounds, [vec!["乙".to_string()], vec!["乙".to_string()]]); +} + +/// 压过了时限：客户端先拿到 `200` 和保活，重试到期限都没成，流里按它的格式报最后那个错误 +#[tokio::test] +async fn past_the_hold_the_retries_go_on_under_keepalives_and_the_last_error_ends_the_stream() { + let up = upstream(Vec::new(), overloaded_in_the_stream()).await; + let (gw, _state, log) = + serve_with(anthropic(&[&up], 2), TICK, Duration::from_millis(300)).await; + let t = Instant::now(); + let r = ask_stream(gw, "压过时限还过载").send().await.unwrap(); + let headers_at = t.elapsed(); + assert_eq!(r.status(), 200); + assert!( + headers_at >= Duration::from_millis(300) && headers_at < Duration::from_millis(1000), + "{headers_at:?}" + ); + let body = r.text().await.unwrap(); + assert!(body.starts_with(": keep-alive\n\n"), "{body}"); + assert!(body.matches(": keep-alive").count() >= 3, "{body}"); + let tail = &body[body.find("event: error").expect("流里报了错")..]; + assert!(tail.contains("overloaded_error"), "{tail}"); + assert!( + !body.contains("message_start"), + "被放弃的开头一个字节都没交出去：{body}" + ); + // 退避 200、400、800 毫秒之后剩下的不够 1.6 秒 + assert_eq!(up.hits(), 4); + let chain = log.attempts("压过时限还过载").await; + assert_eq!(chain.len(), 4); +} + +#[tokio::test] +async fn past_the_hold_a_retry_that_answers_continues_the_stream() { + let up = upstream( + vec![overloaded_in_the_stream(), overloaded_in_the_stream()], + streamed(), + ) + .await; + let (gw, _state, _log) = serve_with( + anthropic(&[&up], 10), + Duration::from_millis(200), + Duration::from_millis(300), + ) + .await; + let r = ask_stream(gw, "压过时限后答了").send().await.unwrap(); + assert_eq!(r.status(), 200); + let body = r.text().await.unwrap(); + assert!(body.starts_with(": keep-alive\n\n"), "{body}"); + assert!(body.contains("\"ok\""), "{body}"); + assert!(!body.contains("overloaded_error"), "{body}"); + assert_eq!(body.matches("event: message_start").count(), 1, "{body}"); + assert_eq!(up.hits(), 3); +} diff --git a/crates/tw-gateway/tests/sealed.rs b/crates/tw-gateway/tests/sealed.rs index 9c0bc194..909a86b0 100644 --- a/crates/tw-gateway/tests/sealed.rs +++ b/crates/tw-gateway/tests/sealed.rs @@ -208,25 +208,39 @@ fn served_by(o: &Outcome) -> Vec<(&str, AttemptOutcome)> { #[tokio::test] async fn sealed_agent_messages_go_back_to_the_account_that_sealed_them() { let (a, b) = (account("甲", false).await, account("乙", false).await); - let state = tw_gateway::AppState::new(cfg(&[("甲", &a), ("乙", &b)], "甲")).unwrap(); + // 一个请求最多等一秒；退避的一秒是一百毫秒，不浮动 + let waiting = |to: &str| { + let mut c = cfg(&[("甲", &a), ("乙", &b)], to); + c.failover.wait_secs = 1; + c + }; + let mut state = tw_gateway::AppState::new(waiting("甲")).unwrap(); + state.retry_backoff = tw_gateway::retry::Backoff { + tick: Duration::from_millis(100), + jitter: 0.0, + }; let (gw, mut rx) = serve(state.clone()).await; let o = ask(gw, &mut rx, body(1, "甲", false)).await; assert_eq!(served_by(&o), [("甲", AttemptOutcome::Served)]); // 改了配置：规则现在送去乙。带着甲封存的、子代理交回来的话的这段对话照旧回甲，乙不做备选 - state.reload(cfg(&[("甲", &a), ("乙", &b)], "乙")).unwrap(); + state.reload(waiting("乙")).unwrap(); let o = ask(gw, &mut rx, body(2, "甲", true)).await; assert_eq!(o.status, 200); assert_eq!(served_by(&o), [("甲", AttemptOutcome::Served)]); assert_eq!(o.stayed, Some(Stay::Sealed)); assert_eq!(b.hits.load(Ordering::SeqCst), 0); - // 甲发不出去：报甲的错，不转给解不开的乙 + // 甲发不出去：退避着重试的也只是甲，到期限了报甲的错，不转给解不开的乙 a.down.store(true, Ordering::SeqCst); let o = ask(gw, &mut rx, body(3, "甲", true)).await; assert_ne!(o.status, 200); - assert_eq!(o.attempts.len(), 1, "{:#?}", o.attempts); - assert_eq!(o.attempts[0].provider, "甲"); + assert!(o.attempts.len() > 1, "{:#?}", o.attempts); + assert!( + o.attempts.iter().all(|a| a.provider == "甲"), + "{:#?}", + o.attempts + ); assert_eq!(b.hits.load(Ordering::SeqCst), 0); // 新的对话照新的规则走 diff --git a/crates/tw-gateway/tests/traffic.rs b/crates/tw-gateway/tests/traffic.rs index 110cc4dc..def9df98 100644 --- a/crates/tw-gateway/tests/traffic.rs +++ b/crates/tw-gateway/tests/traffic.rs @@ -60,6 +60,7 @@ async fn gateway(cfg: Config) -> (SocketAddr, tokio::sync::broadcast::Receiver) -> Config { Config { version: 1, @@ -69,6 +70,10 @@ fn cfg(providers: Vec) -> Config { ..Default::default() }], providers, + failover: tw_config::Failover { + wait_secs: 0, + ..Default::default() + }, ..Default::default() } } diff --git a/crates/tw-observe/src/bus.rs b/crates/tw-observe/src/bus.rs index 690667ff..1955b049 100644 --- a/crates/tw-observe/src/bus.rs +++ b/crates/tw-observe/src/bus.rs @@ -102,7 +102,9 @@ fn about_the_request(ev: &tw_api::Event) -> bool { | E::ToolCallFlagged { .. } // 在排队：半路才来的一方要看得出它在等哪几家，和它是不是已经不排了 | E::RequestQueued { .. } - | E::RequestDequeued { .. } => true, + | E::RequestDequeued { .. } + // 在退避、之后重试：同上 + | E::RequestRetrying { .. } => true, // 开始和三种结局由 `track_at` 自己管 E::RequestStarted { .. } | E::RequestFinished { .. } @@ -508,6 +510,8 @@ mod tests { queued_ms: None, skipped: None, pause_ms: None, + retry: None, + backoff_ms: None, proxy: None, }, tw_api::AttemptView { @@ -521,6 +525,8 @@ mod tests { queued_ms: None, skipped: None, pause_ms: None, + retry: None, + backoff_ms: None, proxy: None, }, ], diff --git a/crates/tw-store/src/db.rs b/crates/tw-store/src/db.rs index e70539b6..683fa0eb 100644 --- a/crates/tw-store/src/db.rs +++ b/crates/tw-store/src/db.rs @@ -3637,6 +3637,8 @@ mod cost_state_tests { queued_ms: None, skipped: None, pause_ms: None, + retry: None, + backoff_ms: None, proxy: None, }], ..Default::default() @@ -4147,6 +4149,8 @@ mod cost { queued_ms: None, skipped: None, pause_ms: None, + retry: None, + backoff_ms: None, proxy: None, } }) diff --git a/crates/tw-store/src/recorder.rs b/crates/tw-store/src/recorder.rs index dac196c8..8dbbdfb6 100644 --- a/crates/tw-store/src/recorder.rs +++ b/crates/tw-store/src/recorder.rs @@ -697,6 +697,8 @@ impl Recorder { // 一个请求开始排队：它等了多久在尝试链（`queued_ms`）里，这一条不用落库 | Event::RequestQueued { .. } | Event::RequestDequeued { .. } + // 退避重试：退避了多久在尝试链（`backoff_ms`）里 + | Event::RequestRetrying { .. } | Event::ModelsChanged { .. } | Event::ProxyChanged { .. } | Event::AuthChanged { .. } @@ -991,6 +993,8 @@ mod tests { queued_ms: None, skipped: None, pause_ms: None, + retry: None, + backoff_ms: None, proxy: None, }, tw_api::AttemptView { @@ -1004,6 +1008,8 @@ mod tests { queued_ms: None, skipped: None, pause_ms: None, + retry: None, + backoff_ms: None, proxy: Some("机场".into()), }, ], @@ -1070,6 +1076,8 @@ mod tests { queued_ms: None, skipped: None, pause_ms: None, + retry: None, + backoff_ms: None, proxy: None, }], billing: tw_api::Billing::PerToken, @@ -1105,6 +1113,8 @@ mod tests { queued_ms: None, skipped: None, pause_ms: None, + retry: None, + backoff_ms: None, proxy: None, }, tw_api::AttemptView { @@ -1118,6 +1128,8 @@ mod tests { queued_ms: None, skipped: None, pause_ms: None, + retry: None, + backoff_ms: None, proxy: None, }, ], @@ -1172,6 +1184,8 @@ mod tests { queued_ms: None, skipped: None, pause_ms: None, + retry: None, + backoff_ms: None, proxy: None, }], billing: tw_api::Billing::PerToken, @@ -1301,6 +1315,8 @@ mod tests { queued_ms: None, skipped: None, pause_ms: None, + retry: None, + backoff_ms: None, proxy: None, }], billing: tw_api::Billing::PerToken, @@ -1628,6 +1644,8 @@ mod tests { queued_ms: None, skipped: None, pause_ms: None, + retry: None, + backoff_ms: None, proxy: None, }], billing: tw_api::Billing::Free, @@ -1671,6 +1689,8 @@ mod tests { queued_ms: None, skipped: None, pause_ms: None, + retry: None, + backoff_ms: None, proxy: None, }], billing: tw_api::Billing::Free, @@ -1753,6 +1773,8 @@ mod billing_tests { queued_ms: None, skipped: None, pause_ms: None, + retry: None, + backoff_ms: None, proxy: None, }], billing, @@ -1846,6 +1868,8 @@ mod billing_tests { queued_ms: None, skipped: None, pause_ms: None, + retry: None, + backoff_ms: None, proxy: None, }], billing, @@ -2293,6 +2317,8 @@ mod security_detail_tests { queued_ms: None, skipped: None, pause_ms: None, + retry: None, + backoff_ms: None, proxy: None, }], billing: tw_api::Billing::PerToken, @@ -2589,6 +2615,8 @@ mod translation_tests { queued_ms: None, skipped: None, pause_ms: None, + retry: None, + backoff_ms: None, proxy: None, }) .collect(), @@ -3282,6 +3310,8 @@ mod settle_hook_tests { queued_ms: None, skipped: None, pause_ms: None, + retry: None, + backoff_ms: None, proxy: None, }], billing: tw_api::Billing::PerToken, diff --git a/docs/config.md b/docs/config.md index 95371c77..c1131bd7 100644 --- a/docs/config.md +++ b/docs/config.md @@ -1134,6 +1134,21 @@ Before the first content of a streamed answer reaches the client, an error the upstream sends in the stream moves the request to the next candidate, the same as an error status would. +When every candidate has been tried and the last one failed temporarily +before any content reached the client — overloaded (503, 529, or an overload +error in the stream), another 5xx, unreachable, or its stream broke — the +request is retried after a pause instead of passing the error on. The pause +is 2, 4 and 8 seconds, then 16 seconds each time, varied by up to a fifth, or +as long as the upstream says (`retry-after-ms`, `Retry-After`). Each retry +goes again to the candidates that failed temporarily, in order, skipping those +set aside unless all of them are; a conversation that stays on an upstream +retries only that one. Retrying stops when the next pause would end after +`wait_secs`, the time a request may wait in all; the client then gets the last +error. Rate limits, used-up quotas, an insufficient balance, rejected +credentials, missing models, errors in the request itself and idle timeouts +are not retried. Each retry appears among the request's attempts with its +number and the pause before it. `wait_secs: 0` turns retrying off. + An upstream can also go quiet: it accepts the request and then sends nothing, or stops partway through. After `idle_timeout_secs` (300 by default) without content, the gateway stops waiting for it. The time counts from the moment the @@ -1149,9 +1164,10 @@ to the complete answer. outcome `idle_timeout`, and the request moves to the next candidate. Until then an upstream's answer is held back from the client, so the next upstream starts it afresh. With no candidate left, the client gets a timeout error - (504) in its own format. The last candidate's stream is passed on as it - arrives, so once its response has started, a timeout there ends it with an - error event instead. + (504) in its own format. The last candidate's answer is held the same way + while the request can still be retried within `wait_secs`; otherwise its + stream is passed on as it arrives, and once its response has started, a + timeout there ends it with an error event instead. - A streamed answer is held for at most 15 seconds. If no content has arrived by then, the client receives `200` and the streaming headers, so that its own wait for headers does not run out, followed by an SSE comment @@ -1183,9 +1199,9 @@ pause is at most `rate_limit_max_pause_secs`, and the circuit breaker and `load-balance` do not count it as a failure. The request that got the 429 moves to the next candidate, or waits for the pause to end; the client sees nothing of it. A streamed answer that reports a rate limit before any content -counts as a 429 too; only on a request's last candidate that has not been rate -limited in the last five minutes does the client get that error (the upstream -is paused all the same). When the pause ends, one request goes first, and the +counts as a 429 too; only on a request's last candidate, once the request can +no longer wait (`wait_secs`), does the client get that error (the upstream is +paused all the same). When the pause ends, one request goes first, and the others follow once the upstream has accepted it (for a streamed answer, once its content starts), or at once if that request gets no answer, times out or finds the upstream's quota used up. A pause lasts until twcore restarts at @@ -1194,10 +1210,10 @@ most. A request waits for upstreams that cannot take it yet for at most `wait_secs` in all: an upstream at its `max_concurrent`, one paused after a 429, and one at its `minute` or `hour` usage limit. The same time also covers waiting for -a key's `minute` or `hour` limit, so a request never waits longer than -`wait_secs` for all of them together. An ongoing conversation waits for the -upstream it stays on and, if it does not become available in time, moves on -to the next one, where its cache starts over. A new conversation skips such an +a key's `minute` or `hour` limit and the pauses before retries, so a request +never waits longer than `wait_secs` for all of them together. An ongoing +conversation waits for the upstream it stays on and, if it does not become +available in time, moves on to the next one, where its cache starts over. A new conversation skips such an upstream at once. An upstream that has used up a day, week or month usage limit is skipped. When no candidate is left, the request waits for whichever becomes available first; streamed answers keep the client's connection open @@ -1219,7 +1235,7 @@ receive the request and failed, the client gets that failure instead. | `quota_pause_secs` | integer | `3600` | Seconds to set aside an upstream whose quota is used up when it does not say when the quota resets. When it does, the upstream is set aside until then. | | `rate_limit_max_pause_secs` | integer | `3600` | Longest pause, in seconds, after an upstream answers 429 (rate limited). The pause lasts as long as the upstream says (`retry-after-ms`, `Retry-After`, or a delay in the error body); when it says nothing, it backs off by the upstream's consecutive 429s: 5, 10, 20 and 40 seconds, then 60 seconds each time. A pause is not a failure: requests wait for the upstream within `wait_secs`, and the circuit breaker does not count it. | | `idle_timeout_secs` | integer | `300` | Seconds an upstream may go without sending content before the gateway stops waiting for it. Counted from the moment the request is sent and started again by every piece of content: text, reasoning and tool calls count, keep-alives do not. A whole (non-streamed) answer counts from sending to the complete answer. Before any content has reached the client, the upstream counts as failed and the request moves to the next one; with none left, the client gets a timeout error. After content has reached the client, the answer ends with an error. From 30 to 3600. | -| `wait_secs` | integer | `120` | Seconds a request waits in all, counted once the key's own `max_concurrent` lets it in: for a key's `minute` or `hour` limit to free up, and for an upstream that cannot take it yet — at its `max_concurrent`, paused after answering 429, or at one of its `minute` or `hour` usage limits. A key limit that does not free up in time refuses the request. An upstream that cannot take it is skipped for the next one, and when none is left the request waits for the first to become available; a conversation that stays on an upstream waits for it. When nothing became available in time, the client gets 429 with `Retry-After`. `0`: never wait. From 0 to 600. | +| `wait_secs` | integer | `120` | Seconds a request waits in all, counted once the key's own `max_concurrent` lets it in: for a key's `minute` or `hour` limit to free up, for an upstream that cannot take it yet — at its `max_concurrent`, paused after answering 429, or at one of its `minute` or `hour` usage limits — and before retrying upstreams that failed temporarily before any content (overloaded, another 5xx, unreachable). A key limit that does not free up in time refuses the request. An upstream that cannot take it is skipped for the next one, and when none is left the request waits for the first to become available; a conversation that stays on an upstream waits for it. When nothing became available in time, the client gets 429 with `Retry-After`; when retries did not succeed in time, it gets the last upstream error. `0`: never wait or retry. From 0 to 600. | ### `aliases` diff --git a/docs/config.zh-CN.md b/docs/config.zh-CN.md index 6579a094..ecbd1bf5 100644 --- a/docs/config.zh-CN.md +++ b/docs/config.zh-CN.md @@ -852,6 +852,15 @@ security: 流式回答在第一段内容交给客户端之前，上游在流里报的错误和错误状态码一样，会把 请求换到下一个候选。 +候选都试过之后，最后一个候选在内容交给客户端之前暂时出错的 —— 过载（503、529 或流里的 +过载错误）、其他 5xx、连不上、流断了 —— 请求不把错误交给客户端，而是等待一段时间后重试。 +等待 2、4、8 秒，之后每次 16 秒，上下浮动不超过两成；上游给出了等待时间的（`retry-after-ms`、 +`Retry-After`）按它给的。每次重试再发给暂时出错的那几个候选，按原来的顺序，停用中的跳过， +都在停用中时照样重试；留在某一家的对话只重试这一家。下一次等待会超出 `wait_secs`（一个请求 +合计最多等的时间）时不再重试，客户端收到最后一次的错误。限流、额度用完、余额不足、凭据被拒、 +模型不存在、请求本身的错误和无响应超时都不重试。每次重试在请求的尝试链上单列一跳，带着它是 +第几次重试和之前等了多久。`wait_secs: 0` 关闭重试。 + 上游也可能不出声：收下请求之后一直不发内容，或者发到一半停住。连续 `idle_timeout_secs` 秒（默认 300）没有内容，网关就不再等它。计时从请求发给这家上游的 那一刻起，等空位的时间不算在内；每来一段内容重新计时：正文、推理、工具调用都算，心跳 @@ -861,7 +870,8 @@ security: - 还没有内容交给客户端时，这家上游记一次失败（和 5xx 一样计入 `failures_to_pause`）， 尝试链上这一跳的结果是 `idle_timeout`，请求换到下一个候选。在此之前上游的回答不交给 客户端，下一家从头开始回答。没有候选了，客户端收到它自己格式的超时错误（504）。 - 最后一个候选的流是边收边交给客户端的，它的响应开始之后再超时，回答以一条错误事件收尾。 + 请求还能在 `wait_secs` 之内重试时，最后一个候选的回答同样暂存；否则它的流边收边交给 + 客户端，响应开始之后再超时，回答以一条错误事件收尾。 - 流式回答最多暂存 15 秒。到时还没有内容，客户端先收到 `200` 和流式响应头，以免它自己 等响应头的时限先到，之后每 15 秒收到一行 SSE 注释（`: keep-alive`）；Gemini 的客户端 不发注释。上游的事件照旧暂存到第一段内容，故障转移在同一个 `200` 之下照常进行：下一家 @@ -882,14 +892,14 @@ failover: 暂停 5、10、20、40 秒，之后每次 60 秒；上游回了别的答复就重新计数。暂停最长 `rate_limit_max_pause_secs` 秒，熔断和 `load-balance` 都不把它算作失败。收到 429 的 请求换下一个候选，或者等暂停结束，客户端察觉不到。流式回答在内容之前报的限流也按 429 -处理；只有请求的最后一个候选、而且这家五分钟内没被限流过时，客户端才会收到这个错误（这家照样 -暂停）。暂停结束时先放一个请求过去，这家接受了它（流式回答要等到内容开始），其余的请求再 +处理；只有请求的最后一个候选、而且这个请求已经不能再等（`wait_secs`）时，客户端才会收到这个 +错误（这家照样暂停）。暂停结束时先放一个请求过去，这家接受了它（流式回答要等到内容开始），其余的请求再 跟上；这个请求没有得到答复、超时，或者遇到这家额度用完，其余的请求也照常发出。暂停只保留 到 twcore 重启为止。 一个请求等暂时不能发的上游合计最多 `wait_secs` 秒：并发数满了（`max_concurrent`）的、 回了 429 正在暂停的、到了 `minute` 或 `hour` 用量上限的都算。等密钥的 `minute`、`hour` -上限也算在这段时间里，几样加起来不超过 `wait_secs`。进行中的对话等它留在的那一家，到时 +上限和重试之前的等待也算在这段时间里，几样加起来不超过 `wait_secs`。进行中的对话等它留在的那一家，到时 还没有恢复就换下一家，缓存在那边从头建；新的对话遇到这样的上游直接跳过。用完了天、周、 月用量上限的上游也跳过。没有候选了，请求等最先恢复的那一家；流式回答照上文先收到 `200` 和保活，连接不会因此断开。都没有恢复，客户端收到 429，`Retry-After` 和 `retry-after-ms` @@ -908,7 +918,7 @@ failover: | `quota_pause_secs` | 整数 | `3600` | 上游报告额度用完、但没有给出重置时间时停用的秒数。给出了重置时间的，停用到那一刻。 | | `rate_limit_max_pause_secs` | 整数 | `3600` | 上游回 429（限流）后暂停这家的最长秒数。上游给出了等待时间的（`retry-after-ms`、`Retry-After` 或错误正文里的等待时间）按它给的暂停；没有给出的，按这家连续被限流的次数退避：5、10、20、40 秒，之后每次 60 秒。暂停不算失败：请求在 `wait_secs` 之内等这家恢复，熔断也不计入。 | | `idle_timeout_secs` | 整数 | `300` | 上游多少秒没有发出内容，网关就不再等它。从请求发出的那一刻算起，每来一段内容重新计时：正文、推理、工具调用都算，心跳不算。整包（非流式）的回答从发出算到整份回来。还没有内容交给客户端时，这家上游记一次失败，请求换到下一家；没有下一家了，客户端收到超时错误。已经有内容交给客户端的，回答以一条错误收尾。取值 30 到 3600。 | -| `wait_secs` | 整数 | `120` | 一个请求合计最多等的秒数，从过了密钥自己的 `max_concurrent` 时算起：等密钥的 `minute`、`hour` 上限空出名额，和等暂时不能发的上游 —— 并发数满了（`max_concurrent`）、回了 429 正在暂停、到了这家 `minute` 或 `hour` 的用量上限 —— 都算在里面。密钥的上限到时空不出来就拒绝。上游暂时不能发的先换下一家，没有下一家了就等最先恢复的那一家；留在某一家的对话等这一家。到时都没有恢复，客户端收到带 `Retry-After` 的 429。`0`：不等。取值 0 到 600。 | +| `wait_secs` | 整数 | `120` | 一个请求合计最多等的秒数，从过了密钥自己的 `max_concurrent` 时算起：等密钥的 `minute`、`hour` 上限空出名额，等暂时不能发的上游 —— 并发数满了（`max_concurrent`）、回了 429 正在暂停、到了这家 `minute` 或 `hour` 的用量上限 —— 和上游在内容之前暂时出错（过载、其他 5xx、连不上）后重试之前的等待，都算在里面。密钥的上限到时空不出来就拒绝。上游暂时不能发的先换下一家，没有下一家了就等最先恢复的那一家；留在某一家的对话等这一家。到时都没有恢复，客户端收到带 `Retry-After` 的 429；到时重试仍未成功，客户端收到最后一次的上游错误。`0`：不等，也不重试。取值 0 到 600。 | ### `aliases` From 4ab031a8359bf144588f10b5da2289be35fa6251 Mon Sep 17 00:00:00 2001 From: fylorn <249551762+fylorn@users.noreply.github.com> Date: Sat, 24 Oct 2026 23:17:06 +0800 Subject: [PATCH 2/3] Tests: start the unreachable upstream only after the first retry is scheduled Refused connections take a while to fail on Windows, so the test no longer guesses when the first attempt fails. Co-Authored-By: Claude Opus 5.5 --- crates/tw-gateway/tests/retry.rs | 37 ++++++++++++++++++-------------- 1 file changed, 21 insertions(+), 16 deletions(-) diff --git a/crates/tw-gateway/tests/retry.rs b/crates/tw-gateway/tests/retry.rs index 583b1fc1..cec9b7a8 100644 --- a/crates/tw-gateway/tests/retry.rs +++ b/crates/tw-gateway/tests/retry.rs @@ -488,26 +488,31 @@ async fn an_immediate_503_is_retried_until_it_answers() { async fn an_upstream_that_cannot_be_reached_yet_is_retried() { let port = common::spare_port(); let addr: SocketAddr = ([127, 0, 0, 1], port).into(); - let hits = Arc::new(Mutex::new(Vec::new())); - let later = hits.clone(); - // 一开始没人听这个端口，过一阵才起来 - tokio::spawn(async move { - tokio::time::sleep(Duration::from_millis(450)).await; - let l = tokio::net::TcpListener::bind(addr).await.unwrap(); - axum::serve(l, app(Vec::new(), ok(), later)).await.unwrap(); - }); let (gw, _state, log) = serve(config(&[(addr, "anthropic")], 10)).await; - let r = ask(gw, "连不上").send().await.unwrap(); + let r = tokio::spawn(ask(gw, "连不上").send()); + // 一开始没人听这个端口：连不上、开始退避之后才起来（Windows 上被拒的连接要重试一阵才报错， + // 不能按时间猜它什么时候失败） + let id = log.id("连不上").await; + log.until("退避", |evs| { + evs.iter() + .any(|e| matches!(e, Event::RequestRetrying { id: i, .. } if *i == id)) + .then_some(()) + }) + .await; + let hits = Arc::new(Mutex::new(Vec::new())); + let l = tokio::net::TcpListener::bind(addr).await.unwrap(); + let up = app(Vec::new(), ok(), hits.clone()); + tokio::spawn(async move { axum::serve(l, up).await.unwrap() }); + let r = r.await.unwrap().unwrap(); assert_eq!(r.status(), 200); assert_eq!(hits.lock().unwrap().len(), 1); let chain = log.attempts("连不上").await; - assert!(chain.len() >= 3, "{chain:#?}"); - assert_eq!(chain.last().unwrap().outcome, AttemptOutcome::Served); - assert!( - chain[..chain.len() - 1] - .iter() - .all(|a| a.outcome == AttemptOutcome::Error), - "{chain:#?}" + assert_eq!( + shape(&chain), + [ + (AttemptOutcome::Error, None, false), + (AttemptOutcome::Served, Some(1), true) + ] ); } From 22cf8e289f46e6f918e4a54f2485194f50d319c7 Mon Sep 17 00:00:00 2001 From: fylorn <249551762+fylorn@users.noreply.github.com> Date: Sun, 25 Oct 2026 00:23:18 +0800 Subject: [PATCH 3/3] Retry: narrow what is retried, floor the hint, keep the last whole answer unheld - The upstream's retry-after-ms / Retry-After can only lengthen the backoff, never shorten it. - Same-upstream retry only for 500, 502, 503, 504 and 529, and not when the error body names a request error. Other 5xx still fail over as before. - In-stream errors are retried only for listed kinds (server_error, api_error, overloaded_error, server_is_overloaded, slow_down, Gemini UNAVAILABLE / INTERNAL and 5xx numeric codes, Bedrock serviceUnavailableException / internalServerException), in response.failed and in the top-level Responses error event alike. Unlisted codes still fail over as before. - The last candidate's opening is held again only for streamed answers: when the client receives an SSE stream (early head and keep-alives), or when the upstream may be rate limited (Throttles::wary, restored). Non-streamed answers from the last candidate are passed on as they come. - Before discarding the last candidate's own in-stream error, check that a retry can actually happen (time left for the next wait, pinned upstream); otherwise hand the upstream's response over unchanged. - A stream that ends cleanly after only opening events counts as broken before content (new code gw.upstream.stream_ended_early). - Hits in plugin-written content (SecretsFound, ContentMatched) are reported once per request, not once per attempt. - Log retried requests as retried rather than failed over. Co-Authored-By: Claude Opus 5.5 --- crates/tw-api/msg-codes.txt | 1 + crates/tw-api/src/lib.rs | 5 +- crates/tw-config/tests/manual/schema.rs | 4 +- crates/tw-gateway/src/guard.rs | 67 ++++++ crates/tw-gateway/src/retry.rs | 103 ++++++++- crates/tw-gateway/src/server/pipeline/hop.rs | 125 +++++++++-- crates/tw-gateway/src/server/pipeline/idle.rs | 2 +- .../tw-gateway/src/server/pipeline/opening.rs | 143 ++++++++++--- crates/tw-gateway/src/server/pipeline/plug.rs | 31 ++- .../tw-gateway/src/server/pipeline/relay.rs | 2 +- crates/tw-gateway/src/throttle.rs | 27 +++ crates/tw-gateway/tests/plugins_security.rs | 50 +++++ crates/tw-gateway/tests/retry.rs | 198 ++++++++++++++++-- docs/config.md | 28 +-- docs/config.zh-CN.md | 18 +- 15 files changed, 702 insertions(+), 102 deletions(-) diff --git a/crates/tw-api/msg-codes.txt b/crates/tw-api/msg-codes.txt index 0f7bd9f0..05055290 100644 --- a/crates/tw-api/msg-codes.txt +++ b/crates/tw-api/msg-codes.txt @@ -450,6 +450,7 @@ gw.upstream.sealed_elsewhere gw.upstream.sign_failed gw.upstream.status gw.upstream.status_message +gw.upstream.stream_ended_early gw.upstream.stream_error gw.upstream.stream_exception gw.upstream.stream_opening_error diff --git a/crates/tw-api/src/lib.rs b/crates/tw-api/src/lib.rs index 5760632c..5dea51e0 100644 --- a/crates/tw-api/src/lib.rs +++ b/crates/tw-api/src/lib.rs @@ -1071,7 +1071,9 @@ pub const MSG_CODES: &str = include_str!("../msg-codes.txt"); /// 连不上、流断了，请求在等待期限（`wait_secs`，等上游和重试共用）之内退避一阵，把暂时出错的 /// 那几家再试一轮。新事件 [`Event::RequestRetrying`]（开始退避：再试哪几家、第几次、什么时候 /// 发出去），退避结束发 [`Event::RequestDequeued`]，两条都在 `InFlightRequest::events` 里。 -/// [`AttemptView`] 多了 `retry`（这一跳是第几次重试）和 `backoff_ms`（发出之前退避了多久） +/// [`AttemptView`] 多了 `retry`（这一跳是第几次重试）和 `backoff_ms`（发出之前退避了多久）。 +/// 新码：`gw.upstream.stream_ended_early`（流只发了开头的例行事件就收了尾，算在 +/// [`FAILED_AFTER_SENDING`] 里） pub const CONTROL_API_VERSION: u32 = 53; /// `/live` 的速度看最近多久（见 [`LiveView::tokens_per_sec`]），毫秒 @@ -2024,6 +2026,7 @@ pub const FAILED_AFTER_SENDING: &[&str] = &[ "gw.upstream.forward_failed", "gw.upstream.body_broken", "gw.upstream.stream_opening_error", + "gw.upstream.stream_ended_early", ]; /// 手动中止的请求报的码（[`FailureSource::Aborted`] 的 `message`，尝试链上 `aborted` 那一跳的 diff --git a/crates/tw-config/tests/manual/schema.rs b/crates/tw-config/tests/manual/schema.rs index 06a23081..7e51af7f 100644 --- a/crates/tw-config/tests/manual/schema.rs +++ b/crates/tw-config/tests/manual/schema.rs @@ -1579,8 +1579,8 @@ pub fn sections() -> Vec
{ Kind::Int, Def::Is("120"), t( - "Seconds a request waits in all, counted once the key's own `max_concurrent` lets it in: for a key's `minute` or `hour` limit to free up, for an upstream that cannot take it yet — at its `max_concurrent`, paused after answering 429, or at one of its `minute` or `hour` usage limits — and before retrying upstreams that failed temporarily before any content (overloaded, another 5xx, unreachable). A key limit that does not free up in time refuses the request. An upstream that cannot take it is skipped for the next one, and when none is left the request waits for the first to become available; a conversation that stays on an upstream waits for it. When nothing became available in time, the client gets 429 with `Retry-After`; when retries did not succeed in time, it gets the last upstream error. `0`: never wait or retry. From 0 to 600.", - "一个请求合计最多等的秒数，从过了密钥自己的 `max_concurrent` 时算起：等密钥的 `minute`、`hour` 上限空出名额，等暂时不能发的上游 —— 并发数满了（`max_concurrent`）、回了 429 正在暂停、到了这家 `minute` 或 `hour` 的用量上限 —— 和上游在内容之前暂时出错（过载、其他 5xx、连不上）后重试之前的等待，都算在里面。密钥的上限到时空不出来就拒绝。上游暂时不能发的先换下一家，没有下一家了就等最先恢复的那一家；留在某一家的对话等这一家。到时都没有恢复，客户端收到带 `Retry-After` 的 429；到时重试仍未成功，客户端收到最后一次的上游错误。`0`：不等，也不重试。取值 0 到 600。", + "Seconds a request waits in all, counted once the key's own `max_concurrent` lets it in: for a key's `minute` or `hour` limit to free up, for an upstream that cannot take it yet — at its `max_concurrent`, paused after answering 429, or at one of its `minute` or `hour` usage limits — and before retrying upstreams that failed temporarily before any content (overloaded, a server error, unreachable). A key limit that does not free up in time refuses the request. An upstream that cannot take it is skipped for the next one, and when none is left the request waits for the first to become available; a conversation that stays on an upstream waits for it. When nothing became available in time, the client gets 429 with `Retry-After`; when retries did not succeed in time, it gets the last upstream error. `0`: never wait or retry. From 0 to 600.", + "一个请求合计最多等的秒数，从过了密钥自己的 `max_concurrent` 时算起：等密钥的 `minute`、`hour` 上限空出名额，等暂时不能发的上游 —— 并发数满了（`max_concurrent`）、回了 429 正在暂停、到了这家 `minute` 或 `hour` 的用量上限 —— 和上游在内容之前暂时出错（过载、服务端出错、连不上）后重试之前的等待，都算在里面。密钥的上限到时空不出来就拒绝。上游暂时不能发的先换下一家，没有下一家了就等最先恢复的那一家；留在某一家的对话等这一家。到时都没有恢复，客户端收到带 `Retry-After` 的 429；到时重试仍未成功，客户端收到最后一次的上游错误。`0`：不等，也不重试。取值 0 到 600。", ), ), ], diff --git a/crates/tw-gateway/src/guard.rs b/crates/tw-gateway/src/guard.rs index abad35ae..1af05693 100644 --- a/crates/tw-gateway/src/guard.rs +++ b/crates/tw-gateway/src/guard.rs @@ -182,6 +182,35 @@ pub fn more_found(before: &[Finding], after: Vec) -> Vec { .collect() } +/// 一个请求已经报过的命中。插件改过的请求每一跳都再查一遍内容过滤、再找一遍密钥（见 +/// `server::pipeline::plug`）：重试、故障转移时插件写进来的还是那几处，**一个请求只报一次** —— +/// 安全日志里一处一行，不因为多试了几跳多出几行 +#[derive(Default)] +pub struct Reported { + /// 出站脱敏报过的值：规则和打了码的那个值 + secrets: HashSet<(String, String)>, + /// 内容过滤报过的命中：规则和摘录 + content: HashSet<(String, String)>, +} + +impl Reported { + /// `found` 里这个请求还没报过的，记下来 + pub fn fresh_secrets(&mut self, found: Vec) -> Vec { + found + .into_iter() + .filter(|f| { + self.secrets + .insert((f.rule.id().to_string(), f.masked.clone())) + }) + .collect() + } + + /// 先前几跳已经报过几个值。一个请求加起来最多报 [`REPORTED_MAX`] 个 + pub fn secrets(&self) -> usize { + self.secrets.len() + } +} + /// 一个请求报出去的值最多几个。 /// /// 安全日志一个值一行（[`tw_api::Event::SecretsFound`] 的一项就是一行）。一个请求里贴进来 @@ -319,6 +348,44 @@ pub fn report_with( sc.refusal().map(|r| refusal(&r.hit)) } +/// [`report_with`]，这个请求先前几跳报过的命中不再报（见 [`Reported`]）。拒绝照旧按这一次查下来 +/// 的结论：拒绝的那一条先前报过的话，请求那时就被拒了，到不了这一跳 +pub fn report_once( + bus: &tw_observe::EventBus, + id: u64, + provider: &str, + sc: &Screening, + details: Vec, + reported: &mut Reported, +) -> Option { + if sc.hits.is_empty() { + return None; + } + let refused = sc.refusal().map(|r| refusal(&r.hit)); + let refusing = sc.refused; + let mut details = details.into_iter(); + let mut fresh = Screening { + body: sc.body.clone(), + ..Default::default() + }; + let mut kept = Vec::new(); + for (i, h) in sc.hits.iter().enumerate() { + let detail = details.next().unwrap_or_default(); + if reported + .content + .insert((h.hit.rule.clone(), h.hit.snippet.clone())) + { + if refusing == Some(i) { + fresh.refused = Some(fresh.hits.len()); + } + fresh.hits.push(h.clone()); + kept.push(detail); + } + } + report_with(bus, id, provider, &fresh, kept); + refused +} + /// 回答里的命中报出去：每条规则一条 [`tw_api::Event::ContentMatched`]（方向是回答），挂在请求 /// `id` 上。片段和细节按这个请求的脱敏规则和账本打码（[`detail::answer`]）。 pub fn report_answer( diff --git a/crates/tw-gateway/src/retry.rs b/crates/tw-gateway/src/retry.rs index 2af3dca0..196ff626 100644 --- a/crates/tw-gateway/src/retry.rs +++ b/crates/tw-gateway/src/retry.rs @@ -7,16 +7,19 @@ //! //! 几条规矩（做法见 `server::pipeline::hop`）： //! -//! - **只重试暂时的错**（[`transient`]），也就是说不出原因的失败。限流有自己的队（见 -//! [`crate::throttle`]）；额度用完、没钱了、凭据被拒、没有这个模型、请求本身的问题，再发一遍 -//! 也一样；没有内容超时的已经等过一整个无响应超时，不再等。 +//! - **只重试暂时的错**：状态码只认 500、502、503、504、529（[`transient_status`]），流里报的错 +//! 只认名单里的码（[`transient_kind`]），正文说是请求本身的问题的不算（[`request_fault`]）； +//! 连不上、流在内容之前断了也算。限流有自己的队（见 [`crate::throttle`]）；额度用完、没钱了、 +//! 凭据被拒、没有这个模型、请求本身的问题，再发一遍也一样；没有内容超时的已经等过一整个 +//! 无响应超时，不再等。别的 5xx（501、505、Cloudflare 的 52x……）照旧换下一家，只是不再试同一家 //! - **在这个请求的等待期限之内**（`failover.wait_secs`，和排队共用一段）：剩下的时间不够下一次 //! 退避就不等了，交出最后那个错误。`0` 是不重试。已经发出去的一跳不受期限限制。 //! - **把暂时出错的那几家再试一轮**，按候选的顺序，在冷却的跳过（都在冷却的照样试，和开头的 //! 熔断过滤一样）；留在一家的对话（会话粘性、封存的历史）只再试那一家。每一跳和故障转移的 //! 一跳一样：插件、内容过滤、脱敏、参数改写都照常跑，熔断的账照常记。 //! - **退避 2、4、8 秒，之后每次 16 秒**，上下浮动两成，免得一起失败的请求一起再来；上游说了 -//! 多久之后再来的（`retry-after-ms`、`Retry-After`）按它说的。 +//! 多久之后再来的（`retry-after-ms`、`Retry-After`）比这长就按它说的，**比这短的不算**：一家 +//! 回 `retry-after-ms: 5` 的中转不能让一个请求变成几千个。 use std::time::Duration; @@ -56,11 +59,60 @@ impl Backoff { } } +/// 第 `n` 次重试之前退避多久：`hint` 是上游说的多久之后再来，只能比退避长，不能比它短 +pub fn wait(backoff: Duration, hint: Option) -> Duration { + hint.map_or(backoff, |h| h.max(backoff)) +} + /// 这个原因是不是暂时的错：说不出原因的失败（5xx、过载）。别的见模块说明 pub fn transient(cause: Cause) -> bool { cause == Cause::Unexplained } +/// 这个状态码是不是暂时的错：服务端出错、网关出错、过载、网关超时。别的 5xx（没实现、版本 +/// 不支持、Cloudflare 的 52x）再发一遍多半也一样 +pub fn transient_status(status: u16) -> bool { + matches!(status, 500 | 502 | 503 | 504 | 529) +} + +/// 流里报的错，码或者类型是这几个的才是暂时的：OpenAI、Anthropic 的服务端出错和过载，Gemini +/// 的 `UNAVAILABLE`、`INTERNAL`，Bedrock 的两种服务端异常。**认不出的一律不算** —— 图片解不开、 +/// 内容被拦这类错同一家再发一遍也一样 +const TRANSIENT_KINDS: &[&str] = &[ + "server_error", + "api_error", + "overloaded_error", + "server_is_overloaded", + "slow_down", + "UNAVAILABLE", + "INTERNAL", + "serviceUnavailableException", + "internalServerException", +]; + +/// 上游在流里报的这个码（或类型）是不是暂时的错（见 [`TRANSIENT_KINDS`]） +pub fn transient_kind(kind: &str) -> bool { + TRANSIENT_KINDS.contains(&kind) +} + +/// 错误正文说的是请求本身的问题（`invalid_request_error` 这一类）：状态码是 5xx 也不重试 —— +/// 同一家再发一遍也一样 +pub fn request_fault(body: &[u8]) -> bool { + const FAULTS: &[&str] = &["invalid_request_error", "invalid_request", "bad_request"]; + let Ok(v) = serde_json::from_slice::(body) else { + return false; + }; + [ + "/error/type", + "/error/code", + "/response/error/type", + "/response/error/code", + ] + .iter() + .filter_map(|p| v.pointer(p).and_then(|t| t.as_str())) + .any(|t| FAULTS.contains(&t)) +} + #[cfg(test)] mod tests { use super::*; @@ -87,6 +139,49 @@ mod tests { } } + #[test] + fn a_hint_lengthens_the_backoff_and_never_shortens_it() { + let two = Duration::from_secs(2); + assert_eq!(wait(two, None), two); + assert_eq!(wait(two, Some(Duration::from_millis(5))), two); + assert_eq!( + wait(two, Some(Duration::from_secs(9))), + Duration::from_secs(9) + ); + } + + #[test] + fn only_some_statuses_and_kinds_are_temporary() { + for s in [500, 502, 503, 504, 529] { + assert!(transient_status(s), "{s}"); + } + for s in [501, 505, 520, 522, 524, 527, 599] { + assert!(!transient_status(s), "{s}"); + } + assert!(transient_kind("server_is_overloaded")); + assert!(transient_kind("UNAVAILABLE")); + for k in [ + "invalid_image", + "content_filter", + "invalid_encrypted_content", + "", + ] { + assert!(!transient_kind(k), "{k}"); + } + } + + #[test] + fn a_request_error_in_the_body_is_not_temporary() { + let refused = br#"{"type":"error","error":{"type":"invalid_request_error","code":"invalid_encrypted_content","message":"x"}}"#; + assert!(request_fault(refused)); + let failed = br#"{"type":"response.failed","response":{"error":{"code":"invalid_request","message":"x"}}}"#; + assert!(request_fault(failed)); + let overloaded = + br#"{"error":{"type":"service_unavailable_error","code":"server_is_overloaded"}}"#; + assert!(!request_fault(overloaded)); + assert!(!request_fault(b"bad gateway")); + } + #[test] fn only_unexplained_failures_are_retried() { assert!(transient(Cause::Unexplained)); diff --git a/crates/tw-gateway/src/server/pipeline/hop.rs b/crates/tw-gateway/src/server/pipeline/hop.rs index de11fb8e..2334e34c 100644 --- a/crates/tw-gateway/src/server/pipeline/hop.rs +++ b/crates/tw-gateway/src/server/pipeline/hop.rs @@ -52,7 +52,7 @@ pub(super) struct Served<'a> { pub(super) slot: crate::slots::Slot, /// 这一跳发出去的那一刻。这一家的快慢样本从这里算起（见 [`crate::latency`]） pub(super) sent_at: std::time::Instant, - /// 这一家答没答上还不知道：最后一家的开头没压（这个请求等不起了），或者压着时到点了还没有 + /// 这一家答没答上还不知道：最后一家的开头没压（整包的、等不起的），或者压着时到点了还没有 /// 内容、在开头报了限流以外的错，原样交了出去。成败的账等回程看到第一段内容才记（见 /// `relay`）。没有内容就超时了记失败，和压着开头时一样 pub(super) unconfirmed: bool, @@ -193,6 +193,12 @@ pub(super) async fn try_upstreams<'a>( // 退避重试的账（见 `crate::retry`）：最后一个失败是不是暂时的错、这一轮里暂时出错的是哪几家、 // 重试过的那几轮 let mut retries = Retries::default(); + // 客户端收的是 SSE 流：压着开头过了时限先交响应头、发保活（见 `super::commit`）。最后一家的 + // 开头只在这时为重试压着 + let early = super::commit::early(state, req, reading).is_some(); + // 插件写进来的、这个请求已经报过的命中：每一跳再查一遍，同样的只报一次（见 + // `crate::guard::Reported`） + let mut reported = crate::guard::Reported::default(); loop { stamp_queued(&mut chain, queued.take()); @@ -203,7 +209,7 @@ pub(super) async fn try_upstreams<'a>( else { break; }; - let n = retries.rounds.len() as u32 + 1; + let n = retries.next(); retrying(state, id, &round, n, wait, deadline_ms); let t = std::time::Instant::now(); let woke = tokio::select! { @@ -550,6 +556,7 @@ pub(super) async fn try_upstreams<'a>( provider, &sent, chain.len(), + &mut reported, ) .await { @@ -923,7 +930,12 @@ pub(super) async fn try_upstreams<'a>( &provider.name, state.health.record_cause(&provider.name, cause), ); - if crate::retry::transient(cause) { + // 同一家再试只认几个状态码（见 `crate::retry::transient_status`），正文说是 + // 请求本身的问题的不算 + if crate::retry::transient(cause) + && crate::retry::transient_status(status) + && !crate::retry::request_fault(&head) + { let hint = crate::throttle::pause_hint(&headers, &head, now_ms()); retries.shaky(name, hint); } @@ -995,12 +1007,20 @@ pub(super) async fn try_upstreams<'a>( // 回答的开头：**第一段内容到之前**上游在流里报的错照样换下一家（报的是限流就排队 // 等它，和回 429 一样），一直没有内容也换（见 `opening`）。 // - // 最后一家在这个请求还等得起时也压：开头报的暂时的错（过载、5xx）、流在内容之前断了， - // 退避一阵再试一遍（见 `crate::retry`）；报的限流暂停这一家、排队等它恢复。客户端都 - // 察觉不到 —— 不压的话那个错误直接交给了客户端。等不起的不压：没有下一家可换、也不 - // 能再试，客户端该当场拿到响应头和开头的事件，由回程按无响应超时收尾（见 `relay`） + // 最后一家只压流式的、这个请求又还等得起的，而且是两种时候：客户端收的是 SSE 流（过了 + // 时限先拿到响应头和保活，见 `super::commit`）—— 开头报的暂时的错（过载、5xx）、流在 + // 内容之前断了，退避一阵再试一遍（见 `crate::retry`）；或者这一家可能在限流（探路的、 + // 最近被限流过的，见 `throttle::Throttles::wary`）。流的开头报的限流暂停这一家、排队 + // 等它恢复。客户端都察觉不到 —— 不压的话那个错误直接交给了客户端。别的时候不压： + // 整包的回答压着的话，客户端到正文的第一个字节才拿到响应头，中转站的空格保活也被 + // 吞掉；等不起的没有下一家可换、也不能再试。不压的由回程按无响应超时收尾（见 `relay`） let hold = hold_of(req, reading, provider, &out, &r); - let held = hold.is_some() && (!last || tokio::time::Instant::now() < until); + let held = hold.as_ref().is_some_and(|h| { + !last + || (matches!(h, Hold::Stream(..)) + && tokio::time::Instant::now() < until + && (early || state.throttles.wary(&provider.name))) + }); // 不压开头的最后一家：它答没答上（成败的账）要等回程看到第一段内容才知道 let mut unconfirmed = last && hold.is_some() && !held; let r = match hold.filter(|_| held) { @@ -1084,6 +1104,7 @@ pub(super) async fn try_upstreams<'a>( body, kind, message, + transient, response, }) => { let verdict = @@ -1093,12 +1114,17 @@ pub(super) async fn try_upstreams<'a>( if !limited { to_throttle(state, &provider.name, sent_at, verdict); } - // 最后一家报的暂时的错，这个请求又还等得起：和换下一家时一样记下， - // 候选都试过了就退避重试（见 `crate::retry`）。说的是请求本身的问题的 - // 不算（见 `opening::faults_the_request`） - let shaky = matches!(verdict, Verdict::Failed(c) if crate::retry::transient(c)) - && !super::opening::faults_the_request(&body); - let retried = shaky && tokio::time::Instant::now() < until; + // 暂时的错（名单里的码，见 `opening::transient_in`）同一家可以再试。 + // 最后一家报的、又真的再试得了（等待期限够下一次退避、留在一家的 + // 对话留的就是能再试的那一家，同 `retry_round`）：和换下一家时一样 + // 记下，候选都试过了就退避重试（见 `crate::retry`）。再试不了的原样 + // 交出去，不丢上游自己的那句话 + let shaky = transient + && matches!(verdict, Verdict::Failed(c) if crate::retry::transient(c)); + let hint = crate::throttle::pause_hint(&headers, &body, now_ms()); + let retried = shaky + && retries + .feasible(state, started, name, hint, counting, until); if verdict == Verdict::ClientError || (last && !limited && !retried) { // 请求本身的问题，或者最后一家报了限流以外、不再重试的错：原样 @@ -1154,11 +1180,6 @@ pub(super) async fn try_upstreams<'a>( state.health.record_cause(&provider.name, cause), ); if shaky { - let hint = crate::throttle::pause_hint( - &headers, - &body, - now_ms(), - ); retries.shaky(name, hint); } chain.push(a); @@ -1396,7 +1417,14 @@ pub(super) async fn try_upstreams<'a>( // `source` 也是这个错误自己的（被限流的就是 `rate_limited`）。 return Err(err); }; - if attempts.len() > 1 { + if !retries.rounds.is_empty() { + tracing::info!( + chain = %attempts.join(" → "), + retries = retries.rounds.len(), + "retried after a temporary failure; {} answered in the end", + served.provider.name + ); + } else if attempts.len() > 1 { tracing::info!( chain = %attempts.join(" → "), "failed over; {} answered in the end", @@ -1461,6 +1489,9 @@ struct Retries<'n> { shaky: Vec<&'n String>, /// 重试过的每一轮：在尝试链上从哪一跳起、是第几次、发出去之前退避了多少毫秒 rounds: Vec<(usize, u32, u64)>, + /// 下一次重试之前按退避该等多久，算过一次就定了（浮动是随机的）：先看再试不试得了 + /// （[`Self::feasible`]）、再真的退避（[`retry_round`]），两处用的是同一个数 + drawn: Option, } impl<'n> Retries<'n> { @@ -1471,6 +1502,38 @@ impl<'n> Retries<'n> { self.shaky.push(name); } } + + /// 下一次是第几次重试，从 1 数 + fn next(&self) -> u32 { + self.rounds.len() as u32 + 1 + } + + /// 下一次重试之前按退避该等多久（见 [`Self::drawn`]） + fn backoff(&mut self, state: &AppState) -> std::time::Duration { + let n = self.next(); + *self + .drawn + .get_or_insert_with(|| state.retry_backoff.before(n)) + } + + /// 这一跳（最后一家）暂时出错的话，再试得了吗：和 [`retry_round`] 同一个判据，只是不动账。 + /// 再试不了的，上游自己的回答原样交出去 + fn feasible( + &mut self, + state: &AppState, + started: &'n Started, + name: &'n String, + hint: Option, + counting: bool, + until: tokio::time::Instant, + ) -> bool { + let mut shaky = self.shaky.clone(); + if !shaky.contains(&name) { + shaky.push(name); + } + let backoff = self.backoff(state); + plan(state, started, &shaky, backoff, hint, counting, until).is_some() + } } /// 候选都试过了，要不要退避重试（见 [`crate::retry`]）：要的话，交回这一轮再试哪几家、先退避 @@ -1486,11 +1549,29 @@ fn retry_round<'n>( ) -> Option<(Vec<&'n String>, std::time::Duration)> { let hint = retries.fault.take()?; let shaky = std::mem::take(&mut retries.shaky); + let backoff = retries.backoff(state); + let planned = plan(state, started, &shaky, backoff, hint, counting, until); + if planned.is_some() { + retries.drawn = None; + } + planned +} + +/// 暂时出错的是 `shaky` 这几家，这一次按退避该等 `backoff`、上游说的是 `hint`：再试哪几家、先 +/// 等多久（见 [`retry_round`]）。等多久取两者里长的（[`crate::retry::wait`]） +fn plan<'n>( + state: &AppState, + started: &'n Started, + shaky: &[&'n String], + backoff: std::time::Duration, + hint: Option, + counting: bool, + until: tokio::time::Instant, +) -> Option<(Vec<&'n String>, std::time::Duration)> { if counting { return None; } - let n = retries.rounds.len() as u32 + 1; - let wait = hint.unwrap_or_else(|| state.retry_backoff.before(n)); + let wait = crate::retry::wait(backoff, hint); if until.saturating_duration_since(tokio::time::Instant::now()) < wait { return None; } diff --git a/crates/tw-gateway/src/server/pipeline/idle.rs b/crates/tw-gateway/src/server/pipeline/idle.rs index 4274ade6..71cf82b1 100644 --- a/crates/tw-gateway/src/server/pipeline/idle.rs +++ b/crates/tw-gateway/src/server/pipeline/idle.rs @@ -13,7 +13,7 @@ //! 流式的回答压过 [`crate::OPENING_HOLD`] 之后响应头已经交出去了，它收到的只是保活注释（见 //! [`super::commit`]）。没有下一家了，回一个超时错误（504，见 //! [`crate::error::Source::Timeout`]），尝试链带着；响应头已经交出去的，在流里报这个错。 -//! - **最后一家的响应已经交出去了**（等不起了的最后一家不压开头）、或者**已经有内容交给了客户端**： +//! - **最后一家的响应已经交出去了**（最后一家不压开头时）、或者**已经有内容交给了客户端**： //! 换不了 —— 换一家会把开头再发一遍。回答按客户端的格式以一条错误收尾，请求记成失败 //! （见 `relay`）。还没有内容的照样记这一家一次失败；内容出到一半才停的不记，和流在半路 //! 断了一样。 diff --git a/crates/tw-gateway/src/server/pipeline/opening.rs b/crates/tw-gateway/src/server/pipeline/opening.rs index 580c4622..3a64e98e 100644 --- a/crates/tw-gateway/src/server/pipeline/opening.rs +++ b/crates/tw-gateway/src/server/pipeline/opening.rs @@ -11,10 +11,11 @@ //! 和无响应超时是同一个判据（见 [`crate::pulse`]）。 //! //! 等到无响应超时那一刻（`failover.idle_timeout_secs`，从请求发出去算起）还没有内容是 -//! [`Opening::Slow`]：调用方放弃这一家、换下一家。**最后一家在请求还等得起时也在这里等**（见 -//! `hop`）：开头报的暂时的错（过载、5xx）、流在内容之前断了，退避一阵再试（见 [`crate::retry`]）； -//! 报的限流暂停这一家、排队等它恢复；别的错、到点了还没有内容，没有下一家可换，照原样交给回程。 -//! 等不起的最后一家不等：压着它只会让客户端晚一点看到同样的东西。 +//! [`Opening::Slow`]：调用方放弃这一家、换下一家。**最后一家只在请求还等得起时等流式的**（客户端 +//! 收 SSE 流、或者这一家可能在限流，见 `hop`）：开头报的暂时的错（过载、5xx）、流在内容之前断了， +//! 退避一阵再试（见 [`crate::retry`]）；报的限流暂停这一家、排队等它恢复；别的错、到点了还没有 +//! 内容，没有下一家可换，照原样交给回程。别的时候最后一家不等：压着它只会让客户端晚一点看到 +//! 同样的东西。 //! 开头压得太多（[`HOLD_LIMIT`]）也不再压，交出去由回程照常计时。压着的这段时间里客户端等的是响应头：流式的回答过了 //! [`crate::OPENING_HOLD`] 先拿到 `200` 和保活注释，上游的事件照旧压在这里（见 //! [`super::commit`]）。 @@ -27,6 +28,7 @@ use futures::StreamExt; use crate::error::GatewayError; use tw_dialect::ir::Dialect; +use tw_types::msg; /// 开头最多压多少字节。前导事件都很小，攒到这么多还没有内容，就不是在等开头了 pub(super) const HOLD_LIMIT: usize = 1024 * 1024; @@ -46,16 +48,18 @@ pub(super) enum Opening { /// 上游的原话，交给 [`crate::failure::classify`] 判断换不换。 /// /// **读过的字节也还在 `response` 里**：判断下来是请求本身的问题（换一家也一样 - /// 被拒）时，照常把它交给客户端 + /// 被拒）时，照常把它交给客户端。`transient` 是这个错是不是暂时的（见 [`transient_in`]）： + /// 最后一家报的暂时的错退避重试（见 [`crate::retry`]） Failed { status: u16, headers: http::HeaderMap, body: Bytes, kind: String, message: String, + transient: bool, response: reqwest::Response, }, - /// 流在第一段内容之前断了 + /// 流在第一段内容之前断了，或者只发了开头的例行事件就好好地收了尾 Broken(GatewayError), } @@ -87,7 +91,8 @@ pub(super) async fn watch( } }; let chunk = match next { - None => break Judge::Content, + // 只有开头的例行事件就收了尾：一段内容都没有，和断了一样 + None => return Opening::Broken(ended()), Some(Ok(c)) => c, Some(Err(e)) => { return Opening::Broken(crate::forward::map_reqwest_error(e)); @@ -132,12 +137,14 @@ pub(super) async fn watch( body, kind, message, + transient, } => Opening::Failed { status, headers, body, kind, message, + transient, response, }, Judge::Content | Judge::Preamble if slow => Opening::Slow { @@ -203,6 +210,8 @@ enum Judge { body: Bytes, kind: String, message: String, + /// 暂时的错（见 [`transient_in`]） + transient: bool, }, } @@ -223,7 +232,10 @@ fn judge(dialect: Dialect, event: Option<&str>, data: &str) -> Judge { .map(str::to_string) .or_else(|| v.get("type").and_then(|t| t.as_str()).map(str::to_string)) .unwrap_or_default(); - if let Some(e) = error_in(dialect, &ty, &v, data) { + if let Some(mut e) = error_in(dialect, &ty, &v, data) { + if let Judge::Error { transient, .. } = &mut e { + *transient = transient_in(dialect, &ty, &v); + } return e; } if crate::pulse::advances(dialect, &ty, &v) { @@ -301,17 +313,44 @@ pub(super) fn stream_error(dialect: Dialect, data: &str) -> Option<(u16, Bytes)> } } -/// 流里报的这个错（[`Opening::Failed`] 的 `body`）说的是请求本身的问题：错误的类型是 -/// `invalid_request_error`。OpenAI 在流里报的码有的认不出（`invalid_encrypted_content`），按状态码 -/// 算是说不出原因的失败，照样换下一家；**可它不是暂时的错**，同一家再发一遍也一样（见 -/// [`crate::retry`]） -pub(super) fn faults_the_request(body: &[u8]) -> bool { - let Ok(v) = serde_json::from_slice::(body) else { - return false; +/// 上游在流里报的这个错是不是暂时的（过载、服务端出错）：码或者类型在名单里（见 +/// [`crate::retry::transient_kind`]），Chat、Gemini 的数字码是暂时的状态码（见 +/// [`crate::retry::transient_status`]）。**认不出的一律不算**：图片解不开、内容被拦这类错同一家 +/// 再发一遍也一样。只管同一家要不要再试，换不换下一家照旧看状态码（[`error_in`]） +fn transient_in(dialect: Dialect, ty: &str, v: &serde_json::Value) -> bool { + let named = |e: &serde_json::Value, keys: &[&str]| { + keys.iter() + .any(|k| crate::retry::transient_kind(&text_of(e, k))) + }; + let numbered = |e: &serde_json::Value| { + e.get("code") + .and_then(|c| c.as_u64()) + .and_then(|c| u16::try_from(c).ok()) + .is_some_and(crate::retry::transient_status) }; - ["/error/type", "/response/error/type"] - .iter() - .any(|p| v.pointer(p).and_then(|t| t.as_str()) == Some("invalid_request_error")) + match dialect { + Dialect::Anthropic => named(v.get("error").unwrap_or(v), &["type"]), + Dialect::Responses => { + let e = match ty { + "response.failed" => v.pointer("/response/error"), + _ => v.get("error"), + }; + named(e.unwrap_or(v), &["code", "type"]) + } + Dialect::Chat | Dialect::Gemini => v + .get("error") + .is_some_and(|e| named(e, &["code", "type", "status"]) || numbered(e)), + // Bedrock 的异常不是 SSE 事件，见 [`bedrock_exception`] + Dialect::Bedrock => false, + } +} + +/// 流只发了开头的例行事件就收了尾：一段内容都没有 +fn ended() -> GatewayError { + GatewayError::upstream(msg!( + "gw.upstream.stream_ended_early" => + "The upstream ended the stream before sending any content." + )) } fn error(status: u16, data: &str, kind: String, message: String) -> Judge { @@ -320,6 +359,7 @@ fn error(status: u16, data: &str, kind: String, message: String) -> Judge { body: Bytes::copy_from_slice(data.as_bytes()), kind, message, + transient: false, } } @@ -369,6 +409,7 @@ fn bedrock_exception(kind: String, message: String) -> Judge { Judge::Error { status, body: Bytes::from(body), + transient: crate::retry::transient_kind(&kind), kind, message, } @@ -514,13 +555,67 @@ mod tests { )); } + fn transient(dialect: Dialect, text: &str) -> bool { + match first_verdict(dialect, text) { + Some(Judge::Error { transient, .. }) => transient, + other => panic!("{other:?}"), + } + } + + /// 只有名单里的码是暂时的错；认不出的码（图片解不开、内容被拦）状态码照旧是 500、照旧换 + /// 下一家，可同一家不再试 #[test] - fn an_invalid_request_in_the_stream_faults_the_request() { - let refused = r#"{"type":"error","error":{"type":"invalid_request_error","code":"invalid_encrypted_content","message":"x"}}"#; - assert!(faults_the_request(refused.as_bytes())); - let overloaded = r#"{"type":"error","error":{"type":"service_unavailable_error","code":"server_is_overloaded","message":"x"}}"#; - assert!(!faults_the_request(overloaded.as_bytes())); - assert!(!faults_the_request(b"not json")); + fn only_listed_kinds_in_the_stream_are_temporary() { + let overloaded = "event: error\ndata: {\"type\":\"error\",\"error\":{\"type\":\"service_unavailable_error\",\"code\":\"server_is_overloaded\",\"message\":\"x\"}}\n\n"; + assert!(transient(Dialect::Responses, overloaded)); + let top = "event: error\ndata: {\"type\":\"error\",\"code\":\"server_is_overloaded\",\"message\":\"x\"}\n\n"; + assert!(transient(Dialect::Responses, top)); + let refused = "event: error\ndata: {\"type\":\"error\",\"error\":{\"type\":\"invalid_request_error\",\"code\":\"invalid_encrypted_content\",\"message\":\"x\"}}\n\n"; + assert!(!transient(Dialect::Responses, refused)); + let image = "event: response.failed\ndata: {\"type\":\"response.failed\",\"response\":{\"error\":{\"code\":\"invalid_image\",\"message\":\"x\"}}}\n\n"; + assert!(matches!( + first_verdict(Dialect::Responses, image), + Some(Judge::Error { + status: 500, + transient: false, + .. + }) + )); + let server = "event: response.failed\ndata: {\"type\":\"response.failed\",\"response\":{\"error\":{\"code\":\"server_error\",\"message\":\"x\"}}}\n\n"; + assert!(transient(Dialect::Responses, server)); + let anthropic = "event: error\ndata: {\"type\":\"error\",\"error\":{\"type\":\"overloaded_error\",\"message\":\"Overloaded\"}}\n\n"; + assert!(transient(Dialect::Anthropic, anthropic)); + let gemini = + "data: {\"error\":{\"code\":503,\"status\":\"UNAVAILABLE\",\"message\":\"x\"}}\n\n"; + assert!(transient(Dialect::Gemini, gemini)); + let numbered = "data: {\"error\":{\"code\":503,\"message\":\"busy\"}}\n\n"; + assert!(transient(Dialect::Chat, numbered)); + let filtered = "data: {\"error\":{\"code\":\"content_filter\",\"message\":\"x\"}}\n\n"; + assert!(!transient(Dialect::Chat, filtered)); + assert!(matches!( + bedrock_exception("serviceUnavailableException".into(), "x".into()), + Judge::Error { + transient: true, + .. + } + )); + assert!(matches!( + bedrock_exception("validationException".into(), "x".into()), + Judge::Error { + transient: false, + .. + } + )); + } + + /// 只发了开头的例行事件就收了尾：和断了一样 + #[tokio::test] + async fn a_stream_that_ends_after_its_opening_is_broken() { + let body = "event: message_start\ndata: {\"type\":\"message_start\"}\n\nevent: ping\ndata: {\"type\":\"ping\"}\n\n"; + match watch(served(body).await, Dialect::Anthropic, false, soon(5)).await { + Opening::Broken(e) => assert_eq!(e.detail.code, "gw.upstream.stream_ended_early"), + _ => panic!("该算断了"), + } } #[test] diff --git a/crates/tw-gateway/src/server/pipeline/plug.rs b/crates/tw-gateway/src/server/pipeline/plug.rs index eba78218..02071375 100644 --- a/crates/tw-gateway/src/server/pipeline/plug.rs +++ b/crates/tw-gateway/src/server/pipeline/plug.rs @@ -83,6 +83,7 @@ pub(super) async fn attempt( provider: &tw_config::Provider, model: &str, attempt: usize, + reported: &mut crate::guard::Reported, ) -> Result { let to = crate::plugin::request::Target { upstream: &provider.name, @@ -114,7 +115,7 @@ pub(super) async fn attempt( None => None, }; // 内容过滤：只报插件加进来的。处置档下删过的话，后面一律用删过的那一份 - let (body, value) = rescreen(state, rt, req, started, &provider.name, &c)?; + let (body, value) = rescreen(state, rt, req, started, &provider.name, &c, reported)?; // 生成回答的请求重新解码：格式转换用改过的这一份。数 token、压缩这些不转换（只发给 // 同格式的上游），不用解 let decoded = req.api.filter(|_| reading.generates).map(|api| { @@ -132,9 +133,11 @@ pub(super) async fn attempt( let shape = crate::client_api::ClientApi::written(&c.path); let look = crate::guard::look_hits_from(mode, &rt.redact, &body, shape, seed); let (found, ledger) = (look.found, look.ledger); - let more = crate::guard::more_found(&started.found, found); - // 和开头那一条加起来，一个请求报的有上限（见 `crate::guard::REPORTED_MAX`）。位置是插件 - // 改过的这一份里的 + // 先前几跳报过的不再报（重试、故障转移时插件写进来的还是那几个，见 `crate::guard::Reported`） + let already = started.found.len() + reported.secrets(); + let more = reported.fresh_secrets(crate::guard::more_found(&started.found, found)); + // 和开头那一条、先前几跳的加起来，一个请求报的有上限（见 `crate::guard::REPORTED_MAX`）。 + // 位置是插件改过的这一份里的 let redaction = super::redaction(rt, ledger.clone()); let seen = crate::guard::detail::Seen { body: &body, @@ -143,7 +146,7 @@ pub(super) async fn attempt( redaction: &redaction, replaced: mode.acts(), }; - let items = crate::guard::items(&more, started.found.len(), &seen); + let items = crate::guard::items(&more, already, &seen); if !items.is_empty() { state.bus.emit(tw_api::Event::SecretsFound { id: started.id, @@ -181,6 +184,7 @@ fn rescreen( started: &Started, provider: &str, c: &crate::plugin::request::Changed, + reported: &mut crate::guard::Reported, ) -> Result<(Bytes, Result), Msg> { let screen = crate::guard::Screen::of(rt); let unchanged = || Ok((c.body.clone(), Ok(c.value.clone()))); @@ -188,7 +192,7 @@ fn rescreen( return unchanged(); } if let Some(inputs) = &c.inputs { - return rescreen_inputs(state, rt, &screen, started, provider, c, inputs); + return rescreen_inputs(state, rt, &screen, started, provider, c, inputs, reported); } let screened = crate::client_api::ClientApi::screened(req.uri.path()); let Some(api) = req.api.filter(|_| screened) else { @@ -202,7 +206,14 @@ fn rescreen( rules: &screen.rules, redaction: &redaction, }; - if let Some(why) = crate::guard::report(&state.bus, started.id, provider, &sc, &src) { + let details = if sc.hits.is_empty() { + Vec::new() + } else { + crate::guard::detail::content(&src, &sc) + }; + let refused = + crate::guard::report_once(&state.bus, started.id, provider, &sc, details, reported); + if let Some(why) = refused { return Err(why); } Ok(match sc.body { @@ -221,6 +232,7 @@ fn rescreen( /// /// 改过的每一项写成一条调用方的消息（一段 Chat 格式的对话），改前、改后各查一遍，只报、 /// 只按插件加进来的拒绝；删过的话，删过的文字写回原文的那几项 +#[allow(clippy::too_many_arguments)] fn rescreen_inputs( state: &AppState, rt: &Runtime, @@ -229,6 +241,7 @@ fn rescreen_inputs( provider: &str, c: &crate::plugin::request::Changed, inputs: &crate::plugin::request::Inputs, + reported: &mut crate::guard::Reported, ) -> Result<(Bytes, Result), Msg> { use crate::plugin::view::inputs::{rewrite_texts, texts}; let as_chat = |texts: &[&String]| { @@ -286,7 +299,9 @@ fn rescreen_inputs( _ => key.to_string(), }; } - if let Some(why) = crate::guard::report_with(&state.bus, started.id, provider, &sc, details) { + let refused = + crate::guard::report_once(&state.bus, started.id, provider, &sc, details, reported); + if let Some(why) = refused { return Err(why); } let Some(stripped) = sc.body else { diff --git a/crates/tw-gateway/src/server/pipeline/relay.rs b/crates/tw-gateway/src/server/pipeline/relay.rs index c5068948..eef3144f 100644 --- a/crates/tw-gateway/src/server/pipeline/relay.rs +++ b/crates/tw-gateway/src/server/pipeline/relay.rs @@ -216,7 +216,7 @@ pub(super) fn respond( let mut said = false; // 被手动中止了（结局报手动中止，不说「流断了」） let mut aborted = false; - // 这一家的成败还没记（等不起了的最后一家不压开头，见 `hop`）：第一段真内容到了记成功 + // 这一家的成败还没记（最后一家不压开头时，见 `hop`）：第一段真内容到了记成功 let mut unconfirmed = unconfirmed; loop { // 补心跳的那一刻。不补的格式给一个不会用到的时刻，那一支不参与 diff --git a/crates/tw-gateway/src/throttle.rs b/crates/tw-gateway/src/throttle.rs index 0fe9144f..a12127bc 100644 --- a/crates/tw-gateway/src/throttle.rs +++ b/crates/tw-gateway/src/throttle.rs @@ -61,6 +61,9 @@ const MIN_HINT: Duration = Duration::from_millis(1); /// 每次 60 秒 const BACKOFF_SECS: [u64; 5] = [5, 10, 20, 40, 60]; +/// 一家最近一次被限流之后这么久之内，它还「可能在限流」（[`Throttles::wary`]） +const WARY_FOR: Duration = Duration::from_secs(300); + /// 两个时钟走的差，到了这么多才当是系统时间被拨过（见 [`Span::end_ms`]）；两个时钟换算出来的 /// 时刻差不到这么多的当是同一刻（见 [`Throttles::limit_state`]）。先后读两个时钟，差会抖一两毫秒 const SLACK_MS: u64 = 1_000; @@ -580,6 +583,16 @@ impl Throttles { (Peek::Open, None) } + /// 这一家此刻**可能在限流**：在探路、连续被限流还没有别的答复，或者最近 + /// （[`WARY_FOR`]）被限流过。最后一家的流式回答这时也压着开头（见 `hop`），客户端不收流 + /// 也一样：流的开头报的限流要排队等它恢复，不交给客户端 + pub fn wary(&self, name: &str) -> bool { + let now = self.moment(); + self.lock().get(name).is_some_and(|up| { + up.probing || up.strikes > 0 || up.limited_at.is_some_and(|t| t.since(now) < WARY_FOR) + }) + } + /// 留在这一家的对话还留不留（见 [`crate::affinity`]）：这一期用完了的、限流暂停要到 /// `deadline_ms`（这个请求的等待期限）之后才结束的，放开 —— 和熔断着的一样，留在一家发不出去 /// 的上游上，省下的缓存一分钱都兑现不了。等得到的照旧留，到时候等它 @@ -1941,6 +1954,20 @@ mod tests { assert!(matches!(t.gate("甲", &[], None), Gate::Go(Some(_)))); } + #[test] + fn an_upstream_is_wary_while_probing_and_for_a_while_after_a_429() { + let (t, c) = clocked(); + assert!(!t.wary("甲"), "没被限流过"); + t.limited("甲", c.at(T0), None); + assert!(t.wary("甲")); + c.advance(5_000); + t.answered("甲", c.at(T0 + 5_000)); + // 探路的过了、连续的次数归零：最近被限流过，还要当心 + assert!(t.wary("甲")); + c.advance(295_000); + assert!(!t.wary("甲")); + } + #[tokio::test] async fn a_renamed_upstream_keeps_its_pause_and_its_queue() { let (t, c) = clocked(); diff --git a/crates/tw-gateway/tests/plugins_security.rs b/crates/tw-gateway/tests/plugins_security.rs index f64b9e39..e0fe5550 100644 --- a/crates/tw-gateway/tests/plugins_security.rs +++ b/crates/tw-gateway/tests/plugins_security.rs @@ -212,6 +212,56 @@ async fn a_key_written_by_a_plugin_is_replaced_before_the_upstream_when_enforcin assert!(up.raw(0).contains(WRITTEN), "{}", up.raw(0)); } +/// 唯一的上游先过载一次（503），请求退避之后重试同一家：插件每一跳都跑、写进来的还是那把 key +/// 和那句话，**安全日志里各只一行** —— 重试不让同一处命中报两遍 +#[tokio::test] +async fn a_retry_does_not_report_what_a_plugin_wrote_twice() { + const WRITTEN: &str = "sk-ant-api03-PLUGINWROTEITAAAAAAAAAAAAA"; + let adds = r#" +export const manifest = { name: "加一句", api: 1, permissions: ["system", "messages"] }; +export function onRequest(req) { + req.system = `${req.system}\n备用密钥：sk-ant-api03-PLUGINWROTEITAAAAAAAAAAAAA`; + req.messages.push({ role: "user", parts: [{ type: "text", text: "Project Falcon 的细节" }] }); + return req; +}"#; + let up = Upstream::start(vec![Answer::Status(503), Answer::Text("好的".into())]).await; + let mut security = redact(SecurityMode::Enforce); + security.content = ContentPolicy { + mode: SecurityMode::Enforce, + custom: vec![CustomContentRule { + name: "内部代号".into(), + pattern: "Project Falcon".into(), + scope: vec![tw_config::ContentScope::Request], + case_sensitive: false, + replacement: "***".into(), + matching: Default::default(), + action: ContentAction::Record, + disabled: false, + }], + ..Default::default() + }; + let gw = Gateway::start(config(&up, security), vec![Plug::new("add", adds)]).await; + let mut rx = gw.events(); + let r = gw.ask(plain("你好", false)).await; + assert_eq!(r.status, 200, "{}", r.body); + assert_eq!(up.hits(), 2, "退避之后重试了同一家"); + assert_eq!(gw.calls("add"), 2, "插件每一跳都跑"); + let (mut secrets, mut matched) = (Vec::new(), Vec::new()); + while let Ok(Ok(ev)) = tokio::time::timeout(Duration::from_secs(10), rx.recv()).await { + match ev { + tw_api::Event::SecretsFound { items, .. } => { + secrets.extend(items.into_iter().map(|i| i.masked)) + } + tw_api::Event::ContentMatched { rule, .. } => matched.push(rule), + tw_api::Event::RequestFinished { .. } | tw_api::Event::RequestFailed { .. } => break, + _ => {} + } + } + assert_eq!(secrets.len(), 1, "{secrets:?}"); + assert!(!secrets[0].contains(WRITTEN), "{secrets:?}"); + assert_eq!(matched.len(), 1, "{matched:?}"); +} + fn tools(mode: SecurityMode) -> Security { Security { inspect_tools: ToolPolicy { diff --git a/crates/tw-gateway/tests/retry.rs b/crates/tw-gateway/tests/retry.rs index cec9b7a8..ee5ec417 100644 --- a/crates/tw-gateway/tests/retry.rs +++ b/crates/tw-gateway/tests/retry.rs @@ -29,6 +29,8 @@ struct Reply { /// 给完 `chunks` 之后把连接断掉，不好好收尾 broken: bool, content_type: &'static str, + /// 头一段之后每一段之前等多久 + slow_ms: u64, } /// 整包地照常答 @@ -40,6 +42,7 @@ fn ok() -> Reply { chunks: Vec::new(), broken: false, content_type: "application/json", + slow_ms: 0, } } @@ -149,9 +152,14 @@ fn app(script: Vec, then: Reply, hits: Arc>>) -> Route if r.chunks.is_empty() { return b.body(axum::body::Body::from(r.body)).unwrap(); } - let broken = r.broken; - let chunks = futures::stream::iter(r.chunks) - .map(|c| Ok::<_, std::io::Error>(bytes::Bytes::from(c))) + let (broken, slow) = (r.broken, r.slow_ms); + let chunks = futures::stream::iter(r.chunks.into_iter().enumerate()) + .then(move |(i, c)| async move { + if i > 0 { + tokio::time::sleep(Duration::from_millis(slow)).await; + } + Ok::<_, std::io::Error>(bytes::Bytes::from(c)) + }) .chain(futures::stream::iter( broken.then(|| Err(std::io::Error::other("cut"))), )); @@ -668,26 +676,40 @@ async fn wait_secs_zero_never_retries() { } #[tokio::test] -async fn retry_after_from_the_upstream_replaces_the_backoff() { +async fn a_retry_after_longer_than_the_backoff_is_waited_for() { let said = Reply { - headers: vec![("retry-after-ms", "300".into())], + headers: vec![("retry-after-ms", "700".into())], ..overloaded_503() }; let up = upstream(vec![said], ok()).await; - // 退避的一秒是十秒：按退避算要等二十秒，比等待期限还长 - let (gw, _state, log) = serve_with( - anthropic(&[&up], 10), - Duration::from_secs(10), - Duration::from_secs(10), - ) - .await; + let (gw, _state, log) = serve(anthropic(&[&up], 10)).await; let t = Instant::now(); let r = ask(gw, "上游说了等多久").send().await.unwrap(); assert_eq!(r.status(), 200); - assert!(t.elapsed() >= Duration::from_millis(300)); - assert!(t.elapsed() < Duration::from_secs(3), "{:?}", t.elapsed()); + assert!( + t.elapsed() >= Duration::from_millis(700), + "{:?}", + t.elapsed() + ); let chain = log.attempts("上游说了等多久").await; - assert!(chain[1].backoff_ms.unwrap() >= 290, "{chain:#?}"); + assert!(chain[1].backoff_ms.unwrap() >= 690, "{chain:#?}"); +} + +/// 上游说的比退避短：不算。回 `retry-after-ms: 5` 的中转不能让一个请求变成几千个 +#[tokio::test] +async fn a_retry_after_shorter_than_the_backoff_does_not_shorten_it() { + let said = Reply { + headers: vec![("retry-after-ms", "5".into())], + ..overloaded_503() + }; + let up = upstream(Vec::new(), said).await; + let (gw, _state, log) = serve(anthropic(&[&up], 1)).await; + let r = ask(gw, "上游说只等五毫秒").send().await.unwrap(); + assert_eq!(r.status(), 502); + // 退避 200、400 毫秒，剩下的不够 800 毫秒：一共三次 + assert_eq!(up.hits(), 3); + let chain = log.attempts("上游说只等五毫秒").await; + assert!(chain[1].backoff_ms.unwrap() >= 190, "{chain:#?}"); } #[tokio::test] @@ -812,7 +834,8 @@ async fn a_conversation_that_stays_on_an_upstream_retries_only_that_one() { assert_eq!(rounds, [vec!["乙".to_string()], vec!["乙".to_string()]]); } -/// 压过了时限：客户端先拿到 `200` 和保活，重试到期限都没成，流里按它的格式报最后那个错误 +/// 压过了时限：客户端先拿到 `200` 和保活，重试到期限都没成。最后那一次再试不了，它的流原样 +/// 接在保活后面 —— 上游自己报的那个错就在流里 #[tokio::test] async fn past_the_hold_the_retries_go_on_under_keepalives_and_the_last_error_ends_the_stream() { let up = upstream(Vec::new(), overloaded_in_the_stream()).await; @@ -831,9 +854,10 @@ async fn past_the_hold_the_retries_go_on_under_keepalives_and_the_last_error_end assert!(body.matches(": keep-alive").count() >= 3, "{body}"); let tail = &body[body.find("event: error").expect("流里报了错")..]; assert!(tail.contains("overloaded_error"), "{tail}"); - assert!( - !body.contains("message_start"), - "被放弃的开头一个字节都没交出去：{body}" + assert_eq!( + body.matches("event: message_start").count(), + 1, + "被放弃的那几次的开头一个字节都没交出去：{body}" ); // 退避 200、400、800 毫秒之后剩下的不够 1.6 秒 assert_eq!(up.hits(), 4); @@ -863,3 +887,139 @@ async fn past_the_hold_a_retry_that_answers_continues_the_stream() { assert_eq!(body.matches("event: message_start").count(), 1, "{body}"); assert_eq!(up.hits(), 3); } + +/// 同一家再试只认 500、502、503、504、529：别的 5xx 和正文说是请求本身问题的 500 当场交出去 +#[tokio::test] +async fn other_5xx_and_a_500_that_blames_the_request_are_not_retried() { + let cases = [ + ("没实现", status(501, "not implemented")), + ("Cloudflare 超时", status(524, "a timeout occurred")), + ( + "500 说请求错了", + status( + 500, + r#"{"type":"error","error":{"type":"invalid_request_error","message":"bad"}}"#, + ), + ), + ]; + for (model, reply) in cases { + let up = upstream(Vec::new(), reply).await; + let (gw, _state, log) = serve(anthropic(&[&up], 10)).await; + let r = ask(gw, model).send().await.unwrap(); + assert_eq!(r.status(), 502, "{model}"); + assert_eq!(up.hits(), 1, "{model}"); + assert!(log.waits(model).await.is_empty(), "{model}"); + } +} + +/// 不在名单里的码（图片解不开）：同一家不再试，原样交出去；有下一家时照旧换下一家 +#[tokio::test] +async fn an_unlisted_error_in_the_stream_is_not_retried_but_still_fails_over() { + let image = || { + sse(vec![ + event( + "response.created", + r#"{"type":"response.created","sequence_number":0,"response":{"id":"resp_1"}}"#, + ), + event( + "response.failed", + r#"{"type":"response.failed","sequence_number":1,"response":{"id":"resp_1","error":{"code":"invalid_image","message":"The image could not be read."}}}"#, + ), + ]) + }; + let answered = sse(vec![ + event( + "response.output_text.delta", + r#"{"type":"response.output_text.delta","sequence_number":0,"item_id":"m1","output_index":0,"content_index":0,"delta":"ok"}"#, + ), + event( + "response.completed", + r#"{"type":"response.completed","sequence_number":1,"response":{"id":"resp_1","status":"completed","usage":{"input_tokens":3,"output_tokens":1,"total_tokens":4}}}"#, + ), + ]); + let post = |gw: SocketAddr, model: &str| { + client() + .post(format!("http://{gw}/v1/responses")) + .header("authorization", "Bearer tw-k") + .header("content-type", "application/json") + .body(format!( + r#"{{"model":"{model}","stream":true,"input":"hi"}}"# + )) + .send() + }; + let only = upstream(Vec::new(), image()).await; + let (gw, _state, log) = serve(config(&[(only.addr, "openai-responses")], 10)).await; + let body = post(gw, "图片解不开").await.unwrap().text().await.unwrap(); + assert!(body.contains("invalid_image"), "{body}"); + assert_eq!(only.hits(), 1); + assert!(log.waits("图片解不开").await.is_empty()); + + let a = upstream(Vec::new(), image()).await; + let b = upstream(Vec::new(), answered).await; + let (gw, _state, log) = serve(config( + &[(a.addr, "openai-responses"), (b.addr, "openai-responses")], + 10, + )) + .await; + let body = post(gw, "换一家").await.unwrap().text().await.unwrap(); + assert!(body.contains("\"delta\":\"ok\""), "{body}"); + assert_eq!((a.hits(), b.hits()), (1, 1)); + let chain = log.attempts("换一家").await; + assert!(chain.iter().all(|a| a.retry.is_none()), "{chain:#?}"); +} + +#[tokio::test] +async fn a_stream_that_ends_after_its_opening_is_retried() { + let up = upstream(vec![sse(vec![message_start()])], streamed()).await; + let (gw, _state, log) = serve(anthropic(&[&up], 10)).await; + let r = ask_stream(gw, "开了头就收尾").send().await.unwrap(); + assert_eq!(r.status(), 200); + assert!(r.text().await.unwrap().contains("\"ok\"")); + assert_eq!(up.hits(), 2); + let chain = log.attempts("开了头就收尾").await; + assert_eq!( + chain[0].error.as_ref().unwrap().code, + "gw.upstream.stream_ended_early" + ); + assert_eq!(chain[1].retry, Some(1)); +} + +/// 最后一家报的暂时的错再试不了（上游要等的比剩下的时间长）：**上游自己的那句话原样交出去**， +/// 不换成网关的 502 +#[tokio::test] +async fn an_error_that_cannot_be_retried_in_time_is_handed_over_as_the_upstream_said_it() { + let said = Reply { + headers: vec![("retry-after", "5".into())], + ..overloaded_in_the_stream() + }; + let up = upstream(Vec::new(), said).await; + let (gw, _state, log) = serve(anthropic(&[&up], 2)).await; + let r = ask_stream(gw, "来不及再试").send().await.unwrap(); + assert_eq!(r.status(), 200); + let body = r.text().await.unwrap(); + assert!(body.contains("message_start"), "上游的原话：{body}"); + assert!(body.contains("overloaded_error"), "{body}"); + assert_eq!(up.hits(), 1); + assert!(log.waits("来不及再试").await.is_empty()); +} + +/// 最后一家整包的回答不压：响应头当场交出去，中转站先发的空格保活照样到客户端 +#[tokio::test] +async fn the_last_whole_answer_is_not_held() { + let whole = Reply { + chunks: vec![" ".into(), ok().body], + slow_ms: 800, + ..ok() + }; + let up = upstream(Vec::new(), whole).await; + let (gw, _state, _log) = serve(anthropic(&[&up], 10)).await; + let t = Instant::now(); + let r = ask(gw, "整包").send().await.unwrap(); + assert!( + t.elapsed() < Duration::from_millis(500), + "响应头等到了正文：{:?}", + t.elapsed() + ); + assert_eq!(r.status(), 200); + assert!(r.text().await.unwrap().contains("\"ok\"")); +} diff --git a/docs/config.md b/docs/config.md index c1131bd7..aba312de 100644 --- a/docs/config.md +++ b/docs/config.md @@ -1135,18 +1135,20 @@ the upstream sends in the stream moves the request to the next candidate, the same as an error status would. When every candidate has been tried and the last one failed temporarily -before any content reached the client — overloaded (503, 529, or an overload -error in the stream), another 5xx, unreachable, or its stream broke — the -request is retried after a pause instead of passing the error on. The pause -is 2, 4 and 8 seconds, then 16 seconds each time, varied by up to a fifth, or -as long as the upstream says (`retry-after-ms`, `Retry-After`). Each retry +before any content reached the client — 500, 502, 503, 504 or 529, an +overload or server error reported in a streamed answer, unreachable, or its +stream broke or ended before any content — the request is retried after a +pause instead of passing the error on. The pause is 2, 4 and 8 seconds, then +16 seconds each time, varied by up to a fifth; when the upstream asks for a +longer one (`retry-after-ms`, `Retry-After`), it is used instead. Each retry goes again to the candidates that failed temporarily, in order, skipping those set aside unless all of them are; a conversation that stays on an upstream retries only that one. Retrying stops when the next pause would end after `wait_secs`, the time a request may wait in all; the client then gets the last error. Rate limits, used-up quotas, an insufficient balance, rejected -credentials, missing models, errors in the request itself and idle timeouts -are not retried. Each retry appears among the request's attempts with its +credentials, missing models, errors in the request itself, other errors +reported in the stream, other 5xx statuses and idle timeouts are not retried; +they still move the request to the next candidate as before. Each retry appears among the request's attempts with its number and the pause before it. `wait_secs: 0` turns retrying off. An upstream can also go quiet: it accepts the request and then sends nothing, @@ -1164,10 +1166,12 @@ to the complete answer. outcome `idle_timeout`, and the request moves to the next candidate. Until then an upstream's answer is held back from the client, so the next upstream starts it afresh. With no candidate left, the client gets a timeout error - (504) in its own format. The last candidate's answer is held the same way - while the request can still be retried within `wait_secs`; otherwise its - stream is passed on as it arrives, and once its response has started, a - timeout there ends it with an error event instead. + (504) in its own format. When the client receives a stream, the last + candidate's stream is held the same way while the request can still be + retried within `wait_secs`; otherwise it is passed on as it arrives, and + once its response has started, a timeout there ends it with an error event + instead. A non-streamed answer from the last candidate is always passed on + as it arrives. - A streamed answer is held for at most 15 seconds. If no content has arrived by then, the client receives `200` and the streaming headers, so that its own wait for headers does not run out, followed by an SSE comment @@ -1235,7 +1239,7 @@ receive the request and failed, the client gets that failure instead. | `quota_pause_secs` | integer | `3600` | Seconds to set aside an upstream whose quota is used up when it does not say when the quota resets. When it does, the upstream is set aside until then. | | `rate_limit_max_pause_secs` | integer | `3600` | Longest pause, in seconds, after an upstream answers 429 (rate limited). The pause lasts as long as the upstream says (`retry-after-ms`, `Retry-After`, or a delay in the error body); when it says nothing, it backs off by the upstream's consecutive 429s: 5, 10, 20 and 40 seconds, then 60 seconds each time. A pause is not a failure: requests wait for the upstream within `wait_secs`, and the circuit breaker does not count it. | | `idle_timeout_secs` | integer | `300` | Seconds an upstream may go without sending content before the gateway stops waiting for it. Counted from the moment the request is sent and started again by every piece of content: text, reasoning and tool calls count, keep-alives do not. A whole (non-streamed) answer counts from sending to the complete answer. Before any content has reached the client, the upstream counts as failed and the request moves to the next one; with none left, the client gets a timeout error. After content has reached the client, the answer ends with an error. From 30 to 3600. | -| `wait_secs` | integer | `120` | Seconds a request waits in all, counted once the key's own `max_concurrent` lets it in: for a key's `minute` or `hour` limit to free up, for an upstream that cannot take it yet — at its `max_concurrent`, paused after answering 429, or at one of its `minute` or `hour` usage limits — and before retrying upstreams that failed temporarily before any content (overloaded, another 5xx, unreachable). A key limit that does not free up in time refuses the request. An upstream that cannot take it is skipped for the next one, and when none is left the request waits for the first to become available; a conversation that stays on an upstream waits for it. When nothing became available in time, the client gets 429 with `Retry-After`; when retries did not succeed in time, it gets the last upstream error. `0`: never wait or retry. From 0 to 600. | +| `wait_secs` | integer | `120` | Seconds a request waits in all, counted once the key's own `max_concurrent` lets it in: for a key's `minute` or `hour` limit to free up, for an upstream that cannot take it yet — at its `max_concurrent`, paused after answering 429, or at one of its `minute` or `hour` usage limits — and before retrying upstreams that failed temporarily before any content (overloaded, a server error, unreachable). A key limit that does not free up in time refuses the request. An upstream that cannot take it is skipped for the next one, and when none is left the request waits for the first to become available; a conversation that stays on an upstream waits for it. When nothing became available in time, the client gets 429 with `Retry-After`; when retries did not succeed in time, it gets the last upstream error. `0`: never wait or retry. From 0 to 600. | ### `aliases` diff --git a/docs/config.zh-CN.md b/docs/config.zh-CN.md index ecbd1bf5..7f8d133c 100644 --- a/docs/config.zh-CN.md +++ b/docs/config.zh-CN.md @@ -852,13 +852,14 @@ security: 流式回答在第一段内容交给客户端之前，上游在流里报的错误和错误状态码一样，会把 请求换到下一个候选。 -候选都试过之后，最后一个候选在内容交给客户端之前暂时出错的 —— 过载（503、529 或流里的 -过载错误）、其他 5xx、连不上、流断了 —— 请求不把错误交给客户端，而是等待一段时间后重试。 -等待 2、4、8 秒，之后每次 16 秒，上下浮动不超过两成；上游给出了等待时间的（`retry-after-ms`、 -`Retry-After`）按它给的。每次重试再发给暂时出错的那几个候选，按原来的顺序，停用中的跳过， +候选都试过之后，最后一个候选在内容交给客户端之前暂时出错的 —— 500、502、503、504、529， +流式回答里报的过载或服务端错误，连不上，流在内容之前断了或收尾 —— 请求不把错误交给客户端， +而是等待一段时间后重试。等待 2、4、8 秒，之后每次 16 秒，上下浮动不超过两成；上游要求等得 +更久的（`retry-after-ms`、`Retry-After`），按它要求的。每次重试再发给暂时出错的那几个候选，按原来的顺序，停用中的跳过， 都在停用中时照样重试；留在某一家的对话只重试这一家。下一次等待会超出 `wait_secs`（一个请求 合计最多等的时间）时不再重试，客户端收到最后一次的错误。限流、额度用完、余额不足、凭据被拒、 -模型不存在、请求本身的错误和无响应超时都不重试。每次重试在请求的尝试链上单列一跳，带着它是 +模型不存在、请求本身的错误、流里报的其他错误、其他 5xx 和无响应超时都不重试，照旧换到下一个 +候选。每次重试在请求的尝试链上单列一跳，带着它是 第几次重试和之前等了多久。`wait_secs: 0` 关闭重试。 上游也可能不出声：收下请求之后一直不发内容，或者发到一半停住。连续 @@ -870,8 +871,9 @@ security: - 还没有内容交给客户端时，这家上游记一次失败（和 5xx 一样计入 `failures_to_pause`）， 尝试链上这一跳的结果是 `idle_timeout`，请求换到下一个候选。在此之前上游的回答不交给 客户端，下一家从头开始回答。没有候选了，客户端收到它自己格式的超时错误（504）。 - 请求还能在 `wait_secs` 之内重试时，最后一个候选的回答同样暂存；否则它的流边收边交给 - 客户端，响应开始之后再超时，回答以一条错误事件收尾。 + 客户端收的是流、请求还能在 `wait_secs` 之内重试时，最后一个候选的流同样暂存；否则它的 + 流边收边交给客户端，响应开始之后再超时，回答以一条错误事件收尾。最后一个候选的整包 + 回答一律边收边交。 - 流式回答最多暂存 15 秒。到时还没有内容，客户端先收到 `200` 和流式响应头，以免它自己 等响应头的时限先到，之后每 15 秒收到一行 SSE 注释（`: keep-alive`）；Gemini 的客户端 不发注释。上游的事件照旧暂存到第一段内容，故障转移在同一个 `200` 之下照常进行：下一家 @@ -918,7 +920,7 @@ failover: | `quota_pause_secs` | 整数 | `3600` | 上游报告额度用完、但没有给出重置时间时停用的秒数。给出了重置时间的，停用到那一刻。 | | `rate_limit_max_pause_secs` | 整数 | `3600` | 上游回 429（限流）后暂停这家的最长秒数。上游给出了等待时间的（`retry-after-ms`、`Retry-After` 或错误正文里的等待时间）按它给的暂停；没有给出的，按这家连续被限流的次数退避：5、10、20、40 秒，之后每次 60 秒。暂停不算失败：请求在 `wait_secs` 之内等这家恢复，熔断也不计入。 | | `idle_timeout_secs` | 整数 | `300` | 上游多少秒没有发出内容，网关就不再等它。从请求发出的那一刻算起，每来一段内容重新计时：正文、推理、工具调用都算，心跳不算。整包（非流式）的回答从发出算到整份回来。还没有内容交给客户端时，这家上游记一次失败，请求换到下一家；没有下一家了，客户端收到超时错误。已经有内容交给客户端的，回答以一条错误收尾。取值 30 到 3600。 | -| `wait_secs` | 整数 | `120` | 一个请求合计最多等的秒数，从过了密钥自己的 `max_concurrent` 时算起：等密钥的 `minute`、`hour` 上限空出名额，等暂时不能发的上游 —— 并发数满了（`max_concurrent`）、回了 429 正在暂停、到了这家 `minute` 或 `hour` 的用量上限 —— 和上游在内容之前暂时出错（过载、其他 5xx、连不上）后重试之前的等待，都算在里面。密钥的上限到时空不出来就拒绝。上游暂时不能发的先换下一家，没有下一家了就等最先恢复的那一家；留在某一家的对话等这一家。到时都没有恢复，客户端收到带 `Retry-After` 的 429；到时重试仍未成功，客户端收到最后一次的上游错误。`0`：不等，也不重试。取值 0 到 600。 | +| `wait_secs` | 整数 | `120` | 一个请求合计最多等的秒数，从过了密钥自己的 `max_concurrent` 时算起：等密钥的 `minute`、`hour` 上限空出名额，等暂时不能发的上游 —— 并发数满了（`max_concurrent`）、回了 429 正在暂停、到了这家 `minute` 或 `hour` 的用量上限 —— 和上游在内容之前暂时出错（过载、服务端出错、连不上）后重试之前的等待，都算在里面。密钥的上限到时空不出来就拒绝。上游暂时不能发的先换下一家，没有下一家了就等最先恢复的那一家；留在某一家的对话等这一家。到时都没有恢复，客户端收到带 `Retry-After` 的 429；到时重试仍未成功，客户端收到最后一次的上游错误。`0`：不等，也不重试。取值 0 到 600。 | ### `aliases`