English | 简体中文
Important
The gateway supports SGLang and vLLM as the inference engine, selected with
CONTINUATION_ENGINE. Within this project, each engine + model combination has its own
continuation strategy. The combinations verified so far:
Status: ✅ verified,
SGLang (NVIDIA GPU)
| Model | Engine version | Status | Notes |
|---|---|---|---|
| Kimi-K3 | v0.5.19 | ✅ Verified | |
| Kimi-K3.1-Pro | dev-kimi-k31-pro | ✅ Verified | |
| Qwen3.6-35B-A3B | v0.5.19 | ✅ Verified | |
| Qwen3.8-27B | v0.5.19 | ✅ Verified | Tested with the FP8 weights |
| GLM-5.3 | v0.5.19 | ✅ Verified | |
| Nemotron-3.5-Lightning-30B-A3B | dev-dsv41 | ✅ Verified | |
| MiMo-V2.5 | dev-dsv41 | ✅ Verified | |
| DeepSeek-V4.1-Flash-1017 | dev-dsv41 | ✅ Verified | |
| Step-3.5-Flash | dev-dsv41 | ✅ Verified | |
| Qwen3.8-Flash-Next | qwen38flashnext | ✅ Verified | |
| Ling-3.0-flash-VL | dev-Ling-3.0-flash-VL | ✅ Verified | |
| Nex-N2.5-Pro | dev-dsv41 | ✅ Verified | |
| Muse-Glimmer-30B | v0.5.19 | When a request is continued after being cut off during reasoning, the continued reasoning shows up in content (without tools, the <|eom|> control token appears there too). Tool calls and the final answer are not affected. SGLang's Muse parser does not take the continuation prefix into account when deciding whether it is inside reasoning, so this cannot be avoided by how the prefix is built |
|
| gpt-oss-20b | v0.5.19 | ❌ Not supported | |
| gpt-oss-120b | v0.5.19 | ❌ Not supported |
vLLM (NVIDIA GPU)
| Model | Engine version | Status | Notes |
|---|---|---|---|
| Qwen3.6-35B-A3B | v0.31.0 | ✅ Verified | |
| GLM-5.3 | v0.31.0 | ✅ Verified | |
| DeepSeek-V4-Flash-0731-nvfp4 | v0.31.0 | ✅ Verified | |
| Kimi-K3 | v0.31.0 | ✅ Verified | vLLM must be started with --enable-scale-out, otherwise continuation does not work. Multimodal requests are not continued |
vLLM (Ascend 910B4) — tested via vllm-ascend; the engine version column below is the
vllm-ascend release.
| Model | Engine version | Status | Notes |
|---|---|---|---|
| Qwen3.6-35B-A3B | v0.30.0 | ✅ Verified | Tested with the W8A8 weights |
| Qwen3.6-27B | v0.30.0 | ✅ Verified | Tested with the W8A8 weights |
| Qwen3-30B-A3B | v0.30.0 | ✅ Verified | Tested with the W8A8 weights |
| Qwen3-14B | v0.30.0 | ✅ Verified | |
| Qwen3-8B | v0.30.0 | ✅ Verified | |
| GLM-5.3-Flash | v0.30.0 | ✅ Verified | Tested with the W8A8 weights |
| Gemma-4-31B-it | v0.30.0 | ✅ Verified | The model itself is occasionally unstable: it may emit its control tokens as plain text in content (for example </think>, <|tool_call>) so that the tool_call is not parsed and is lost. This comes from the model and its parser, not from how the continuation prefix is built |
| Gemma-4-26B-A4B-it | v0.30.0 | ✅ Verified | Same as Gemma-4-31B-it. Its reasoning may also occasionally end up in content |
| DeepSeek-V4.1-Flash-1017 | v0.30.0 | Tested with the W4A8 weights. Requests are continued and tool_calls are produced, but the model's internal special tokens may leak into the continued output as plain text (for example <ds_safety>…</ds_safety>, <DS_STREAMING>, @@INLINE_ICON@@), in which case the answer may be cut short or end up in reasoning instead of content. A continuation from a short prefix may also drift off topic or switch language. The seam can occasionally lose or gain a character |
|
| Qwen3-Omni-30B-A3B-Instruct | v0.30.0 | ❌ Not supported |
The model names in the tables are exactly the strings CONTINUATION_MODEL should be set to
(case-insensitive, exact match). A quantized or fine-tuned variant of a listed model, such as
GLM-5.3-FP8, should use the listed name GLM-5.3; a different size or version
(Qwen3-32B, GLM-5.4) is a different model and is not matched.
Other models and engines have not been verified; they may work, but please verify them yourself before relying on them. We are continuing to adapt more engine + model combinations.
A crash-continuation gateway for LLM inference services. It sits between clients and an OpenAI-compatible inference service (SGLang or vLLM — see the note above).
An inference instance can crash in the middle of a streaming generation, have its connection
reset, or stall (the process is alive but stops emitting tokens). The client is then left with
a truncated reply that never received a finish_reason, and usually has no choice but to retry
the whole request — the content already generated and the tokens already spent are wasted.
When this gateway detects that a streaming response has stalled (idle timeout) or has been
disconnected, it automatically issues one continuation request: the reasoning/content
already sent to the client is assembled into an assistant prefix, and the model is asked to
carry on from there. To the client, it is still a single, continuous, complete streaming
response.
The way the prefix is assembled is determined by CONTINUATION_MODEL (the model this
deployment actually serves; the model field of the request is not consulted), see
continuation_gateway/reconstruct.py. Two formats are built in: Kimi-K3 (XTML) and a generic
<think>...</think> format. Supporting a model with another format only takes adding a
builder there. A model that matches no verified model is handled with the generic format, and
the gateway logs a warning at startup. On vLLM, some models (such as Qwen3.6-35B-A3B) additionally split
the content-done partial message into separate reasoning / content fields.
leg1: original request
client ──▶ continuation_gateway ──────────────────▶ downstream router ──▶ inference instance
▲ │ forwards chunk by chunk, parses SSE (SGLang / vLLM)
│ │ accumulates reasoning / content
│ │ detects stall / disconnect (no finish_reason seen)
│ ▼
│ builds the prefix, issues leg2: continuation request ──▶ downstream router
│ │ (can be set separately via CONTINUATION_URL)
└─────────┘ leg2's output continues right after what leg1 already forwarded;
id / reasoning field name / usage are rewritten before forwarding
- Orchestration at the gateway layer; the inference engine is untouched. Continuation
requires parsing SSE semantics, remembering what has already been forwarded, issuing an
extra request on its own initiative, and rewriting
usagein place. So within a single request the gateway is stateful rather than a transparent byte-forwarder. - Conservative first: when in doubt, pass through. A request that is out of scope, or does not meet the conditions for continuation, behaves exactly as if this gateway were not there: it does not hang up the connection, swallow exceptions, or rewrite the response. The gateway steps in only when a meaningful continuation can actually be made.
- One continuation, at most two legs. A single client request involves at most leg1 +
leg2. If leg2 itself fails, no third leg is issued and the request ends as it stands — no
worse than having no gateway at all. This is also why the
usagecorrection formula can assume "exactly two legs". - The gateway must not become a point of failure itself. Any unexpected error during the continuation phase is only logged, never propagated; content already forwarded to the client is unaffected.
- Transparent to the client. The
idand the reasoning field name stay consistent across the whole stream. The recovered portion is counted in this request'susage, and the continuation request'smax_tokensis reduced by what has already been generated, so the original budget is never exceeded. - Billing is addition-only and does not aim for absolute precision. The recovered content
is converted to a token count by character-based estimation and added to the usage reported
by leg2. A small deviation is accepted in preference to the risk of a completely wrong
measurement, and nothing depends on the downstream's
/v1/tokenize. - No cross-request state. All state lives in the coroutine of a single request. Replicas share nothing, so the gateway scales horizontally and needs no sticky sessions.
Out of scope for this gateway: general request retry (any problem before the first chunk arrives is propagated as-is), choosing or avoiding failed inference instances (left to the downstream router), and admission control and rate limiting (left to the upstream ingress).
Only free-text scenarios are handled, where no tool_call chunk had appeared at the time of
the crash and the request is not constrained by response_format to json_object /
json_schema. The following cases are passed through without continuation; see the comments
on should_intervene() in continuation_gateway/server.py for details:
- A crash after a tool_call chunk has already appeared: under the standard
continue_final_messagesemantics there is no "tool call still being generated" state, so supporting it would require changing the SGLang source.BUFFER_TOOL_CALLS=true(see below) works around this limitation: tool_call chunks are held in gateway memory and released to the client only once complete; on a crash the whole held batch is discarded and the continuation starts from the state before the tool_call appeared. The client never received a tool_call, so there is no "cannot be continued" state. - Structured output with
response_formatset tojson_object/json_schema: the constrained JSON content is mixed into ordinary content chunks, and there is no separate signal to recognize it. - Requests that already carry
continue_final_messagefrom the client: continuation is not stacked on top. - With
CONTINUATION_ENGINE=vllm, requests whose messages carry non-text content (images, audio, video). - Request bodies larger than
MAX_CONTINUATION_BODY_MB(by default, large multimodal bodies).
The repository root includes a Dockerfile (the only dependency is aiohttp). The base image
and the PyPI index are build args:
| build arg | default | description |
|---|---|---|
BUILDER_IMAGE |
python:3.12-slim |
Base image used to build and run |
PIP_INDEX_URL |
empty (direct PyPI) | Optional PyPI mirror |
docker build -t continuation-gateway:latest .
# If Docker Hub / PyPI are not reachable, use your own registry and PyPI mirror:
docker build -t continuation-gateway:latest \
--build-arg BUILDER_IMAGE=<registry>/python:3.12-slim \
--build-arg PIP_INDEX_URL=https://pypi.tuna.tsinghua.edu.cn/simple .Run the container (environment variables are described in the next section):
docker run --rm -p 8000:8000 \
-e DOWNSTREAM_URL=http://<router>:<port> \
-e CONTINUATION_MODEL=<actual model> \
-e CONTINUATION_ENGINE=<sglang|vllm> \
continuation-gateway:latestWithout a container, run it directly with Python 3.12 (pip install -r requirements.txt).
The downstream is expected to be a datacenter/cluster router (not a specific SGLang instance directly). The original request and the continuation request go to the same URL by default:
DOWNSTREAM_URL=http://<router>:<port> CONTINUATION_MODEL=<actual model> \
CONTINUATION_ENGINE=<sglang|vllm> python -m continuation_gateway.serverCONTINUATION_MODEL is the model the downstream of this deployment actually serves (for
example kimi-k3, case-insensitive). It is not the model field of the client's request,
nor the inference engine's --served-model-name (which can be set to anything and need not
match the real model). The gateway assumes a deployment's downstream serves a single model;
this value decides how the continuation prefix is built and applies uniformly to every request
it receives — requests are no longer filtered by their model field. When it is not set, the
gateway triggers no continuation at all and passes every request through unchanged.
CONTINUATION_ENGINE is the inference engine behind the downstream, sglang (default) or
vllm. See the compatibility table at the top for which engine + model combinations are
verified.
Optional environment variables (defaults are in continuation_gateway/config.py): PORT,
CONTINUATION_URL (send the continuation request to a different address; if unset it is the
same as DOWNSTREAM_URL), CONTINUATION_ENABLED (default true; when set to false the
gateway still does all of leg1's monitoring, TRIGGERED detection and logging, but does not
actually issue the second leg — useful for verifying the monitoring path itself without adding
real continuation load to the downstream), STALL_IDLE_TIMEOUT_SECONDS,
CONNECT_TIMEOUT_SECONDS, MAX_CONTINUATION_BODY_MB, MAX_REQUEST_BODY_MB,
CJK_CHARS_PER_TOKEN, OTHER_CHARS_PER_TOKEN.
DERENDER_BATCH_CHUNKS (default 8, minimum 1) only applies to the combination whose
continuation leg uses render-generate (vLLM + Kimi-K3): that leg runs render, tokenize,
/inference/v1/generate and derender against CONTINUATION_URL, so the address must expose
all four endpoints (the vLLM server needs --enable-scale-out). Each derender call replays all
previously generated tokens, so the generated chunks are batched, DERENDER_BATCH_CHUNKS of
them per call; the remainder is flushed when the final chunk (the one carrying finish_reason)
arrives. A larger value means fewer derender calls but a more bursty stream.
BUFFER_TOOL_CALLS (default false; behavior is unchanged when off; it only takes effect
when CONTINUATION_ENABLED=true, see below): once a chunk carrying tool_calls appears on
leg1, that chunk and every line after it are held in gateway memory until finish_reason (or
[DONE]) arrives, and are then forwarded all at once. If the stream stalls or disconnects
while holding, the held content is discarded as a whole, and the continuation starts from the
thinking-partial / content-done state before the first tool_call chunk (the continuation leg
regenerates, and may produce a tool_call again). When there is no recoverable content, the
held content is released to the client unchanged. Cost: tool_call arguments no longer arrive
chunk by chunk but all at once after the tool_call completes.
CONTINUATION_ENABLED=true is a prerequisite for BUFFER_TOOL_CALLS to take effect: the only
purpose of holding is to keep continuation feasible. Once the continuation itself is turned
off, holding would buy no recovery and would only make the tool_call arguments reach the client
a round later for nothing, so with this combination the gateway skips holding and treats the
request as a plain pass-through, the same as when BUFFER_TOOL_CALLS is not enabled.
tests/smoke_test.py: does not connect to a real cluster. It starts a local fake downstream
(simulating SGLang's streaming responses) plus the real continuation_gateway service, and
uses real HTTP requests to verify the gateway's own forwarding and orchestration logic
(continuation triggered by stall/disconnect, scenarios such as tool_call / response_format /
oversized body being left alone, usage rewriting, and prefix reconstruction dispatched by
CONTINUATION_MODEL). It does not verify whether the continued content is semantically coherent — that has
to be verified on a real cluster.
python tests/smoke_test.pyThis project is open-sourced under the Apache License 2.0.