Repository navigation
feat(dory): add asynchronous self-speculation and speculative CUDA Graphs - #197
Open
XinhaoTheo wants to merge 5 commits into
Open
XinhaoTheo wants to merge 5 commits into
XinhaoTheo wants to merge 5 commits into
Conversation
Asynchronous block and pipeline rounds now give PRELUDE and CODA their depth-0 KV. The positions of these rows exist only on the device. Thus the runner builds device metadata for the boundary stages, as it does for the recurrent loops. A pipeline step runs both boundary stages on all rows. Write masks keep the input-stage KV of retained rows and the output-stage KV of rows between the draft and target depths. Each step stores the same KV as a synchronous step. The engine accepts asynchronous eager rounds for boundary-KV models. CUDA Graphs, preemption and sampled requests stay rejected.
Asynchronous rounds capture bucketed PRELUDE and CODA graphs (DeviceBoundaryGraphs) next to the recurrent graphs. Positions, write addresses, write masks and context lengths are graph inputs. Thus the graphs keep the device-owned progress of the rounds. The engine captures all buckets at construction. Synchronous rounds of a boundary-KV model use the host-planned boundary graphs of ordinary decode (PreludeGraphs, and CodaGraphs with the cache). The previous synchronous CODA graph had no KV input, and Dory cannot run it. The engine accepts CUDA Graphs for boundary-KV self-speculation. Preemption stays rejected.
The asynchronous pipeline step updated the device state of each request with its own small operations, about 35 kernel launches for each request in each step. At 64 requests, these launches made asynchronous eager rounds 27 % slower than synchronous eager rounds. They also ran outside the CUDA Graphs. The pipeline state of all live requests is now one pool, indexed by the slot table that block rounds already use. A step gathers the rows of its requests, updates them in one batch and scatters them back. Thus the number of operations does not change with the request count. The rows, their order and the results do not change. Preemption snapshots and restoration use the pool slot of the request.
Describe how asynchronous rounds give PRELUDE and CODA their depth-0 KV, the pipeline write masks, the sampling limit, the shared pipeline slot pool and the graph families of each mode. Add the validation status: the CUDA test file, the real-checkpoint FP32 token agreement at 1 to 2, 1 to 4 and 2 to 4 loops, GSM8K accuracy, and BF16 wall time, first-token time and time per output token on one H100.
XinhaoTheo
force-pushed
the
feat/rfc176-dory-async-speculation
branch
from
November 18, 2026 04:55
8fbe505 to
a6e2fe2
Compare
XinhaoTheo
marked this pull request as ready for review
November 18, 2026 06:44
Open
1 of 2 tasks
The previous wall times compared speculation with ordinary async Graph decode that used the default graph cache of 16 shapes. At 64 requests that baseline ran eager in most calls, so the reported gain was too large. Replace the results with a concurrency sweep at a fixed prompt length (1, 4, 16 and 64 requests) and a separate long-context run. Every Graph arm now captures up to 128 shapes and has no graph fallback. Add the time to the first token and the time per output token, the GSM8K accuracy, and the known issues: async Graph block at 64 requests and eager async rounds at low concurrency.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Required human contributor answers
1. What outcome are you trying to achieve?
Dory self-speculation that runs with asynchronous scheduling and CUDA Graphs. This is item 3B of RFC #176.
2. What is the highest-leverage variable for that outcome?
Host-side cost per committed token. Eager Dory is bound by kernel launches, so the draft and verify steps must run as graph replays without a host read.
3. How will this work move that variable rather than create busywork?
PRELUDE and CODA now replay bucketed graphs with device-owned positions, and the async pipeline updates all requests in one batch. At 1 to 16 concurrent requests, the async Graph pipeline is 10–29 % faster than ordinary async Graph decode. At 64 requests the gain is 3 %. Tokens and GSM8K accuracy do not change.
4. Who is the customer or user, and what do they actually need?
People who serve Dory with this runtime at low or medium concurrency and want fast greedy decode. Also the owner of 3C, which needs async rounds for its async qualification.
PR necessity
A PR is necessary. Before this change, the engine rejected Dory self-speculation with async scheduling or CUDA Graphs. This is a feature, not a bugfix.
Modification
DeviceBoundaryGraphsreplays PRELUDE and CODA in the same buckets as the recurrent graphs. Positions, write masks and context lengths are graph inputs.PreludeGraphsandCodaGraphsfrom 2D.Out of scope: sampled async rounds, preemption (3C),
last_loopKV, HTTP serving and open-loop arrivals.Related PR
async_speculative.pyandcuda_graph.py. The merge order must be agreed.Accuracy Test
GPU tests passed on one H100 (torch 2.15.0+cu132, Triton). The code at
a6e2fe2is the code that was tested.tests/test_dory_speculative_gpu.py(97 cases): async eager, async Graph and sync Graph against the dense oracle; 1→2, 1→4 and 2→4; rejection at each draft position; the KV of rows in flight; cancellation and failure cleanup.Benchmark and Profiling
Real checkpoint, BF16, 2 loops, a one-loop draft, K = 3, greedy, WikiText-103 prompts. All requests are sent at the same time. Each arm runs one warmup, then five timed repetitions, all in one H100 container. Wall times are the mean whole-run time in seconds, prefill included. Every Graph arm uses
cuda_graph_max_graphs=128, so no arm runs eager because its graph cache is full; every Graph arm had zero graph fallbacks. BF16 acceptance is 0.85–0.88.Table 1: With CUDA Graphs, concurrency sweep. Prompt 1024 tokens, 128 generated tokens for each request. Percentages are against the baseline.
Table 2: Latency with CUDA Graphs. Same workload. TTFT is the time to the first token; TPOT is the time per output token after the first token. Percentages are against the baseline at the same concurrency.
Table 3: Without CUDA Graphs (eager), sync against async. Same workload. There is no eager baseline without speculation, so each async percentage is against the sync value of the same mode.
Table 4: Long context. 8 requests, prompt 8192 tokens, 128 generated tokens for each request.
Reading the numbers:
Known issues
1. Async Graph block is slower than the baseline at 64 requests (+5 %). Its time per output token is 11 % lower, but its time to the first token is 3–9 % higher, so the whole run is longer. The async Graph pipeline is still 3 % faster at 64 requests. At high concurrency, the pipeline mode is the better choice.
2. Eager async rounds at low concurrency. Without CUDA Graphs, async against sync varies from −6 % to +6 % at 1 and 4 requests (Table 3). A dedicated run kept a sync and an async engine alive and alternated them in ABBA order, 10 timed repetitions each, at 1 request in pipeline mode:
The profiler counts 5503 kernel launches per generated token for sync and 5626 for async (+124, +2.2 %), the same in both containers. Eager Dory is bound by host-side launch cost, so the extra launches give the extra time. Async rounds keep positions on the GPU. Thus the GPU computes the KV metadata, the PRELUDE and CODA write masks, and the slot-pool gather, update and scatter. Sync rounds do this work on the CPU, without kernel launches. At 16 and 64 requests, async eager is 0–3 % faster than sync eager.
Options, for a maintainer decision:
Conclusion
Dory self-speculation now runs async and with CUDA Graphs, and tokens and GSM8K accuracy do not change. With CUDA Graphs, the async pipeline is 10–29 % faster at 1 to 16 requests and 3 % faster at 64 requests; block mode is 5 % slower at 64 requests. The Known issues section gives the data and the open decisions.