Skip to content

feat(dory): add asynchronous self-speculation and speculative CUDA Graphs - #197

Open
XinhaoTheo wants to merge 5 commits into
InferMatrixLab:mainfrom
XinhaoTheo:feat/rfc176-dory-async-speculation
Open

XinhaoTheo wants to merge 5 commits into
InferMatrixLab:mainfrom
XinhaoTheo:feat/rfc176-dory-async-speculation

Conversation

@XinhaoTheo

@XinhaoTheo XinhaoTheo commented Nov 18, 2026 •

Copy link
Copy Markdown

Required human contributor answers

1. What outcome are you trying to achieve?
Dory self-speculation that runs with asynchronous scheduling and CUDA Graphs. This is item 3B of RFC #176.

2. What is the highest-leverage variable for that outcome?
Host-side cost per committed token. Eager Dory is bound by kernel launches, so the draft and verify steps must run as graph replays without a host read.

3. How will this work move that variable rather than create busywork?
PRELUDE and CODA now replay bucketed graphs with device-owned positions, and the async pipeline updates all requests in one batch. At 1 to 16 concurrent requests, the async Graph pipeline is 10–29 % faster than ordinary async Graph decode. At 64 requests the gain is 3 %. Tokens and GSM8K accuracy do not change.

4. Who is the customer or user, and what do they actually need?
People who serve Dory with this runtime at low or medium concurrency and want fast greedy decode. Also the owner of 3C, which needs async rounds for its async qualification.

PR necessity
A PR is necessary. Before this change, the engine rejected Dory self-speculation with async scheduling or CUDA Graphs. This is a feature, not a bugfix.

Modification

  • Async rounds give PRELUDE and CODA depth-0 device KV metadata. A pipeline step runs both stages on all rows. Write masks keep the KV of rows that do not use a stage.
  • DeviceBoundaryGraphs replays PRELUDE and CODA in the same buckets as the recurrent graphs. Positions, write masks and context lengths are graph inputs.
  • Sync Graph rounds use the KV-aware PreludeGraphs and CodaGraphs from 2D.
  • The async pipeline keeps the state of all requests in one slot pool. A step does one gather, one update and one scatter. Before this change, a step used about 35 operations per request, and async eager was up to 27 % slower than sync eager at 64 requests. This change also applies to Ouro.
  • The engine accepts async and Graph modes for Dory. Sampled async requests and preemption stay rejected.

Out of scope: sampled async rounds, preemption (3C), last_loop KV, HTTP serving and open-loop arrivals.

Related PR

Accuracy Test

GPU tests passed on one H100 (torch 2.15.0+cu132, Triton). The code at a6e2fe2 is the code that was tested.

  • tests/test_dory_speculative_gpu.py (97 cases): async eager, async Graph and sync Graph against the dense oracle; 1→2, 1→4 and 2→4; rejection at each draft position; the KV of rows in flight; cancellation and failure cleanup.
  • The existing async/sync speculation, Dory and CUDA Graph tests.
  • Real checkpoint, FP32, 32 requests × 512 prompt tokens + 128 generated tokens, at 1→2, 1→4 and 2→4: all 7 speculative modes give tokens identical to sync decode without speculation.
  • GSM8K, BF16, first 200 test questions, 8-shot: baseline 161/200, async Graph pipeline 164/200, async Graph block 163/200.

Benchmark and Profiling

Real checkpoint, BF16, 2 loops, a one-loop draft, K = 3, greedy, WikiText-103 prompts. All requests are sent at the same time. Each arm runs one warmup, then five timed repetitions, all in one H100 container. Wall times are the mean whole-run time in seconds, prefill included. Every Graph arm uses cuda_graph_max_graphs=128, so no arm runs eager because its graph cache is full; every Graph arm had zero graph fallbacks. BF16 acceptance is 0.85–0.88.

Table 1: With CUDA Graphs, concurrency sweep. Prompt 1024 tokens, 128 generated tokens for each request. Percentages are against the baseline.

Requests Baseline (async Graph, no spec) Baseline, default graph cache Sync Graph pipeline Async Graph pipeline Async Graph block
1 2.50 2.49 (−0.1 %) 1.83 (−27 %) 1.78 (−29 %) 1.91 (−23 %)
4 3.17 3.18 (+0.4 %) 2.66 (−16 %) 2.63 (−17 %) 2.95 (−7 %)
16 5.89 5.90 (+0.1 %) 5.51 (−6 %) 5.30 (−10 %) 5.63 (−5 %)
64 15.60 25.72 (+65 %) 16.00 (+3 %) 15.10 (−3 %) 16.45 (+5 %)

Table 2: Latency with CUDA Graphs. Same workload. TTFT is the time to the first token; TPOT is the time per output token after the first token. Percentages are against the baseline at the same concurrency.

Requests Arm TTFT p50 (s) TTFT p90 (s) TPOT p50 (ms) TPOT p90 (ms)
1 Baseline (async Graph, no spec) 0.18 0.18 18.3 18.3
1 Async Graph pipeline 0.17 (−2 %) 0.20 (+13 %) 12.6 (−31 %) 12.6 (−31 %)
1 Async Graph block 0.17 (−1 %) 0.19 (+7 %) 13.7 (−25 %) 13.7 (−25 %)
4 Baseline (async Graph, no spec) 0.67 0.67 20.0 20.0
4 Async Graph pipeline 0.66 (−1 %) 0.67 (−0.8 %) 16.6 (−17 %) 17.4 (−13 %)
4 Async Graph block 0.69 (+3 %) 0.69 (+3 %) 18.1 (−10 %) 19.0 (−5 %)
16 Baseline (async Graph, no spec) 2.01 2.75 35.2 40.9
16 Async Graph pipeline 1.68 (−17 %) 2.70 (−2 %) 28.8 (−18 %) 36.1 (−12 %)
16 Async Graph block 1.79 (−11 %) 2.93 (+6 %) 30.1 (−15 %) 36.1 (−12 %)
64 Baseline (async Graph, no spec) 6.21 10.58 76.6 108.5
64 Async Graph pipeline 5.78 (−7 %) 10.02 (−5 %) 70.9 (−7 %) 102.5 (−5 %)
64 Async Graph block 6.43 (+3 %) 11.54 (+9 %) 68.1 (−11 %) 96.4 (−11 %)

Table 3: Without CUDA Graphs (eager), sync against async. Same workload. There is no eager baseline without speculation, so each async percentage is against the sync value of the same mode.

Requests Pipeline sync Pipeline async Block sync Block async
1 6.80 6.42 (−6 %) 6.83 7.16 (+5 %)
4 7.80 8.24 (+6 %) 9.64 9.71 (+0.8 %)
16 10.74 10.57 (−2 %) 13.24 13.25 (+0.1 %)
64 21.22 20.49 (−3 %) 28.02 27.29 (−3 %)

Table 4: Long context. 8 requests, prompt 8192 tokens, 128 generated tokens for each request.

Arm Wall (s) TTFT p50 (s) TTFT p90 (s) TPOT p50 (ms) TPOT p90 (ms)
Baseline (async Graph, no spec) 23.73 18.60 20.71 53.2 68.9
Sync Graph pipeline 23.52 (−0.9 %) 17.85 (−4 %) 20.73 (+0.1 %) 51.2 (−4 %) 72.5 (+5 %)
Async Graph pipeline 23.18 (−2 %) 17.69 (−5 %) 20.58 (−0.6 %) 49.4 (−7 %) 70.3 (+2 %)
Async Graph block 23.13 (−3 %) 17.60 (−5 %) 20.42 (−1 %) 49.9 (−6 %) 67.9 (−1 %)
Sync eager pipeline 28.83 18.04 21.08 86.1 111.4
Async eager pipeline 28.52 (−1 % vs sync) 18.10 (+0.3 % vs sync) 21.15 (+0.3 % vs sync) 87.2 (+1 % vs sync) 108.0 (−3 % vs sync)

Reading the numbers:

  • The gain of speculation falls as concurrency rises. At one request the GPU has spare capacity, so draft and verify work is almost free. At 64 requests the GPU is busy, and the extra verify rows cancel most of the saved steps.
  • At 8 × 8192, prefill takes about 18 of 24 seconds. Speculation shortens only decode, so the whole run is 2–3 % faster.
  • The baseline with the default graph cache (16 shapes) runs eager in most calls at 64 requests and takes 25.72 s instead of 15.60 s. This is a property of ordinary async Graph decode, not of this PR. Comparisons against that baseline overstate the gain of speculation.

Known issues

1. Async Graph block is slower than the baseline at 64 requests (+5 %). Its time per output token is 11 % lower, but its time to the first token is 3–9 % higher, so the whole run is longer. The async Graph pipeline is still 3 % faster at 64 requests. At high concurrency, the pipeline mode is the better choice.

2. Eager async rounds at low concurrency. Without CUDA Graphs, async against sync varies from −6 % to +6 % at 1 and 4 requests (Table 3). A dedicated run kept a sync and an async engine alive and alternated them in ABBA order, 10 timed repetitions each, at 1 request in pipeline mode:

Container Sync Async Difference Rounds with async slower
1 5.34 s (5.26–5.44) 5.49 s (5.38–5.68) +2.7 % 9 of 10
2 6.27 s (5.80–7.18) 6.51 s (6.05–7.01) +3.9 % 6 of 10

The profiler counts 5503 kernel launches per generated token for sync and 5626 for async (+124, +2.2 %), the same in both containers. Eager Dory is bound by host-side launch cost, so the extra launches give the extra time. Async rounds keep positions on the GPU. Thus the GPU computes the KV metadata, the PRELUDE and CODA write masks, and the slot-pool gather, update and scatter. Sync rounds do this work on the CPU, without kernel launches. At 16 and 64 requests, async eager is 0–3 % faster than sync eager.

Options, for a maintainer decision:

  1. Accept the gap. Use async rounds with CUDA Graphs; eager low-concurrency decode is not a target configuration.
  2. Small change in this PR: PRELUDE and CODA share one depth-0 metadata and differ only in the write mask, and the mask operations merge. Estimate: 20–30 fewer launches per token, about a quarter of the gap.
  3. Separate PR: fuse the per-step state update and metadata into one or two Triton kernels. This changes the shared async runtime, so Ouro also needs validation.

Conclusion

Dory self-speculation now runs async and with CUDA Graphs, and tokens and GSM8K accuracy do not change. With CUDA Graphs, the async pipeline is 10–29 % faster at 1 to 16 requests and 3 % faster at 64 requests; block mode is 5 % slower at 64 requests. The Known issues section gives the data and the open decisions.

  • Async eager rounds with boundary KV
  • Async and sync speculative CUDA Graphs
  • Batched async pipeline step
  • GPU tests on H100
  • Real-checkpoint FP32 token agreement at 1→2, 1→4 and 2→4
  • GSM8K accuracy, BF16
  • Real-checkpoint BF16 benchmark: concurrency sweep, long context, TTFT and TPOT
  • Maintainer decision on eager async rounds at low concurrency

Asynchronous block and pipeline rounds now give PRELUDE and CODA their
depth-0 KV. The positions of these rows exist only on the device. Thus the
runner builds device metadata for the boundary stages, as it does for the
recurrent loops.

A pipeline step runs both boundary stages on all rows. Write masks keep the
input-stage KV of retained rows and the output-stage KV of rows between the
draft and target depths. Each step stores the same KV as a synchronous step.

The engine accepts asynchronous eager rounds for boundary-KV models. CUDA
Graphs, preemption and sampled requests stay rejected.
Asynchronous rounds capture bucketed PRELUDE and CODA graphs
(DeviceBoundaryGraphs) next to the recurrent graphs. Positions, write
addresses, write masks and context lengths are graph inputs. Thus the graphs
keep the device-owned progress of the rounds. The engine captures all buckets
at construction.

Synchronous rounds of a boundary-KV model use the host-planned boundary
graphs of ordinary decode (PreludeGraphs, and CodaGraphs with the cache). The
previous synchronous CODA graph had no KV input, and Dory cannot run it.

The engine accepts CUDA Graphs for boundary-KV self-speculation. Preemption
stays rejected.
The asynchronous pipeline step updated the device state of each request with
its own small operations, about 35 kernel launches for each request in each
step. At 64 requests, these launches made asynchronous eager rounds 27 %
slower than synchronous eager rounds. They also ran outside the CUDA Graphs.

The pipeline state of all live requests is now one pool, indexed by the slot
table that block rounds already use. A step gathers the rows of its requests,
updates them in one batch and scatters them back. Thus the number of
operations does not change with the request count. The rows, their order and
the results do not change. Preemption snapshots and restoration use the pool
slot of the request.
Describe how asynchronous rounds give PRELUDE and CODA their depth-0 KV, the
pipeline write masks, the sampling limit, the shared pipeline slot pool and
the graph families of each mode.

Add the validation status: the CUDA test file, the real-checkpoint FP32 token
agreement at 1 to 2, 1 to 4 and 2 to 4 loops, GSM8K accuracy, and BF16 wall
time, first-token time and time per output token on one H100.
@XinhaoTheo
XinhaoTheo force-pushed the feat/rfc176-dory-async-speculation branch from 8fbe505 to a6e2fe2 Compare November 18, 2026 04:55
@XinhaoTheo
XinhaoTheo marked this pull request as ready for review November 18, 2026 06:44
The previous wall times compared speculation with ordinary async Graph decode
that used the default graph cache of 16 shapes. At 64 requests that baseline
ran eager in most calls, so the reported gain was too large.

Replace the results with a concurrency sweep at a fixed prompt length (1, 4,
16 and 64 requests) and a separate long-context run. Every Graph arm now
captures up to 128 shapes and has no graph fallback. Add the time to the first
token and the time per output token, the GSM8K accuracy, and the known issues:
async Graph block at 64 requests and eager async rounds at low concurrency.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant