Repository navigation
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Required human contributor answers
The contributor explicitly confirmed these answers and the decision to open this PR before submission.
1. What outcome are you trying to achieve?
Reduce Dory inference overhead while preserving model outputs, loop behavior, and KV semantics.
2. What is the highest-leverage variable for that outcome?
The number of redundant rotary sine/cosine evaluations and GPU launches per stage traversal.
3. How will this work move that variable rather than create busywork?
Share one rotary embedding pair per attention type within a traversal. On the default two-loop model, this reduces 56 evaluations to 8. Matched BF16 measurements show improved throughput, with exact output and selected state/KV equivalence.
4. Who is the customer or user, and what do they actually need?
Users deploying Dory with vllm-rlt need lower inference latency and better throughput while retaining the checkpoint's numerical behavior and existing execution modes.
PR necessity and bugfix evidence
A PR is useful because redundant per-layer work is in the model implementation and cannot be removed through an existing deployment option. This is a performance optimization; no model-correctness bug is claimed.
Purpose
Dory currently recomputes rotary sine/cosine values in every attention sublayer, although layers of the same attention type use the same positions, head dimension, and rotary parameters during a traversal.
Compute each pair once in DoryBackbone and pass it to the attention sublayers. Reuse is local to PRELUDE, each recurrent invocation, and CODA; full and sliding attention retain separate frequency parameters. There is no cross-request or cross-replay result cache.
For the default two-loop checkpoint, the count changes from
8 + 20 + 20 + 8 = 56to2 + 2 + 2 + 2 = 8. Q/K rotation still runs in every attention layer. The FP32 angle calculation, dtype conversions, attention/KV behavior, parameter names, checkpoint loading, sampling, and loop policy are preserved.A full-context lookup table would add persistent storage and startup work. Cross-stage sharing would require carrying values through independently scheduled batches. Traversal-local reuse removes the measured redundancy with a smaller ownership change. No new flag or usage change is required.
Test Plan
vllm-rlt Version / Commit: baseline
bed34006ea8dc5b6ba1a3abd99a9fd81bd91a31b; candidate99509928f237d2e011d2f4b4181f74cbf07d0a27.Test Result
key_scale. This is not a full-GPU-suite pass.7.2494e-6). No tolerance was changed.git diff --checkpass.pre-commit run --all-filesalso passes all three hooks on Linux. Windows full-suite execution does not pass with its local dependency/runtime differences; the Linux run is the submission check.Real-checkpoint correctness
Both arms use NVIDIA Dory 4B BF16, revision
ee517dd388a9b348890ae1e00dacc5eb4d0c750d, with the weight file verified by SHA256.Warmed engine-side performance
One RTX 5090; full BF16 weights in VRAM; PyTorch 2.11.0+cu128, CUDA 12.8, Triton 3.6.0. Two recurrent loops, 768 KV blocks of 16 tokens, max sequences 4, token budget/prefill chunk 128, greedy generation, 32 output tokens/request, ignore EOS. Text is repeated/truncated to exact input lengths. All requests in a batch arrive together.
Values are medians of three unprofiled runs. Throughput is generated output tokens divided by total engine drain time. p95 is the median of per-run p95 intervals between actual per-token engine outputs; tail samples are small and descriptive. TTFT is engine-side and excludes networking. Loading, compilation/capture, and warmup are excluded.
All 15 three-run throughput ranges are separated in this sample. Gains are specific to the tested model, GPU, and workloads; no universal speedup is claimed. Both prefill and decode benefit.
Peak allocated memory increases by approximately 0.12–0.25 MiB because reused sin/cos tensors stay live until the end of the traversal. No persistent full-context table is allocated.
Profiling and qualification limits
A warmed real-weight recurrent traversal reduces RoPE evaluations from 20 to 2 and CUDA kernel launches from 2877 to 2751. Separate ops-only and full captures include CPU and CUDA activity; the full captures contain shapes, memory events, and Python source/parent call events. Folded
export_stacksfiles are empty in this runtime.Additional complete-request profiles use the same BF16 model and engine settings with 128 input / 4 output tokens to keep diagnostic captures bounded. Both arms complete successfully with identical token IDs, depths, and finish reasons. They include prefill through request completion, independent ops-only/full captures, matching CPU/runtime/GPU stream tracks, and equal-scale timelines. This shorter diagnostic is not included in the 32-output unprofiled performance table.
These engine schedule counts include the work performed by complete request execution; they are distinct from the per-token 56 → 8 count. Runtime synchronization counts (1 prefill / 18 decode) and memcpy counts (4 / 33) remain unchanged. The trace supports removal of repeated short operators and host launches; non-kernel timeline gaps are not automatically eliminable work. Profiled durations are diagnostic, not production speedup measurements.
Reproduction, controls, limitations, and evidence index · Sanitized traces, raw timing/correctness JSON, scripts, and test logs
Published traces use relative timestamps, remapped process/thread IDs, and allow-listed diagnostic fields; private host paths, identities, concrete tensor inputs, and memory addresses are removed. Shape, memory-byte, operator/correlation, and Python call/parent events remain available. Both temporary experiment directories were removed and absence verified after retrieval. No task-accuracy score or speedup outside the tested matrix is claimed.
Current real-weight tests do not qualify other GPUs/backends, FP16 GPU inference, real-weight self-speculation, or every KV layout. Other model implementations and shared RoPE helpers are unchanged.