Skip to content

perf(dory): reuse rotary embeddings within each stage traversal - #199

Open
cecilsd wants to merge 1 commit into
InferMatrixLab:mainfrom
cecilsd:codex/dory-rope-reuse
Open

cecilsd wants to merge 1 commit into
InferMatrixLab:mainfrom
cecilsd:codex/dory-rope-reuse

Conversation

@cecilsd

@cecilsd cecilsd commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Required human contributor answers

The contributor explicitly confirmed these answers and the decision to open this PR before submission.

1. What outcome are you trying to achieve?

Reduce Dory inference overhead while preserving model outputs, loop behavior, and KV semantics.

2. What is the highest-leverage variable for that outcome?

The number of redundant rotary sine/cosine evaluations and GPU launches per stage traversal.

3. How will this work move that variable rather than create busywork?

Share one rotary embedding pair per attention type within a traversal. On the default two-loop model, this reduces 56 evaluations to 8. Matched BF16 measurements show improved throughput, with exact output and selected state/KV equivalence.

4. Who is the customer or user, and what do they actually need?

Users deploying Dory with vllm-rlt need lower inference latency and better throughput while retaining the checkpoint's numerical behavior and existing execution modes.

PR necessity and bugfix evidence

A PR is useful because redundant per-layer work is in the model implementation and cannot be removed through an existing deployment option. This is a performance optimization; no model-correctness bug is claimed.

Purpose

Dory currently recomputes rotary sine/cosine values in every attention sublayer, although layers of the same attention type use the same positions, head dimension, and rotary parameters during a traversal.

Compute each pair once in DoryBackbone and pass it to the attention sublayers. Reuse is local to PRELUDE, each recurrent invocation, and CODA; full and sliding attention retain separate frequency parameters. There is no cross-request or cross-replay result cache.

For the default two-loop checkpoint, the count changes from 8 + 20 + 20 + 8 = 56 to 2 + 2 + 2 + 2 = 8. Q/K rotation still runs in every attention layer. The FP32 angle calculation, dtype conversions, attention/KV behavior, parameter names, checkpoint loading, sampling, and loop policy are preserved.

A full-context lookup table would add persistent storage and startup work. Cross-stage sharing would require carrying values through independently scheduled batches. Traversal-local reuse removes the measured redundancy with a smaller ownership change. No new flag or usage change is required.

Test Plan

vllm-rlt Version / Commit: baseline bed34006ea8dc5b6ba1a3abd99a9fd81bd91a31b; candidate 99509928f237d2e011d2f4b4181f74cbf07d0a27.

  • CPU regressions: repeated full/sliding layers in all three stages, changing positions, FP32/BF16, per-loop and last-loop KV layouts, and exact logits/hidden/KV equality against per-layer recomputation. The new computation-count regression fails on the original implementation.
  • GPU regressions: synchronous/asynchronous eager and CUDA Graph execution, changing positions, actual graph replays, and greedy outputs against an independent dense reference.
  • Matched official Dory BF16 checkpoint on one RTX 5090, using Triton, fixed KV capacity, fixed loop depth, identical prompts and generation controls.
  • Three warmed unprofiled rounds per arm, interleaved A/B, B/A, A/B; independent ops-only and full diagnostic profiles.

Test Result

  • Linux full CPU collection: 1225 passed, 873 skipped, including 834 GPU tests.
  • Related supported GPU selection: 201 passed, 15 deselected.
  • Expanded GPU selection reproduces the same 15 failures on untouched main and the candidate: 12 explicit FlashAttention configurations rejected because the pinned Paged FlashAttention-4 release does not support SM120; 3 existing test fixtures missing key_scale. This is not a full-GPU-suite pass.
  • Latest Dory/speculative/shared-RoPE CPU selection: 122 passed.
  • Other-model/shared-RoPE selection on Windows: both baseline and candidate have 98 passed, 18 skipped, 1 identical Huginn tolerance failure (one of 320 logits, absolute difference 7.2494e-6). No tolerance was changed.
  • Ruff lint, Ruff format, and git diff --check pass. pre-commit run --all-files also passes all three hooks on Linux. Windows full-suite execution does not pass with its local dependency/runtime differences; the Linux run is the submission check.

Real-checkpoint correctness

Both arms use NVIDIA Dory 4B BF16, revision ee517dd388a9b348890ae1e00dacc5eb4d0c750d, with the weight file verified by SHA256.

  • Loops 1/2/4 across sync/async and eager/graph: all 36 matched requests / 432 generated token IDs agree, including exit depths and finish reasons.
  • Performance runs: all 90 matched requests / 2880 generated token IDs agree.
  • Teacher-forced logits, stage hidden states, and populated boundary/recurrent K/V: 8,294,400 tensor elements exactly equal across loops 1/2/4.
  • These are equivalence checks on the tested inputs, not a general task-quality score.

Warmed engine-side performance

One RTX 5090; full BF16 weights in VRAM; PyTorch 2.11.0+cu128, CUDA 12.8, Triton 3.6.0. Two recurrent loops, 768 KV blocks of 16 tokens, max sequences 4, token budget/prefill chunk 128, greedy generation, 32 output tokens/request, ignore EOS. Text is repeated/truncated to exact input lengths. All requests in a batch arrive together.

Values are medians of three unprofiled runs. Throughput is generated output tokens divided by total engine drain time. p95 is the median of per-run p95 intervals between actual per-token engine outputs; tail samples are small and descriptive. TTFT is engine-side and excludes networking. Loading, compilation/capture, and warmup are excluded.

Mode Input tokens / requests Output tokens/s: main → change Change p95 token interval ms: main → change
sync_eager 128 / 1 18.81 → 19.74 +4.90% 54.29 → 52.46
sync_eager 2048 / 1 12.12 → 12.57 +3.73% 52.87 → 50.63
sync_eager 128 / 4 50.67 → 53.17 +4.94% 71.67 → 69.56
sync_graph 128 / 1 50.74 → 51.86 +2.22% 18.42 → 18.05
sync_graph 2048 / 1 19.16 → 19.59 +2.21% 21.11 → 20.65
sync_graph 128 / 4 110.38 → 113.14 +2.50% 28.74 → 28.38
async_eager 128 / 1 19.01 → 19.92 +4.75% 52.84 → 50.05
async_eager 2048 / 1 11.91 → 12.58 +5.61% 54.02 → 50.12
async_eager 128 / 4 51.18 → 53.38 +4.30% 70.05 → 67.29
async_graph 128 / 1 53.22 → 54.49 +2.38% 17.46 → 17.12
async_graph 2048 / 1 19.50 → 19.95 +2.30% 20.16 → 19.72
async_graph 128 / 4 138.16 → 141.56 +2.46% 23.55 → 23.05
sync_packed 128 / 1 18.81 → 19.66 +4.54% 53.03 → 50.91
sync_packed 2048 / 1 12.85 → 13.46 +4.69% 52.75 → 50.70
sync_packed 128 / 4 51.00 → 53.29 +4.49% 72.28 → 67.90

All 15 three-run throughput ranges are separated in this sample. Gains are specific to the tested model, GPU, and workloads; no universal speedup is claimed. Both prefill and decode benefit.

Peak allocated memory increases by approximately 0.12–0.25 MiB because reused sin/cos tensors stay live until the end of the traversal. No persistent full-context table is allocated.

Profiling and qualification limits

A warmed real-weight recurrent traversal reduces RoPE evaluations from 20 to 2 and CUDA kernel launches from 2877 to 2751. Separate ops-only and full captures include CPU and CUDA activity; the full captures contain shapes, memory events, and Python source/parent call events. Folded export_stacks files are empty in this runtime.

Additional complete-request profiles use the same BF16 model and engine settings with 128 input / 4 output tokens to keep diagnostic captures bounded. Both arms complete successfully with identical token IDs, depths, and finish reasons. They include prefill through request completion, independent ops-only/full captures, matching CPU/runtime/GPU stream tracks, and equal-scale timelines. This shorter diagnostic is not included in the 32-output unprofiled performance table.

Complete warmed request timeline

Matching first-decode zoom

Diagnostic phase RoPE evaluations: main → change CUDA kernels: main → change GPU kernel-time sum: main → change
Prefill through first token 64 → 10 9421 → 9043 31.49 → 31.17 ms
Remaining decode 168 → 24 24186 → 23178 52.16 → 51.09 ms

These engine schedule counts include the work performed by complete request execution; they are distinct from the per-token 56 → 8 count. Runtime synchronization counts (1 prefill / 18 decode) and memcpy counts (4 / 33) remain unchanged. The trace supports removal of repeated short operators and host launches; non-kernel timeline gaps are not automatically eliminable work. Profiled durations are diagnostic, not production speedup measurements.

Reproduction, controls, limitations, and evidence index · Sanitized traces, raw timing/correctness JSON, scripts, and test logs

Published traces use relative timestamps, remapped process/thread IDs, and allow-listed diagnostic fields; private host paths, identities, concrete tensor inputs, and memory addresses are removed. Shape, memory-byte, operator/correlation, and Python call/parent events remain available. Both temporary experiment directories were removed and absence verified after retrieval. No task-accuracy score or speedup outside the tested matrix is claimed.

Current real-weight tests do not qualify other GPUs/backends, FP16 GPU inference, real-weight self-speculation, or every KV layout. Other model implementations and shared RoPE helpers are unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant