Repository navigation
Add DSv4 cudnn sparse attention kernel adapters to TE - #3566
sudhakarsingh27 wants to merge 41 commits into
Conversation
DSv4 HCA and CSA need joint attention over local and compressed KV rows, while CSA also selects compressed rows. Expose the cuDNN Frontend compressor, selector, and sparse attention operations through a small parameter-free TE module so model-owned projections and positional handling remain reusable. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Build the BSD-to-BSD CSA/HCA layer over the existing cuDNN-backed compressor, indexer, and attention operations so callers can supply hidden states directly. Keep the shared RoPE preparation and indexer in small modules, and cache FP32 frequencies across forwards. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Megatron trains its CSA indexer with a separate dense objective, while discrete top-k selection leaves the indexer without language-model gradients. Expose the live indexer tensors and narrow cuDNN score/backward calls so model code can retain ownership of KL reduction and loss attachment. Allow a caller RoPE module to match compressed YaRN without copying that policy into TE; preserve the default BSD return. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Keep the DSv4 API branch mergeable as attention exports evolve upstream. Preserve both the existing linear attention exports and the DSv4 sparse attention exports in the public package. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
The repository Black configuration reformats the newly added DSv4 files. Apply it so lint can assess the implementation without style-only failures; this commit changes no behavior. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
for more information, see https://pre-commit.ci
The PR Python check reports DSv4 docstring, literal, unused-argument, and local-initialization findings. Resolve them without changing the attention or indexer contracts so CI can validate the implementation. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Megatron passes sequence-major hidden states to attention. Accept and return that layout by default so the batch-one path can enter the existing packed-row core through a view without an extra public BSD round trip. Keep explicit BSD input for batch-major callers; larger batches still repack for the present core layout. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
|
A few initial comments - I know the PR is still in draft.
Thanks! |
The eager FP32 square/mean path materializes full-size query intermediates. At B4/S32768/BF16 on B200, weightless F.rms_norm lowers HCA and CSA TE peak allocated memory by about 45 GiB and cuts forward/backward time by 26-28%, while mapped-weight output and gradient cosines remain above 0.999999. Keep the unweighted normalization contract without adding a learned scale. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
cuDNN requires max_seqlen_q <= max_seqlen_k * ratio. Keep the compressed prefixes at the number of complete blocks, but pass a ceil-sized compile bound so non-divisible sequence lengths work. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Name each module after the implementation it owns, and merge the parameter-free DSv4Attention core with the high-level attention layer. Remove the redundant dsv4 wrapper and legacy alias while preserving tensor contracts, state layout, and execution order. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
The module adapts PyTorch and autograd calls to cuDNN DSA APIs rather than implementing kernels, so dsa_cudnn is the shorter and more accurate boundary name. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Python-side CuTe launches dominate the otherwise fast rotary kernels. Register shape-specialized CuTe callables through the common TVM-FFI cache so steady-state SM100 execution dispatches from C++ while unsupported inputs keep the eager fallback. Compile the weightless query normalization and let its fresh output own the in-place query rotation. This removes one full-width query copy without changing the public attention or state-dict contract; other RoPE sites retain out-of-place semantics required by their autograd boundaries. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
for more information, see https://pre-commit.ci
Follow the existing TE packed-attention contract at the high-level layer so callers can pass [T,D] hidden states with CUDA cumulative lengths. Build compressed prefixes on device to avoid a per-step host synchronization, and clear unused capacity rows before normalization because cuDNN only guarantees rows below the final compressed prefix. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Use the upstream forward/backward kernels from Megatron-LM c4129afff4867ca89b1fb7b4b911128424cdfbe8 while retaining TE tensor ownership and gradient semantics. Keeping the existing out-of-place KV/output and in-place fresh-query boundaries isolates the kernel-backend change and preserves autograd correctness. Remove the DSv4-specific CuTe implementation and native bridge so this branch has a single optimized RoPE backend. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Remove the parameter-free module that only stored window/ratio and delegated to the backend function. Call that function directly from the high-level layer and retain its FP8 rejection, giving DSv4Attention one clear hidden-state contract without an additional wrapper or compatibility alias. Prepared-tensor callers continue to use dsa_cudnn.attention. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Use qkv_format with sbhd, bshd, and thd as in MultiheadAttention and DotProductAttention. The names describe the attention layout while high-level hidden states omit the head axis. Preserve the same dense and packed computation without aliases for the retired spelling. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Name the prepared-tensor operation sparse_attention to distinguish it from the hidden-state layer. Update callers and tests together, and document the module hierarchy and CSA/HCA usage without repeating implementation detail. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Use a named launch-grid function and document the rotary autograd methods. Split packed-prefix type validation from tensor checks to satisfy the Boolean-expression limit while preserving the same validation and error message. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
5876d2c to
06013ce
Compare
[Medium impact] The PR adds a new attention path and changes shared RoPE code used by MLA callers. The PR still needs the learned-RoPE failure fixed before merging. Findings
SummaryAdds experimental
No new findings were accepted. Diagram%%{init: {'theme': 'neutral'}}%%
flowchart TD
X[Hidden states] --> Q[Query and local KV projections]
X --> C[Compressor]
X --> I[CSA Indexer]
Q --> R[RoPE]
C --> R
I --> K[Selected compressed indices]
R --> A[cuDNN sparse attention]
K --> A
A --> O[Inverse RoPE and output projection]
O --> Y[Hidden states]
I -. Live indexer tensors .-> L[Caller-owned auxiliary loss]
Reviews (9) · Last reviewed commit: "Merge branch 'main' into sudhakars/dsv4-..." · Reviewed by Greptile |
|
|
||
|
|
||
| @pytest.mark.parametrize("heads", [1, 64]) | ||
| def test_dsa_triton_rope_batched_forward_backward(monkeypatch, heads): |
There was a problem hiding this comment.
Have there been some tests for these kernels in #3033?
Same for test_dsa_rope_non_power_of_two_forward_backward.
There was a problem hiding this comment.
PR #3033 covers MLA RoPE through the MXFP8 attention path. The consolidated RoPE suite checks both rotary output layouts, ragged sequence boundaries, and DSA’s in-place and out-of-place paths. We removed some extraneous tests.
| ) | ||
|
|
||
|
|
||
| def test_dsa_packed_rope_eager_forward_backward(): |
There was a problem hiding this comment.
Can we move this closer to test_dsa_triton_rope_batched_forward_backward in the file, so we have 4-ish groups of tests, dsa_rope, compressor, indexer, dsv4_attention? (IF this makes sense)
There was a problem hiding this comment.
attention/rope.pycontains the DSA and MLA RoPE wrappers, with one shared Triton Q adapter.common/triton/dsa_mla_rope.pycontains the Triton kernels.test_fused_rope.pynow includes two DSA adapter cases, two packed Q-kernel layout cases, and the MLA KV forward/backward case.test_dsv4_attention.pygroups the Compressor, Indexer, sparse attention, and full-layer tests.test_fused_mla.pycovers fused MLA projection and quantization.
cyanguwa
left a comment
There was a problem hiding this comment.
Left a few comments - let me know what you think! Thanks!
Remove duplicated GPU computation while preserving distinct DSA interleaved and MLA split-half/KV adapters, inverse rotation and packed-position behavior. Keep framework dependencies out of the common layer and add independent layout, context-position and gradient regressions. Validation: 35 DSv4 and 21 focused MLA cases passed on each of SM103 and SM107, including the shared-kernel regressions. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Admit SM103 and SM107 after exercising compression, sparse attention and indexer training on the actual architectures. Retain BF16 activation checks because the compression and indexer dependencies do not support FP16. Centralize support and qualification requirements instead of repeating hardware restrictions throughout operation docstrings. Validation: 35 DSv4 and 21 focused MLA cases passed on each architecture; outputs, applicable gradients and checkpoint loading match the original implementation with only its architecture guard relaxed. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Remove the private high-level unfused comparison branch to avoid maintaining two checkpoint layouts. Preserve default fused parameter names and shapes, and keep standalone Compressor and Indexer projection options available. Validation: strict checkpoint loading, outputs and applicable gradients match the original fused path on SM103 and SM107. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Make the reusable classes discoverable without advertising implementation modules as package exports. Explain packed addressing, exact dense capacity, position bias and caller-owned indexer gradients so users can compose the operations without assuming that discrete selection trains the indexer. Rename the Compressor input to hidden_states without changing computation. Validation: syntax, formatter, changed-production pylint and whitespace checks pass; independent review checked related callers and contracts. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Include the sparse tests in the standard PyTorch launcher with its existing JUnit and failure handling. Group cases by component and remove redundant forward_backward suffixes while retaining the packed-layout distinction. Validation: AST comparison preserves all 13 test functions, assertions and 35 parametrized cases from the GPU-validated candidate. Shell syntax and pre-commit checks pass; this editorial change does not add a GPU run. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Apply fused projection composition consistently to Compressor and Indexer instead of retaining unused unfused branches and passing a constant flag. Remove the obsolete mode-specific rejection and update the capacity fixture to exercise the fused split. The prior standalone unfused parameter layout is intentionally retired; existing fused and high-level DSv4 parameter keys and row order are preserved. Validation: the existing Compressor CPU output/gradient regression passes with its fused fixture; pre-commit and changed-production pylint pass. A separate review checked related source, callers, documentation and tests. No fresh GPU integration run was performed for this branch simplification. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Describe query, local KV and compressed KV token indices explicitly so selection indices cannot be confused with indices into concatenated KV. Distinguish incomplete input blocks from packed output padding and retain the normalization cleanup rationale. Preserve physical weight/table rows. Only docstrings, comments, one error message and a private local variable name change. Executable AST equivalence, pre-commit, changed-production pylint and whitespace checks pass; no numerical behavior or public call contract changes. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Document existing operation and constructor contracts consistently, using explicit batched/packed shapes and cumulative token counts. Explain the TE output allocation separately from valid compressed tokens and illustrate local/compressed index conversion beside the actual implementation. Preserve user prose while resolving inline notes. Changes are documentation, comments and the clarified full-sequence shape-error text; numerical code and public call contracts are unchanged. Validation: pre-commit, pylint and executable AST checks pass. Twelve CUDA checks of actual source helpers and instrumented packing/padding/test boundaries match expected results. Numerical backends were intercepted; this is not a new native cuDNN attention validation. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
|
|
||
|
|
||
| @pytest.mark.skipif(not HAVE_TRITON, reason="MLA RoPE requires Triton") | ||
| def test_mla_rope_kv_equations_backward(): |
There was a problem hiding this comment.
test_mla_rope_kv compares K/V outputs and input gradients against independent PyTorch equations. It uses three heads in a four-head block to check that the gradients sum correctly into the positional key shared across heads.
| torch.testing.assert_close(query_grad, reference_grad, atol=0.03125, rtol=0.002) | ||
|
|
||
|
|
||
| def test_dsa_rope_non_power_of_two(monkeypatch): |
There was a problem hiding this comment.
Can we have a look at moving RoPE tests to mla_rope_utils.py and focus on sparse attention/compressor/indexer tests in this file?
There was a problem hiding this comment.
Moved the two DSA adapter cases, two packed Q-kernel layout cases, and the MLA KV forward/backward case to test_fused_rope.py. Test layout
|
|
||
|
|
||
| @triton.jit | ||
| def _mla_rope_bwd_inplace_kernel( |
There was a problem hiding this comment.
Rename these two to rotary_fwd/bwd_q_kernel?
There was a problem hiding this comment.
Renamed the entry points to rotary_fwd_q_kernel and rotary_bwd_q_kernel. How DSv4 and MLA use them
| ) | ||
|
|
||
|
|
||
| def _apply_rotary_eager(x, cos, sin, cu_seqlens=None): |
There was a problem hiding this comment.
Can we check if rotary_embeddings or _apply_rotary_eager implementations already exist in rope.py? If not, we can keep these (and possibly rename them to yarn_rope); if so, we can remove the two functions here. Thanks.
There was a problem hiding this comment.
The two paths use the same rotation math. The existing API takes angles and rotates the leading channels. DSA takes precomputed cosine/sine tables, rotates the trailing interleaved channels, and handles packed compressed positions. I’m keeping those adapters separate for this PR.
| return _TritonQueryRotary.apply(query, _rope_tables(cos), _rope_tables(sin), cu_seqlens, module) | ||
|
|
||
|
|
||
| def apply_rotary(x, cos, sin, cu_seqlens=None): |
There was a problem hiding this comment.
I renamed the helper to apply_dsa_rope. It applies the supplied cosine/sine tables, including YaRN tables when the caller provides them.
| restore_value=["DO"], | ||
| ) | ||
| @triton.jit | ||
| def rotary_bwd_q_kernel( |
There was a problem hiding this comment.
Do we need to remove rotary_fwd_q_kernel from this file?
There was a problem hiding this comment.
MLA’s local rotary_bwd_q_kernel was removed because MLA and DSv4 now use the same Triton RoPE backward implementation. MLA’s fused query projection still applies RoPE in cuDNN during forward.
DSv4 uses the Triton Q kernel pair for both Q and KV. The separate MLA KV kernels are called through the MLA wrapper in attention/rope.py, whose current callers are tests. They reuse one positional key per token across all attention heads and sum its gradients across heads in backward.
| __all__ = [ | ||
| "DSv4Attention", | ||
| "Compressor", | ||
| "Indexer", |
There was a problem hiding this comment.
Can we bring dense_indexer_loss_scores and dense_indexer_loss_backward to the same level as these three APIs so people can easily access them? Also, can add the loss ones in PR description.
There was a problem hiding this comment.
dense_indexer_loss_scores and dense_indexer_loss_backward are now importable from transformer_engine.pytorch alongside the three sparse attention classes. Both are listed in the API reference and PR description.
Keep representative RoPE and packed-attention cases to avoid repeated test work. Align RoPE entry point names with their callers and export the existing loss helpers so model code can train the indexer through the public API. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Bring the branch up to date with upstream before publishing the review changes. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Use one RoPE adapter and autograd function so callers can opt into reusing a disposable query buffer while preserving inputs by default. Expose the table cache for standalone sparse attention components and honor the requested device on cache hits. Keep the existing adapter parity cases and exercise cache migration in the CSA integration test. Simplify the MLA KV test name and explain how it assembles keys and values. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Share table construction and the Triton Q launch across DSA and MLA while preserving each path's layout and mutation contract. Keep standalone RoPE coverage together and zero-fill masked MLA heads so partial blocks contribute correctly to the positional-key gradient. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Use ape for the learned additive position embedding in the module, cuDNN wrapper, and examples. The experimental compressor state-dict key now matches the reference implementation. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
The standalone Indexer tests check numerical backward. Exercise the single-sequence context returned by the full layer to verify that the dense loss helpers train the Indexer without sending gradients to the input or other layer parameters. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
| # Query-up norm is unweighted; TE RMSNorm would add a learned scale. | ||
| q = _compiled_q_rms_norm(q, self.rms_norm_eps) | ||
| q = ( | ||
| apply_dsa_rope(q, *token_rope, cu, allow_inplace=True) |
There was a problem hiding this comment.
When a custom rope module returns tables that require gradients, this query call now raises before sparse attention runs. It always sets allow_inplace=True, but apply_dsa_rope rejects that request when either table requires gradients. Previously, _apply_rotary_query used eager RoPE for these tables.
Keep the eager fallback here instead of requiring the in-place path.
There was a problem hiding this comment.
The in-place query RoPE path currently requires cosine and sine tables with gradients disabled. We’ll extend support for trainable tables in a follow-up.
Keep Q kernel attribution in the shared module. Name the forward and backward entry points in the DSv4 and MLA documentation and identify the separate KV kernel used by the MLA reference tests. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
|
/te-ci pytorch L0 |
|
Following up on the initial feedback.
|
Description
Add experimental DSv4 attention (HCA/CSA) and reusable building blocks for other sparse attention layers.
Type of change
Changes
DSv4Attentionfrom TE projections/norms, publicCompressorandIndexermodules, cuDNN compression/sparse-attention forward and backward kernels, and top-k selection. Thedense_indexer_loss_scoresanddense_indexer_loss_backwardhelpers return score buffers and explicit gradients for a caller-owned auxiliary loss.sbhd,bshdand packedthdformats, fused shared-input projections, injectable RoPE embeddings and optional CSA indexer context. Current support and limitations are documented in the sparse-attention README.Validation
Checklist: