Skip to content

Add DSv4 cudnn sparse attention kernel adapters to TE - #3566

Open
sudhakarsingh27 wants to merge 41 commits into
NVIDIA:mainfrom
sudhakarsingh27:sudhakars/dsv4-cudnn-wrapper
Open

sudhakarsingh27 wants to merge 41 commits into
NVIDIA:mainfrom
sudhakarsingh27:sudhakars/dsv4-cudnn-wrapper

Conversation

@sudhakarsingh27

@sudhakarsingh27 sudhakarsingh27 commented Sep 23, 2026 •

Copy link
Copy Markdown
Member

Description

Add experimental DSv4 attention (HCA/CSA) and reusable building blocks for other sparse attention layers.

Type of change

  • Documentation change (change only to the documentation, either a fix or a new content)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Infra/Build change
  • Code refactoring

Changes

  • Compose DSv4Attention from TE projections/norms, public Compressor and Indexer modules, cuDNN compression/sparse-attention forward and backward kernels, and top-k selection. The dense_indexer_loss_scores and dense_indexer_loss_backward helpers return score buffers and explicit gradients for a caller-owned auxiliary loss.
  • Support hidden states in sbhd, bshd and packed thd formats, fused shared-input projections, injectable RoPE embeddings and optional CSA indexer context. Current support and limitations are documented in the sparse-attention README.
  • Consolidate shared Megatron-derived Triton RoPE kernels for DSA and existing MLA callers; add API documentation, numerical/backward tests and the DSv4 QA test entry.

Validation

  • Exact-source native build on B200 with cuDNN Frontend 1.30.0: 35 DSv4 tests and 21 selected MLA tests passed, with no failures or skips. Performance and multi-GPU cases were deselected.
  • Native TE forward/backward canary and documented index examples passed.
  • Local formatting/pre-commit checks and pylint passed.

Checklist:

  • I have read and followed the contributing guidelines
  • The functionality is complete
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

DSv4 HCA and CSA need joint attention over local and compressed KV rows, while CSA also selects compressed rows. Expose the cuDNN Frontend compressor, selector, and sparse attention operations through a small parameter-free TE module so model-owned projections and positional handling remain reusable.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Build the BSD-to-BSD CSA/HCA layer over the existing cuDNN-backed compressor, indexer, and attention operations so callers can supply hidden states directly. Keep the shared RoPE preparation and indexer in small modules, and cache FP32 frequencies across forwards.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
@sudhakarsingh27 sudhakarsingh27 changed the title Add DSv4 sparse attention bindings to TE Add DSv4 sparse attention layers to TE Oct 1, 2026
sudhakarsingh27 and others added 6 commits October 1, 2026 13:04
Megatron trains its CSA indexer with a separate dense objective, while discrete top-k selection leaves the indexer without language-model gradients. Expose the live indexer tensors and narrow cuDNN score/backward calls so model code can retain ownership of KL reduction and loss attachment. Allow a caller RoPE module to match compressed YaRN without copying that policy into TE; preserve the default BSD return.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Keep the DSv4 API branch mergeable as attention exports evolve upstream. Preserve both the existing linear attention exports and the DSv4 sparse attention exports in the public package.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
The repository Black configuration reformats the newly added DSv4 files. Apply it so lint can assess the implementation without style-only failures; this commit changes no behavior.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
The PR Python check reports DSv4 docstring, literal, unused-argument, and local-initialization findings. Resolve them without changing the attention or indexer contracts so CI can validate the implementation.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Megatron passes sequence-major hidden states to attention. Accept and return that layout by default so the batch-one path can enter the existing packed-row core through a view without an extra public BSD round trip. Keep explicit BSD input for batch-major callers; larger batches still repack for the present core layout.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
@cyanguwa

cyanguwa commented Oct 6, 2026

Copy link
Copy Markdown
Collaborator

A few initial comments - I know the PR is still in draft.

  • Should we rename DSv4Attention to DSv4CoreAttention, and DSv4HybridAttention to DSv4Attention? Also, maybe we can tighten up the file naming/structure a bit more so it's more obvious to users what modules we're trying to offer. I feel maybe these ones (without the v4 denotation, given that they can be used by v4.1, Qwen as well): ds_rope, ds_compressor, ds_indexer, ds_core_attention, ds_attention?
  • You've included some explanation in README.md about the structure/usage of this PR, but I feel it's still a bit confusing. Maybe we could reuse the sketch in Po's presentation, and replace certain components with our module names?
  • How easy is it to add your Megatron/Hugging Face validator to our tests/? It could be a good way to show the usage of our PR, but this can be punted to a future PR as well (as we discussed offline) if it requires too many changes.

Thanks!

@sudhakarsingh27 sudhakarsingh27 changed the title Add DSv4 sparse attention layers to TE Add DSv4 cudnn sparse attention kernel adapters to TE Oct 7, 2026
sudhakarsingh27 and others added 12 commits October 13, 2026 11:50
The eager FP32 square/mean path materializes full-size query intermediates. At B4/S32768/BF16 on B200, weightless F.rms_norm lowers HCA and CSA TE peak allocated memory by about 45 GiB and cuts forward/backward time by 26-28%, while mapped-weight output and gradient cosines remain above 0.999999. Keep the unweighted normalization contract without adding a learned scale.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
cuDNN requires max_seqlen_q <= max_seqlen_k * ratio. Keep the compressed prefixes at the number of complete blocks, but pass a ceil-sized compile bound so non-divisible sequence lengths work.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Name each module after the implementation it owns, and merge the parameter-free DSv4Attention core with the high-level attention layer. Remove the redundant dsv4 wrapper and legacy alias while preserving tensor contracts, state layout, and execution order.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
The module adapts PyTorch and autograd calls to cuDNN DSA APIs rather than implementing kernels, so dsa_cudnn is the shorter and more accurate boundary name.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Python-side CuTe launches dominate the otherwise fast rotary kernels. Register shape-specialized CuTe callables through the common TVM-FFI cache so steady-state SM100 execution dispatches from C++ while unsupported inputs keep the eager fallback.

Compile the weightless query normalization and let its fresh output own the in-place query rotation. This removes one full-width query copy without changing the public attention or state-dict contract; other RoPE sites retain out-of-place semantics required by their autograd boundaries.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Follow the existing TE packed-attention contract at the high-level layer so callers can pass [T,D] hidden states with CUDA cumulative lengths. Build compressed prefixes on device to avoid a per-step host synchronization, and clear unused capacity rows before normalization because cuDNN only guarantees rows below the final compressed prefix.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Use the upstream forward/backward kernels from Megatron-LM c4129afff4867ca89b1fb7b4b911128424cdfbe8 while retaining TE tensor ownership and gradient semantics. Keeping the existing out-of-place KV/output and in-place fresh-query boundaries isolates the kernel-backend change and preserves autograd correctness. Remove the DSv4-specific CuTe implementation and native bridge so this branch has a single optimized RoPE backend.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Remove the parameter-free module that only stored window/ratio and delegated to the backend function. Call that function directly from the high-level layer and retain its FP8 rejection, giving DSv4Attention one clear hidden-state contract without an additional wrapper or compatibility alias. Prepared-tensor callers continue to use dsa_cudnn.attention.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Use qkv_format with sbhd, bshd, and thd as in MultiheadAttention and DotProductAttention. The names describe the attention layout while high-level hidden states omit the head axis. Preserve the same dense and packed computation without aliases for the retired spelling.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Name the prepared-tensor operation sparse_attention to distinguish it from the hidden-state layer. Update callers and tests together, and document the module hierarchy and CSA/HCA usage without repeating implementation detail.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Use a named launch-grid function and document the rotary autograd methods. Split packed-prefix type validation from tensor checks to satisfy the Boolean-expression limit while preserving the same validation and error message.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
@sudhakarsingh27
sudhakarsingh27 force-pushed the sudhakars/dsv4-cudnn-wrapper branch from 5876d2c to 06013ce Compare October 13, 2026 18:50
@sudhakarsingh27
sudhakarsingh27 marked this pull request as ready for review October 13, 2026 18:51
@greptile-apps

greptile-apps Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 4/5

[Medium impact] The PR adds a new attention path and changes shared RoPE code used by MLA callers.

The PR still needs the learned-RoPE failure fixed before merging.

Findings

  1. P1 Learned RoPE stops attention ▶

Summary

Adds experimental DSv4Attention with reusable Compressor and Indexer modules.

  • DSv4 attention turns hidden states into HCA or CSA outputs.
  • DSv4 and MLA use shared RoPE code in the attention module.

No new findings were accepted.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  X[Hidden states] --> Q[Query and local KV projections]
  X --> C[Compressor]
  X --> I[CSA Indexer]
  Q --> R[RoPE]
  C --> R
  I --> K[Selected compressed indices]
  R --> A[cuDNN sparse attention]
  K --> A
  A --> O[Inverse RoPE and output projection]
  O --> Y[Hidden states]
  I -. Live indexer tensors .-> L[Caller-owned auxiliary loss]
Loading

Reviews (9) · Last reviewed commit: "Merge branch 'main' into sudhakars/dsv4-..." · Reviewed by Greptile

Comment thread tests/pytorch/test_dsv4_attention.py Outdated


@pytest.mark.parametrize("heads", [1, 64])
def test_dsa_triton_rope_batched_forward_backward(monkeypatch, heads):

@cyanguwa cyanguwa Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have there been some tests for these kernels in #3033?

Same for test_dsa_rope_non_power_of_two_forward_backward.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR #3033 covers MLA RoPE through the MXFP8 attention path. The consolidated RoPE suite checks both rotary output layouts, ragged sequence boundaries, and DSA’s in-place and out-of-place paths. We removed some extraneous tests.

Comment thread tests/pytorch/test_dsv4_attention.py Outdated
)


def test_dsa_packed_rope_eager_forward_backward():

@cyanguwa cyanguwa Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we move this closer to test_dsa_triton_rope_batched_forward_backward in the file, so we have 4-ish groups of tests, dsa_rope, compressor, indexer, dsv4_attention? (IF this makes sense)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • attention/rope.py contains the DSA and MLA RoPE wrappers, with one shared Triton Q adapter.
  • common/triton/dsa_mla_rope.py contains the Triton kernels.
  • test_fused_rope.py now includes two DSA adapter cases, two packed Q-kernel layout cases, and the MLA KV forward/backward case.
  • test_dsv4_attention.py groups the Compressor, Indexer, sparse attention, and full-layer tests.
  • test_fused_mla.py covers fused MLA projection and quantization.

Comment thread tests/pytorch/test_dsv4_attention.py Resolved

@cyanguwa cyanguwa left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a few comments - let me know what you think! Thanks!

Remove duplicated GPU computation while preserving distinct DSA interleaved
and MLA split-half/KV adapters, inverse rotation and packed-position behavior.
Keep framework dependencies out of the common layer and add independent
layout, context-position and gradient regressions.

Validation: 35 DSv4 and 21 focused MLA cases passed on each of SM103 and
SM107, including the shared-kernel regressions.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Admit SM103 and SM107 after exercising compression, sparse attention and
indexer training on the actual architectures. Retain BF16 activation checks
because the compression and indexer dependencies do not support FP16.
Centralize support and qualification requirements instead of repeating
hardware restrictions throughout operation docstrings.

Validation: 35 DSv4 and 21 focused MLA cases passed on each architecture;
outputs, applicable gradients and checkpoint loading match the original
implementation with only its architecture guard relaxed.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Remove the private high-level unfused comparison branch to avoid maintaining
two checkpoint layouts. Preserve default fused parameter names and shapes,
and keep standalone Compressor and Indexer projection options available.

Validation: strict checkpoint loading, outputs and applicable gradients
match the original fused path on SM103 and SM107.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Make the reusable classes discoverable without advertising implementation
modules as package exports. Explain packed addressing, exact dense capacity,
position bias and caller-owned indexer gradients so users can compose the
operations without assuming that discrete selection trains the indexer.
Rename the Compressor input to hidden_states without changing computation.

Validation: syntax, formatter, changed-production pylint and whitespace
checks pass; independent review checked related callers and contracts.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Include the sparse tests in the standard PyTorch launcher with its existing
JUnit and failure handling. Group cases by component and remove redundant
forward_backward suffixes while retaining the packed-layout distinction.

Validation: AST comparison preserves all 13 test functions, assertions and
35 parametrized cases from the GPU-validated candidate. Shell syntax and
pre-commit checks pass; this editorial change does not add a GPU run.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Apply fused projection composition consistently to Compressor and Indexer
instead of retaining unused unfused branches and passing a constant flag.
Remove the obsolete mode-specific rejection and update the capacity fixture
to exercise the fused split.

The prior standalone unfused parameter layout is intentionally retired;
existing fused and high-level DSv4 parameter keys and row order are preserved.

Validation: the existing Compressor CPU output/gradient regression passes
with its fused fixture; pre-commit and changed-production pylint pass.
A separate review checked related source, callers, documentation and tests.
No fresh GPU integration run was performed for this branch simplification.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Describe query, local KV and compressed KV token indices explicitly so
selection indices cannot be confused with indices into concatenated KV.
Distinguish incomplete input blocks from packed output padding and retain
the normalization cleanup rationale. Preserve physical weight/table rows.

Only docstrings, comments, one error message and a private local variable
name change. Executable AST equivalence, pre-commit, changed-production
pylint and whitespace checks pass; no numerical behavior or public
call contract changes.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Document existing operation and constructor contracts consistently, using
explicit batched/packed shapes and cumulative token counts. Explain the TE
output allocation separately from valid compressed tokens and illustrate
local/compressed index conversion beside the actual implementation.

Preserve user prose while resolving inline notes. Changes are documentation,
comments and the clarified full-sequence shape-error text; numerical code
and public call contracts are unchanged.

Validation: pre-commit, pylint and executable AST checks pass. Twelve CUDA
checks of actual source helpers and instrumented packing/padding/test
boundaries match expected results. Numerical backends were intercepted;
this is not a new native cuDNN attention validation.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>


@pytest.mark.skipif(not HAVE_TRITON, reason="MLA RoPE requires Triton")
def test_mla_rope_kv_equations_backward():

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does this test do?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test_mla_rope_kv compares K/V outputs and input gradients against independent PyTorch equations. It uses three heads in a four-head block to check that the gradients sum correctly into the positional key shared across heads.

Comment thread tests/pytorch/test_dsv4_attention.py Outdated
torch.testing.assert_close(query_grad, reference_grad, atol=0.03125, rtol=0.002)


def test_dsa_rope_non_power_of_two(monkeypatch):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we have a look at moving RoPE tests to mla_rope_utils.py and focus on sparse attention/compressor/indexer tests in this file?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved the two DSA adapter cases, two packed Q-kernel layout cases, and the MLA KV forward/backward case to test_fused_rope.py. Test layout



@triton.jit
def _mla_rope_bwd_inplace_kernel(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rename these two to rotary_fwd/bwd_q_kernel?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed the entry points to rotary_fwd_q_kernel and rotary_bwd_q_kernel. How DSv4 and MLA use them

)


def _apply_rotary_eager(x, cos, sin, cu_seqlens=None):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we check if rotary_embeddings or _apply_rotary_eager implementations already exist in rope.py? If not, we can keep these (and possibly rename them to yarn_rope); if so, we can remove the two functions here. Thanks.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The two paths use the same rotation math. The existing API takes angles and rotates the leading channels. DSA takes precomputed cosine/sine tables, rotates the trailing interleaved channels, and handles packed compressed positions. I’m keeping those adapters separate for this PR.

return _TritonQueryRotary.apply(query, _rope_tables(cos), _rope_tables(sin), cu_seqlens, module)


def apply_rotary(x, cos, sin, cu_seqlens=None):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"apply_yarn_rope"?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I renamed the helper to apply_dsa_rope. It applies the supplied cosine/sine tables, including YaRN tables when the caller provides them.

restore_value=["DO"],
)
@triton.jit
def rotary_bwd_q_kernel(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to remove rotary_fwd_q_kernel from this file?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MLA’s local rotary_bwd_q_kernel was removed because MLA and DSv4 now use the same Triton RoPE backward implementation. MLA’s fused query projection still applies RoPE in cuDNN during forward.

DSv4 uses the Triton Q kernel pair for both Q and KV. The separate MLA KV kernels are called through the MLA wrapper in attention/rope.py, whose current callers are tests. They reuse one positional key per token across all attention heads and sum its gradients across heads in backward.

__all__ = [
"DSv4Attention",
"Compressor",
"Indexer",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we bring dense_indexer_loss_scores and dense_indexer_loss_backward to the same level as these three APIs so people can easily access them? Also, can add the loss ones in PR description.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dense_indexer_loss_scores and dense_indexer_loss_backward are now importable from transformer_engine.pytorch alongside the three sparse attention classes. Both are listed in the API reference and PR description.

Keep representative RoPE and packed-attention cases to avoid repeated test work. Align RoPE entry point names with their callers and export the existing loss helpers so model code can train the indexer through the public API.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Bring the branch up to date with upstream before publishing the review changes.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Use one RoPE adapter and autograd function so callers can opt into reusing a disposable query buffer while preserving inputs by default. Expose the table cache for standalone sparse attention components and honor the requested device on cache hits.

Keep the existing adapter parity cases and exercise cache migration in the CSA integration test. Simplify the MLA KV test name and explain how it assembles keys and values.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Share table construction and the Triton Q launch across DSA and MLA while preserving each path's layout and mutation contract. Keep standalone RoPE coverage together and zero-fill masked MLA heads so partial blocks contribute correctly to the positional-key gradient.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Use ape for the learned additive position embedding in the module, cuDNN wrapper, and examples. The experimental compressor state-dict key now matches the reference implementation.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
The standalone Indexer tests check numerical backward. Exercise the single-sequence context returned by the full layer to verify that the dense loss helpers train the Indexer without sending gradients to the input or other layer parameters.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
# Query-up norm is unweighted; TE RMSNorm would add a learned scale.
q = _compiled_q_rms_norm(q, self.rms_norm_eps)
q = (
apply_dsa_rope(q, *token_rope, cu, allow_inplace=True)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Learned RoPE stops attention

When a custom rope module returns tables that require gradients, this query call now raises before sparse attention runs. It always sets allow_inplace=True, but apply_dsa_rope rejects that request when either table requires gradients. Previously, _apply_rotary_query used eager RoPE for these tables.

Keep the eager fallback here instead of requiring the in-place path.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The in-place query RoPE path currently requires cosine and sine tables with gradients disabled. We’ll extend support for trainable tables in a follow-up.

Keep Q kernel attribution in the shared module. Name the forward and backward entry points in the DSv4 and MLA documentation and identify the separate KV kernel used by the MLA reference tests.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
@sudhakarsingh27

Copy link
Copy Markdown
Member Author

/te-ci pytorch L0

@sudhakarsingh27

Copy link
Copy Markdown
Member Author

Following up on the initial feedback.

DSv4Attention is the hidden-state layer, and dsa_cudnn.sparse_attention is the entry point for prepared tensors. The README shows the module hierarchy and data flow, with CSA and HCA examples. Megatron and Hugging Face integration tests can follow in a separate PR.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants