Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
628485c
Add DSv4 sparse attention bindings to TE
sudhakarsingh27 Sep 23, 2026
07b210f
Compose DSv4 hybrid attention from sparse primitives
sudhakarsingh27 Oct 1, 2026
73a2c33
Expose cuDNN dense indexer training primitives for CSA
sudhakarsingh27 Oct 1, 2026
3c9a948
Merge main into the DSv4 attention branch
sudhakarsingh27 Oct 1, 2026
0c9f73e
Format DSv4 Python additions for repository checks
sudhakarsingh27 Oct 1, 2026
39d4c95
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 1, 2026
540507e
Clear DSv4 Python lint findings
sudhakarsingh27 Oct 1, 2026
8b1bf89
Default DSv4 hybrid attention to sequence-major tensors
sudhakarsingh27 Oct 1, 2026
4b01ff2
Use weightless RMS norm for DSv4 query projection
sudhakarsingh27 Oct 2, 2026
52bf9b7
Fix DSv4 indexer bound for partial windows
sudhakarsingh27 Oct 7, 2026
5e8ea3b
Refactor DSv4 sparse attention modules
sudhakarsingh27 Oct 7, 2026
6470ae1
Name the cuDNN DSA integration directly
sudhakarsingh27 Oct 7, 2026
96f9f9c
Accelerate DSv4 RoPE through native CuTe dispatch
sudhakarsingh27 Oct 8, 2026
f6fc4f7
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 8, 2026
865842e
Add packed THD support to DSv4 attention
sudhakarsingh27 Oct 9, 2026
3c7d4f2
Replace DSv4 CuTe RoPE with Megatron Triton kernels
sudhakarsingh27 Oct 9, 2026
31fc926
Make DSv4Attention the hidden-state attention layer
sudhakarsingh27 Oct 12, 2026
17e4b0d
Align DSv4 layout naming with TE attention
sudhakarsingh27 Oct 12, 2026
69661db
Clarify the DSv4 API stack and sparse attention entry point
sudhakarsingh27 Oct 13, 2026
06013ce
Resolve DSv4 Python lint findings
sudhakarsingh27 Oct 13, 2026
e02f6c1
Expose DSA building blocks and validate indexer gradients
sudhakarsingh27 Oct 13, 2026
a5d597f
Clarify sparse attention contracts and naming
sudhakarsingh27 Oct 15, 2026
0003c12
Clarify sparse attention building blocks and examples
sudhakarsingh27 Oct 15, 2026
77f68aa
Reject unsupported Compressor weight projection settings
sudhakarsingh27 Oct 16, 2026
9754d21
Merge branch 'main' into sudhakars/dsv4-cudnn-wrapper
sudhakarsingh27 Oct 19, 2026
3fe1c73
Share RoPE kernels across sparse attention and MLA
sudhakarsingh27 Nov 11, 2026
3e4865e
Validate sparse attention on additional Blackwell GPUs
sudhakarsingh27 Nov 11, 2026
e35f1b0
Keep one fused projection path in DSv4Attention
sudhakarsingh27 Nov 11, 2026
ef2106a
Clarify sparse attention building-block contracts
sudhakarsingh27 Nov 11, 2026
47d7fe5
Register and organize sparse attention regression tests
sudhakarsingh27 Nov 11, 2026
3521f19
Remove redundant fusion options from sparse attention helpers
sudhakarsingh27 Nov 11, 2026
7aeb880
Use token terminology consistently in sparse attention
sudhakarsingh27 Nov 16, 2026
02630ed
Clarify sparse attention APIs and worked index examples
sudhakarsingh27 Nov 17, 2026
8f9b598
Refine DSv4 tests and expose indexer loss helpers
sudhakarsingh27 Nov 19, 2026
b515e8c
Merge upstream main into DSv4 attention branch
sudhakarsingh27 Nov 19, 2026
8aeb32d
Consolidate DSA RoPE application and expose its cache
sudhakarsingh27 Nov 19, 2026
f51dfb8
Consolidate DSA and MLA RoPE adapters
sudhakarsingh27 Nov 20, 2026
dfe286a
Use the DeepSeek APE name in Compressor
sudhakarsingh27 Nov 20, 2026
b6b5b71
Test DSv4 auxiliary Indexer gradient isolation
sudhakarsingh27 Nov 20, 2026
ce449d7
Clarify shared DSA and MLA RoPE kernel usage
sudhakarsingh27 Nov 20, 2026
cf3290a
Merge branch 'main' into sudhakars/dsv4-cudnn-wrapper
sudhakarsingh27 Nov 20, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions docs/api/pytorch.rst
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,19 @@ PyTorch
.. autoapiclass:: transformer_engine.pytorch.DotProductAttention(num_attention_heads, kv_channels, **kwargs)
:members: forward, set_context_parallel_group

.. autoapiclass:: transformer_engine.pytorch.DSv4Attention(hidden_size, q_lora_rank, **kwargs)
:members: forward

.. autoapiclass:: transformer_engine.pytorch.Compressor(hidden_size, width, ratio, overlap, eps, device, params_dtype, **kwargs)
:members: forward

.. autoapiclass:: transformer_engine.pytorch.Indexer(hidden_size, q_lora_rank, head_dim, n_heads, top_k, ratio, eps, device, params_dtype, **kwargs)
:members: forward

.. autoapifunction:: transformer_engine.pytorch.dense_indexer_loss_scores

.. autoapifunction:: transformer_engine.pytorch.dense_indexer_loss_backward

.. autoapiclass:: transformer_engine.pytorch.GatedDeltaNetAttention(num_attention_heads, kv_channels, **kwargs)
:members: forward

Expand Down
1 change: 1 addition & 0 deletions qa/L0_pytorch_unittest/test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ python3 -m pytest --tb=auto --junitxml=$XML_LOG_DIR/pytest_test_identity_quantiz
NVTE_ALLOW_UNSAFE_PICKLE_EXTRA_STATE=1 python3 -m pytest --tb=auto --junitxml=$XML_LOG_DIR/pytest_test_attention.xml $TE_PATH/tests/pytorch/attention/test_attention.py || test_fail "test_attention.py"
python3 -m pytest --tb=auto --junitxml=$XML_LOG_DIR/pytest_test_attention_backend_selection.xml $TE_PATH/tests/pytorch/attention/test_attention_backend_selection.py || test_fail "test_attention_backend_selection.py"
python3 -m pytest --tb=auto --junitxml=$XML_LOG_DIR/pytest_test_flex_attention.xml $TE_PATH/tests/pytorch/attention/test_flex_attention.py || test_fail "test_flex_attention.py"
python3 -m pytest --tb=auto --junitxml=$XML_LOG_DIR/pytest_test_dsv4_attention.xml $TE_PATH/tests/pytorch/test_dsv4_attention.py || test_fail "test_dsv4_attention.py"
NVTE_GDN_TEST_REQUIRED=1 python3 -m pytest --tb=auto --junitxml=$XML_LOG_DIR/pytest_test_gdn_attention.xml $TE_PATH/tests/pytorch/attention/test_gdn_attention.py || test_fail "test_gdn_attention.py"
NVTE_GDN2_TEST_REQUIRED=1 python3 -m pytest --tb=auto --junitxml=$XML_LOG_DIR/pytest_test_gdn2_attention.xml $TE_PATH/tests/pytorch/attention/test_gdn2_attention.py || test_fail "test_gdn2_attention.py"
NVTE_GDP_TEST_REQUIRED=1 python3 -m pytest --tb=auto --junitxml=$XML_LOG_DIR/pytest_test_gdp_attention.xml $TE_PATH/tests/pytorch/attention/test_gdp_attention.py || test_fail "test_gdp_attention.py"
Expand Down
Loading
Loading