Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
77 commits
Select commit Hold shift + click to select a range
45ed767
[PyTorch] Add DeepSeekV3Layer skeleton (MLA + MoE)
pggPL Aug 18, 2026
c306c6f
Move DeepSeekV3 skeleton to models/deepseek_v3 subpackage
pggPL Aug 18, 2026
f73d04e
Add DeepSeekV3 layer entries to PyTorch API docs
pggPL Aug 18, 2026
09f28a9
[PyTorch] Implement DeepSeekV3Layer: MLA + MoE from TE building blocks
pggPL Aug 18, 2026
e23100b
Add distributed EP test for DeepSeekV3 MoE/layer
pggPL Aug 18, 2026
4c6e1e8
Fix EP wgrad test collective + zero EP recv/grad buffers
pggPL Aug 18, 2026
aa17c37
Use fused MLA RoPE kernels in MultiLatentAttention
pggPL Aug 18, 2026
c713af7
Add HF transformers numeric reference test for DeepSeekV3Layer
pggPL Aug 21, 2026
88028a9
Docstring cleanups for lint and docs build
pggPL Aug 21, 2026
5f68c9b
Move model-specific layers to a dedicated docs page
pggPL Aug 21, 2026
9db495a
Drop HF-transformers comparison test from the repo
pggPL Aug 21, 2026
4883b17
Docs: reduce models page to a plain API listing
pggPL Aug 21, 2026
2f52af1
Rename distributed DeepSeek EP tests to generic test_models
pggPL Sep 3, 2026
e841f96
Rename test_deepseek.py to test_models.py and add models tests to QA …
pggPL Sep 3, 2026
935e475
Add YaRN RoPE scaling to DeepSeek V3 MLA
pggPL Sep 3, 2026
7eaefd9
Drop tests/pytorch/attention/mla_rope_utils.py shim; use models.deeps…
pggPL Sep 3, 2026
cef2e39
Distributed models test: single full DeepSeekV3Layer EP-vs-local nume…
pggPL Sep 3, 2026
5453fea
run_models.py: plain main() instead of unittest, simplify launcher
pggPL Sep 3, 2026
f24835b
run_models.py: fail hard instead of swallowing symm-mem/cleanup errors
pggPL Sep 3, 2026
d66fc1c
DeepSeekV3MoE docstring: ep_bootstrap must precede construction
pggPL Sep 3, 2026
80041fa
Distributed models test: launch torchrun directly from pytest, drop s…
pggPL Sep 3, 2026
2475a91
Add DeepSeekV3Layer to test_sanity; pad per-expert rows for quantized…
pggPL Sep 3, 2026
babc5e7
Tests: drop fwd/bwd smoke tests covered by sanity, trim sanity combos…
pggPL Sep 3, 2026
3354906
Rewrite DeepSeekV3MoE class docstring
pggPL Sep 3, 2026
397733e
DeepSeekV3MoE: drop ep_recv_capacity_per_rank and ep_alignment parame…
pggPL Sep 3, 2026
cc10354
DeepSeekV3MoE: build shared expert with the same SwiGLU MLP helper as…
pggPL Sep 3, 2026
2c1cd4c
DeepSeekV3MoE EP path: count tokens per expert with scatter_add inste…
pggPL Sep 3, 2026
46c065c
Merge remote-tracking branch 'origin/main' into deepseek_v3_layer
pggPL Sep 3, 2026
6bae1ba
Docs: list model-specific layers inline on the PyTorch API page; grou…
pggPL Sep 3, 2026
610a1e2
Lint: use dict literals in models.deepseek_v3
pggPL Sep 3, 2026
5c41a3b
Merge remote-tracking branch 'origin/main' into deepseek_v3_layer
pggPL Sep 8, 2026
b58a7ae
Merge remote-tracking branch 'upstream/main' into deepseek_v3_layer
pggPL Sep 8, 2026
93a40df
[PyTorch] DeepSeekV3MoE EP: drop full-buffer zeroing of recv/grad buf…
pggPL Sep 9, 2026
5e36ef1
[PyTorch] Add DeepSeekV3Layer expert-parallel example
pggPL Sep 9, 2026
3612879
[PyTorch] DeepSeekV3Layer example: naive PyTorch MoE and dense baselines
pggPL Sep 9, 2026
eb36d7d
[PyTorch] DeepSeekV3Layer example: two-level plain PyTorch MoE baseline
pggPL Sep 9, 2026
7a1693a
[PyTorch] DeepSeekV3: MLA TP gradient reduction, RoPE guards, per-for…
pggPL Sep 9, 2026
49ecaff
[PyTorch] DeepSeekV3Layer example: argument validation, gradient chec…
pggPL Sep 9, 2026
a131eb3
[PyTorch] Drop example-dependent tests from test_models.py
pggPL Sep 9, 2026
b5605bc
[Docs] Organize DeepSeekV3 example README around results
pggPL Sep 9, 2026
e9fac4f
[Docs] Keep naive MoE comparison in headline results
pggPL Sep 9, 2026
00bcbd9
[PyTorch] DeepSeekV3Layer example README: remeasure all variants on t…
pggPL Sep 9, 2026
4ffa639
[PyTorch] DeepSeekV3Layer example README: split NCCL EP permute from …
pggPL Sep 9, 2026
0367781
[PyTorch] DeepSeekV3MoE: optional caller-owned EpBuffer
pggPL Sep 9, 2026
da81d1d
[PyTorch] DeepSeekV3Layer example README: medians of three runs, brea…
pggPL Sep 9, 2026
2f7859a
[PyTorch] Validate MoE grouping and check synchronized expert bias up…
pggPL Sep 10, 2026
5371619
Merge upstream/main into deepseek_v3_layer
pggPL Sep 17, 2026
db03bd8
[PyTorch] Move MLA RoPE helpers into attention
pggPL Sep 17, 2026
e90cbd7
[PyTorch] Document MLA RoPE helpers
pggPL Sep 17, 2026
7030c91
[Docs] Nest MLA RoPE helpers under Other
pggPL Sep 17, 2026
793b46b
Merge upstream/main into deepseek_v3_layer
pggPL Sep 28, 2026
939c9db
Fix DeepSeekV3 layer review findings
pggPL Sep 30, 2026
00dbae6
Use bounded inputs and RHT in quantized MoE regression test
pggPL Sep 30, 2026
271a1be
Merge remote-tracking branch 'upstream/main' into deepseek_v3_layer
pggPL Sep 30, 2026
16eb75d
Refresh DeepSeekV3 benchmarks on GB200 and GB300
pggPL Sep 30, 2026
b32a4f8
Remove benchmark commit provenance from DeepSeek README
pggPL Oct 1, 2026
85612d4
Merge upstream main for MoE Sequential ops
pggPL Oct 1, 2026
0343fab
Compose DeepSeek expert parallel path with MoE ops
pggPL Oct 1, 2026
72abef7
Let MoE EP ops allocate their data buffers
pggPL Oct 5, 2026
b14c09f
Fix DeepSeek sanity coverage and grouped FP8 requirements
pggPL Oct 6, 2026
21781c3
Remove kernel error handling changes from the DeepSeek PR
pggPL Oct 7, 2026
c60ad6c
[Common] MXFP8 dequantization CuTeDSL kernels (#3842)
kainzhong Nov 12, 2026
464df75
Add user to TE CI (#3855)
ksivaman Nov 12, 2026
fe4f7c5
Reduce autocast CPU overhead for non-delayed_scaling recipes (#3859)
vthumbe1503 Nov 13, 2026
94f2e01
[JAX] Benchmark the attention tutorial on a shorter sequence in its t…
aybchan Nov 13, 2026
ab4b922
[PyTorch] Preserve GEMM-ready scale metadata for empty grouped tensor…
vthumbe1503 Nov 13, 2026
28f6b2a
[Common, PyTorch] Write both NVFP4 amax outputs in the reduction (#3750)
kimchi-developer Nov 13, 2026
9137e0f
[Common] Add MXFP8 2D quantization CuTeDSL kernels (#3843)
kainzhong Nov 14, 2026
154ee78
[PyTorch] Grouped MLP: emit FP32 scale gradient with no cast to BF16 …
sraman-rgb Nov 14, 2026
4f5c08e
[PyTorch] Fuse NVFP4 global scales and GEMM alpha in the grouped MLP …
sraman-rgb Nov 15, 2026
e5d9b0e
Merge upstream/main into deepseek_v3_layer
pggPL Nov 16, 2026
55144aa
Address review comments on DeepSeek-V3 docs and example README
pggPL Nov 16, 2026
fd47803
Keep autocast and surrounding docs entries unchanged
pggPL Nov 16, 2026
4801988
Document MLA RoPE in docstrings instead of the API page
pggPL Nov 16, 2026
a77a4be
Fix docs heading levels for the new API sections
pggPL Nov 16, 2026
5751393
Drop grouped GEMM split accumulator override and mark its test xfail
pggPL Nov 16, 2026
24c987b
Remove grouped GEMM split accumulator test
pggPL Nov 16, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions docs/api/pytorch.rst
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,9 @@
PyTorch
=======

General-purpose layers
----------------------

.. autoapiclass:: transformer_engine.pytorch.Linear(in_features, out_features, **kwargs)
:members: forward, set_tensor_parallel_group

Expand Down Expand Up @@ -43,6 +46,24 @@ PyTorch
.. autoapiclass:: transformer_engine.pytorch.TransformerLayer(hidden_size, ffn_hidden_size, num_attention_heads, **kwargs)
:members: forward, set_context_parallel_group, set_tensor_parallel_group

Model layers
------------

DeepSeek-V3
~~~~~~~~~~~

.. autoapiclass:: transformer_engine.pytorch.models.DeepSeekV3Layer(hidden_size, num_attention_heads, **kwargs)
:members: forward

.. autoapiclass:: transformer_engine.pytorch.models.DeepSeekV3MoE(hidden_size, moe_ffn_hidden_size, num_experts, **kwargs)
:members: forward, update_expert_bias

.. autoapiclass:: transformer_engine.pytorch.models.MultiLatentAttention(hidden_size, num_attention_heads, **kwargs)
:members: forward

Other
-----

.. autoapiclass:: transformer_engine.pytorch.dot_product_attention.inference.InferenceParams(max_batch_size, max_sequence_length)
:members: reset, allocate_memory, pre_step, get_seqlens_pre_step, convert_paged_to_nonpaged, step

Expand Down Expand Up @@ -71,6 +92,15 @@ PyTorch

.. autoapifunction:: transformer_engine.pytorch.deinterleave_glu_tensor

MLA rotary position embeddings
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. autoapifunction:: transformer_engine.pytorch.attention.mla_rope.build_rope_tables

.. autoapifunction:: transformer_engine.pytorch.attention.mla_rope.apply_mla_rope_q

.. autoapifunction:: transformer_engine.pytorch.attention.mla_rope.apply_mla_rope_kv

Data types
----------

Expand Down
13 changes: 12 additions & 1 deletion docs/examples/jax/test_attention.py
Original file line number Diff line number Diff line change
Expand Up @@ -59,9 +59,20 @@ def test_bshd_gqa_swa_matches_baseline():


@requires_hopper_or_newer
def test_single_gpu_benchmark():
def test_single_gpu_benchmark(monkeypatch):
import attention

# The benchmark synchronizes on the native baseline, which has the same memory problem
# as test_bshd_gqa_swa_matches_baseline. Benchmark the same smaller prefix; the tutorial script itself
# still runs at the full sequence length.
seq = 1024
monkeypatch.setattr(attention, "qkv", tuple(t[:, :seq] for t in attention.qkv))
monkeypatch.setattr(attention, "dout", attention.dout[:, :seq])
monkeypatch.setattr(
attention,
"sequence_descriptor",
SequenceDescriptor.from_seqlens(jnp.full((attention.batch,), seq, dtype=jnp.int32)),
)
attention.run_single_gpu_bench()


Expand Down
124 changes: 124 additions & 0 deletions examples/pytorch/deepseek_v3/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
# DeepSeekV3Layer with expert parallelism

A forward + backward benchmark of one `DeepSeekV3Layer`: RMSNorm, multi-latent attention,
and MoE with a shared expert. Routed experts are sharded across GPUs using NCCL EP.

## Quick start

Requires SM90+ GPUs with NVLink and an NCCL EP-enabled TE build;
see [requirements](#requirements). From this directory:

```bash
bash run_deepseek_v3_layer_ep.sh --dsv3
```

For MXFP8 with the fused grouped MLP (SM100-class GPUs):

```bash
NVTE_CUTEDSL_FUSED_GROUPED_MLP=1 bash run_deepseek_v3_layer_ep.sh --dsv3 --recipe mxfp8
```

## Requirements

The following NCCL EP requirements apply to `--impl te`. The naive variants use
ordinary NCCL collectives and can also run on older GPUs without NVLink.

- SM90 or newer GPUs connected with NVLink.
- NCCL >= 2.30.4, PyTorch >= 2.11 (symmetric memory), Transformer Engine built with NCCL EP
enabled (`NVTE_WITH_NCCL_EP`, on by default when an SM90+ architecture is targeted;
`NVTE_WITH_NCCL_EP=0` disables it).
- `NVTE_CUTEDSL_FUSED_GROUPED_MLP=1` additionally needs SM100-class GPUs and the CuTe DSL
(`nvidia-cutlass-dsl`) for the fused MXFP8 grouped MLP.

## Running

Run these commands from this directory. Single node, all local GPUs:

```bash
bash run_deepseek_v3_layer_ep.sh # small dims, bf16
bash run_deepseek_v3_layer_ep.sh --dsv3 # DeepSeek-V3 layer dims, bf16
bash run_deepseek_v3_layer_ep.sh --dsv3 --recipe mxfp8 # MXFP8 experts (unfused grouped GEMM)
NVTE_CUTEDSL_FUSED_GROUPED_MLP=1 bash run_deepseek_v3_layer_ep.sh --dsv3 --recipe mxfp8
bash run_deepseek_v3_layer_ep.sh --dsv3 --impl naive # all_to_all + loop over experts
bash run_deepseek_v3_layer_ep.sh --dsv3 --impl naive_grouped # all_to_all + TE grouped GEMM
```

`--impl` selects the MoE block inside the same layer (attention and norms are identical):

- `te` (default): `DeepSeekV3MoE`, NCCL EP dispatch/combine, experts as one grouped GEMM.
- `naive`: MoE written with plain PyTorch, no TE MoE code: sigmoid top-k router with expert
bias, `all_to_all_single` dispatch and combine (two host syncs per layer for the split sizes),
a Python loop over the local experts with dense `F.linear` SwiGLU MLPs, `index_copy` /
`index_add` to place results, and a shared expert. This is what an EP MoE looks like before
any fused kernels.
- `naive_grouped`: the same all_to_all dispatch and combine, but the received rows are sorted by
local expert and run through one `te.ops.GroupedLinear` / `ScaledSwiGLU` / `GroupedLinear`
stack. Isolates the cost of the Python loop from the cost of the communication path.

Multi-node: launch `torchrun` yourself, EP spans every rank:

```bash
torchrun --nnodes=2 --nproc-per-node=4 --rdzv-backend=c10d --rdzv-endpoint=<head>:29500 \
deepseek_v3_layer_ep.py --dsv3 --recipe mxfp8
```

Every rank owns `--num-local-experts` experts (default 8), so the expert count is
`8 * world_size`. Other knobs: `--tokens-per-rank`, `--topk`, `--hidden`, `--num-heads`,
`--moe-ffn`, the MLA dims (`--q-lora-rank`, `--kv-lora-rank`, `--qk-nope-head-dim`,
`--qk-rope-head-dim`, `--v-head-dim`), `--warmup`, `--iters`.
`--warmup 0` is supported; `--iters` must be positive and `--tokens-per-rank` must be a
positive multiple of four. MXFP8 requires `--impl te`.

## Results

GB200 and GB300 GPUs, 4096 tokens per rank, top-k 8, 8 local experts per GPU.
Times cover one layer's forward + backward; throughput is global, in millions of tokens/s.
Every number is the median of three independent runs (spread within 0.3 ms).
See [benchmark setup](#benchmark-setup) below for the full configuration.

### TE vs. plain PyTorch (`--dsv3`)

Same layer and dimensions, BF16 unless noted.

All variants run the router backward and clear parameter gradients before each step.

| MoE implementation | 4 GB200 (32 experts) | 4 GB300 (32 experts) | 8 GB300 (64 experts) |
|---|---:|---:|---:|
| `naive`: all_to_all + loop over experts | 28.61 ms | 27.21 ms | 27.70 ms |
| `naive_grouped`: all_to_all + TE grouped GEMM | 17.79 ms | 16.72 ms | 17.13 ms |
| `te`: NCCL EP + grouped GEMM | 13.68 ms | 12.37 ms | 14.09 ms |
| `te`, mxfp8 (unfused grouped GEMM) | 10.99 ms | 10.32 ms | 12.11 ms |
| `te`, mxfp8 fused | 10.11 ms | 9.49 ms | 10.45 ms |

### TE throughput (`--dsv3`)

| Precision | 4 GB200 · Mtok/s | 4 GB300 · Mtok/s | 8 GB300 · Mtok/s |
|---|---:|---:|---:|
| BF16 | 1.20 | 1.33 | 2.32 |
| MXFP8 | 1.49 | 1.59 | 2.71 |
| MXFP8 fused | 1.62 | 1.73 | 3.14 |

4 GPUs = 1 node / 32 experts; 8 GPUs = 2 nodes / 64 experts.
Both nodes share one NVLink domain (MNNVL).

“Fused” enables `NVTE_CUTEDSL_FUSED_GROUPED_MLP=1`.

### Benchmark setup

| Setting | Value |
|---|---|
| hardware | GB200 (SM100), one node; GB300 (SM103), one or two nodes; 4 GPUs per node, both GB300 nodes in one NVLink domain (MNNVL) |
| software | CUDA 13.3, NCCL 2.30.7, PyTorch 2.13.0a0+9186a08b2c.nv26.07, cuDNN 9.26, cuDNN Frontend 1.29.0, CUTLASS DSL 4.6.2 |
| NVIDIA driver | GB200: 580.173.02; GB300: 580.173.10 |
| `--dsv3` dims | hidden 7168, 128 heads, MLA q_lora 1536 / kv_lora 512 / nope 128 / rope 64 / v 128, expert ffn 2048, shared expert ffn 2048 |
| MoE | 8 local experts per rank (32 on 4 GPUs, 64 on 8), top-k 8, 4096 tokens per rank |
| precision | bf16 params and activations; `--recipe mxfp8` = MXFP8 block scaling for the expert GEMMs and dense projections |
| timing | fwd + bwd, 5 warmup, 10 timed iterations per run, median of 3 runs; wall-clock timing with CUDA synchronization, no CUDA graphs or profiler |
| environment | `OMP_NUM_THREADS=8`, `NVTE_GROUPED_LINEAR_SINGLE_PARAM=0`, `NVTE_ALLOW_NONDETERMINISTIC_ALGO=1`, `NVTE_FLASH_ATTN_V2=1`, `NVTE_FLASH_ATTN_V3=0`, `NVTE_FLASH_ATTN_V4=0`; EP buffer reused |

Each run reuses a normally distributed random input and backpropagates an all-ones
output gradient. Parameter and input gradients are cleared before each iteration.

`ep_bootstrap` is given the recv capacity returned by
`DeepSeekV3MoE.ep_recv_capacity(ep_size, tokens_per_rank, topk, num_local_experts)`, with
per-expert zones aligned to 256 rows.
Loading