Repository navigation
Conversation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
MultiLatentAttention: low-rank q/kv latents (RMSNorm fused into LayerNormLinear up-projections), decoupled RoPE/NoPE head split with a shared key rope head, DotProductAttention with kv_channels=(qk, v) for the cuDNN fused backend. DeepSeekV3MoE: fused sigmoid router with aux-loss-free expert bias and grouped top-k, routed experts as te.ops GroupedLinear+ScaledSwiGLU+ GroupedLinear (CuTe fused grouped MLP on supported HW), probs applied per-token in the activation, local permute/unpermute or NCCL expert parallelism via ep_dispatch/ep_combine, optional shared expert. DeepSeekV3Layer: pre-RMSNorm + MLA and dense LayerNormMLP (RMSNorm, swiglu) or MoE with residual connections. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
run_deepseek_ep.py checks the EP path against the all-experts-local path numerically (forward, input/gate grads, all-reduced expert wgrads) and smoke-tests the full layer with EP. Also size the default EP recv capacity for per-expert alignment padding and the fused grouped MLP's row-count requirement. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
The per-expert wgrad check called all_reduce on different tensors per rank (rank-local experts), corrupting the reference grads; reduce every expert's grad on every rank instead. Also pass zero-filled recv/grad buffers to ep_dispatch/ep_combine so alignment-padding rows inside the grouped-GEMM m_splits can never poison expert wgrads. Verified on lyris (4x GB300, arm64): run_test_deepseek_ep.sh passes on all ranks (EP forward/dgrad/gate-grad/expert-wgrad match the all-local reference; full-layer EP smoke passes). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Move the Triton MLA RoPE kernels (Megatron-LM fused_mla_yarn_rope_apply port) from tests/pytorch/attention/ mla_rope_utils.py into models/deepseek_v3/mla_rope.py and use them in MultiLatentAttention: the q kernel rotates the rope slice in place and the kv kernel assembles key/value in a single pass, removing the torch.cat/expand/contiguous copies (~10% of layer GPU time). PyTorch fallback (same convention) covers missing Triton and bshd. Fix a latent bug from the test util: the q backward kernel assumed a contiguous incoming gradient, but cuDNN attention backward can hand over a strided one (allocator-state dependent IMA). The old test file stays as a compat shim. Add a Triton-vs-PyTorch parity test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Maps HF DeepseekV3DecoderLayer weights into DeepSeekV3Layer (GLU interleave for routed experts, fused latent norms) and checks forward and input grads match within bf16 tolerance. Expose layernorm_epsilon on MultiLatentAttention (HF latent RMSNorms use 1e-6). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
docs/api/pytorch_models.rst: usage (local and EP), fused-path notes, HF checkpoint weight mapping, and the class API; linked from the PyTorch API page via a toctree entry. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Keep the verified weight-mapping table in the docs; the comparison itself stays as an out-of-tree script. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
…scripts Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
…eek_v3.mla_rope directly Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
…rical comparison Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
…hell launcher Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
… grouped GEMM in local MoE path Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
…; expose MLA softmax_scale; clean docstrings Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
…ters Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
… routed experts Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
…ad of syncing bincount Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
…p standard layers, autocast and other utilities Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Move reusable MLA RoPE kernels and YaRN helpers alongside attention primitives. Update model and MXFP8 attention imports, and move standalone RoPE tests into the attention suite while retaining L0 coverage. Validation: 46 passed, 3 skipped on RTX 5880 Ada using the installed TE primitives; Black, targeted pylint and git diff --check passed. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Make MLA query RoPE allocate by default and preserve incoming gradients. Keep explicit in-place execution limited to eager autograd. Align MXFP8 and NVFP4 experts to 256 rows independently of fusion, remove the EP tail margin, and cover fused and unfused padding in numerical tests. Simplify model APIs, documentation and the EP benchmark following review. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Remove caller-supplied receive and gradient tensors from the DeepSeek Sequential path. Keep per-call EpBuffer routing state for overlapping forward/backward calls. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Use supported MLA and expert dimensions in DeepSeek sanity tests and check finite outputs and gradients. Restrict NVFP4 MoE coverage to the BF16 dtype supported by grouped Hadamard transforms. Align Float8BlockScaling expert rows to 128 for grouped quantization, enforce its split-accumulator requirement in legacy grouped GEMM, and abort invalid grouped block-scaling row layouts even when device assertions are disabled. Add numerical MoE/GEMM coverage and isolated invalid-layout regressions. Validated on B100 and H100 with targeted normal/debug PyTorch tests. Rebuilt the common library in Release on H100: 44 grouped-quantization PyTorch cases and 144 existing C++ cases pass. Python L0 lint, formatting and pre-commit checks pass. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Restore grouped FP8 block-scaling device error handling and remove the associated invalid-row subprocess regression to keep this PR focused on the DeepSeek layer. Retain the MoE padding, grouped GEMM accumulation and sanity coverage fixes. Both restored files match their pre-b14c09ff versions exactly. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
* [Common] More CuTeDSL MXFP8 special kernels Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * wip Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * wip Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * refactor Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * refactor Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * refactor Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * disable invariant temporarily Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * fix Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * refactor Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * fix Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * also support swizzled scale for g2r bidim Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * lint Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * add tests Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * fix Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * nit Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * refactor Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * refactor Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * more refactor Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * g2r bidim can also support colwise only Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * make g2r support activations Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * fix Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * fix Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * +0 and -0 should both legal Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * nit Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * fix Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * fix Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * close fp32 perf gap Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * can always use static tile num Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * nit Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * nit Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * the scale is so annoying Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * refactor Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * fix Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * Fix SMEM bank conflicts in rowwise MXFP8 CuTeDSL dequantize The rowwise thread layout was column-major, so consecutive threads accessed different 128 B SMEM rows that map to the same banks (22x more load and 5.7x more store bank conflicts than the CUDA kernel). Lay threads out row-major along X as the CUDA kernel does. Bit-exact; rowwise goes from 0.70x to 1.06x of CUDA on GB200. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Kaining Zhong <kainingz@nvidia.com> --------- Signed-off-by: Kaining Zhong <kainingz@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Teddy Do <tdophung@nvidia.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
…3859) * Reduce autocast CPU overhead for non-delayed recipes Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> * Test autocast reuse after failed entry Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> --------- Signed-off-by: Varun Thumbe <vthumbe@nvidia.com>
…est (NVIDIA#3847) * [JAX] Benchmark the attention tutorial on a shorter sequence in its test The attention tutorial's native baseline materializes fp32 scores, so at the tutorial's 4K sequence its forward/backward step needs a 33 GiB temporary buffer. That does not fit when CI runs several test workers per GPU. test_single_gpu_benchmark used to pass anyway because the speedometer never waited for the results; since the speedometer synchronizes (NVIDIA#3830) the test fails with RESOURCE_EXHAUSTED. Benchmark the same 1K prefix that test_bshd_gqa_swa_matches_baseline compares on. The tutorial script itself keeps its full sequence length. Signed-off-by: Alex Y. Chan <alechan@nvidia.com> * Update comment Signed-off-by: Alex Y. Chan <alechan@nvidia.com> --------- Signed-off-by: Alex Y. Chan <alechan@nvidia.com>
NVIDIA#3850) * [PyTorch] Preserve GEMM-ready scale metadata for empty grouped tensors Treat swizzling as a completed no-op for logically empty MXFP8 grouped tensors and propagate the layout flag to Python even when no scale buffers are allocated. This prevents grouped GEMM from rejecting an NCCL EP rank that receives zero tokens. Cover empty prequantized inputs with and without columnwise storage. Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> * [PyTorch] Share logical emptiness checks for grouped swizzling Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> --------- Signed-off-by: Varun Thumbe <vthumbe@nvidia.com>
…DIA#3750) * [Common, PyTorch] Write both NVFP4 amax outputs in the reduction Plain NVFP4 quantization with row-wise and column-wise outputs reduced the amax into one buffer and then copied it into the other with a separate device-to-device copy. Let the amax kernels zero and update an optional second output, used for the column-wise amax of NVFP4 outputs, and drop the copy. Other callers of the amax reduction keep one output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: ceo <ceo@logictree.app> * [Common, PyTorch] Address dual amax review feedback Name the zeroing kernel outputs rowwise and colwise. Fold decreasing-amax output reuse checks into the existing NVFP4 quantization tests, including columnwise-only 1D and 2D coverage. Signed-off-by: ceo <ceo@logictree.app> * Update transformer_engine/pytorch/csrc/quantizer.cpp Signed-off-by: vthumbe1503 <vthumbe@nvidia.com> * Update transformer_engine/common/include/transformer_engine/recipe.h Signed-off-by: vthumbe1503 <vthumbe@nvidia.com> * Update transformer_engine/common/recipe/current_scaling.cu Signed-off-by: vthumbe1503 <vthumbe@nvidia.com> * Update transformer_engine/common/recipe/current_scaling.cu Signed-off-by: vthumbe1503 <vthumbe@nvidia.com> * Update transformer_engine/common/recipe/current_scaling.cu Signed-off-by: vthumbe1503 <vthumbe@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update transformer_engine/common/recipe/current_scaling.cu Signed-off-by: vthumbe1503 <vthumbe@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update transformer_engine/common/recipe/current_scaling.cu Signed-off-by: vthumbe1503 <vthumbe@nvidia.com> * Update transformer_engine/common/recipe/current_scaling.cu Signed-off-by: vthumbe1503 <vthumbe@nvidia.com> * Update transformer_engine/common/include/transformer_engine/recipe.h Signed-off-by: vthumbe1503 <vthumbe@nvidia.com> * Update amax kernel to use columnwise_amax Signed-off-by: vthumbe1503 <vthumbe@nvidia.com> * Test Cleanup 1. check_amax_reset instead of reuse_output 2. We can check amax reset for any kind of quantized buffer cpp allocated or not Signed-off-by: vthumbe1503 <vthumbe@nvidia.com> --------- Signed-off-by: ceo <ceo@logictree.app> Signed-off-by: vthumbe1503 <vthumbe@nvidia.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: vthumbe1503 <vthumbe@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* [Common] More CuTeDSL MXFP8 special kernels Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * wip Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * wip Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * refactor Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * refactor Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * refactor Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * disable invariant temporarily Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * fix Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * refactor Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * fix Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * also support swizzled scale for g2r bidim Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * lint Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * add tests Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * fix Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * nit Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * refactor Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * refactor Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * more refactor Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * g2r bidim can also support colwise only Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * make g2r support activations Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * fix Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * fix Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * +0 and -0 should both legal Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * nit Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * fix Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * fix Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * close fp32 perf gap Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * can always use static tile num Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * [Common] Add MXFP8 2D quantization in CuTeDSL Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix nan Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Kaining Zhong <kainingz@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…VIDIA#3870) * [PyTorch] Grouped MLP: emit FP32 scale gradient in one kernel for FP32 scales The block-scaled fused grouped MLP backward rounds the FP32 scale (router probability) gradient to the compute dtype with `.to(dtype)`. When the scales input is FP32 (e.g. FP32 dispatched router probabilities), autograd then casts that BF16 gradient back to FP32: two cast kernels per layer and microbatch. Return float32(round(grad, compute dtype)) directly from one Triton kernel (`round_through_dtype`) when the scales are FP32. The values are bitwise identical to the previous `.to(bf16)` + autograd `.to(fp32)` pair; BF16/FP16 scales keep the old path. Signed-off-by: Siddhartha Raman S <sraman@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [PyTorch] Grouped MLP: keep the FP32 scale gradient, without rounding it Review feedback: the block-scaled fused grouped MLP rounded the FP32 scale (router probability) gradient from the cuDNN kernels to the compute dtype with .to(dtype), a leftover from when the scales were always BF16. With FP32 scales, autograd then cast it back to FP32, so the gradient lost precision for no reason and cost two cast kernels. Return the gradient in the dtype of the scales input instead, as the BF16 fused grouped MLP path already does: FP32 scales get the kernels' FP32 values unchanged (no cast), BF16 scales get one cast as before. This drops the round_through_dtype Triton kernel. The test checks the FP32 gradient is returned bitwise unchanged with no cast in the backward, and the BF16 case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Siddhartha Raman S <sraman@nvidia.com> * [PyTorch] Grouped MLP: FP32 router-probability scales end to end, no casts Review feedback: the scale gradient from the cuDNN kernels is already FP32, so return it as is (no helper; autograd casts it for lower-precision scales), and pass the scales to the cuDNN kernels in FP32 in the forward for every recipe, as the NVFP4 path already did, instead of rounding them to the compute dtype for MXFP8. Drops the cast-counting test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Siddhartha Raman S <sraman@nvidia.com> * [PyTorch] Test: the MXFP8 GLU forward now gets FP32 probabilities Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Siddhartha Raman S <sraman@nvidia.com> * [PyTorch] Document why the grouped MLP FC1 prob is always FP32 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * Preserve MXFP8 forward router probability dtype Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> --------- Signed-off-by: Siddhartha Raman S <sraman@nvidia.com> Signed-off-by: Varun Thumbe <vthumbe@nvidia.com> Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: vthumbe1503 <vthumbe@nvidia.com> Co-authored-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
…VIDIA#3756) Compute both operands' per-group NVFP4 global scales and their product (or its square root for dGLU) in one Triton launch instead of several eager elementwise kernels. Bit-identical to the eager ops; falls back to them for unsupported inputs. Signed-off-by: Siddhartha Raman S <sraman@nvidia.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Restore autocast to its original docs section, describe MLA RoPE with a formula, and shorten the example README: drop profiling and kernel breakdown sections, merge the benchmark configuration into Results and point to NVTE_WITH_NCCL_EP instead of the NCCL EP submodule. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds
transformer_engine.pytorch.models— a namespace for model-specific layers composed from TE modules — with a DeepSeek-V3 transformer layer analogous toTransformerLayer, exercising TE's MoE and MLA features end to end.MultiLatentAttention(models/deepseek_v3/multi_latent_attention.py): low-rank q/kv latents with RMSNorm fused into the up-projections (LayerNormLinear(normalization="RMSNorm")), decoupled RoPE/NoPE head split with a single shared key rope head, core attention throughDotProductAttentionwith asymmetric head dimskv_channels=(qk_nope+qk_rope, v)so the cuDNN fused attention backend is used where supported. RoPE runs through fused Triton kernels (models/deepseek_v3/mla_rope.py): in-place rotation of the query rope slice and single-pass key/value assembly with the broadcast shared rope head, with a pure-PyTorch fallback. Optional YaRN context extension (rope_scaling_factor,original_max_position_embeddings,beta_fast/beta_slow,mscale/mscale_all_dim) with the matchingsoftmax_scale; verified against the Megatron-Core YaRN implementation. TP via column/row parallel projections.DeepSeekV3MoE(models/deepseek_v3/moe.py): fused sigmoid router with aux-loss-free expert bias and node-limited grouped top-k (fused_topk_with_score_function) plus anupdate_expert_bias()helper; routed experts aste.ops.Sequential(GroupedLinear, ScaledSwiGLU, GroupedLinear), which auto-fuses into the CuTe grouped-GEMM MLP on supported hardware and runs the identical unfused path elsewhere; routing probs applied per-token inside the activation, so merge is plain accumulation. Token routing either local (moe_permute_with_probs/moe_unpermute, with per-expert row alignment under quantization) or expert-parallel over NCCL (ep_dispatch/ep_combine). Optional shared expert built with the same SwiGLU MLP helper.DeepSeekV3Layer(models/deepseek_v3/transformer_layer.py): pre-RMSNorm + MLA, then denseLayerNormMLP(first dense layers) orDeepSeekV3MoE, with residual connections.Type of change
Changes
transformer_engine/pytorch/models/namespace withmodels/deepseek_v3/subpackage (MultiLatentAttention,DeepSeekV3MoE,DeepSeekV3Layer, fused MLA RoPE kernels), exported viatransformer_engine.pytorch.modelsModel-specific layerssection on the PyTorch API page (docs/api/pytorch.rst), with the page regrouped into standard layers / model-specific layers / othertests/pytorch/test_models.py: Triton MLA RoPE vs PyTorch reference (fwd/bwd), YaRN tables andsoftmax_scale, MoE vs dense reference overtopk× grouped routing × shared experttests/pytorch/test_sanity.py::test_sanity_deepseek_v3_layer: dense/MoE layer fwd+bwd across dtypes and all quantization recipestests/pytorch/distributed/test_models.py+run_models.py(added to L1): fullDeepSeekV3Layerwith EP vs all-experts-local, comparing output, input grad, non-expert param grads and all-reduced expert wgrads; verified on 4x GB300tests/pytorch/attention/mla_rope_utils.pyremoved;test_linear_mxfp8_attention.pyusesmodels.deepseek_v3.mla_ropedirectlyChecklist:
🤖 Generated with Claude Code