Skip to content

[PyTorch] DeepSeekV3Layer: full MoE transformer layer (MLA + DeepSeek MoE) - #36

Draft
pggPL wants to merge 77 commits into
mainfrom
deepseek_v3_layer
Draft

pggPL wants to merge 77 commits into
mainfrom
deepseek_v3_layer

Conversation

@pggPL

@pggPL pggPL commented Aug 18, 2026 •

Copy link
Copy Markdown
Owner

Description

Adds transformer_engine.pytorch.models — a namespace for model-specific layers composed from TE modules — with a DeepSeek-V3 transformer layer analogous to TransformerLayer, exercising TE's MoE and MLA features end to end.

MultiLatentAttention (models/deepseek_v3/multi_latent_attention.py): low-rank q/kv latents with RMSNorm fused into the up-projections (LayerNormLinear(normalization="RMSNorm")), decoupled RoPE/NoPE head split with a single shared key rope head, core attention through DotProductAttention with asymmetric head dims kv_channels=(qk_nope+qk_rope, v) so the cuDNN fused attention backend is used where supported. RoPE runs through fused Triton kernels (models/deepseek_v3/mla_rope.py): in-place rotation of the query rope slice and single-pass key/value assembly with the broadcast shared rope head, with a pure-PyTorch fallback. Optional YaRN context extension (rope_scaling_factor, original_max_position_embeddings, beta_fast/beta_slow, mscale/mscale_all_dim) with the matching softmax_scale; verified against the Megatron-Core YaRN implementation. TP via column/row parallel projections.

DeepSeekV3MoE (models/deepseek_v3/moe.py): fused sigmoid router with aux-loss-free expert bias and node-limited grouped top-k (fused_topk_with_score_function) plus an update_expert_bias() helper; routed experts as te.ops.Sequential(GroupedLinear, ScaledSwiGLU, GroupedLinear), which auto-fuses into the CuTe grouped-GEMM MLP on supported hardware and runs the identical unfused path elsewhere; routing probs applied per-token inside the activation, so merge is plain accumulation. Token routing either local (moe_permute_with_probs/moe_unpermute, with per-expert row alignment under quantization) or expert-parallel over NCCL (ep_dispatch/ep_combine). Optional shared expert built with the same SwiGLU MLP helper.

DeepSeekV3Layer (models/deepseek_v3/transformer_layer.py): pre-RMSNorm + MLA, then dense LayerNormMLP (first dense layers) or DeepSeekV3MoE, with residual connections.

Type of change

  • Documentation change (change only to the documentation, either a fix or a new content)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Infra/Build change
  • Code refactoring

Changes

  • New transformer_engine/pytorch/models/ namespace with models/deepseek_v3/ subpackage (MultiLatentAttention, DeepSeekV3MoE, DeepSeekV3Layer, fused MLA RoPE kernels), exported via transformer_engine.pytorch.models
  • Docs: Model-specific layers section on the PyTorch API page (docs/api/pytorch.rst), with the page regrouped into standard layers / model-specific layers / other
  • Tests:
    • tests/pytorch/test_models.py: Triton MLA RoPE vs PyTorch reference (fwd/bwd), YaRN tables and softmax_scale, MoE vs dense reference over topk × grouped routing × shared expert
    • tests/pytorch/test_sanity.py::test_sanity_deepseek_v3_layer: dense/MoE layer fwd+bwd across dtypes and all quantization recipes
    • tests/pytorch/distributed/test_models.py + run_models.py (added to L1): full DeepSeekV3Layer with EP vs all-experts-local, comparing output, input grad, non-expert param grads and all-reduced expert wgrads; verified on 4x GB300
    • tests/pytorch/attention/mla_rope_utils.py removed; test_linear_mxfp8_attention.py uses models.deepseek_v3.mla_rope directly
  • TODO (follow-up): optional seq-wise aux loss

Checklist:

  • I have read and followed the contributing guidelines
  • The functionality is complete
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

🤖 Generated with Claude Code

pggPL and others added 30 commits August 18, 2026 12:37
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
MultiLatentAttention: low-rank q/kv latents (RMSNorm fused into
LayerNormLinear up-projections), decoupled RoPE/NoPE head split with a
shared key rope head, DotProductAttention with kv_channels=(qk, v) for
the cuDNN fused backend.

DeepSeekV3MoE: fused sigmoid router with aux-loss-free expert bias and
grouped top-k, routed experts as te.ops GroupedLinear+ScaledSwiGLU+
GroupedLinear (CuTe fused grouped MLP on supported HW), probs applied
per-token in the activation, local permute/unpermute or NCCL expert
parallelism via ep_dispatch/ep_combine, optional shared expert.

DeepSeekV3Layer: pre-RMSNorm + MLA and dense LayerNormMLP (RMSNorm,
swiglu) or MoE with residual connections.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
run_deepseek_ep.py checks the EP path against the all-experts-local
path numerically (forward, input/gate grads, all-reduced expert wgrads)
and smoke-tests the full layer with EP. Also size the default EP recv
capacity for per-expert alignment padding and the fused grouped MLP's
row-count requirement.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
The per-expert wgrad check called all_reduce on different tensors per
rank (rank-local experts), corrupting the reference grads; reduce every
expert's grad on every rank instead. Also pass zero-filled recv/grad
buffers to ep_dispatch/ep_combine so alignment-padding rows inside the
grouped-GEMM m_splits can never poison expert wgrads.

Verified on lyris (4x GB300, arm64): run_test_deepseek_ep.sh passes on
all ranks (EP forward/dgrad/gate-grad/expert-wgrad match the all-local
reference; full-layer EP smoke passes).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Move the Triton MLA RoPE kernels (Megatron-LM
fused_mla_yarn_rope_apply port) from tests/pytorch/attention/
mla_rope_utils.py into models/deepseek_v3/mla_rope.py and use them in
MultiLatentAttention: the q kernel rotates the rope slice in place and
the kv kernel assembles key/value in a single pass, removing the
torch.cat/expand/contiguous copies (~10% of layer GPU time). PyTorch
fallback (same convention) covers missing Triton and bshd.

Fix a latent bug from the test util: the q backward kernel assumed a
contiguous incoming gradient, but cuDNN attention backward can hand
over a strided one (allocator-state dependent IMA). The old test file
stays as a compat shim. Add a Triton-vs-PyTorch parity test.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Maps HF DeepseekV3DecoderLayer weights into DeepSeekV3Layer (GLU
interleave for routed experts, fused latent norms) and checks forward
and input grads match within bf16 tolerance. Expose layernorm_epsilon
on MultiLatentAttention (HF latent RMSNorms use 1e-6).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
docs/api/pytorch_models.rst: usage (local and EP), fused-path notes,
HF checkpoint weight mapping, and the class API; linked from the
PyTorch API page via a toctree entry.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Keep the verified weight-mapping table in the docs; the comparison
itself stays as an out-of-tree script.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
…scripts

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
…eek_v3.mla_rope directly

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
…rical comparison

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
…hell launcher

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
… grouped GEMM in local MoE path

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
…; expose MLA softmax_scale; clean docstrings

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
…ters

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
… routed experts

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
…ad of syncing bincount

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
…p standard layers, autocast and other utilities

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
pggPL and others added 30 commits September 17, 2026 11:45
Move reusable MLA RoPE kernels and YaRN helpers alongside attention primitives. Update model and MXFP8 attention imports, and move standalone RoPE tests into the attention suite while retaining L0 coverage.

Validation: 46 passed, 3 skipped on RTX 5880 Ada using the installed TE primitives; Black, targeted pylint and git diff --check passed.
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Make MLA query RoPE allocate by default and preserve incoming gradients. Keep explicit in-place execution limited to eager autograd.

Align MXFP8 and NVFP4 experts to 256 rows independently of fusion, remove the EP tail margin, and cover fused and unfused padding in numerical tests.

Simplify model APIs, documentation and the EP benchmark following review.

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Remove caller-supplied receive and gradient tensors from the DeepSeek
Sequential path. Keep per-call EpBuffer routing state for overlapping
forward/backward calls.

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Use supported MLA and expert dimensions in DeepSeek sanity tests and check
finite outputs and gradients. Restrict NVFP4 MoE coverage to the BF16 dtype
supported by grouped Hadamard transforms.

Align Float8BlockScaling expert rows to 128 for grouped quantization, enforce
its split-accumulator requirement in legacy grouped GEMM, and abort invalid
grouped block-scaling row layouts even when device assertions are disabled.

Add numerical MoE/GEMM coverage and isolated invalid-layout regressions.

Validated on B100 and H100 with targeted normal/debug PyTorch tests.
Rebuilt the common library in Release on H100: 44 grouped-quantization
PyTorch cases and 144 existing C++ cases pass. Python L0 lint, formatting
and pre-commit checks pass.

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Restore grouped FP8 block-scaling device error handling and remove the
associated invalid-row subprocess regression to keep this PR focused on
the DeepSeek layer. Retain the MoE padding, grouped GEMM accumulation
and sanity coverage fixes.

Both restored files match their pre-b14c09ff versions exactly.

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
* [Common] More CuTeDSL MXFP8 special kernels

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* wip

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* wip

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* refactor

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* refactor

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* refactor

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* disable invariant temporarily

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* fix

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* refactor

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* fix

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* also support swizzled scale for g2r bidim

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* lint

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* add tests

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* fix

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* nit

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* refactor

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* refactor

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* more refactor

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* g2r bidim can also support colwise only

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* make g2r support activations

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* fix

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* fix

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* +0 and -0 should both legal

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* nit

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* fix

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* fix

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* close fp32 perf gap

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* can always use static tile num

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* nit

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* nit

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* the scale is so annoying

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* refactor

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* fix

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* Fix SMEM bank conflicts in rowwise MXFP8 CuTeDSL dequantize

The rowwise thread layout was column-major, so consecutive threads accessed
different 128 B SMEM rows that map to the same banks (22x more load and 5.7x more
store bank conflicts than the CUDA kernel). Lay threads out row-major along X as the
CUDA kernel does. Bit-exact; rowwise goes from 0.70x to 1.06x of CUDA on GB200.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

---------

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Teddy Do <tdophung@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
…3859)

* Reduce autocast CPU overhead for non-delayed recipes

Signed-off-by: Varun Thumbe <vthumbe@nvidia.com>

* Test autocast reuse after failed entry

Signed-off-by: Varun Thumbe <vthumbe@nvidia.com>

---------

Signed-off-by: Varun Thumbe <vthumbe@nvidia.com>
…est (NVIDIA#3847)

* [JAX] Benchmark the attention tutorial on a shorter sequence in its test

The attention tutorial's native baseline materializes fp32 scores, so
at the tutorial's 4K sequence its forward/backward step needs a 33 GiB
temporary buffer. That does not fit when CI runs several test workers
per GPU. test_single_gpu_benchmark used to pass anyway because the
speedometer never waited for the results; since the speedometer
synchronizes (NVIDIA#3830) the test fails with RESOURCE_EXHAUSTED.

Benchmark the same 1K prefix that test_bshd_gqa_swa_matches_baseline
compares on. The tutorial script itself keeps its full sequence length.

Signed-off-by: Alex Y. Chan <alechan@nvidia.com>

* Update comment

Signed-off-by: Alex Y. Chan <alechan@nvidia.com>

---------

Signed-off-by: Alex Y. Chan <alechan@nvidia.com>
NVIDIA#3850)

* [PyTorch] Preserve GEMM-ready scale metadata for empty grouped tensors

Treat swizzling as a completed no-op for logically empty MXFP8 grouped tensors and propagate the layout flag to Python even when no scale buffers are allocated. This prevents grouped GEMM from rejecting an NCCL EP rank that receives zero tokens.

Cover empty prequantized inputs with and without columnwise storage.

Signed-off-by: Varun Thumbe <vthumbe@nvidia.com>

* [PyTorch] Share logical emptiness checks for grouped swizzling

Signed-off-by: Varun Thumbe <vthumbe@nvidia.com>

---------

Signed-off-by: Varun Thumbe <vthumbe@nvidia.com>
…DIA#3750)

* [Common, PyTorch] Write both NVFP4 amax outputs in the reduction

Plain NVFP4 quantization with row-wise and column-wise outputs reduced
the amax into one buffer and then copied it into the other with a
separate device-to-device copy. Let the amax kernels zero and update an
optional second output, used for the column-wise amax of NVFP4 outputs,
and drop the copy. Other callers of the amax reduction keep one output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: ceo <ceo@logictree.app>

* [Common, PyTorch] Address dual amax review feedback

Name the zeroing kernel outputs rowwise and colwise. Fold decreasing-amax output reuse checks into the existing NVFP4 quantization tests, including columnwise-only 1D and 2D coverage.

Signed-off-by: ceo <ceo@logictree.app>

* Update transformer_engine/pytorch/csrc/quantizer.cpp

Signed-off-by: vthumbe1503 <vthumbe@nvidia.com>

* Update transformer_engine/common/include/transformer_engine/recipe.h

Signed-off-by: vthumbe1503 <vthumbe@nvidia.com>

* Update transformer_engine/common/recipe/current_scaling.cu

Signed-off-by: vthumbe1503 <vthumbe@nvidia.com>

* Update transformer_engine/common/recipe/current_scaling.cu

Signed-off-by: vthumbe1503 <vthumbe@nvidia.com>

* Update transformer_engine/common/recipe/current_scaling.cu

Signed-off-by: vthumbe1503 <vthumbe@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update transformer_engine/common/recipe/current_scaling.cu

Signed-off-by: vthumbe1503 <vthumbe@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update transformer_engine/common/recipe/current_scaling.cu

Signed-off-by: vthumbe1503 <vthumbe@nvidia.com>

* Update transformer_engine/common/recipe/current_scaling.cu

Signed-off-by: vthumbe1503 <vthumbe@nvidia.com>

* Update transformer_engine/common/include/transformer_engine/recipe.h

Signed-off-by: vthumbe1503 <vthumbe@nvidia.com>

* Update amax kernel to use columnwise_amax

Signed-off-by: vthumbe1503 <vthumbe@nvidia.com>

* Test Cleanup

1. check_amax_reset instead of reuse_output
2. We can check amax reset for any kind of quantized buffer cpp allocated or not

Signed-off-by: vthumbe1503 <vthumbe@nvidia.com>

---------

Signed-off-by: ceo <ceo@logictree.app>
Signed-off-by: vthumbe1503 <vthumbe@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: vthumbe1503 <vthumbe@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* [Common] More CuTeDSL MXFP8 special kernels

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* wip

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* wip

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* refactor

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* refactor

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* refactor

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* disable invariant temporarily

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* fix

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* refactor

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* fix

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* also support swizzled scale for g2r bidim

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* lint

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* add tests

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* fix

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* nit

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* refactor

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* refactor

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* more refactor

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* g2r bidim can also support colwise only

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* make g2r support activations

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* fix

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* fix

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* +0 and -0 should both legal

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* nit

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* fix

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* fix

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* close fp32 perf gap

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* can always use static tile num

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* [Common] Add MXFP8 2D quantization in CuTeDSL

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix nan

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…VIDIA#3870)

* [PyTorch] Grouped MLP: emit FP32 scale gradient in one kernel for FP32 scales

The block-scaled fused grouped MLP backward rounds the FP32 scale (router
probability) gradient to the compute dtype with `.to(dtype)`. When the scales
input is FP32 (e.g. FP32 dispatched router probabilities), autograd then casts
that BF16 gradient back to FP32: two cast kernels per layer and microbatch.

Return float32(round(grad, compute dtype)) directly from one Triton kernel
(`round_through_dtype`) when the scales are FP32. The values are bitwise
identical to the previous `.to(bf16)` + autograd `.to(fp32)` pair; BF16/FP16
scales keep the old path.

Signed-off-by: Siddhartha Raman S <sraman@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [PyTorch] Grouped MLP: keep the FP32 scale gradient, without rounding it

Review feedback: the block-scaled fused grouped MLP rounded the FP32 scale (router
probability) gradient from the cuDNN kernels to the compute dtype with .to(dtype), a
leftover from when the scales were always BF16. With FP32 scales, autograd then cast it back
to FP32, so the gradient lost precision for no reason and cost two cast kernels.

Return the gradient in the dtype of the scales input instead, as the BF16 fused grouped MLP
path already does: FP32 scales get the kernels' FP32 values unchanged (no cast), BF16 scales
get one cast as before. This drops the round_through_dtype Triton kernel. The test checks the
FP32 gradient is returned bitwise unchanged with no cast in the backward, and the BF16 case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Siddhartha Raman S <sraman@nvidia.com>

* [PyTorch] Grouped MLP: FP32 router-probability scales end to end, no casts

Review feedback: the scale gradient from the cuDNN kernels is already FP32, so return it as is
(no helper; autograd casts it for lower-precision scales), and pass the scales to the cuDNN
kernels in FP32 in the forward for every recipe, as the NVFP4 path already did, instead of
rounding them to the compute dtype for MXFP8. Drops the cast-counting test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Siddhartha Raman S <sraman@nvidia.com>

* [PyTorch] Test: the MXFP8 GLU forward now gets FP32 probabilities

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Siddhartha Raman S <sraman@nvidia.com>

* [PyTorch] Document why the grouped MLP FC1 prob is always FP32

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

* Preserve MXFP8 forward router probability dtype

Signed-off-by: Varun Thumbe <vthumbe@nvidia.com>

---------

Signed-off-by: Siddhartha Raman S <sraman@nvidia.com>
Signed-off-by: Varun Thumbe <vthumbe@nvidia.com>
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: vthumbe1503 <vthumbe@nvidia.com>
Co-authored-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
…VIDIA#3756)

Compute both operands' per-group NVFP4 global scales and their product
(or its square root for dGLU) in one Triton launch instead of several
eager elementwise kernels. Bit-identical to the eager ops; falls back to
them for unsupported inputs.

Signed-off-by: Siddhartha Raman S <sraman@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Restore autocast to its original docs section, describe MLA RoPE with a
formula, and shorten the example README: drop profiling and kernel
breakdown sections, merge the benchmark configuration into Results and
point to NVTE_WITH_NCCL_EP instead of the NCCL EP submodule.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants