Repository navigation
[PyTorch] Run NVFP4 FC2 fprop and FC1 dgrad of the fused grouped MLP on CuTe DSL - #3871
xrennvidia wants to merge 5 commits into
Conversation
[Medium impact] The changes since the previous review appear safe to merge. SummaryThe PR moves NVFP4 FC2 forward and FC1 input-gradient GEMMs to CuTe DSL when supported.
Reviews (6) · Last reviewed commit: "[PyTorch] Parametrize the grouped MLP CU..." · Reviewed by Greptile |
…on CuTe DSL The fused grouped MLP runs the NVFP4 FC2 forward and FC1 input-gradient GEMMs on cuBLAS. Run them on the cuDNN frontend CuTe DSL grouped GEMM kernel instead, like the MXFP8 path. Add NVTE_DISABLE_CUTEDSL_PLAIN_GROUPED_MLP_GEMM to move the GEMMs without a fused activation that both backends implement back to cuBLAS: the weight gradient GEMMs and the NVFP4 FC2 forward and FC1 input-gradient GEMMs. It replaces NVTE_DISABLE_CUTEDSL_WGRAD_FUSED_GROUPED_MLP, which covered the weight-gradient GEMMs only and stays as a deprecated alias, honored when the new variable is unset. A single-group MLP on the dense single-group path (every activation but the SReLU ones) keeps its dense GEMMs. NVFP4 with single_grouped_weight=True is not supported by the fused op (the cuBLAS path already rejects its unswizzled weight scales); the new paths raise NotImplementedError for it instead of failing in a kernel. Tests: backend routing for grouped and single-group MLPs, the environment variable and its alias, CUDA graph capture with NVFP4, and the single-grouped-weight error. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Xiaowei Ren <xren@nvidia.com>
81aab0c to
f20e8ad
Compare
| return buffer | ||
|
|
||
|
|
||
| def _convert_to_cudnn_grouped_gemm_tensor_format( |
There was a problem hiding this comment.
cuDNN grouped GEMM logic belongs with the rest of the cuDNN FE logic in grouped_mlp.py. We previously had some helper functions in this file to workaround a cuBLAS limitation, but they were removed in #3827.
Also, can we avoid these reshapes by relying on cuDNN's support for canonical data layouts (see #3641 and #3820)? Choosing between cuDNN and cuBLAS grouped GEMM is an advanced feature, so we could just ignore the user's hint if the cuDNN FE version is too old.
There was a problem hiding this comment.
Done in 506da5c. The helper is gone and gemm.py is back to what's on main, so all the cuDNN grouped GEMM logic lives in grouped_mlp.py.
The NVFP4 FC2 fprop and FC1 dgrad GEMMs now pass canonical layouts directly (A as (sum(m), k), the packed scale buffer as-is, 1-D prob, plain output buffer), with no reshapes. If the quant wrapper doesn't advertise supports_canonical_layouts (cuDNN FE < 1.32.1), these two GEMMs ignore the hint and run on cuBLAS.
One side effect: I moved the "NVFP4 does not support single_grouped_weight=True" error to a single check at the top of fuser_forward, so it no longer depends on which backend is picked.
| When set to ``1``, overrides the operation's declared split alignment | ||
| to 256, enabling grouped MLP fusion when its other requirements are met. | ||
|
|
||
| .. envvar:: NVTE_DISABLE_CUTEDSL_PLAIN_GROUPED_MLP_GEMM |
There was a problem hiding this comment.
"Plain grouped MLP GEMM" is not a particularly clear name. Maybe NVTE_PREFER_CUBLAS_GEMM_IN_GROUPED_MLP or NVTE_DISABLE_CUDNN_UNFUSED_GEMM_IN_GROUPED_MLP?
There was a problem hiding this comment.
Agreed, renamed to NVTE_DISABLE_CUDNN_UNFUSED_GEMM_IN_GROUPED_MLP in 9db1cc4, along with the helper, tests and docs. NVTE_DISABLE_CUTEDSL_WGRAD_FUSED_GROUPED_MLP is still honored as a deprecated alias when the new variable is unset.
| if use_nvfp4: | ||
| # A single dense group keeps its dense GEMM. | ||
| use_cutedsl_fc2_gemm = _use_cutedsl_plain_gemm() and not use_dense_single_group | ||
| if use_nvfp4 and not use_cutedsl_fc2_gemm: |
There was a problem hiding this comment.
Can we make MXFP8 respect the cuBLAS/cuDNN choice as well?
| if use_nvfp4 and not use_cutedsl_fc2_gemm: | |
| if not use_cutedsl_fc2_gemm: |
There was a problem hiding this comment.
Done in 1ed26b2. MXFP8 FC2 fprop now follows the variable as well, so with it set all four unfused GEMMs (FC2 fprop, FC1 dgrad, both wgrads) run on cuBLAS for both recipes.
I couldn't take the suggestion verbatim: after the canonical-layout change the flag was NVFP4-only, so the bare if not use_cutedsl_fc2_gemm: would have sent MXFP8 to cuBLAS unconditionally. The flag now covers both recipes. A dense single group takes the shared branch and keeps its dense GEMM, so I dropped the unreachable dense case from the MXFP8 kernel branch — most of the diff there is re-indentation.
test_grouped_mlp_unfused_gemm_backend now checks the backend of each GEMM for NVFP4 and for MXFP8 with discrete and single grouped weights. The full test_grouped_mlp.py passes with the variable unset and with it set to 1.
One remaining asymmetry: on cuDNN FE 1.31.x–1.32.0, NVFP4 fprop/dgrad fall back to cuBLAS while MXFP8 stays on cuDNN through its legacy layouts. Raising the minimum FE to 1.32.1 would remove that; happy to do it here or in a follow-up if you prefer.
| dtype=dtype, | ||
| ) | ||
| elif use_nvfp4: | ||
| elif use_nvfp4 and not _use_cutedsl_plain_gemm(): |
There was a problem hiding this comment.
Can we make MXFP8 respect the cuBLAS/cuDNN choice?
| elif use_nvfp4 and not _use_cutedsl_plain_gemm(): | |
| elif not _use_cutedsl_plain_gemm(): |
There was a problem hiding this comment.
Done in 1ed26b2, together with the FC2 fprop one above. The dgrad branch is now elif not _use_cutedsl_fprop_dgrad_gemm(..., use_nvfp4=use_nvfp4):, so MXFP8 FC1 dgrad runs on cuBLAS when NVTE_DISABLE_CUDNN_UNFUSED_GEMM_IN_GROUPED_MLP=1. The suggestion itself no longer applies since the line and the helper name changed.
| split_alignment: int = 256, | ||
| glu_interleave_size: int = 32, | ||
| token_padding: int = 2048, | ||
| quantization: str = "mxfp8", |
There was a problem hiding this comment.
Nice how test_grouped_mlp_cuda_graph_safe_mxfp8 might not test MXFP8.
There was a problem hiding this comment.
Fair point, fixed in 5403276. The test is now test_grouped_mlp_cuda_graph_safe, with quantization as a real parameter rather than a default hidden in an MXFP8-named test.
|
|
||
| @pytest.mark.parametrize("accumulate_into_main_grad", (False, True)) | ||
| @pytest.mark.skipif(not nvfp4_available, reason=reason_for_no_nvfp4) | ||
| def test_grouped_mlp_cuda_graph_safe_nvfp4( |
There was a problem hiding this comment.
This is a weird structure, isn't it? Claude is more interested in minimizing the diff than writing good code. It would be much more logical to refactor test_grouped_mlp_cuda_graph_safe_mxfp8 into a general graph safety test:
@pytest.mark.parametrize("single_grouped_weight", (False, True))
@pytest.mark.parametrize("accumulate_into_main_grad", (False, True))
@pytest.mark.parametrize("activation", ("scaled_swiglu", "scaled_clamped_qgeglu"))
@pytest.mark.parametrize("quantization", ("mxfp8", "nvfp4"))
def test_grouped_mlp_cuda_graph_safe(There was a problem hiding this comment.
Agreed, refactored in 5403276 along the lines you suggested. There is now a single test_grouped_mlp_cuda_graph_safe parametrized over single_grouped_weight, accumulate_into_main_grad, activation and quantization, and the NVFP4 wrapper is removed.
Two small deviations from your snippet: I used "nvfp4_rht" instead of "nvfp4" since the fused path needs RHT for NVFP4, and NVFP4 with single_grouped_weight=True is skipped as unsupported. As a bonus, NVFP4 now also covers scaled_clamped_qgeglu under CUDA graphs.
|
|
||
|
|
||
| @functools.lru_cache(maxsize=1) | ||
| def _use_cutedsl_plain_gemm() -> bool: |
There was a problem hiding this comment.
"Plain GEMM" is not an intuitive term. Something like "use_unfused_cudnn_gemm" is more descriptive.
There was a problem hiding this comment.
Agreed, renamed in 9db1cc4 together with the env var. The helper is now _cudnn_unfused_gemm_disabled(), mirroring NVTE_DISABLE_CUDNN_UNFUSED_GEMM_IN_GROUPED_MLP, and "plain GEMM" is gone from the code, tests and docs.
… dgrad Pass the operands of the NVFP4 FC2 fprop and FC1 dgrad CuTe DSL GEMMs in cuDNN's canonical layouts instead of reshaping them into the kernel-facing strided views, and drop the reshape helper from cpp_extensions/gemm.py. These two GEMMs fall back to cuBLAS when the cuDNN frontend predates canonical layouts, whatever the plain-GEMM backend setting. Reject NVFP4 with single_grouped_weight=True once, up front, so the error does not depend on the GEMM backend. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Xiaowei Ren <xren@nvidia.com>
Rename NVTE_DISABLE_CUTEDSL_PLAIN_GROUPED_MLP_GEMM to NVTE_DISABLE_CUDNN_UNFUSED_GEMM_IN_GROUPED_MLP, since "plain GEMM" does not say which GEMMs are meant, and rename the helpers and tests to match. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Xiaowei Ren <xren@nvidia.com>
Run the MXFP8 FC2 fprop and FC1 dgrad GEMMs of the fused grouped MLP on cuBLAS when NVTE_DISABLE_CUDNN_UNFUSED_GEMM_IN_GROUPED_MLP is set, as is already done for NVFP4 and for the wgrad GEMMs. A dense single group now takes the branch shared with cuBLAS for both recipes, so drop the unreachable dense case from the MXFP8 kernel branch. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Xiaowei Ren <xren@nvidia.com>
Turn test_grouped_mlp_cuda_graph_safe_mxfp8 into test_grouped_mlp_cuda_graph_safe, parametrized over MXFP8 and NVFP4, and drop the NVFP4 wrapper that called the MXFP8 test. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Xiaowei Ren <xren@nvidia.com>
Description
The fused grouped MLP runs the NVFP4 FC2 forward and FC1 input-gradient GEMMs on cuBLAS,
while every other GEMM in the op already uses a cuDNN frontend CuTe DSL kernel. This PR
runs those two on the CuTe DSL grouped GEMM kernel as well, like the MXFP8 path, so that
with NVFP4 all grouped GEMMs of the op use CuTe DSL by default.
It also adds one environment variable that moves the unfused GEMMs of the op (FC2 fprop,
FC1 dgrad and wgrad) back to cuBLAS, for both NVFP4 and MXFP8.
Fixes # (issue)
Type of change
Changes
grouped_gemm_quant_wrapper_sm100.The operands are passed in cuDNN's canonical layouts ([PyTorch] Pass canonical layouts to cuDNN grouped GEMM in fused grouped MLP #3641, [PyTorch] Pass canonical layouts to cuDNN for the NVFP4 fused grouped MLP #3820), with no reshapes.
The GEMM alpha uses
_nvfp4_global_scale_pair([PyTorch] Fuse NVFP4 global scales and GEMM alpha in the grouped MLP #3756).supports_canonical_layouts(1.32.1 or later). On an older frontend they run on cuBLAS.NVTE_DISABLE_CUDNN_UNFUSED_GEMM_IN_GROUPED_MLP(default0). Set to1, it movesthe GEMMs without a fused activation back to cuBLAS: the FC2 fprop, FC1 dgrad and
weight-gradient GEMMs, for both NVFP4 and MXFP8.
NVTE_DISABLE_CUTEDSL_WGRAD_FUSED_GROUPED_MLP, which covered the weight-gradient GEMMsonly, stays as a deprecated alias: honored when the new variable is unset, with a
DeprecationWarning.ones) keeps its dense GEMMs either way.
single_grouped_weight=Trueis not supported by the fused op. It now raisesNotImplementedErrorat the start of the forward pass, whichever backend is selected,instead of failing inside a kernel.
docs/envvars.rst.Testing
On GB300 (SM 10.3), cuDNN frontend 1.34.0:
tests/pytorch/test_grouped_mlp.py: 1394 passed, 3948 skipped, both with the variableunset and with
NVTE_DISABLE_CUDNN_UNFUSED_GEMM_IN_GROUPED_MLP=1.tests/pytorch/nvfp4/test_nvfp4_global_scale.py: 112 passed.supports_canonical_layoutsfrom the quant wrapper, not on a real older frontend.New and changed tests:
test_grouped_mlp_unfused_gemm_backend: which backend each of the four GEMMs uses, withthe variable unset and set, for grouped and single-group MLPs and three activations, for
NVFP4 and for MXFP8 with discrete and single grouped weights.
test_cudnn_unfused_gemm_disabled_env_var: the variable, its deprecated alias andprecedence.
test_grouped_mlp_cuda_graph_safe: the formertest_grouped_mlp_cuda_graph_safe_mxfp8,now parametrized over MXFP8 and NVFP4.
test_grouped_mlp_nvfp4_rejects_single_grouped_weight.The existing
test_grouped_mlpNVFP4 matrix now runs through the new paths by default.Checklist: