Repository navigation
Conversation
Wrap packed high-precision grouped inputs without copying payload data or reading splits on the host. Carry optional row-alignment metadata through aliases and saved storage, preserve input gradients, and accept these inputs in fusible grouped linear and MLP operations. Validate the layout contract and cover numerical gradients, activation checkpointing, CUDA Graph replay, and compatibility with prequantized inputs. Fusion selection and EP return types remain unchanged. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Decline cuDNN grouped MLP fusion when input layout metadata cannot prove 256-row alignment for the split buffer consumed by both linears. Infer the same guarantee from an immediately preceding MoeDispatch only when both split channels come from that dispatch. Include eligibility in the plan cache so alternating layouts cannot reuse an incompatible fused plan. Keep existing tensor signatures and fusion callback contracts. Cover plain/high-precision/quantized inputs, split aliases, dispatch provenance, cache transitions, numerical fallback, and changing-split CUDA Graph replay. Update fused-kernel tests to declare their existing 256-row padding. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
zhongbozhu
reviewed
Oct 7, 2026
zhongbozhu
left a comment
There was a problem hiding this comment.
Could you clarify how row_alignment stays consistent with the alignment used by the dispatcher? For example, a caller could dispatch with EpBuffer(alignment=128) and then wrap the output with GroupedTensor.from_tensor(..., row_alignment=256). The planner would trust the declaration and could select fusion even though the actual splits do not satisfy its requirements.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Enabling grouped MLP fusion can select cuDNN kernels requiring 256 rows per expert for inputs that were valid with 128-row padding on the unfused path. The fusion planner currently has no layout guarantee to check.
Add
GroupedTensor.from_tensor(x, m_splits, row_alignment=...)to wrap packed FP32/FP16/BF16 inputs without copying their data or reading split values on the host. The wrapper preserves gradients tox; existing constructors accept optionalrow_alignment=1metadata.The planner now selects cuDNN grouped MLP fusion only when a declared alignment is divisible by 256 and both grouped linears use the matching split buffer. An immediately preceding
MoeDispatchcan supply the guarantee fromEpConfig.alignmentwhen both split inputs are bound to its tokens-per-expert channel. Eligibility is part of the fusion-cache key, so alternating inputs select the appropriate cached plan.Unknown alignment, insufficient guarantees, mismatched split sources and unsupported intermediate operations fall back to separate grouped operations. Plain-tensor signatures, dispatch return types and fusion callback signatures remain unchanged. Existing callers can lose this fusion until they supply a sufficient guarantee. Alignment is a caller assertion about an already padded layout; this change does not insert padding.
Related to NVIDIA/TransformerEngine#3640. Automatic propagation through other producers and intermediate operations remains outside this change.
Type of change
Changes
Validation
Exact-source native build on B100/SM100, PyTorch 2.15 nightly, CUDA 13.4, cuDNN frontend 1.29, NCCL EP disabled.
Confirmed validation (incremental runs):
The internal-dispatch coverage checks planning and channel provenance without EP communication. No distributed or performance claims.
Known baseline limitations: full Python L0 lint reports the unchanged C0411 import-order error in
attention/fused_mla_q_uproj.py:19. WithNVTE_CUTEDSL_FUSED_GROUPED_MLP=1, the unrelated custom-op compile tests fail while tracing cachedsuper().is_supported(); restoring the pre-change fuser reproduces the same failure. The standard-setting compile selection passes as reported above.This draft targets a snapshot of upstream
6eb41bd20f16c14b9c83c45e72b2f14e75072930on the fork so the diff contains only the two implementation commits.Checklist