Skip to content

Add GroupedTensor alignment metadata and guard grouped MLP fusion - #58

Draft
pggPL wants to merge 2 commits into
grouped-tensor-base-20261007from
grouped_tensor_from_tensor
Draft

pggPL wants to merge 2 commits into
grouped-tensor-base-20261007from
grouped_tensor_from_tensor

Conversation

@pggPL

@pggPL pggPL commented Oct 7, 2026

Copy link
Copy Markdown
Owner

Description

Enabling grouped MLP fusion can select cuDNN kernels requiring 256 rows per expert for inputs that were valid with 128-row padding on the unfused path. The fusion planner currently has no layout guarantee to check.

Add GroupedTensor.from_tensor(x, m_splits, row_alignment=...) to wrap packed FP32/FP16/BF16 inputs without copying their data or reading split values on the host. The wrapper preserves gradients to x; existing constructors accept optional row_alignment=1 metadata.

The planner now selects cuDNN grouped MLP fusion only when a declared alignment is divisible by 256 and both grouped linears use the matching split buffer. An immediately preceding MoeDispatch can supply the guarantee from EpConfig.alignment when both split inputs are bound to its tokens-per-expert channel. Eligibility is part of the fusion-cache key, so alternating inputs select the appropriate cached plan.

Unknown alignment, insufficient guarantees, mismatched split sources and unsupported intermediate operations fall back to separate grouped operations. Plain-tensor signatures, dispatch return types and fusion callback signatures remain unchanged. Existing callers can lose this fusion until they supply a sufficient guarantee. Alignment is a caller assertion about an already padded layout; this change does not insert padding.

Related to NVIDIA/TransformerEngine#3640. Automatic propagation through other producers and intermediate operations remains outside this change.

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)

Changes

  • Add the autograd-preserving packed-data factory, row-alignment storage metadata, and high-precision grouped input support in fusible grouped linear/MLP operations.
  • Preserve alias/detach semantics across gradient and inference modes.
  • Gate grouped MLP fusion using metadata and split provenance, and include eligibility in cached plans.
  • Document the compatibility/performance impact and add numerical, cache, alias, checkpointing and CUDA Graph regressions.

Validation

Exact-source native build on B100/SM100, PyTorch 2.15 nightly, CUDA 13.4, cuDNN frontend 1.29, NCCL EP disabled.

Confirmed validation (incremental runs):

  • Grouped-tensor and initial alignment suites: 274 passed, 71 skipped. Five additional CUDA Graph/interleaved-backward regressions passed, including both weight layouts.
  • Existing planner-cache tests: 2 passed.
  • Existing prequantized input/gradient regressions: 12 passed, 2 existing frozen-weight skips.
  • Selected existing fused-MLP numerics, caller buffers and CUDA Graphs, including NVFP4: 80 passed, 58 skipped.
  • Existing custom-op/fuser compile regressions with the standard fusion setting: 21 passed.
  • Changed-source pylint (10.00/10), license, pre-commit and documentation build passed.

The internal-dispatch coverage checks planning and channel provenance without EP communication. No distributed or performance claims.

Known baseline limitations: full Python L0 lint reports the unchanged C0411 import-order error in attention/fused_mla_q_uproj.py:19. With NVTE_CUTEDSL_FUSED_GROUPED_MLP=1, the unrelated custom-op compile tests fail while tracing cached super().is_supported(); restoring the pre-change fuser reproduces the same failure. The standard-setting compile selection passes as reported above.

This draft targets a snapshot of upstream 6eb41bd20f16c14b9c83c45e72b2f14e75072930 on the fork so the diff contains only the two implementation commits.

Checklist

  • I have read and followed the contributing guidelines.
  • I have made corresponding changes to the documentation.
  • I have added tests for the new functionality.
  • Focused validation completed; baseline limitations documented above.
  • Human author review before marking ready for maintainer review.

pggPL added 2 commits October 7, 2026 14:33
Wrap packed high-precision grouped inputs without copying payload data or reading splits on the host. Carry optional row-alignment metadata through aliases and saved storage, preserve input gradients, and accept these inputs in fusible grouped linear and MLP operations.

Validate the layout contract and cover numerical gradients, activation checkpointing, CUDA Graph replay, and compatibility with prequantized inputs. Fusion selection and EP return types remain unchanged.

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Decline cuDNN grouped MLP fusion when input layout metadata cannot prove
256-row alignment for the split buffer consumed by both linears. Infer the
same guarantee from an immediately preceding MoeDispatch only when both
split channels come from that dispatch. Include eligibility in the plan
cache so alternating layouts cannot reuse an incompatible fused plan.

Keep existing tensor signatures and fusion callback contracts. Cover
plain/high-precision/quantized inputs, split aliases, dispatch provenance,
cache transitions, numerical fallback, and changing-split CUDA Graph replay.
Update fused-kernel tests to declare their existing 256-row padding.

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>

@zhongbozhu zhongbozhu left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you clarify how row_alignment stays consistent with the alignment used by the dispatcher? For example, a caller could dispatch with EpBuffer(alignment=128) and then wrap the output with GroupedTensor.from_tensor(..., row_alignment=256). The planner would trust the declaration and could select fusion even though the actual splits do not satisfy its requirements.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants