Skip to content

[PyTorch] [torch.compile] torch.compile support for LayerNormLinear and LayerNormMLP - #3534

Open
pggPL wants to merge 17 commits into
NVIDIA:mainfrom
pggPL:layernorm_compile
Open

pggPL wants to merge 17 commits into
NVIDIA:mainfrom
pggPL:layernorm_compile

Conversation

@pggPL

@pggPL pggPL commented Sep 17, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Add full-graph torch.compile support for LayerNormLinear and LayerNormMLP, including backward, dynamic batch dimensions and normalization outputs.

The single late fallback check relies on the Dynamo fix in pggPL/pytorch#4, related to pytorch/pytorch#144820. Without that fix, a graph break inside forward's try/finally can disable compilation for subsequent supported instances of the same module class.

Targets main. This branch includes the NVFP4 packed-buffer, TensorSpec and Bias correctness fixes from NVIDIA/TransformerEngine#3628. Uses the GEMM scale-layout contract merged in NVIDIA/TransformerEngine#3817, so fake outputs follow quantizer metadata without querying native kernel layout rules.

Changes

  • Describe forward outputs, saved tensors, aliases and backward gradients for both LayerNorm modules; retain eager fallbacks for unsupported configurations.
  • Decide fallback eligibility once after preparing quantizers in LayerNormLinear and LayerNormMLP; reuse the prepared eager path and preserve end_forward cleanup.
  • Route gradients from both normalization outputs through an optional backward hook. Share quantizer validation between the LayerNorm modules.
  • Support compiled FP8 weight caching in both LayerNorm modules, following Linear: replace the cache on the first microbatch and reuse it on subsequent microbatches, including both MLP weights. GPU-controlled cache updates and fused wgrad accumulation retain eager fallbacks.
  • Share the cache regression across Linear, LayerNormLinear and LayerNormMLP, checking independent eager outputs/gradients, cache refresh/reuse, optimizer updates, immediate/deferred backward and CUDA Graph replay.
  • Describe normalization statistics and activation outputs directly from quantizer metadata, including gathered LayerNormMLP output shapes. Remove layout-query bindings and backend-specific layout prediction from fake implementations.
  • Parameterize module compile tests across recipes, shapes and modes, with LayerNorm-specific outputs, gradients and eager fallbacks. Retain the non-matrix Linear regressions and expanded cold CUDA Graph recipe coverage inherited from [PyTorch] Fix NVFP4 metadata in compiled Linear operations #3628.

@pggPL
pggPL marked this pull request as ready for review September 17, 2026 17:13
@pggPL
pggPL requested a review from ksivaman as a code owner September 17, 2026 17:13
@greptile-apps

greptile-apps Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 3/5

[Medium impact] The changes affect compiled PyTorch execution for Transformer Engine modules and shared autograd plumbing.

The PR needs fixes before merging because recent Linear changes can change forward results and break compiled weight-cache reuse.

Summary

Adds full-graph torch.compile support for LayerNormLinear and LayerNormMLP, including backward, normalization outputs, and FP8 weight caching.

  • LayerNormLinear runs as a compiled operation with backward support.
  • LayerNormMLP runs as a compiled operation with backward support.
  • Shared compile helpers describe quantized tensors and route output gradients.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Module forward] --> B[Prepare quantizers]
    B --> C{Compiled path supported?}
    C -- No --> D[Eager forward]
    C -- Yes --> E[Fake describes outputs and saved tensors]
    E --> F[Custom op runs forward]
    F --> G[Return outputs and update weight cache]
    G --> H[Backward restores saved tensors]
    H --> I[Assign output gradients]
    I --> J[Custom op runs backward]
Loading

Reviews (40) · Last reviewed commit: "Keep existing Linear behavior in LayerNo..." · Reviewed by Greptile

Comment thread tests/pytorch/test_torch_compile.py Resolved
@ptrendx ptrendx self-assigned this Sep 28, 2026
@greptile-apps

greptile-apps Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Comments Outside Diff

These findings could not be posted inline.

  • P1 Recipe changes reuse unsupported plans transformer_engine/pytorch/ops/fuser.py:709 ▶

    The cache key now includes only recipe type, backward boundary, and backward_override. Fusion also depends on disable_rht, MXFP8 format, and enable_cutedsl_fused_grouped_mlp. A same-class recipe change returns the old plan without checking those settings. For example, a plan built with NVFP4 RHT stays selected after RHT is disabled, although the fused path requires it. Include the fusion-relevant recipe settings in the key.

  • P1 Quantized fake tensors fail transformer_engine/pytorch/dynamo/tensor_spec.py:102 ▶

    When a quantized TensorSpec does not explicitly set with_gemm_swizzled_scales, these lines omit that value from the metadata passed to storage reconstruction. BasicLinear’s fake quantized forward and dgrad paths create such specs, but the MXFP8 and NVFP4 storage constructors require this argument. Tracing therefore fails while assembling the fake tensor, before the compiled operation can run. Please preserve the quantizer-derived default when the field is unspecified while still allowing fake paths to override a known native layout.

@pggPL
pggPL changed the base branch from pgadzinski/fix_nvfp4_compile_nd to main October 12, 2026 15:43
Comment thread tests/cpp/test_common.h Resolved
Comment thread transformer_engine/common/util/rtc.cpp Resolved
Comment thread transformer_engine/pytorch/router.py Resolved
Register LayerNorm module forward and backward custom ops with matching fake tensor metadata, shared eligibility checks, and compiled weight caches. Extend the existing compile tests to cover the LayerNorm modules.

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
@pggPL
pggPL force-pushed the layernorm_compile branch from 70e4b7d to 83641e8 Compare October 29, 2026 12:07

std::optional<size_t> mxfp8_norm_fused_alignment(bool rmsnorm, bool with_2d_quantization,
bool use_cudnn) {
if (rmsnorm ? with_2d_quantization : !use_cudnn) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this checking specifically for RMSNorm? This sounds like the original logic had some bug where only RMSNorm was checked rather than any normalization.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, it seems we have but it is hard to obtain it in real training since it is about layernorming 2d scaled tensor (weights) which does not happen. I ll fix in separate PR

@pggPL pggPL removed the 2.21 label Oct 30, 2026
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Comment thread tests/pytorch/test_torch_compile.py Resolved
pggPL and others added 11 commits October 30, 2026 13:42
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
pggPL added 4 commits November 4, 2026 15:11
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Keep the merged optimize_for_gemm implementation and adapt the LayerNormLinear compile path to the current check_gemm_dims signature.

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Restore Linear and fused-attention backward to main. Make normalization gradient setup optional and remove the unused SHAPE field kind and its test.

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
yravan added a commit to yravan/TransformerEngine that referenced this pull request Nov 19, 2026
yravan added a commit to yravan/TransformerEngine that referenced this pull request Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants