Skip to content

[PyTorch] [torch.compile] torch.compile support for LayerNormLinear and LayerNormMLP - #43

Open
pggPL wants to merge 17 commits into
mainfrom
layernorm_compile
Open

pggPL wants to merge 17 commits into
mainfrom
layernorm_compile

Conversation

@pggPL

@pggPL pggPL commented Sep 3, 2026 •

Copy link
Copy Markdown
Owner

Description

Add full-graph torch.compile support for LayerNormLinear and LayerNormMLP, including backward, dynamic batch dimensions and normalization outputs.

Targets main. This branch includes the NVFP4 packed-buffer, TensorSpec and Bias correctness fixes from NVIDIA/TransformerEngine#3628.

Changes

  • Describe forward outputs, saved tensors, aliases and backward gradients for both LayerNorm modules; retain eager fallbacks for unsupported configurations.

  • Route gradients from both normalization outputs through an optional backward hook. Share quantizer validation between the LayerNorm modules.

  • Validate both MLP GEMMs, including gated activations and sequence-parallel dimensions.

  • Support compiled FP8 weight caching in both LayerNorm modules, following Linear: replace the cache on the first microbatch and reuse it on subsequent microbatches, including both MLP weights. GPU-controlled cache updates and fused wgrad accumulation retain eager fallbacks.

  • Share the cache regression across Linear, LayerNormLinear and LayerNormMLP, checking independent eager outputs/gradients, cache refresh/reuse, optimizer updates, immediate/deferred backward and CUDA Graph replay.

  • Describe LayerNorm/RMSNorm statistics and activation outputs through dynamo.fake, using the quantizer layout contract merged in NVIDIA/TransformerEngine#3817. Preserve gathered LayerNormMLP output shapes.

  • Parameterize module compile tests across recipes, shapes and modes, with LayerNorm-specific outputs, gradients and eager fallbacks. Retain the non-matrix Linear regressions and expanded cold CUDA Graph recipe coverage inherited from [PyTorch] Fix NVFP4 metadata in compiled Linear operations NVIDIA/TransformerEngine#3628.

  • Cover metadata reconstruction and CustomRecipe MXFP8 2D cases in the compiled normalization scale-layout matrix.

Validation

2026-10-23 RMSNorm follow-up: the new metadata regression detects two failures on the old helper. The corrected selection passes 113 tests, with 83 hardware skips on Ada. The final helper also passes all 32 metadata cases after the lint-only condition refactor. Native MXFP8 2D compiled forward/backward cases are added for supported hardware but remain unverified on Ada. Existing native libraries reused.

2026-10-23, workstation RTX 5880 Ada, PyTorch 2.15.0a0+875d815502.nvinternal.main; existing native libraries reused, no fresh native build:

  • After merging NVIDIA/main dd02f3f7: fake-versus-real outputs, LayerNorm variants and inherited Linear cache selection: 85 passed, 15 skipped.
  • Final cache and eager-fallback selection: 21 passed, 36 hardware skips. The cache tests cover all three modules, five minibatches with optimizer updates, three microbatches each, both compile modes and immediate/deferred backward. Both MLP caches are checked. Float8CurrentScaling passes; Float8BlockScaling, MXFP8 and NVFP4 cache cases require supported hardware and remain unverified for this change.
  • Black 24.4.2, changed-module pylint, license, AST and whitespace checks pass. Full Python lint retains the pre-existing fused_mla_q_uproj.py:19 C0411 import-order diagnostic.

Historical validation before this merge/cache extension:

B100/SM100, PyTorch 2.15.0a0+875d815502.nvinternal.main, CUDA 13.4; existing October 7 native libraries reused, no fresh native build.

  • Focused LayerNorm compile selection: 180 passed, 4 expected skips for unsupported LayerNormMLP backward overrides. Includes full-graph forward/backward, non-matrix inputs, normalization output gradients, variants and eager fallbacks.
  • The selection includes 60 normalization scale-layout cases: both modules, LayerNorm/RMSNorm, MXFP8 and NVFP4, cuDNN settings, shapes, returned norm output, and NVFP4 RHT fusion enabled/disabled.
  • The inherited cold CUDA Graph regression is identical to the version validated in [PyTorch] Fix NVFP4 metadata in compiled Linear operations NVIDIA/TransformerEngine#3628: 48 passed with an independent eager model and no implicit recipe construction for unquantized inputs.
  • Pre-commit, license, syntax and whitespace checks passed. Full PyTorch Python lint retains an unchanged C0411 import-order diagnostic in attention/fused_mla_q_uproj.py in the GPU container; no unrelated source cleanup is included. Pylint on all changed production files passed.

No complete compile-suite rerun or distributed TP/SP/Userbuffers validation. The gathered-output test emulates the collective and does not establish multi-GPU correctness.

@pggPL
pggPL force-pushed the layernorm_compile branch from 90f05e3 to 9422845 Compare September 3, 2026 12:42
@pggPL
pggPL changed the base branch from layernorm_compile_refactor to layernorm_compile_prep September 3, 2026 12:42
@pggPL
pggPL force-pushed the layernorm_compile branch from 9422845 to 4cfffaf Compare September 3, 2026 12:49
@pggPL
pggPL requested a review from cyanguwa as a code owner September 17, 2026 11:00
@pggPL
pggPL changed the base branch from layernorm_compile_prep to main September 17, 2026 11:33
@pggPL
pggPL changed the base branch from main to fix_nvfp4_compile_nd October 12, 2026 15:32
@pggPL
pggPL changed the base branch from fix_nvfp4_compile_nd to main October 12, 2026 15:43
Register LayerNorm module forward and backward custom ops with matching fake tensor metadata, shared eligibility checks, and compiled weight caches. Extend the existing compile tests to cover the LayerNorm modules.

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
@pggPL
pggPL force-pushed the layernorm_compile branch from 70e4b7d to 83641e8 Compare October 29, 2026 12:07
pggPL and others added 16 commits October 30, 2026 13:18
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Keep the merged optimize_for_gemm implementation and adapt the LayerNormLinear compile path to the current check_gemm_dims signature.

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Restore Linear and fused-attention backward to main. Make normalization gradient setup optional and remove the unused SHAPE field kind and its test.

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant