Skip to content

[PyTorch] Compile activation operations and fused bias gradients - #57

Draft
pggPL wants to merge 36 commits into
mainfrom
ops_activations_compile
Draft

pggPL wants to merge 36 commits into
mainfrom
ops_activations_compile

Conversation

@pggPL

@pggPL pggPL commented Oct 6, 2026 •

Copy link
Copy Markdown
Owner

Description

Enable full-graph compilation of te.ops.Sequential MLPs containing ordinary activations, including SwiGLU, and the fused activation gradient and bias reduction used by quantized GELU/ReLU MLPs. Support NVFP4 forward/backward, optimizer steps and CUDA Graph replay with matching eager results.

Targets main, including merged upstream #3817, which honors optimize_for_gemm in quantized outputs. The feature diff adds activation compilation in five files.

Type of change

  • New feature (non-breaking change which adds functionality)
  • Code refactoring

Changes

  • Add activation forward/backward and BackwardActivationBias custom ops using the existing argument packing, compute/fake and context setup contract.
  • Support quantized saved inputs, SwiGLU interleaving and dynamic batches.
  • Share activation output descriptions through dynamo.fake, using the quantizer layout guaranteed by main for forward and backward outputs. Preserve compute-dtype overrides for fused bias gradients.
  • Test representative activation/dtype/input-layout/cache combinations, AMP, dynamic batches, MLP training and CUDA Graphs. The activation matrix uses three pytest entry points with a shared checker.

Validation

Earlier validation before the October 20 main merge: B100/SM100, PyTorch 2.15.0a0+875d815502.nvinternal.main, CUDA 13.4; existing October 7 native libraries reused, no fresh native build.

  • Activation compile selection: 133 passed, including forward/backward, dtype/layout/cache combinations, AMP and quantized MLP training/CUDA Graph cases.
  • NVFP4 MLP cases compare outputs and gradients exactly to eager across four SGD steps, with RHT on/off and default/reduce-overhead modes. Stochastic rounding is disabled for these comparisons.
  • Before the merge, the inherited cold CUDA Graph regression passed 48 cases. The merge preserves the final, reduced NVFP4 regression matrix now on main; that matrix was not rerun in this update.
  • Pre-commit, license, syntax and whitespace checks passed. Full PyTorch Python lint retains an unchanged C0411 import-order diagnostic in attention/fused_mla_q_uproj.py in the GPU container; no unrelated source cleanup is included. Pylint on the changed production file passed. Removing the unused normalization helper leaves activation implementations unchanged.

November 12 update: merged upstream main at bb3be523 after NVIDIA#3817 landed and returned the PR base to main. Kept the merged native implementation and preserved the five-file activation feature diff, including compute-dtype overrides and the plain-gradient alias branch in fake_bgrad. Syntax, pre-commit and source-preservation checks passed. Runtime tests remain paused and were not rerun. CPU offload, delayed scaling, additional activation variants, collectives and extra tensor channels remain follow-ups.

Checklist:

  • I have read and followed the contributing guidelines
  • The functionality is complete
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

Draft pending author review.

@pggPL
pggPL force-pushed the ops_activations_compile branch from 3bc53ed to 7b65a95 Compare October 9, 2026 11:01
@pggPL
pggPL changed the base branch from ops_linear_bias_compile to main October 9, 2026 11:01
@pggPL
pggPL changed the base branch from main to fix_nvfp4_compile_nd October 12, 2026 14:25
@pggPL
pggPL changed the base branch from fix_nvfp4_compile_nd to main October 12, 2026 15:43
Add activation forward/backward and fused bias-gradient custom ops with shared fake output descriptions. Cover quantized saved inputs, gated and dynamic activations, AMP, and MLP training with CUDA Graphs.

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
@pggPL
pggPL force-pushed the ops_activations_compile branch from 6e34d17 to 3a60ea4 Compare October 20, 2026 16:09
pggPL and others added 23 commits October 26, 2026 06:57
Use the same fake_activation implementation as LayerNormMLP, including optional separate quantization. Preserve the default fused activation path and incorporate current upstream changes.

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
…to quantizer_gemm_ready_outputs

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>

# Conflicts:
#	transformer_engine/pytorch/csrc/common.h
#	transformer_engine/pytorch/csrc/extensions/cast.cpp
#	transformer_engine/pytorch/csrc/extensions/pybind.cpp
#	transformer_engine/pytorch/csrc/quantizer.cpp
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Add FUSED_NORM_QUANT_UNSWIZZLED to layernorm_fwd and handle compact
scale views and swizzling in the existing Impl switches of both
normalization functions instead of a separate optional path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Convert a reused GEMM output with the quantizer where the output tensor is
constructed instead of after the GEMM. Limit compact scale views and
batched swizzling to the grouped NVFP4 split-quantize kernel; other
multi-tensor paths swizzle per tensor in quantize().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Quantize NVFP4 outputs whose kernel cannot write GEMM-swizzled scales
into compact scratch views and swizzle them with one kernel in
multi_tensor_quantize and split_quantize, while outputs with in-kernel
swizzling are written directly. Declare the internal swizzle helpers in
util.h next to the existing ones.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
pggPL and others added 5 commits November 5, 2026 19:41
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
@pggPL
pggPL changed the base branch from main to quantizer_gemm_ready_outputs November 9, 2026 09:31
Use the merged optimize_for_gemm implementation while preserving activation custom ops and their fake metadata contracts.

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
@pggPL
pggPL changed the base branch from quantizer_gemm_ready_outputs to main November 12, 2026 14:55
Reduce 137 cases to 73 while retaining each activation and representative layout, cache, autocast, dynamic-shape, quantization, and CUDA Graph coverage. Exercise cached FP8 input in fused GELU bias backward and aligned NVFP4 RHT output shapes.

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant