Skip to content

[PyTorch] Add torch.compile custom ops for GroupedLinear - #32

Draft
pggPL wants to merge 36 commits into
grouped_linear_refactorfrom
grouped_linear_compile
Draft

pggPL wants to merge 36 commits into
grouped_linear_refactorfrom
grouped_linear_compile

Conversation

@pggPL

@pggPL pggPL commented Aug 13, 2026 •

Copy link
Copy Markdown
Owner

Description

Add torch.compile(fullgraph=True) support to te.pytorch.GroupedLinear on top of the eager refactor in #50. Both custom ops reuse the shared forward/backward computation from that PR.

This is the second of two stacked PRs; its base is grouped_linear_refactor.

Type of change

  • New feature (non-breaking change which adds functionality)

Changes

  • Add custom-op registrations and fake implementations for both paths, sharing weight/cache metadata, gradient allocation descriptions and saved-alias restoration.
  • Extend _FieldPlan / _ArgPlan with tensor-list packing, per-element gradients, subclass flattening and SymInt[] inputs. Flatten mixed tensor/storage lists in one pass, and cover empty SymInt lists in autograd with a regression test.
  • Deduplicate saved-tensor aliases and apply compile-specific allocation restrictions. Preallocate the grouped cuBLAS workspace for CUDA-graph capture.
  • Use shared compile eligibility checks and dispatch for both paths, with path-specific restrictions and dimension checks reusing Linear's guards. Compile tests share numerical comparisons, including bias gradients across weight-cache updates.

Validation

RTX Ada workstation, current source:

  • test_torch_compile.py + test_grouped_linear.py: 849 passed, 608 skipped, 2 XPASS.
  • Additional fused BF16 fullgraph=True probe: forward/backward numerics and changing device splits without recompilation pass with grouped GEMM replaced by matmuls. This does not validate native fused kernels or CUDA-graph capture.
  • A temporary mixed plain-tensor/FP8-wrapper/storage/None list probe preserves exact values and buffer ordering.
  • Changed production files pass pylint; Black and diff checks pass. Full Python lint reports the existing import-order diagnostic in unchanged attention/fused_mla_q_uproj.py.

The installed PyTorch 2.14 development build exposes is_opaque_value_type instead of the is_opaque_constant_type name expected by TE main. Tests used a temporary compatibility alias outside the repository.

Native fused grouped GEMM tests require Hopper/Blackwell and were skipped on RTX Ada.

Limitations

Unsupported configurations fall back to eager: debug instrumentation, distributed weights, CPU offloading, delayed wgrad, fused wgrad accumulation, caller-provided output buffers, single grouped parameters and non-value-opaque quantizers. Legacy compiled execution requires host-side splits; fused execution requires CUDA splits. Fused save_original_input=True falls back to eager.

Checklist

  • I have read and followed the contributing guidelines
  • New and existing tests pass on the available GPU
  • Changes introduce no lint diagnostics
  • Native fused compilation revalidated on Hopper/Blackwell

@pggPL
pggPL force-pushed the grouped_linear_compile branch from 926952e to 45a2c8b Compare August 13, 2026 10:27
@pggPL
pggPL changed the base branch from linear_compile_on_main to linear_torch_compile_final_attempt August 13, 2026 10:27
@pggPL pggPL changed the title [PyTorch] [torch.compile] torch.compile support for GroupedLinear (plan) [PyTorch] [torch.compile] torch.compile support for GroupedLinear Aug 13, 2026
@pggPL
pggPL force-pushed the grouped_linear_compile branch from e431045 to 2859669 Compare September 10, 2026 10:33
@pggPL
pggPL changed the base branch from linear_torch_compile_final_attempt to main September 10, 2026 10:33
@pggPL
pggPL force-pushed the grouped_linear_compile branch from 2859669 to 71732da Compare September 10, 2026 10:44
@pggPL
pggPL changed the base branch from main to grouped_linear_refactor September 10, 2026 10:45
@pggPL
pggPL force-pushed the grouped_linear_compile branch from 71732da to 2db50bd Compare September 10, 2026 11:24
@pggPL
pggPL force-pushed the grouped_linear_refactor branch from 50ca413 to d007d1c Compare September 10, 2026 11:24
@pggPL pggPL changed the title [PyTorch] [torch.compile] torch.compile support for GroupedLinear [PyTorch] Add torch.compile custom ops for GroupedLinear Sep 10, 2026
@pggPL
pggPL force-pushed the grouped_linear_compile branch from 2db50bd to c89b414 Compare September 10, 2026 11:31
Reuse eager computation for both grouped paths and add fake implementations, registrations, saved-state serialization and compile dispatch. Extend argument packing and autograd routing for tensor lists and SymInt lists, including empty lists, with compile regression coverage.

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
@pggPL
pggPL force-pushed the grouped_linear_refactor branch from d007d1c to a3c005d Compare September 11, 2026 11:34
@pggPL
pggPL force-pushed the grouped_linear_compile branch from c89b414 to 151f2b2 Compare September 11, 2026 11:34
pggPL and others added 15 commits September 18, 2026 18:52
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
…aving

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Expose grouped buffers through the existing custom-op field plan and preserve their logical metadata when saving for backward. Describe grouped weight allocations and packed gradients in fake implementations, and reuse cached workspaces across compiled microbatches.

Validate fused recipes and parameter layouts under fullgraph and CUDA graphs, primary quantized weights, cache reuse, and gradients and optimizer updates against independent linear operations.

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Resolve saved input and output references in the custom-op framework and use grouped storage metadata for fused inputs. Share real and fake saved-weight handling and cover dynamic shapes, saved-tensor hooks, and autograd version checks.

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Derive backward return schemas from differentiable forward fields, retaining scalar and list gradients across the operator boundary. Remove flat-gradient slicing and cursor bookkeeping while preserving input-shape restoration.

Cover empty gradient lists, missing gradients, and zero or one gradient field through eager and compiled custom-op calls.

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Gather selected sigmoid scores from the saved unbiased intermediate
instead of subtracting expert bias after top-k. Adding the bias can
round away tiny scores, which subtraction cannot recover.

Keep expert selection, QB histograms, and other score functions unchanged.
Cover tiny and mixed scores across top-k sizes, QB modes, and output maps.

Signed-off-by: Philip Monk <pmonk@nvidia.com>
Co-authored-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
* Build FP8 recipe quantizer params per instance

Float8CurrentScaling and Float8BlockScaling created fp8_quant_* QParams at class definition, so constructor values for use_power_2_scales / use_f32_scales were ignored. Construct them in __post_init__ instead.

Signed-off-by: varun-51 <nakarvarun77@gmail.com>

* Rebuild FP8 recipe quantizer params when unpickling older recipes

Recipes pickled by earlier releases lack the fp8_quant_* instance attributes, and unpickling does not call __post_init__. Rebuild them in __setstate__.

Signed-off-by: varun-51 <nakarvarun77@gmail.com>

---------

Signed-off-by: varun-51 <nakarvarun77@gmail.com>
Preserve grouped scale dtype metadata during tensor flattening and pass the forward output width to compiled backward setup.

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
…IDIA#3635)

* [PyTorch] Allow deterministic dprob for the GLU fused grouped MLP

Feature-detect the cuDNN dactivation wrapper's `deterministic` argument in the
base class instead of only for dSReLU, so the GLU path (dGLU) can honor
NVTE_ALLOW_NONDETERMINISTIC_ALGO=0 / torch.use_deterministic_algorithms once the
installed cuDNN front-end carries NVIDIA/cudnn-frontend#1319, rather than raising.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Zhiyu Li <zhiyul@nvidia.com>

* [PyTorch] Report cuDNN's deterministic-dprob refusal as TE's error

Which configurations cuDNN can make deterministic (e.g. dGLU: no dbias, no Rubin kernel,
discrete weights only from cudnn-frontend#1414) is cuDNN's to decide. Instead of mirroring
those rules in TE, keep the capability a pure feature-detect of the wrapper's
`deterministic` argument and translate cuDNN's NotImplementedError into TE's determinism
RuntimeError at the call. Drops the cuDNN FE >= 1.31.0 gate added for the same purpose.

Tests check for TE's error by exact type: cuDNN's NotImplementedError subclasses
RuntimeError. New test_glu_fc1_bias_refuses covers an FC1-bias-only GLU.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Zhiyu Li <zhiyul@nvidia.com>

* [PyTorch] Accept either TE determinism refusal in test_glu_fc1_bias_refuses

On a front-end whose dGLU wrapper lacks `deterministic`, TE refuses in its early check rather
than by re-raising cuDNN's refusal. Match the prefix both errors share instead of the
re-raise wording; cuDNN's own NotImplementedError still fails the test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Zhiyu Li <zhiyul@nvidia.com>

* [PyTorch] Drop implementation-detail determinism tests

Per review: test_capability_follows_the_wrapper and test_glu_fc1_bias_refuses check how
the capability and the error re-raise are implemented rather than deterministic
execution itself. Keep test_dprob_is_bit_exact_across_runs (SReLU and GLU).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Zhiyu Li <zhiyul@nvidia.com>

* Update transformer_engine/pytorch/ops/fused/grouped_mlp.py

Co-authored-by: Tim Moon <4406448+timmoon10@users.noreply.github.com>
Signed-off-by: Zhiyu Li <zhiyul@NVIDIA.com>

* [PyTorch] Accept cuDNN's NotImplementedError in the determinism refusal test

With the re-raise removed, a configuration cuDNN cannot make deterministic (e.g. GLU on
Rubin) now surfaces cuDNN's NotImplementedError directly. Accept it alongside TE's own
refusal; any other RuntimeError still fails the test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Zhiyu Li <zhiyul@nvidia.com>

---------

Signed-off-by: Zhiyu Li <zhiyul@nvidia.com>
Signed-off-by: Zhiyu Li <zhiyul@NVIDIA.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Tim Moon <4406448+timmoon10@users.noreply.github.com>
… fit (NVIDIA#3600)

* [Common] Fall back from gated TMA kernels when shared memory does not fit

The delayed-scaling gated activation dispatch selected the TMA kernel
(cast_fp8_gated_kernel) on every device with compute capability 10.0+,
without checking its dynamic shared memory request against the device
limit. On SM 12.0 (opt-in limit 99 KiB per block) FP32 configurations
request up to 160 KiB, so cudaFuncSetAttribute fails with
"invalid argument" in both forward and backward (issue NVIDIA#3299).

Compute the kernel's shared memory requirement in one helper shared by
dispatch and launch, compare it with the device's cached opt-in
per-block limit, and use the existing non-TMA kernels when it does not
fit. Configurations that fit (e.g. BF16/FP16 on SM 12.0, everything on
SM 10.0) keep using the TMA kernel.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Ravi M <ravitejamajeti@gmail.com>

* [Common] Read gated TMA kernel static shared memory from the compiled kernel

The fit check hard-coded the static shared memory of cast_fp8_gated_kernel
as its mbarrier array (32 B) and missed the staging array declared in
reduce_max (64 B). Query the compiled kernel instance instead via
cudaFuncGetAttributes, exposed as cuda::static_shared_memory_size() and
cached per kernel and device, so __shared__ arrays in called device
functions are always included.

Also replace the file-local get_max_dynamic_smem() in swizzle.cu with
cuda::max_shared_memory_per_block_optin().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Ravi M <ravitejamajeti@gmail.com>

---------

Signed-off-by: Ravi M <ravitejamajeti@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…DIA#3547)

* [PyTorch] Compile Linear and Bias operations with forward fusion

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>

* [PyTorch] Preserve quantized tensors for dequantized backward

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>

* [PyTorch] Share Linear and Bias compute between eager and compile

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>

* [PyTorch] Enable compiled BasicLinear tensor parallelism and address review

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>

* [PyTorch] Simplify ops backward bookkeeping and clarify compile tests

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>

* [PyTorch] Pass custom op context directly to operation compute

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>

* [PyTorch] Fix ops compile regressions and clarify test scenarios

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>

* [PyTorch] Match saved FP8 fake tensor dtypes to sources

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>

* [PyTorch] Specialize BasicLinear compile ops by TP size

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>

* [PyTorch] Move compiled bias coverage into Linear tests

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>

---------

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
pggPL and others added 20 commits October 8, 2026 23:24
* [PyTorch] Split FusedAttnFunc into single-argument forward/backward helpers

Behavior-neutral restructuring of the cuDNN fused attention autograd
function, mirroring what NVIDIA#2967 did for Linear: FusedAttnFwdArgs /
FusedAttnBwdArgs dataclasses, module-level _fused_attn_forward_impl,
_fused_attn_setup_ctx and _fused_attn_backward_impl, and a thin
FusedAttnFunc wrapper taking the differentiable tensors plus one args
object instead of 37 positional arguments.

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>

* [PyTorch] Match the fused attention aux pack split to the C++ conditions

The bias / softmax_offset aux entries exist only when the tensor was
passed, not merely when the bias / softmax type asks for one.

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>

* [PyTorch] torch.compile support for FusedAttention

Register the cuDNN fused attention forward/backward as a torch custom op
through dynamo/custom_op.py, reusing the single-argument helpers behind
FusedAttnFunc, so DotProductAttention with the fused backend traces under
torch.compile(fullgraph=True), including KV caching. F16/BF16 only; FP8,
context parallelism, score_mod, FAv2 backward and CPU offloading keep
falling back to eager via no_torch_dynamo(when=...).

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>

* [PyTorch] Preserve packed gradients in compiled fused attention

Differentiate packed QKV and KV inputs directly and return their native packed gradient buffers without contiguous copies or repacking. Cover packed backward graphs, FP16/BF16 gradients and CUDA graph replay, and rename the custom-op-unavailable fallback test.

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>

* [PyTorch] Simplify compiled fused attention output handling

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>

* [PyTorch] Remove redundant fused attention gradient conversions

DPA routes undeclared packed QKV views to eager before custom-op dispatch. Return the independently allocated gradients directly for the remaining unpacked path.

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>

* [PyTorch] Unify fused attention backward state setup

Store packed gradient metadata in FusedAttnBwdArgs and initialize all backward state in the shared setup function. Reconstruct omitted QKV views in the shared forward and setup, removing the custom-op-specific dataclass and forwarding wrappers.

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>

* Keep fused attention packed inputs in saved backward state

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>

* Consolidate packed attention compile tests into shared configurations

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>

* Declare fused attention RNG effects and align fake statistics

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>

* Address fused attention compile review feedback

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>

---------

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
* Fix TMA shared-memory reuse synchronization

Signed-off-by: Oleg Goncharov <ogoncharov@nvidia.com>

* Fix remaining TMA synchronization gaps

Signed-off-by: Oleg Goncharov <ogoncharov@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Oleg Goncharov <ogoncharov@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Przemyslaw Tredak <ptredak@nvidia.com>
* [Common] Fuse RMSNorm forward with MXFP8 quantization

Signed-off-by: Siddhartha Raman Sundara Raman <sraman@nvidia.com>
Signed-off-by: Tim Moon <4406448+timmoon10@users.noreply.github.com>
Co-authored-by: Tim Moon <4406448+timmoon10@users.noreply.github.com>
Co-authored-by: vthumbe1503 <vthumbe@nvidia.com>
)

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>
* [Common] Enable all activations for MXFP8 C++ Unittest

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* nit

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* also enable other activations for grouped MXFP8

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

---------

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>
… test (NVIDIA#3608)

Reset context-parallel pools before retry

Persistent workers for different world sizes may share GPUs. Restarting only the failed worker can leave CUDA and NCCL state from another pool resident, so the retry starts under the same resource pressure.

Shut down every active pool before retrying. Pool objects remain registered and restart lazily when requested, preserving reuse on the normal path.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
…3656)

Requantize grouped MXFP8 tensors with 128x128 tiles

Signed-off-by: Siddhartha Raman S <sraman@nvidia.com>
Co-authored-by: Phuong Nguyen <phuonguyen@nvidia.com>
…led scale (NVIDIA#3599)

* fix grouped MXFP8 quantization with varying last dim & swizzled scale

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

* nit

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>

---------

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>
Co-authored-by: Oleg Goncharov <64355998+Oleg-Goncharov@users.noreply.github.com>
…VIDIA#3658)

* [Common] Swizzle MXFP8 scaling factors with one CTA per scale tile

The MXFP8 scale swizzle is latency bound. The narrow-K and narrow-M
kernels put up to 32 128x4 scale tiles in one 1024-thread CTA, so the
scaling factors of a 4096-row activation are swizzled by a single CTA.

Give every 128-row M-tile (row-wise) or 128-column x 32-row block
(column-wise) its own CTA, which moves a contiguous chunk with coalesced
16-byte loads and stores through shared memory. Multi-tensor launches
group tiles per CTA only when the grid would otherwise be very large.
The existing kernels remain for NVFP4, unaligned buffers, and row-wise
scales too wide for 48 KB of shared memory.

Signed-off-by: Siddhartha Raman S <sraman@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Test full and clipped 128-row column-wise swizzle chunks

The column-wise multi-tensor case that selects 128-row chunks had only
four scale rows. Add one with 132 scale rows per tensor: a full 128-row
chunk and a clipped last chunk.

Signed-off-by: Siddhartha Raman S <sraman@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [Common] Make the per-tile MXFP8 swizzle kernels opt-in

Select the per-tile MXFP8 scale-swizzle kernels with
NVTE_MXFP8_PER_TILE_SWIZZLE=1; by default, the existing kernels run as
before. Give the per-tile code descriptive names (for example
TileSwizzleDesc -> PerTileSwizzleTensor, tpc -> tiles_per_block,
k4_magic -> words_per_row_magic), document the variable, and run the
single- and multi-tensor swizzle tests with it unset and set.

Signed-off-by: Siddhartha Raman S <sraman@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Signed-off-by: Siddhartha Raman S <sraman@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
NVIDIA#3662)

With one group, the fused grouped MLP passes the weights to cuDNN as
stacked tensors instead of per-group pointers, and two of its NVFP4
views of them were wrong:

- Forward: the FC1 weight scale view divided the logical K by the
  scale-atom divisor, which counts packed bytes for NVFP4 (two values
  per byte). The view asked for twice the scales and raised.
- Backward: the FC2 dgrad viewed the column-wise NVFP4 data as
  (out_features, in_features / 2), but it is the transposed weight,
  (in_features, out_features / 2), and swapped the scale atom axes,
  which cuDNN rejects. Build these operands with
  _convert_to_cudnn_grouped_gemm_tensor_format, as the FC1 dgrad does.

Run the single-group test for every grouped MLP recipe, not only MXFP8.

Signed-off-by: Siddhartha Raman S <sraman@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…#3667)

Six RuntimeError messages in the fusible-op API build one string out of four
fragments and leave the f off the first one, so the brace expression that
names the offending subclass is emitted verbatim:

  {self.__class__.__name__} operation has 1 extra tensor inputs and 0 extra
  tensor outputs. It should override `fuser_forward` instead of `op_forward`.

The two fragments after it are f-strings and do interpolate, so the counts
come out right and only the class name is lost, which is the one part of the
message a caller cannot work out for themselves.

The same four-fragment message appears six more times in the same tree, in
basic/swiglu.py, basic/activation.py and basic/grouped_linear.py, with the f
in place. These six are the ones that were missed.

No test: reaching these lines needs a built extension and a custom
BasicOperation subclass that declares extra tensor inputs or outputs and
implements op_forward rather than fuser_forward, and nothing in the tree
asserts on the text. The behaviour is a property of the language, not of TE.

Signed-off-by: Dmitry Voropaev <dy.voropaev@gmail.com>
* Update FA4 D=256 capability gates

FA4 b31 added seqused_q/k forward support for the SM100 D=256 kernel, while b33 added backward support. Select FA4 by the installed release and actual backward eligibility, retain the deterministic backward guard, and exercise the released b33 dependency stack in L3.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>

* Cover FA4 D=256 seqused paths

Protect the release boundary with focused selector cases and extend the existing no-load-balance CP test to run D=256. This verifies the existing all-gather metadata path without adding strategy-specific production plumbing.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>

* Integrate FA4 window sentinel handling into D=256 coverage

FA4 b33 treats TE's -1 unbounded window sentinel as a literal offset, so
the D=256 coverage in this PR needs the adapter from NVIDIA#3532 at all four FA4
entry points. Bring that final adapter change into this branch instead of
leaving the new FA4 path dependent on another open PR.

Keep the regression tests in test_attention_backend_selection.py alongside
the focused FA4 tests. Cover both window argument forms and preserved
negative offsets, compare every batch and head with a float64 reference,
and run these cases under the FA4 b33 L3 lane.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>

* Shorten FA4 sentinel regression explanations

Keep the adapter comment focused on the sentinel contract and the FA4-only boundary. Name the numerical test for its independent reference and remove the redundant all-zero assertion, which singled out one symptom already caught by the full-output comparison.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>

* Remove redundant FA4 float64 reference test

The standalone D=128 BSHD comparison does not exercise the D=256 THD path added by this PR. Keep the direct window-sentinel regression in L3; existing attention tests provide numerical comparisons.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>

* Restore FA4 numerical sentinel regression

The argument-only sentinel test cannot prove the adapted FA4 call produces correct attention; generic cross-backend comparisons may skip. Restore the independent reference check and keep only representative left, right, and non-sentinel cases for both window argument forms.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>

* Update FA4 L3 dependency and backend coverage

CUTLASS DSL 4.8.0 recognizes SM107 while the previous pin does not, so keep the FA4 L3 dependency and installation guidance aligned. Running the small backend-selection module directly covers its eligibility and window-sentinel cases without a bespoke node list. Clarify that non-all-gather THD metadata depends on pad_between_seqs.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>

* Keep FA4 regression tests focused on model configs

A dedicated D=256 model preserves separate QK and V dimensions without a one-off runner override. Exclude it from the broad CP matrix because the no-load-balance test already exercises the intended path. The L3 causal cross-backend cases cover numerical behavior, so retain the sentinel unit test and remove the redundant standalone reference test.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>

* Keep FA4 expectation explicit in CP test

Query backend availability before deciding whether to skip the D=256 case. Supported FA4 b33 configurations should fail when FA4 is unexpectedly unavailable, while unsupported configurations still skip. Keep the general no-Flash skip separate from the D=256 assertion.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>

* Keep full CP model catalogs for focused tests

The earlier D=256 broad-matrix exclusion kept cp_2_4 in the essential FlashAttention catalog solely for the focused no-load-balance case. That mixes model availability with test selection. Keep the full FlashAttention and FusedAttention catalogs available to the runner and focused tests, and narrow only the essential broad CP parametrizations. Non-essential broad runs continue to use the full catalog.

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>

---------

Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
…NVIDIA#3672)

* [Common][PyTorch] Fuse the MLA key/value RoPE with MXFP8 quantization

For MXFP8 attention, multi-latent attention writes its key and value in
BF16 (the key/value split, with the shared key position embedding
rotated and appended to every head's key) and then reads them again to
quantize them. Add a kernel that reads the key/value up-projection
output and the key position embedding once and writes the MXFP8 key and
value directly: the same bytes as mxfp8_quantize_only applied to the
RoPE'd key and the value, row-wise and column-wise, with compact
scaling factors.

The RoPE is that of Megatron-LM's MLA kernels, with their roundings, for
DeepSeek-V3's head dimensions (128 + 64 key, 128 value) on SM 10.0+. It
is exposed as nvte_fused_mla_kv_rope_mxfp8 and, for PyTorch, as
fused_mla_kv_rope_mxfp8 next to mxfp8_quantize_only, whose re-wrap of
the quantized 2D view in the input's 4D shape moves into a helper that
both use.

At [4096, 1, 128 heads] on VR200, the key/value split and quantization
take 100.1 us instead of 60.8 + 75.6 us.

Signed-off-by: Siddhartha Raman S <sraman@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [PyTorch] Address review of the fused MLA key/value RoPE + MXFP8 kernel

- Run fused_mla_kv_rope_mxfp8 on the inputs' device: check that kv, k_rope,
  cos and sin are on one CUDA device, guard it, and allocate key and value
  there.
- Reject MXFP8 quantizers with 2D quantization, which the kernel does not
  implement.
- Rename k_pos_emb to k_rope: it holds the key's RoPE dimensions, not a
  position embedding table.
- Merge test_fused_mla_q_uproj.py and test_fused_mla_kv_rope_mxfp8.py into
  test_fused_mla.py, and test the device handling and the 2D quantizer check.

Signed-off-by: Siddhartha Raman S <sraman@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Signed-off-by: Siddhartha Raman S <sraman@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
bump nccl ep

Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com>
… down NCCL (NVIDIA#3655)

* Distributed tests set device in NCCL init and manually shuts down NCCL

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Fully destroy NCCL environment when finishing distributed TE ops test

Signed-off-by: Tim Moon <tmoon@nvidia.com>

* Apply suggestion from @timmoon10

Signed-off-by: Tim Moon <4406448+timmoon10@users.noreply.github.com>

* Apply suggestion from @timmoon10

Signed-off-by: Tim Moon <4406448+timmoon10@users.noreply.github.com>

---------

Signed-off-by: Tim Moon <tmoon@nvidia.com>
Signed-off-by: Tim Moon <4406448+timmoon10@users.noreply.github.com>
* [PyTorch] Allow ZeroTensor dummy weight gradients

With fuse_wgrad_accumulation, the modules accumulate the weight gradient
into main_grad and return a dummy weight gradient. When one backward
pass uses a weight twice, autograd adds the two dummies; for an LM head
shared by the main and MTP heads in Megatron-LM that is a [vocab, hidden]
add whose result is discarded (0.44 ms for 129280 x 7168 on VR200).

A parameter with zero_tensor_dummy_wgrad set now gets a ZeroTensor
dummy, which has no storage and costs nothing to add. zero_out_wgrad
keeps precedence and its zero-filled dummy; other parameters get the
dummy they get today.

Signed-off-by: Siddhartha Raman S <sraman@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [PyTorch] Get LayerNormMLP's dummy weight gradients from get_dummy_wgrad

The FC1 and FC2 branches for zero_out_wgrad, zero_tensor_dummy_wgrad and
plain dummies reimplemented get_dummy_wgrad. Call it instead, as Linear,
LayerNormLinear and GroupedLinear do.

Signed-off-by: Siddhartha Raman S <sraman@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [PyTorch] Key the cached dummy weight gradients by device

get_dummy_wgrad cached its dummies by shape and dtype only, so after a
backward pass on one GPU a same-shaped weight on another GPU got the first
GPU's tensor, which autograd rejects ("expected device cuda:1 but got
cuda:0"). This affected Linear, LayerNormLinear, ops.Linear and, since it
now uses get_dummy_wgrad, LayerNormMLP. The dummies are now allocated on
the current device and cached per device.

Test the device of the dummy weight gradient when the same module shape
runs backward on two GPUs. GroupedLinear is left out because its
multi-stream GEMM already fails on a second device in the forward pass.

Signed-off-by: Siddhartha Raman S <sraman@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Signed-off-by: Siddhartha Raman S <sraman@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* [PyTorch] Return the argmax from parallel_cross_entropy

The forward kernel of parallel_cross_entropy already scans every row of
logits for its maximum. With the new keyword-only return_argmax=True it
also returns the index of that maximum, so a caller that needs the
predicted tokens, such as an accuracy metric, does not read the logits a
second time.

The index is the one torch.argmax returns over the unsharded logits: the
first one when several logits are equal. With tensor parallelism, each
rank's local index travels with the statistics in the existing
all-gather (one more FP32 column, as raw bits), and the first rank that
holds the global maximum provides it. Without return_argmax the kernels
compile as before; with it, loss and gradient are bit-identical.

Signed-off-by: Siddhartha Raman S <sraman@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [PyTorch] Count NaN as the maximum in the parallel_cross_entropy argmax

tl.max(..., return_indices=True) is not NaN-safe: its comparisons are false
for NaN, so a block with a NaN could give an index that is neither the
maximum nor the NaN ([8, -10, NaN, -10] gave 1). The argmax now follows
torch.argmax: NaN counts as the maximum and the first NaN wins. Each block
finds its first NaN separately and takes the maximum over the other values,
in a running maximum kept apart from the loss's; with tensor parallelism
each rank also shares that maximum, NaN if its shard holds a NaN. Loss and
gradient are unchanged.

Signed-off-by: Siddhartha Raman S <sraman@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [PyTorch] Test for NaN through one helper in the cross entropy kernels

pylint flags x != x as comparison-with-itself, which failed the lint of
transformer_engine/common. The three NaN tests now call one _is_nan helper
that keeps the comparison and disables that check on its line.

Signed-off-by: Siddhartha Raman S <sraman@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Signed-off-by: Siddhartha Raman S <sraman@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Change to enable CuteDSL DenseGEMM for eval

Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com>

* Add unit test for Single group grouped_gemm for no_grad and frozen_weights use cases

Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com>

---------

Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com>
* [PyTorch] Refactor GroupedLinear eager forward and backward

Extract shared computation and backward state for both split-quantize and grouped-tensor execution. Preserve eager saved-tensor layout and gradient allocation so compile adapters can reuse these functions in a follow-up change.

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Preserve GroupedLinear behavior across the eager refactor

Derive backward feature dimensions from validated forward tensors so local
distributed-weight shards do not determine GEMM buffer shapes. Restore
the fused input-width check and retain disabled weight-gradient stores
so they can be enabled between forward and backward. Capture only the
accumulator flag in deferred fused wgrad callbacks to release cast weights
after dgrad.

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>

* [PyTorch] Preserve GroupedLinear behavior across refactor

Restore the grad-output bulk-allocation and debug dbias conditions from
the original implementation. Preserve quantizers selected from materialized
weights and apply the common quantizer setup on both execution paths.

Read GEMM accumulator settings from the saved recipe when each GEMM runs,
including the default settings for high-precision backward overrides.
Adapt the backward-override test to the extracted backward state holder
without changing its assertions or numerical tolerances.

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>

* [PyTorch] Reduce GroupedLinear Python overhead

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>

---------

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Resolve custom-op tags and graph-safe GroupedLinear conflicts. Keep eager recipe handling and pass scalar GEMM accumulator settings across the compile boundary.

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.