Repository navigation
Conversation
pggPL
force-pushed
the
grouped_linear_compile
branch
from
August 13, 2026 10:27
926952e to
45a2c8b
Compare
pggPL
changed the base branch from
linear_compile_on_main
to
linear_torch_compile_final_attempt
August 13, 2026 10:27
pggPL
force-pushed
the
grouped_linear_compile
branch
from
September 10, 2026 10:33
e431045 to
2859669
Compare
pggPL
changed the base branch from
linear_torch_compile_final_attempt
to
main
September 10, 2026 10:33
pggPL
force-pushed
the
grouped_linear_compile
branch
from
September 10, 2026 10:44
2859669 to
71732da
Compare
5 tasks done
pggPL
force-pushed
the
grouped_linear_compile
branch
from
September 10, 2026 11:24
71732da to
2db50bd
Compare
pggPL
force-pushed
the
grouped_linear_refactor
branch
from
September 10, 2026 11:24
50ca413 to
d007d1c
Compare
pggPL
force-pushed
the
grouped_linear_compile
branch
from
September 10, 2026 11:31
2db50bd to
c89b414
Compare
Reuse eager computation for both grouped paths and add fake implementations, registrations, saved-state serialization and compile dispatch. Extend argument packing and autograd routing for tensor lists and SymInt lists, including empty lists, with compile regression coverage. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
pggPL
force-pushed
the
grouped_linear_refactor
branch
from
September 11, 2026 11:34
d007d1c to
a3c005d
Compare
pggPL
force-pushed
the
grouped_linear_compile
branch
from
September 11, 2026 11:34
c89b414 to
151f2b2
Compare
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
…aving Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
for more information, see https://pre-commit.ci
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Expose grouped buffers through the existing custom-op field plan and preserve their logical metadata when saving for backward. Describe grouped weight allocations and packed gradients in fake implementations, and reuse cached workspaces across compiled microbatches. Validate fused recipes and parameter layouts under fullgraph and CUDA graphs, primary quantized weights, cache reuse, and gradients and optimizer updates against independent linear operations. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Resolve saved input and output references in the custom-op framework and use grouped storage metadata for fused inputs. Share real and fake saved-weight handling and cover dynamic shapes, saved-tensor hooks, and autograd version checks. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Derive backward return schemas from differentiable forward fields, retaining scalar and list gradients across the operator boundary. Remove flat-gradient slicing and cursor bookkeeping while preserving input-shape restoration. Cover empty gradient lists, missing gradients, and zero or one gradient field through eager and compiled custom-op calls. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Gather selected sigmoid scores from the saved unbiased intermediate instead of subtracting expert bias after top-k. Adding the bias can round away tiny scores, which subtraction cannot recover. Keep expert selection, QB histograms, and other score functions unchanged. Cover tiny and mixed scores across top-k sizes, QB modes, and output maps. Signed-off-by: Philip Monk <pmonk@nvidia.com> Co-authored-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
* Build FP8 recipe quantizer params per instance Float8CurrentScaling and Float8BlockScaling created fp8_quant_* QParams at class definition, so constructor values for use_power_2_scales / use_f32_scales were ignored. Construct them in __post_init__ instead. Signed-off-by: varun-51 <nakarvarun77@gmail.com> * Rebuild FP8 recipe quantizer params when unpickling older recipes Recipes pickled by earlier releases lack the fp8_quant_* instance attributes, and unpickling does not call __post_init__. Rebuild them in __setstate__. Signed-off-by: varun-51 <nakarvarun77@gmail.com> --------- Signed-off-by: varun-51 <nakarvarun77@gmail.com>
Preserve grouped scale dtype metadata during tensor flattening and pass the forward output width to compiled backward setup. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
…IDIA#3635) * [PyTorch] Allow deterministic dprob for the GLU fused grouped MLP Feature-detect the cuDNN dactivation wrapper's `deterministic` argument in the base class instead of only for dSReLU, so the GLU path (dGLU) can honor NVTE_ALLOW_NONDETERMINISTIC_ALGO=0 / torch.use_deterministic_algorithms once the installed cuDNN front-end carries NVIDIA/cudnn-frontend#1319, rather than raising. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Zhiyu Li <zhiyul@nvidia.com> * [PyTorch] Report cuDNN's deterministic-dprob refusal as TE's error Which configurations cuDNN can make deterministic (e.g. dGLU: no dbias, no Rubin kernel, discrete weights only from cudnn-frontend#1414) is cuDNN's to decide. Instead of mirroring those rules in TE, keep the capability a pure feature-detect of the wrapper's `deterministic` argument and translate cuDNN's NotImplementedError into TE's determinism RuntimeError at the call. Drops the cuDNN FE >= 1.31.0 gate added for the same purpose. Tests check for TE's error by exact type: cuDNN's NotImplementedError subclasses RuntimeError. New test_glu_fc1_bias_refuses covers an FC1-bias-only GLU. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Zhiyu Li <zhiyul@nvidia.com> * [PyTorch] Accept either TE determinism refusal in test_glu_fc1_bias_refuses On a front-end whose dGLU wrapper lacks `deterministic`, TE refuses in its early check rather than by re-raising cuDNN's refusal. Match the prefix both errors share instead of the re-raise wording; cuDNN's own NotImplementedError still fails the test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Zhiyu Li <zhiyul@nvidia.com> * [PyTorch] Drop implementation-detail determinism tests Per review: test_capability_follows_the_wrapper and test_glu_fc1_bias_refuses check how the capability and the error re-raise are implemented rather than deterministic execution itself. Keep test_dprob_is_bit_exact_across_runs (SReLU and GLU). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Zhiyu Li <zhiyul@nvidia.com> * Update transformer_engine/pytorch/ops/fused/grouped_mlp.py Co-authored-by: Tim Moon <4406448+timmoon10@users.noreply.github.com> Signed-off-by: Zhiyu Li <zhiyul@NVIDIA.com> * [PyTorch] Accept cuDNN's NotImplementedError in the determinism refusal test With the re-raise removed, a configuration cuDNN cannot make deterministic (e.g. GLU on Rubin) now surfaces cuDNN's NotImplementedError directly. Accept it alongside TE's own refusal; any other RuntimeError still fails the test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Zhiyu Li <zhiyul@nvidia.com> --------- Signed-off-by: Zhiyu Li <zhiyul@nvidia.com> Signed-off-by: Zhiyu Li <zhiyul@NVIDIA.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Tim Moon <4406448+timmoon10@users.noreply.github.com>
… fit (NVIDIA#3600) * [Common] Fall back from gated TMA kernels when shared memory does not fit The delayed-scaling gated activation dispatch selected the TMA kernel (cast_fp8_gated_kernel) on every device with compute capability 10.0+, without checking its dynamic shared memory request against the device limit. On SM 12.0 (opt-in limit 99 KiB per block) FP32 configurations request up to 160 KiB, so cudaFuncSetAttribute fails with "invalid argument" in both forward and backward (issue NVIDIA#3299). Compute the kernel's shared memory requirement in one helper shared by dispatch and launch, compare it with the device's cached opt-in per-block limit, and use the existing non-TMA kernels when it does not fit. Configurations that fit (e.g. BF16/FP16 on SM 12.0, everything on SM 10.0) keep using the TMA kernel. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Ravi M <ravitejamajeti@gmail.com> * [Common] Read gated TMA kernel static shared memory from the compiled kernel The fit check hard-coded the static shared memory of cast_fp8_gated_kernel as its mbarrier array (32 B) and missed the staging array declared in reduce_max (64 B). Query the compiled kernel instance instead via cudaFuncGetAttributes, exposed as cuda::static_shared_memory_size() and cached per kernel and device, so __shared__ arrays in called device functions are always included. Also replace the file-local get_max_dynamic_smem() in swizzle.cu with cuda::max_shared_memory_per_block_optin(). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Ravi M <ravitejamajeti@gmail.com> --------- Signed-off-by: Ravi M <ravitejamajeti@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…DIA#3547) * [PyTorch] Compile Linear and Bias operations with forward fusion Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> * [PyTorch] Preserve quantized tensors for dequantized backward Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> * [PyTorch] Share Linear and Bias compute between eager and compile Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> * [PyTorch] Enable compiled BasicLinear tensor parallelism and address review Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> * [PyTorch] Simplify ops backward bookkeeping and clarify compile tests Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> * [PyTorch] Pass custom op context directly to operation compute Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> * [PyTorch] Fix ops compile regressions and clarify test scenarios Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> * [PyTorch] Match saved FP8 fake tensor dtypes to sources Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> * [PyTorch] Specialize BasicLinear compile ops by TP size Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> * [PyTorch] Move compiled bias coverage into Linear tests Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> --------- Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
* [PyTorch] Split FusedAttnFunc into single-argument forward/backward helpers Behavior-neutral restructuring of the cuDNN fused attention autograd function, mirroring what NVIDIA#2967 did for Linear: FusedAttnFwdArgs / FusedAttnBwdArgs dataclasses, module-level _fused_attn_forward_impl, _fused_attn_setup_ctx and _fused_attn_backward_impl, and a thin FusedAttnFunc wrapper taking the differentiable tensors plus one args object instead of 37 positional arguments. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> * [PyTorch] Match the fused attention aux pack split to the C++ conditions The bias / softmax_offset aux entries exist only when the tensor was passed, not merely when the bias / softmax type asks for one. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> * [PyTorch] torch.compile support for FusedAttention Register the cuDNN fused attention forward/backward as a torch custom op through dynamo/custom_op.py, reusing the single-argument helpers behind FusedAttnFunc, so DotProductAttention with the fused backend traces under torch.compile(fullgraph=True), including KV caching. F16/BF16 only; FP8, context parallelism, score_mod, FAv2 backward and CPU offloading keep falling back to eager via no_torch_dynamo(when=...). Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> * [PyTorch] Preserve packed gradients in compiled fused attention Differentiate packed QKV and KV inputs directly and return their native packed gradient buffers without contiguous copies or repacking. Cover packed backward graphs, FP16/BF16 gradients and CUDA graph replay, and rename the custom-op-unavailable fallback test. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> * [PyTorch] Simplify compiled fused attention output handling Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> * [PyTorch] Remove redundant fused attention gradient conversions DPA routes undeclared packed QKV views to eager before custom-op dispatch. Return the independently allocated gradients directly for the remaining unpacked path. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> * [PyTorch] Unify fused attention backward state setup Store packed gradient metadata in FusedAttnBwdArgs and initialize all backward state in the shared setup function. Reconstruct omitted QKV views in the shared forward and setup, removing the custom-op-specific dataclass and forwarding wrappers. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> * Keep fused attention packed inputs in saved backward state Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> * Consolidate packed attention compile tests into shared configurations Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> * Declare fused attention RNG effects and align fake statistics Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> * Address fused attention compile review feedback Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> --------- Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
* Fix TMA shared-memory reuse synchronization Signed-off-by: Oleg Goncharov <ogoncharov@nvidia.com> * Fix remaining TMA synchronization gaps Signed-off-by: Oleg Goncharov <ogoncharov@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Oleg Goncharov <ogoncharov@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Przemyslaw Tredak <ptredak@nvidia.com>
* [Common] Fuse RMSNorm forward with MXFP8 quantization Signed-off-by: Siddhartha Raman Sundara Raman <sraman@nvidia.com> Signed-off-by: Tim Moon <4406448+timmoon10@users.noreply.github.com> Co-authored-by: Tim Moon <4406448+timmoon10@users.noreply.github.com> Co-authored-by: vthumbe1503 <vthumbe@nvidia.com>
* [Common] Enable all activations for MXFP8 C++ Unittest Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * nit Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * also enable other activations for grouped MXFP8 Signed-off-by: Kaining Zhong <kainingz@nvidia.com> --------- Signed-off-by: Kaining Zhong <kainingz@nvidia.com>
… test (NVIDIA#3608) Reset context-parallel pools before retry Persistent workers for different world sizes may share GPUs. Restarting only the failed worker can leave CUDA and NCCL state from another pool resident, so the retry starts under the same resource pressure. Shut down every active pool before retrying. Pool objects remain registered and restart lazily when requested, preserving reuse on the normal path. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
…3656) Requantize grouped MXFP8 tensors with 128x128 tiles Signed-off-by: Siddhartha Raman S <sraman@nvidia.com> Co-authored-by: Phuong Nguyen <phuonguyen@nvidia.com>
…led scale (NVIDIA#3599) * fix grouped MXFP8 quantization with varying last dim & swizzled scale Signed-off-by: Kaining Zhong <kainingz@nvidia.com> * nit Signed-off-by: Kaining Zhong <kainingz@nvidia.com> --------- Signed-off-by: Kaining Zhong <kainingz@nvidia.com> Co-authored-by: Oleg Goncharov <64355998+Oleg-Goncharov@users.noreply.github.com>
…VIDIA#3658) * [Common] Swizzle MXFP8 scaling factors with one CTA per scale tile The MXFP8 scale swizzle is latency bound. The narrow-K and narrow-M kernels put up to 32 128x4 scale tiles in one 1024-thread CTA, so the scaling factors of a 4096-row activation are swizzled by a single CTA. Give every 128-row M-tile (row-wise) or 128-column x 32-row block (column-wise) its own CTA, which moves a contiguous chunk with coalesced 16-byte loads and stores through shared memory. Multi-tensor launches group tiles per CTA only when the grid would otherwise be very large. The existing kernels remain for NVFP4, unaligned buffers, and row-wise scales too wide for 48 KB of shared memory. Signed-off-by: Siddhartha Raman S <sraman@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Test full and clipped 128-row column-wise swizzle chunks The column-wise multi-tensor case that selects 128-row chunks had only four scale rows. Add one with 132 scale rows per tensor: a full 128-row chunk and a clipped last chunk. Signed-off-by: Siddhartha Raman S <sraman@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [Common] Make the per-tile MXFP8 swizzle kernels opt-in Select the per-tile MXFP8 scale-swizzle kernels with NVTE_MXFP8_PER_TILE_SWIZZLE=1; by default, the existing kernels run as before. Give the per-tile code descriptive names (for example TileSwizzleDesc -> PerTileSwizzleTensor, tpc -> tiles_per_block, k4_magic -> words_per_row_magic), document the variable, and run the single- and multi-tensor swizzle tests with it unset and set. Signed-off-by: Siddhartha Raman S <sraman@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Signed-off-by: Siddhartha Raman S <sraman@nvidia.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
NVIDIA#3662) With one group, the fused grouped MLP passes the weights to cuDNN as stacked tensors instead of per-group pointers, and two of its NVFP4 views of them were wrong: - Forward: the FC1 weight scale view divided the logical K by the scale-atom divisor, which counts packed bytes for NVFP4 (two values per byte). The view asked for twice the scales and raised. - Backward: the FC2 dgrad viewed the column-wise NVFP4 data as (out_features, in_features / 2), but it is the transposed weight, (in_features, out_features / 2), and swapped the scale atom axes, which cuDNN rejects. Build these operands with _convert_to_cudnn_grouped_gemm_tensor_format, as the FC1 dgrad does. Run the single-group test for every grouped MLP recipe, not only MXFP8. Signed-off-by: Siddhartha Raman S <sraman@nvidia.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…#3667) Six RuntimeError messages in the fusible-op API build one string out of four fragments and leave the f off the first one, so the brace expression that names the offending subclass is emitted verbatim: {self.__class__.__name__} operation has 1 extra tensor inputs and 0 extra tensor outputs. It should override `fuser_forward` instead of `op_forward`. The two fragments after it are f-strings and do interpolate, so the counts come out right and only the class name is lost, which is the one part of the message a caller cannot work out for themselves. The same four-fragment message appears six more times in the same tree, in basic/swiglu.py, basic/activation.py and basic/grouped_linear.py, with the f in place. These six are the ones that were missed. No test: reaching these lines needs a built extension and a custom BasicOperation subclass that declares extra tensor inputs or outputs and implements op_forward rather than fuser_forward, and nothing in the tree asserts on the text. The behaviour is a property of the language, not of TE. Signed-off-by: Dmitry Voropaev <dy.voropaev@gmail.com>
* Update FA4 D=256 capability gates FA4 b31 added seqused_q/k forward support for the SM100 D=256 kernel, while b33 added backward support. Select FA4 by the installed release and actual backward eligibility, retain the deterministic backward guard, and exercise the released b33 dependency stack in L3. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com> * Cover FA4 D=256 seqused paths Protect the release boundary with focused selector cases and extend the existing no-load-balance CP test to run D=256. This verifies the existing all-gather metadata path without adding strategy-specific production plumbing. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com> * Integrate FA4 window sentinel handling into D=256 coverage FA4 b33 treats TE's -1 unbounded window sentinel as a literal offset, so the D=256 coverage in this PR needs the adapter from NVIDIA#3532 at all four FA4 entry points. Bring that final adapter change into this branch instead of leaving the new FA4 path dependent on another open PR. Keep the regression tests in test_attention_backend_selection.py alongside the focused FA4 tests. Cover both window argument forms and preserved negative offsets, compare every batch and head with a float64 reference, and run these cases under the FA4 b33 L3 lane. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com> * Shorten FA4 sentinel regression explanations Keep the adapter comment focused on the sentinel contract and the FA4-only boundary. Name the numerical test for its independent reference and remove the redundant all-zero assertion, which singled out one symptom already caught by the full-output comparison. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com> * Remove redundant FA4 float64 reference test The standalone D=128 BSHD comparison does not exercise the D=256 THD path added by this PR. Keep the direct window-sentinel regression in L3; existing attention tests provide numerical comparisons. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com> * Restore FA4 numerical sentinel regression The argument-only sentinel test cannot prove the adapted FA4 call produces correct attention; generic cross-backend comparisons may skip. Restore the independent reference check and keep only representative left, right, and non-sentinel cases for both window argument forms. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com> * Update FA4 L3 dependency and backend coverage CUTLASS DSL 4.8.0 recognizes SM107 while the previous pin does not, so keep the FA4 L3 dependency and installation guidance aligned. Running the small backend-selection module directly covers its eligibility and window-sentinel cases without a bespoke node list. Clarify that non-all-gather THD metadata depends on pad_between_seqs. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com> * Keep FA4 regression tests focused on model configs A dedicated D=256 model preserves separate QK and V dimensions without a one-off runner override. Exclude it from the broad CP matrix because the no-load-balance test already exercises the intended path. The L3 causal cross-backend cases cover numerical behavior, so retain the sentinel unit test and remove the redundant standalone reference test. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com> * Keep FA4 expectation explicit in CP test Query backend availability before deciding whether to skip the D=256 case. Supported FA4 b33 configurations should fail when FA4 is unexpectedly unavailable, while unsupported configurations still skip. Keep the general no-Flash skip separate from the D=256 assertion. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com> * Keep full CP model catalogs for focused tests The earlier D=256 broad-matrix exclusion kept cp_2_4 in the essential FlashAttention catalog solely for the focused no-load-balance case. That mixes model availability with test selection. Keep the full FlashAttention and FusedAttention catalogs available to the runner and focused tests, and narrow only the essential broad CP parametrizations. Non-essential broad runs continue to use the full catalog. Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com> --------- Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
…NVIDIA#3672) * [Common][PyTorch] Fuse the MLA key/value RoPE with MXFP8 quantization For MXFP8 attention, multi-latent attention writes its key and value in BF16 (the key/value split, with the shared key position embedding rotated and appended to every head's key) and then reads them again to quantize them. Add a kernel that reads the key/value up-projection output and the key position embedding once and writes the MXFP8 key and value directly: the same bytes as mxfp8_quantize_only applied to the RoPE'd key and the value, row-wise and column-wise, with compact scaling factors. The RoPE is that of Megatron-LM's MLA kernels, with their roundings, for DeepSeek-V3's head dimensions (128 + 64 key, 128 value) on SM 10.0+. It is exposed as nvte_fused_mla_kv_rope_mxfp8 and, for PyTorch, as fused_mla_kv_rope_mxfp8 next to mxfp8_quantize_only, whose re-wrap of the quantized 2D view in the input's 4D shape moves into a helper that both use. At [4096, 1, 128 heads] on VR200, the key/value split and quantization take 100.1 us instead of 60.8 + 75.6 us. Signed-off-by: Siddhartha Raman S <sraman@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [PyTorch] Address review of the fused MLA key/value RoPE + MXFP8 kernel - Run fused_mla_kv_rope_mxfp8 on the inputs' device: check that kv, k_rope, cos and sin are on one CUDA device, guard it, and allocate key and value there. - Reject MXFP8 quantizers with 2D quantization, which the kernel does not implement. - Rename k_pos_emb to k_rope: it holds the key's RoPE dimensions, not a position embedding table. - Merge test_fused_mla_q_uproj.py and test_fused_mla_kv_rope_mxfp8.py into test_fused_mla.py, and test the device handling and the 2D quantizer check. Signed-off-by: Siddhartha Raman S <sraman@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Signed-off-by: Siddhartha Raman S <sraman@nvidia.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
bump nccl ep Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com>
… down NCCL (NVIDIA#3655) * Distributed tests set device in NCCL init and manually shuts down NCCL Signed-off-by: Tim Moon <tmoon@nvidia.com> * Fully destroy NCCL environment when finishing distributed TE ops test Signed-off-by: Tim Moon <tmoon@nvidia.com> * Apply suggestion from @timmoon10 Signed-off-by: Tim Moon <4406448+timmoon10@users.noreply.github.com> * Apply suggestion from @timmoon10 Signed-off-by: Tim Moon <4406448+timmoon10@users.noreply.github.com> --------- Signed-off-by: Tim Moon <tmoon@nvidia.com> Signed-off-by: Tim Moon <4406448+timmoon10@users.noreply.github.com>
* [PyTorch] Allow ZeroTensor dummy weight gradients
With fuse_wgrad_accumulation, the modules accumulate the weight gradient
into main_grad and return a dummy weight gradient. When one backward
pass uses a weight twice, autograd adds the two dummies; for an LM head
shared by the main and MTP heads in Megatron-LM that is a [vocab, hidden]
add whose result is discarded (0.44 ms for 129280 x 7168 on VR200).
A parameter with zero_tensor_dummy_wgrad set now gets a ZeroTensor
dummy, which has no storage and costs nothing to add. zero_out_wgrad
keeps precedence and its zero-filled dummy; other parameters get the
dummy they get today.
Signed-off-by: Siddhartha Raman S <sraman@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* [PyTorch] Get LayerNormMLP's dummy weight gradients from get_dummy_wgrad
The FC1 and FC2 branches for zero_out_wgrad, zero_tensor_dummy_wgrad and
plain dummies reimplemented get_dummy_wgrad. Call it instead, as Linear,
LayerNormLinear and GroupedLinear do.
Signed-off-by: Siddhartha Raman S <sraman@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* [PyTorch] Key the cached dummy weight gradients by device
get_dummy_wgrad cached its dummies by shape and dtype only, so after a
backward pass on one GPU a same-shaped weight on another GPU got the first
GPU's tensor, which autograd rejects ("expected device cuda:1 but got
cuda:0"). This affected Linear, LayerNormLinear, ops.Linear and, since it
now uses get_dummy_wgrad, LayerNormMLP. The dummies are now allocated on
the current device and cached per device.
Test the device of the dummy weight gradient when the same module shape
runs backward on two GPUs. GroupedLinear is left out because its
multi-stream GEMM already fails on a second device in the forward pass.
Signed-off-by: Siddhartha Raman S <sraman@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Signed-off-by: Siddhartha Raman S <sraman@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* [PyTorch] Return the argmax from parallel_cross_entropy The forward kernel of parallel_cross_entropy already scans every row of logits for its maximum. With the new keyword-only return_argmax=True it also returns the index of that maximum, so a caller that needs the predicted tokens, such as an accuracy metric, does not read the logits a second time. The index is the one torch.argmax returns over the unsharded logits: the first one when several logits are equal. With tensor parallelism, each rank's local index travels with the statistics in the existing all-gather (one more FP32 column, as raw bits), and the first rank that holds the global maximum provides it. Without return_argmax the kernels compile as before; with it, loss and gradient are bit-identical. Signed-off-by: Siddhartha Raman S <sraman@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [PyTorch] Count NaN as the maximum in the parallel_cross_entropy argmax tl.max(..., return_indices=True) is not NaN-safe: its comparisons are false for NaN, so a block with a NaN could give an index that is neither the maximum nor the NaN ([8, -10, NaN, -10] gave 1). The argmax now follows torch.argmax: NaN counts as the maximum and the first NaN wins. Each block finds its first NaN separately and takes the maximum over the other values, in a running maximum kept apart from the loss's; with tensor parallelism each rank also shares that maximum, NaN if its shard holds a NaN. Loss and gradient are unchanged. Signed-off-by: Siddhartha Raman S <sraman@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [PyTorch] Test for NaN through one helper in the cross entropy kernels pylint flags x != x as comparison-with-itself, which failed the lint of transformer_engine/common. The three NaN tests now call one _is_nan helper that keeps the comparison and disables that check on its line. Signed-off-by: Siddhartha Raman S <sraman@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Signed-off-by: Siddhartha Raman S <sraman@nvidia.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Change to enable CuteDSL DenseGEMM for eval Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com> * Add unit test for Single group grouped_gemm for no_grad and frozen_weights use cases Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com> --------- Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com>
* [PyTorch] Refactor GroupedLinear eager forward and backward Extract shared computation and backward state for both split-quantize and grouped-tensor execution. Preserve eager saved-tensor layout and gradient allocation so compile adapters can reuse these functions in a follow-up change. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Preserve GroupedLinear behavior across the eager refactor Derive backward feature dimensions from validated forward tensors so local distributed-weight shards do not determine GEMM buffer shapes. Restore the fused input-width check and retain disabled weight-gradient stores so they can be enabled between forward and backward. Capture only the accumulator flag in deferred fused wgrad callbacks to release cast weights after dgrad. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> * [PyTorch] Preserve GroupedLinear behavior across refactor Restore the grad-output bulk-allocation and debug dbias conditions from the original implementation. Preserve quantizers selected from materialized weights and apply the common quantizer setup on both execution paths. Read GEMM accumulator settings from the saved recipe when each GEMM runs, including the default settings for high-precision backward overrides. Adapt the backward-override test to the extracted backward state holder without changing its assertions or numerical tolerances. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> * [PyTorch] Reduce GroupedLinear Python overhead Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> --------- Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Resolve custom-op tags and graph-safe GroupedLinear conflicts. Keep eager recipe handling and pass scalar GEMM accumulator settings across the compile boundary. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Add
torch.compile(fullgraph=True)support tote.pytorch.GroupedLinearon top of the eager refactor in #50. Both custom ops reuse the shared forward/backward computation from that PR.This is the second of two stacked PRs; its base is
grouped_linear_refactor.Type of change
Changes
_FieldPlan/_ArgPlanwith tensor-list packing, per-element gradients, subclass flattening andSymInt[]inputs. Flatten mixed tensor/storage lists in one pass, and cover empty SymInt lists in autograd with a regression test.Validation
RTX Ada workstation, current source:
test_torch_compile.py+test_grouped_linear.py: 849 passed, 608 skipped, 2 XPASS.fullgraph=Trueprobe: forward/backward numerics and changing device splits without recompilation pass with grouped GEMM replaced by matmuls. This does not validate native fused kernels or CUDA-graph capture.attention/fused_mla_q_uproj.py.The installed PyTorch 2.14 development build exposes
is_opaque_value_typeinstead of theis_opaque_constant_typename expected by TE main. Tests used a temporary compatibility alias outside the repository.Native fused grouped GEMM tests require Hopper/Blackwell and were skipped on RTX Ada.
Limitations
Unsupported configurations fall back to eager: debug instrumentation, distributed weights, CPU offloading, delayed wgrad, fused wgrad accumulation, caller-provided output buffers, single grouped parameters and non-value-opaque quantizers. Legacy compiled execution requires host-side splits; fused execution requires CUDA splits. Fused
save_original_input=Truefalls back to eager.Checklist