Repository navigation
Conversation
_load_from_state_dict added '{prefix}bias = None' for a bias-free Linear op; load_state_dict(strict=True)
then reports it as an unexpected key (None-valued registered params are excluded from the module's own
local-state check). Only alias the bias when the op has one. Test: strict round trip with bias on and off.
…olkit The framework wheel (torch) is built against and ships the nvidia-* python packages; loading the system toolkit's copy first makes TE and torch run two different cuDNN versions in one process (cuDNN SDPA then fails with CUDNN_STATUS_BAD_PARAM after 'import transformer_engine'), and the fix was to export CUDNN_HOME and NVRTC_HOME by hand. Prefer the python packages when they are installed; the system search is unchanged as the fallback.
…al block reduce_dbias_kernel ran as a single 256-thread block (one thread per 4 columns, cols=1024 -> one block) looping serially over the [rows/128 x cols] fp32 partials: 0.116 ms per call at 900 x 1024, ~30 GB/s, on one SM, 200+ calls per training step of a ViT-L with biased linears. Add an in-place first stage that spreads the rows over up to 128 blocks per column slab (each block writes its slab sum into the first row of its own range), then run the existing one-block pass over those sums with a row stride.
…s_dgelu_erf gelu_erf and dgelu_erf are PyTorch's CUDA GeLU expressions and constants (F.gelu(approximate="none") and its backward), exposed as nvte_gelu_erf, nvte_dgelu_erf, nvte_geglu_erf, nvte_dgeglu_erf and nvte_quantize_dbias_dgelu_erf, the matching tex bindings, LayerNormMLP activation="gelu_erf"/"geglu_erf" and a te.ops GELUErf op (with the dbias fusion). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
torch.compile(dynamic=True) hands FusedAttention symbolic scalars, and the custom-op packer baked every scalar into the OpaqueValueBundle, which only takes Python constants: dynamo died with 'AsPythonConstantNotImplementedError: SymNodeVariable() is not a constant'. max_seqlen_q / max_seqlen_kv follow the input shape, so they get a SymInt schema slot (SymIntLike) and the fake impl sees them symbolically. Every other scalar is configuration: pack() specializes it through guard_scalar under tracing, so the graph guards on the value instead of failing on it. The softcap check is spelled as comparisons so a symbolic cap guards rather than graph-breaks on math.isfinite. Test: test_dpa_torch_compile_dynamic (flash / fused, dynamic=True). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Two places made dynamo guard a compiled DotProductAttention on state that differs per caller, so callers with two input shapes recompiled the frame on every switch and fell past the recompile limit into eager: - DotProductAttention compared the call's AttentionParams against a process-global cache and dynamo guarded the graph on the cache, so callers with different batch sizes recompiled each other on every switch. _select_attention_backends selects directly under compile; the cache is an eager-only memo. - Backend selection is a constant of the graph (assume_constant_result), which cannot take a symbolic argument. AttentionParams.__post_init__ now specializes every scalar field on construction under compile (batch size, sequence lengths, head count and size, ...), replacing the three ad-hoc guard_scalar calls in forward that covered only batch and sequence lengths; a fused-attention selection under dynamic shapes no longer graph-breaks. Outside a trace every field is concrete already and it returns at once. Tests: test_dpa_torch_compile_dynamic runs with fullgraph=True for both backends; test_dpa_compile_alternating_batch_sizes (2 graphs for 6 alternating calls). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Only BasicLinear had a custom op, so a Linear -> SwiGLU -> Linear fuser could never take the compiled path: it traced its eager implementation, graph-broke at every extension call and, under current scaling, crashed dynamo on the quantizer it called while traced. SwiGLU now has forward and backward custom ops in the BasicLinear pattern (SwiGLUFwdArgs / SwiGLUBwdArgs, forward_compute and backward_compute with their fake twins, pack_*_args, forward_setup_context); the eager path runs the same forward_compute. The input quantized for backward (cache_quantized_input) comes from a quantizer the op builds in its constructor, so the op args carry a value-opaque constant instead of constructing one in a traced region. SiTUGLU keeps its kernels through the same hooks, with its soft-cap parameters in subclass arg dataclasses. Outputs are described with fake_activation; the GLU interleaving helpers replace the three copies of the reshape. A te.ops Linear-SwiGLU-Linear stack compiles as one graph (fullgraph=True) under bf16, current scaling and MXFP8, with the saved input kept or quantized: test_te_ops_swiglu_compiles_fullgraph. The fusible-ops SwiGLU family tests cover the eager path. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… stored ones Under current scaling and MXFP8 a module class compiled per instance (as a transformer compiles its blocks: one code object, N instances) recompiled once per instance and fell past the recompile limit, or raised under fullgraph. Two guards on the stored quantizers caused it, both because a forward mutated them: - te.Linear / LayerNormLinear / LayerNormMLP configured usage, internal and optimize_for_gemm on self.quantizers[...] inside forward, and the backward flipped the usage again. Dynamo guards an opaque constant by equality, and the guard was recorded against an object whose value kept changing, so the next instance (in its post-backward state) never matched. - fake_norm passed that same stored quantizer through assume_constant_result, which guards an object argument read from a module attribute by identity. The stored quantizers now hold recipe state only (scale and amax buffers, reduction group, recipe parameters) and are configured at recipe reset. Every call configures its own copies: the modules take them from _call_quantizer(meta_key, index), so the in-forward usage, layout and amax-reduction-group changes (set_quantizer_amax_reduction_group no longer clobbers the stored one under compile) all land on per-call objects; in te.ops the call's quantizers live on its OperationContext, where the backward already kept them: pre_fuser_forward(ctx) publishes them (BasicLinear and GroupedLinear as configured copies, other ops as their recipe quantizers), the fuser reads the neighbours' from the contexts and the fused linear ops and the grouped MLP read their basic ops' from the same place. SwiGLU builds its saved-input quantizer in the constructor instead of on the first forward. The equality guard on a stored quantizer then holds across instances and across the forward/backward usage flips, and the fake_norm argument is a trace-local copy with no source. Tests: test_te_modules_compile_once_across_instances and test_te_ops_compile_once_across_instances (three instances, two training steps each, one graph) under current scaling and MXFP8. The base Quantizer gets a default shallow copy(): every attribute is shared, so recipe state stays common to the copies and only the usage and layout flags a call sets diverge, which is what the built-in copy() methods do by hand. A custom recipe's quantizers (and the reference quantizers shipped with TE) have no copy() of their own and failed at the first forward otherwise (test_custom_quantizer_default_copy). Not copy.copy, as a quantizer's __getstate__ may drop its reduction group for pickling. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…r's arguments
Every TE op under torch.compile went through two torch.library dispatches per
call: the wrapper op (the one a quantized-tensor subclass can enter) flattened
each subclass argument into the base op's slots and forwarded, and the base op
rebuilt the dataclass from a {slot: value} dict, rebuilt the subclass from its
slots, ran the kernel and packed the result under a torch.device context after
a pytree walk over every argument to find that device. The flattening built an
OpaqueValueBundle per subclass argument on each call, validating, freezing and
hashing its contents for Dynamo guards that only ever see trace-time bundles.
On a ViT-sized te.Linear that framing costs more host time than the op's
kernels take on the GPU (1.6 ms against 0.65 ms of GPU work per fwd+bwd), so a
compiled current-scaling model runs host-bound at 60 % GPU busy while the
delayed-scaling recipe, which cannot compile and runs its modules eagerly,
pays none of it.
The two tiers are a tracing need: a torch.library op cannot take a tensor
subclass, so while a FakeTensorMode is active the wrapper still flattens and
forwards to the base op, whose fake kernel sees the flat schema. A real call
now runs the real kernel directly on the wrapper's own arguments
(_make_slot_forwarder): the kernel takes the subclass whole, so nothing is
flattened, rebuilt, bundled or dispatched a second time. _ArgPlan.unpack reads
the flat arguments by slot position (slot_index, built once at registration);
result packing takes the op device from the first non-CPU tensor argument and
hands it to the None sentinels instead of entering a device context. The real
and fake kernels are built once by _make_kernels; _register_base_op only
registers them.
Verification (fork tests on H100, overlay of this tree): test_torch_compile
553 passed, test_fusible_ops bias/activation/linear 830 passed,
test_float8_current_scaling_exact 5 passed, test_numerics accuracy unchanged
(the 72 fp32 "small" failures are identical on the stock wheel). 12-layer
TEViTLayer stack, bs16, compiled: current scaling 60.3 -> 49.2 ms/step wall,
level with delayed scaling (48.8; GPU 10.1 vs 9.8 ms).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…tion Inductor's generated wrapper spells every opaque constant out as its __fx_repr__ constructor expression inline in the op call (val_to_arg_str -> get_opaque_obj_repr, no hoisting), so each call of a compiled TE op rebuilds its 3-5 bundles and 3-5 quantizers: 432 bundles and 384 quantizers per step on a 12-layer stack. A bundle validated, ordered and hashed its contents in __init__ for the guards Dynamo evaluates on the trace-time instance; on the per-call rebuilds, which are only read, that was the largest Python cost of the op path (17.8 of 102 ms of profiled Python time per step). The bundle now keys itself lazily, through two functools.cached_property members: _key (the validated, ordered, hashable contents) and _hash, computed on the first __hash__ or __eq__ (what the guard does with the trace-time bundle); __fx_repr__ validates on its own, since it formats the raw values and only runs at codegen. A rebuilt copy pays a dict literal and an object. Verification (fork tests on H100, overlay of this tree): test_torch_compile 553 passed, test_fusible_ops bias/activation/linear 830 passed, test_float8_current_scaling_exact 5 passed, test_numerics accuracy unchanged. 12-layer TEViTLayer stack, bs16, compiled under current scaling: cProfile loses OpaqueValueBundle.__init__ and its hashing chain (profiled Python time 102 -> 87 ms/step); wall 49.2 -> 40.0 ms/step. te.Linear fwd+bwd at 256 tokens: compiled 1,581 -> 1,323 us against 739 eager. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…dent bidimensional cast The bidimensional cast kernel keeps the whole 32-row tile in registers, so the activation runs on the registers it already holds: FP32 math (two lanes per f32x2 instruction, util/activation_x2.cuh), one BF16 rounding, then the existing quantization. The bias gradient is reduced per CTA from the same registers. Scales are written in the GEMM-swizzled layout when asked. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…l cast The gated cast keeps the 32-row tile of each half in registers and runs the activation, gate and gradient math on them, then both quantizations. One CTA covers 32 rows by 256 columns; the backward pass quantizes the activation and gate gradients as two planes of the same tile. Used for BF16 to E4M3 SiLU gating with rows % 32 == 0, cols % 256 == 0 and 32-byte aligned operands, otherwise the TMA kernel runs as before. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The backward reads ln_out and act_out only column-wise (weight-gradient GEMMs), but the forward saved both usages, keeping a dead row-wise fp8 copy of each per layer (Linear already trims it). At 86016x1024/4096 that is ~440 MB per layer. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…select, fma.sat in dgelu, swizzle index once per tile, packed dbias sums) Per 86016x4096 call on B200: gelu_erf fwd 0.451 -> 0.342 ms, dgelu+dbias 0.651 -> 0.554 ms. Outputs hash-identical to the previous kernel in all 29 harness configurations. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…per plane/tile 17600x5120 on B200: fwd 0.122 -> 0.115 ms, bwd 0.172 -> 0.156 ms; byte differences vs the generic kernel unchanged. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…s the narrow tile The fused activations are issue-bound (ncu at boost clocks, 86016x4096: 68% issue slots busy, 4.2 TB/s), so the 4-wide cluster that helps the cast-only kernel costs them 5-8%; the dbias+dgelu variant spilled 170 B/thread in the wide tile (two inputs plus 16 column sums in 64 registers) and runs spill-free in the narrow one. Per call at 86016x4096 on B200 (harness, cluster/tile now chosen by the dispatch): gelu_erf fwd 0.342 -> 0.327 ms, dbias+dgelu 0.554 -> 0.509 ms, dgelu 0.515 -> 0.488 ms. Outputs and scales hash-identical to the generic kernel on randn; the all-bf16 input differs only in the pre-existing NaN/Inf rowwise-scale bytes. Two persistent-CTA variants that prefetch the next tile (cp.async into shared memory; a second register buffer at 2 CTAs/SM) hid the load latency (long_scoreboard 2.0 -> 0.4 cycles per issue) but lost more elsewhere: 0.369 ms either way, and the plain cast fell from 0.210 to 0.266-0.282 ms, so the launch-per-tile structure stays. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… cast LayerNorm with an MXFP8 quantizer ran as two kernels: ln_fwd_tuned_kernel into a BF16 tensor, then quantize_mxfp8_kernel_cast_only (row-wise + column-wise, GEMM-swizzled scales). ln_fwd_mxfp8.cu does both in one persistent kernel: tiles of 32 rows arrive in shared memory by bulk copy while the previous tile is processed; each warp replays the tuned kernel's one-warp-per-row arithmetic (same reductions, FP32 ops and BF16 rounding), quantizes its rows from registers with the cast's instructions, parks the BF16 values in place, and the CTA quantizes the tile column-wise from shared memory. Output bytes, scales, mu and rsigma are identical to the two-kernel path for BF16 input and weights, rows a multiple of 128 and 768 or 1024 columns; everything else keeps the previous path. nvte_layernorm_fwd takes the fused kernel for a GEMM-swizzled MXFP8 output (previously rejected) or any supported MXFP8 output unless cuDNN is requested; the new nvte_layernorm_fwd_mxfp8_fused tells the PyTorch extension when to hand it the quantizer's tensor instead of normalizing to BF16 and quantizing. is_aligned_to moves to normalization/common.h, shared with the RMSNorm kernel. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…per lane) Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…cp.approx) Forward is bit-identical to the scalar kernel (0 / 49.9M elements). The backward is not: the scalar kernel used __frcp_rn, and with rcp.approx 0.012% of dact outputs differ by one rounding step (max 2.9e-11 on a 1.1e-8 gradient). Isolated kernel times at 60x448 rows x 4096: fwd 168 -> 117 us, bwd 216 -> 198 us. With __frcp_rn per lane it is bit-exact in both directions at 144 / 204 us. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…SE_FAST_MATH=1 Default (__frcp_rn) is bit-identical to the scalar kernel in forward, input gradient and weight gradients (0 / 49.9M elements; fwd 168 -> 142 us, bwd 218 -> 200 us at 60x448 rows x 4096). NVTE_USE_FAST_MATH=1 selects rcp.approx: fwd 116 us, bwd 199 us, 0.012% of backward elements differ by one rounding step. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The partial-sum reduction ran as two kernels (row blocks in place, then one block summing the slab sums serially): ~20 us per call at our shapes. One kernel now computes each group sum with the same interleave (8 accumulators, then in order) and the final in-order pass from shared memory. Output is bit-identical; 1024 cols 19.2 -> 5.2 us, 3072 cols 23.5 -> 6.4 us, dgelu 4096 cols x 3840 partial rows 36.0 -> 17.1 us (B200, idle GPU). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A swizzled quantize with rows or cols not a multiple of 128 launched one zero_scales_kernel per scale buffer before the cast (2-13 us each, ~400 per step on the 1856-wide backbone). The generic and TMA cast-only kernels now zero only the padding slots themselves; the register-resident paths keep the separate launch. Scale buffers (padding included) are bit-identical. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The two rows a warp owns are reduced as one f32x2 chain (same per-row order), gamma/beta are staged once as fp32 in shared memory, and the column-wise amax runs on bf16x2. Bit-identical outputs, mu and rsigma; 122880x1024 130.5 -> 125.1 us (B200, idle GPU). The kernel stays latency-bound at one CTA per SM. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…radient LayerNormLinear takes qkv_rope=(cos, sin, head_dim) when the caller applies RoPE to the q and k parts of its packed output. Its backward then rotates the q/k output gradient back inside the MXFP8 dbias quantize (nvte_quantize_dbias_rope, tex.bgrad_quantize(rope_cos=, rope_sin=, head_dim=, rope_cols=)): the kernel rotates each staged input tile in shared memory in FP32, rounds to BF16 and quantizes it, so the MXFP8 data, scales and dbias equal a separate rotation kernel followed by the plain dbias quantize. Other backward paths rotate with torch ops before quantizing. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… quantize rejects The fused dbias quantize asserts 'Unsupported RoPE layout' unless head_dim % 8 == 0, 64 % head_dim == 0, rot_dim % 4 == 0 and the q/k width is a multiple of 256. LayerNormLinear(qkv_rope=...) now checks the layout before choosing the fused path and otherwise applies qkv_rope_backward, as it already does outside MXFP8. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
mxfp8: inverse RoPE fused into the dbias quantize of a packed q/k/v gradient
mxfp8: one-launch dbias reduce, in-kernel scale padding, faster LN+MXFP8
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The subset of #1 that bf16 and MXFP8 training of our model needs (sunday-robotics/learning#1306 accepts
fp8_recipe=mxfp8only and pins the wheel built from this branch's tip,34e55a71). Seventeen commits onpr3534-base(a2f424e): ten from #1 (first table) and seven MXFP8 kernel commits (second table), each one change; fixups from #1 are folded into the commit they fix.load_state_dictof the backbone's bias-free te.ops LinearsCUDNN_STATUS_BAD_PARAM(system 9.13 loaded before torch's 9.24)reduce_dbias(quantize_mxfp8.cuh:1039); perf onlyfp8/quantize_fp8.cuh(delayed/current cast path)SymNodeVariable() is not a constant(bf16 jobs 53167 / 53368)MXFP8 kernels (on top of the ten above)
Dropped from #1 (delayed or current scaling only): e0a2977 (LayerNormMLP recompute delayed meta), 2dc6574 + 3a2fe51's module half + 09876a7 + 5527236 (delayed scaling runs outside dynamo), 5d036f3 and 572459a (f32x2 casts on the tensor-scaling path), 1e031a6 + 6a49aa7 + the LayerNormMLP hunk of 54a8270 (current-scaling fused dbias).
Verification
First ten commits (B200, wheel
2.22.0.dev0+972b4187, venv a128794165f53c6f):test_torch_compile856 passed, 2 failed;test_fusible_ops -k "bias or activation or linear or swiglu or state_dict"1574 passed;test_numerics -k "erf or dbias or layernorm_mlp"611 passed, 32 failed;test_custom_recipe51 passed.test_te_ops_bias_nvfp4_layout[rht,2d]: sympyBooleanAtom not allowed;test_layernorm_mlp_accuracy[*-small-*-dtype0], all fp32 "small", 32 on both).All seventeen commits (wheel
34e55a71, learning#1306 on the launch branch):bench_vjepa2_av1, bs56, MXFP8, B200: 1 node 44.1% MFU / 114.6 clips/s/GPU (job 62296); 2 nodes 42.85% (62294); 4 nodes 42.62% (62297).34e55a71.🤖 Generated with Claude Code