Skip to content

lemi fixes for bf16 + MXFP8 on top of #3534 (minimal subset of #1) - #2

Draft
yravan wants to merge 27 commits into
pr3534-basefrom
lemi/mxfp8-minimal
Draft

yravan wants to merge 27 commits into
pr3534-basefrom
lemi/mxfp8-minimal

Conversation

@yravan

@yravan yravan commented Nov 6, 2026 •

Copy link
Copy Markdown
Owner

The subset of #1 that bf16 and MXFP8 training of our model needs (sunday-robotics/learning#1306 accepts fp8_recipe=mxfp8 only and pins the wheel built from this branch's tip, 34e55a71). Seventeen commits on pr3534-base (a2f424e): ten from #1 (first table) and seven MXFP8 kernel commits (second table), each one change; fixups from #1 are folded into the commit they fix.

commit from #1 needed for
ops.Linear: do not inject a None bias key b6e8fcc strict load_state_dict of the backbone's bias-free te.ops Linears
Load cuDNN/NVRTC/cuRAND from the python packages f6d3934 cuDNN attention CUDNN_STATUS_BAD_PARAM (system 9.13 loaded before torch's 9.24)
cast: parallel dbias reduction 884267b MXFP8 dbias calls the same reduce_dbias (quantize_mxfp8.cuh:1039); perf only
activation: exact (erf) GELU ce4d79c, without its f32x2 hunk in fp8/quantize_fp8.cuh (delayed/current cast path) V-JEPA 2's exact GELU
dynamo: fused attention under dynamic shapes dd19e95 SymNodeVariable() is not a constant (bf16 jobs 53167 / 53368)
attention: no per-call-site guards 3a2fe51 (attention half) + 0139fd7 + the guard_scalar import from 54a8270 recompiles on every vision batch-size switch
ops: SwiGLU and SiTUGLU custom ops c312c8e + 9778ed4 backbone MLP as one graph (probe 178677: 8 recompiles, frame skipped)
compile: per-call quantizer copies 53a43ef + 5c8aa8b + the BasicLinear hunk of 54a8270 per-instance recompiles / fullgraph failure (178852)
compile: real custom-op call runs the kernel directly ad6c94d host overhead of every compiled TE op
compile: OpaqueValueBundle keys itself on first hash 9e51bb8 + 3f1bccc host overhead

MXFP8 kernels (on top of the ten above)

commit change measured
293eca9 activation, dactivation and dbias fused into the register-resident bidimensional cast
e818cf4 SwiGLU and dSwiGLU fused into a register-resident bidimensional cast
bd3c328 LayerNormMLP: drop the row-wise MXFP8 copies of the saved FC1/FC2 inputs
376dce7 faster register-resident act-quant (erf both-polynomials-then-select, fma.sat in dgelu, swizzle index once per tile, packed dbias sums) gelu fwd 0.452 to 0.341 ms (-24%), dgelu+dbias 0.652 to 0.542 ms (-17%) at 86016x4096, bit-exact
e6d0764 gated SwiGLU act-quant computes the swizzled scale index once per plane/tile bit-exact (same measurement as 376dce7)
602e958 act-quant launches unclustered; the bias-gradient variant takes the narrow tile
34e55a7 fused LayerNorm forward with MXFP8 output bit-exact with LN + cast (data, both scale tensors, mu/rsigma, edge values); 0.1241 vs 0.1662 ms at 114688x1024 (-25%)

Dropped from #1 (delayed or current scaling only): e0a2977 (LayerNormMLP recompute delayed meta), 2dc6574 + 3a2fe51's module half + 09876a7 + 5527236 (delayed scaling runs outside dynamo), 5d036f3 and 572459a (f32x2 casts on the tensor-scaling path), 1e031a6 + 6a49aa7 + the LayerNormMLP hunk of 54a8270 (current-scaling fused dbias).

Verification

First ten commits (B200, wheel 2.22.0.dev0+972b4187, venv a128794165f53c6f):

  • TE: test_torch_compile 856 passed, 2 failed; test_fusible_ops -k "bias or activation or linear or swiglu or state_dict" 1574 passed; test_numerics -k "erf or dbias or layernorm_mlp" 611 passed, 32 failed; test_custom_recipe 51 passed.
  • The 34 failures are pre-existing: the lemi fixes on top of #3534 (torch.compile for LayerNormLinear/LayerNormMLP) #1 tip wheel (54a8270) fails the same ones (test_te_ops_bias_nvfp4_layout[rht,2d]: sympy BooleanAtom not allowed; test_layernorm_mlp_accuracy[*-small-*-dtype0], all fp32 "small", 32 on both).
  • learning#1306 on this wheel: te_fp8 / te_layers / vjepa2 TE block 43 passed; vjepa2 encoder + video module 23 passed; train_step / utils / ddp_resume 26 passed.

All seventeen commits (wheel 34e55a71, learning#1306 on the launch branch):

  • bench_vjepa2_av1, bs56, MXFP8, B200: 1 node 44.1% MFU / 114.6 clips/s/GPU (job 62296); 2 nodes 42.85% (62294); 4 nodes 42.62% (62297).
  • SOTA two-stage post-train, 1 node, global batch 1024 as bs32 x ga4, same node: MXFP8 1.39 s/step and 89.0 GB vs bf16 1.55 s/step and 104.8 GB (jobs 62399 vs 62395). Loss parity not shown: the MXFP8 run was stopped at step ~2,600 of 18,925.
  • The TE test suites above were not re-run on 34e55a71.

🤖 Generated with Claude Code

Yajvan Ravan and others added 25 commits November 6, 2026 00:03
_load_from_state_dict added '{prefix}bias = None' for a bias-free Linear op; load_state_dict(strict=True)
then reports it as an unexpected key (None-valued registered params are excluded from the module's own
local-state check). Only alias the bias when the op has one. Test: strict round trip with bias on and off.
…olkit

The framework wheel (torch) is built against and ships the nvidia-* python packages; loading the system
toolkit's copy first makes TE and torch run two different cuDNN versions in one process (cuDNN SDPA then
fails with CUDNN_STATUS_BAD_PARAM after 'import transformer_engine'), and the fix was to export CUDNN_HOME
and NVRTC_HOME by hand. Prefer the python packages when they are installed; the system search is unchanged
as the fallback.
…al block

reduce_dbias_kernel ran as a single 256-thread block (one thread per 4 columns, cols=1024 -> one block)
looping serially over the [rows/128 x cols] fp32 partials: 0.116 ms per call at 900 x 1024, ~30 GB/s, on
one SM, 200+ calls per training step of a ViT-L with biased linears. Add an in-place first stage that
spreads the rows over up to 128 blocks per column slab (each block writes its slab sum into the first
row of its own range), then run the existing one-block pass over those sums with a row stride.
…s_dgelu_erf

gelu_erf and dgelu_erf are PyTorch's CUDA GeLU expressions and constants
(F.gelu(approximate="none") and its backward), exposed as nvte_gelu_erf,
nvte_dgelu_erf, nvte_geglu_erf, nvte_dgeglu_erf and
nvte_quantize_dbias_dgelu_erf, the matching tex bindings, LayerNormMLP
activation="gelu_erf"/"geglu_erf" and a te.ops GELUErf op (with the
dbias fusion).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
torch.compile(dynamic=True) hands FusedAttention symbolic scalars, and the
custom-op packer baked every scalar into the OpaqueValueBundle, which only
takes Python constants: dynamo died with
'AsPythonConstantNotImplementedError: SymNodeVariable() is not a constant'.

max_seqlen_q / max_seqlen_kv follow the input shape, so they get a SymInt
schema slot (SymIntLike) and the fake impl sees them symbolically. Every
other scalar is configuration: pack() specializes it through guard_scalar
under tracing, so the graph guards on the value instead of failing on it.
The softcap check is spelled as comparisons so a symbolic cap guards rather
than graph-breaks on math.isfinite.

Test: test_dpa_torch_compile_dynamic (flash / fused, dynamic=True).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Two places made dynamo guard a compiled DotProductAttention on state that
differs per caller, so callers with two input shapes recompiled the frame on
every switch and fell past the recompile limit into eager:

- DotProductAttention compared the call's AttentionParams against a
  process-global cache and dynamo guarded the graph on the cache, so callers
  with different batch sizes recompiled each other on every switch.
  _select_attention_backends selects directly under compile; the cache is an
  eager-only memo.
- Backend selection is a constant of the graph (assume_constant_result), which
  cannot take a symbolic argument. AttentionParams.__post_init__ now specializes
  every scalar field on construction under compile (batch size, sequence
  lengths, head count and size, ...), replacing the three ad-hoc guard_scalar
  calls in forward that covered only batch and sequence lengths; a
  fused-attention selection under dynamic shapes no longer graph-breaks.
  Outside a trace every field is concrete already and it returns at once.

Tests: test_dpa_torch_compile_dynamic runs with fullgraph=True for both
backends; test_dpa_compile_alternating_batch_sizes (2 graphs for 6 alternating
calls).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Only BasicLinear had a custom op, so a Linear -> SwiGLU -> Linear fuser could
never take the compiled path: it traced its eager implementation, graph-broke at
every extension call and, under current scaling, crashed dynamo on the quantizer
it called while traced. SwiGLU now has forward and backward custom ops in the
BasicLinear pattern (SwiGLUFwdArgs / SwiGLUBwdArgs, forward_compute and
backward_compute with their fake twins, pack_*_args, forward_setup_context); the
eager path runs the same forward_compute. The input quantized for backward
(cache_quantized_input) comes from a quantizer the op builds in its constructor,
so the op args carry a value-opaque constant instead of constructing one in a
traced region. SiTUGLU keeps its kernels through the same hooks, with its soft-cap
parameters in subclass arg dataclasses. Outputs are described with
fake_activation; the GLU interleaving helpers replace the three copies of the
reshape.

A te.ops Linear-SwiGLU-Linear stack compiles as one graph (fullgraph=True) under
bf16, current scaling and MXFP8, with the saved input kept or quantized:
test_te_ops_swiglu_compiles_fullgraph. The fusible-ops SwiGLU family tests cover
the eager path.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… stored ones

Under current scaling and MXFP8 a module class compiled per instance (as a
transformer compiles its blocks: one code object, N instances) recompiled once
per instance and fell past the recompile limit, or raised under fullgraph. Two
guards on the stored quantizers caused it, both because a forward mutated them:

- te.Linear / LayerNormLinear / LayerNormMLP configured usage, internal and
  optimize_for_gemm on self.quantizers[...] inside forward, and the backward
  flipped the usage again. Dynamo guards an opaque constant by equality, and the
  guard was recorded against an object whose value kept changing, so the next
  instance (in its post-backward state) never matched.
- fake_norm passed that same stored quantizer through assume_constant_result,
  which guards an object argument read from a module attribute by identity.

The stored quantizers now hold recipe state only (scale and amax buffers,
reduction group, recipe parameters) and are configured at recipe reset. Every
call configures its own copies: the modules take them from
_call_quantizer(meta_key, index), so the in-forward usage, layout and
amax-reduction-group changes (set_quantizer_amax_reduction_group no longer
clobbers the stored one under compile) all land on per-call objects; in te.ops
the call's quantizers live on its OperationContext, where the backward already
kept them: pre_fuser_forward(ctx) publishes them (BasicLinear and GroupedLinear
as configured copies, other ops as their recipe quantizers), the fuser reads the
neighbours' from the contexts and the fused linear ops and the grouped MLP read
their basic ops' from the same place. SwiGLU builds its saved-input quantizer in
the constructor instead of on the first forward. The equality guard on a stored
quantizer then holds across instances and across the forward/backward usage
flips, and the fake_norm argument is a trace-local copy with no source.

Tests: test_te_modules_compile_once_across_instances and
test_te_ops_compile_once_across_instances (three instances, two training steps
each, one graph) under current scaling and MXFP8.

The base Quantizer gets a default shallow copy(): every attribute is shared, so
recipe state stays common to the copies and only the usage and layout flags a
call sets diverge, which is what the built-in copy() methods do by hand. A
custom recipe's quantizers (and the reference quantizers shipped with TE) have
no copy() of their own and failed at the first forward otherwise
(test_custom_quantizer_default_copy). Not copy.copy, as a quantizer's
__getstate__ may drop its reduction group for pickling.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…r's arguments

Every TE op under torch.compile went through two torch.library dispatches per
call: the wrapper op (the one a quantized-tensor subclass can enter) flattened
each subclass argument into the base op's slots and forwarded, and the base op
rebuilt the dataclass from a {slot: value} dict, rebuilt the subclass from its
slots, ran the kernel and packed the result under a torch.device context after
a pytree walk over every argument to find that device. The flattening built an
OpaqueValueBundle per subclass argument on each call, validating, freezing and
hashing its contents for Dynamo guards that only ever see trace-time bundles.
On a ViT-sized te.Linear that framing costs more host time than the op's
kernels take on the GPU (1.6 ms against 0.65 ms of GPU work per fwd+bwd), so a
compiled current-scaling model runs host-bound at 60 % GPU busy while the
delayed-scaling recipe, which cannot compile and runs its modules eagerly,
pays none of it.

The two tiers are a tracing need: a torch.library op cannot take a tensor
subclass, so while a FakeTensorMode is active the wrapper still flattens and
forwards to the base op, whose fake kernel sees the flat schema. A real call
now runs the real kernel directly on the wrapper's own arguments
(_make_slot_forwarder): the kernel takes the subclass whole, so nothing is
flattened, rebuilt, bundled or dispatched a second time. _ArgPlan.unpack reads
the flat arguments by slot position (slot_index, built once at registration);
result packing takes the op device from the first non-CPU tensor argument and
hands it to the None sentinels instead of entering a device context. The real
and fake kernels are built once by _make_kernels; _register_base_op only
registers them.

Verification (fork tests on H100, overlay of this tree): test_torch_compile
553 passed, test_fusible_ops bias/activation/linear 830 passed,
test_float8_current_scaling_exact 5 passed, test_numerics accuracy unchanged
(the 72 fp32 "small" failures are identical on the stock wheel). 12-layer
TEViTLayer stack, bs16, compiled: current scaling 60.3 -> 49.2 ms/step wall,
level with delayed scaling (48.8; GPU 10.1 vs 9.8 ms).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…tion

Inductor's generated wrapper spells every opaque constant out as its
__fx_repr__ constructor expression inline in the op call (val_to_arg_str ->
get_opaque_obj_repr, no hoisting), so each call of a compiled TE op rebuilds
its 3-5 bundles and 3-5 quantizers: 432 bundles and 384 quantizers per step
on a 12-layer stack. A bundle validated, ordered and hashed its contents in
__init__ for the guards Dynamo evaluates on the trace-time instance; on the
per-call rebuilds, which are only read, that was the largest Python cost of
the op path (17.8 of 102 ms of profiled Python time per step).

The bundle now keys itself lazily, through two functools.cached_property
members: _key (the validated, ordered, hashable contents) and _hash, computed
on the first __hash__ or __eq__ (what the guard does with the trace-time
bundle); __fx_repr__ validates on its own, since it formats the raw values and
only runs at codegen. A rebuilt copy pays a dict literal and an object.

Verification (fork tests on H100, overlay of this tree): test_torch_compile
553 passed, test_fusible_ops bias/activation/linear 830 passed,
test_float8_current_scaling_exact 5 passed, test_numerics accuracy unchanged.
12-layer TEViTLayer stack, bs16, compiled under current scaling: cProfile loses
OpaqueValueBundle.__init__ and its hashing chain (profiled Python time
102 -> 87 ms/step); wall 49.2 -> 40.0 ms/step. te.Linear fwd+bwd at 256
tokens: compiled 1,581 -> 1,323 us against 739 eager.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…dent bidimensional cast

The bidimensional cast kernel keeps the whole 32-row tile in registers, so the activation runs on
the registers it already holds: FP32 math (two lanes per f32x2 instruction, util/activation_x2.cuh),
one BF16 rounding, then the existing quantization. The bias gradient is reduced per CTA from the
same registers. Scales are written in the GEMM-swizzled layout when asked.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…l cast

The gated cast keeps the 32-row tile of each half in registers and runs the activation, gate and
gradient math on them, then both quantizations. One CTA covers 32 rows by 256 columns; the backward
pass quantizes the activation and gate gradients as two planes of the same tile. Used for BF16 to
E4M3 SiLU gating with rows % 32 == 0, cols % 256 == 0 and 32-byte aligned operands, otherwise the
TMA kernel runs as before.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The backward reads ln_out and act_out only column-wise (weight-gradient GEMMs),
but the forward saved both usages, keeping a dead row-wise fp8 copy of each per
layer (Linear already trims it). At 86016x1024/4096 that is ~440 MB per layer.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…select, fma.sat in dgelu, swizzle index once per tile, packed dbias sums)

Per 86016x4096 call on B200: gelu_erf fwd 0.451 -> 0.342 ms, dgelu+dbias 0.651 -> 0.554 ms.
Outputs hash-identical to the previous kernel in all 29 harness configurations.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…per plane/tile

17600x5120 on B200: fwd 0.122 -> 0.115 ms, bwd 0.172 -> 0.156 ms; byte differences vs the generic kernel unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…s the narrow tile

The fused activations are issue-bound (ncu at boost clocks, 86016x4096: 68% issue slots busy,
4.2 TB/s), so the 4-wide cluster that helps the cast-only kernel costs them 5-8%; the
dbias+dgelu variant spilled 170 B/thread in the wide tile (two inputs plus 16 column sums in
64 registers) and runs spill-free in the narrow one.

Per call at 86016x4096 on B200 (harness, cluster/tile now chosen by the dispatch):
gelu_erf fwd 0.342 -> 0.327 ms, dbias+dgelu 0.554 -> 0.509 ms, dgelu 0.515 -> 0.488 ms.
Outputs and scales hash-identical to the generic kernel on randn; the all-bf16 input differs
only in the pre-existing NaN/Inf rowwise-scale bytes.

Two persistent-CTA variants that prefetch the next tile (cp.async into shared memory;
a second register buffer at 2 CTAs/SM) hid the load latency (long_scoreboard 2.0 -> 0.4
cycles per issue) but lost more elsewhere: 0.369 ms either way, and the plain cast fell
from 0.210 to 0.266-0.282 ms, so the launch-per-tile structure stays.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… cast

LayerNorm with an MXFP8 quantizer ran as two kernels: ln_fwd_tuned_kernel into a
BF16 tensor, then quantize_mxfp8_kernel_cast_only (row-wise + column-wise, GEMM-swizzled
scales). ln_fwd_mxfp8.cu does both in one persistent kernel: tiles of 32 rows arrive
in shared memory by bulk copy while the previous tile is processed; each warp replays the
tuned kernel's one-warp-per-row arithmetic (same reductions, FP32 ops and BF16 rounding),
quantizes its rows from registers with the cast's instructions, parks the BF16 values
in place, and the CTA quantizes the tile column-wise from shared memory. Output bytes,
scales, mu and rsigma are identical to the two-kernel path for BF16 input and weights,
rows a multiple of 128 and 768 or 1024 columns; everything else keeps the previous path.

nvte_layernorm_fwd takes the fused kernel for a GEMM-swizzled MXFP8 output (previously
rejected) or any supported MXFP8 output unless cuDNN is requested; the new
nvte_layernorm_fwd_mxfp8_fused tells the PyTorch extension when to hand it the quantizer's
tensor instead of normalizing to BF16 and quantizing. is_aligned_to moves to
normalization/common.h, shared with the RMSNorm kernel.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…per lane)

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…cp.approx)

Forward is bit-identical to the scalar kernel (0 / 49.9M elements). The backward is not: the scalar kernel
used __frcp_rn, and with rcp.approx 0.012% of dact outputs differ by one rounding step (max 2.9e-11 on a
1.1e-8 gradient). Isolated kernel times at 60x448 rows x 4096: fwd 168 -> 117 us, bwd 216 -> 198 us.
With __frcp_rn per lane it is bit-exact in both directions at 144 / 204 us.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…SE_FAST_MATH=1

Default (__frcp_rn) is bit-identical to the scalar kernel in forward, input gradient and weight gradients
(0 / 49.9M elements; fwd 168 -> 142 us, bwd 218 -> 200 us at 60x448 rows x 4096). NVTE_USE_FAST_MATH=1
selects rcp.approx: fwd 116 us, bwd 199 us, 0.012% of backward elements differ by one rounding step.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The partial-sum reduction ran as two kernels (row blocks in place, then one
block summing the slab sums serially): ~20 us per call at our shapes. One
kernel now computes each group sum with the same interleave (8 accumulators,
then in order) and the final in-order pass from shared memory. Output is
bit-identical; 1024 cols 19.2 -> 5.2 us, 3072 cols 23.5 -> 6.4 us,
dgelu 4096 cols x 3840 partial rows 36.0 -> 17.1 us (B200, idle GPU).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A swizzled quantize with rows or cols not a multiple of 128 launched one
zero_scales_kernel per scale buffer before the cast (2-13 us each, ~400 per
step on the 1856-wide backbone). The generic and TMA cast-only kernels now
zero only the padding slots themselves; the register-resident paths keep the
separate launch. Scale buffers (padding included) are bit-identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The two rows a warp owns are reduced as one f32x2 chain (same per-row order),
gamma/beta are staged once as fp32 in shared memory, and the column-wise amax
runs on bf16x2. Bit-identical outputs, mu and rsigma; 122880x1024 130.5 ->
125.1 us (B200, idle GPU). The kernel stays latency-bound at one CTA per SM.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…radient

LayerNormLinear takes qkv_rope=(cos, sin, head_dim) when the caller applies RoPE
to the q and k parts of its packed output. Its backward then rotates the q/k
output gradient back inside the MXFP8 dbias quantize (nvte_quantize_dbias_rope,
tex.bgrad_quantize(rope_cos=, rope_sin=, head_dim=, rope_cols=)): the kernel
rotates each staged input tile in shared memory in FP32, rounds to BF16 and
quantizes it, so the MXFP8 data, scales and dbias equal a separate rotation
kernel followed by the plain dbias quantize. Other backward paths rotate with
torch ops before quantizing.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… quantize rejects

The fused dbias quantize asserts 'Unsupported RoPE layout' unless head_dim % 8 == 0,
64 % head_dim == 0, rot_dim % 4 == 0 and the q/k width is a multiple of 256.
LayerNormLinear(qkv_rope=...) now checks the layout before choosing the fused path and
otherwise applies qkv_rope_backward, as it already does outside MXFP8.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
mxfp8: inverse RoPE fused into the dbias quantize of a packed q/k/v gradient
mxfp8: one-launch dbias reduce, in-kernel scale padding, faster LN+MXFP8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant