Skip to content

lemi fixes on top of #3534 (torch.compile for LayerNormLinear/LayerNormMLP) - #1

Draft
yravan wants to merge 33 commits into
pr3534-basefrom
lemi/te-fixes
Draft

yravan wants to merge 33 commits into
pr3534-basefrom
lemi/te-fixes

Conversation

@yravan

@yravan yravan commented Nov 2, 2026 •

Copy link
Copy Markdown
Owner

Nine commits on a2f424e, the merge-base of NVIDIA#3534, as shipped in the 2.22.0.dev0+dd19e95d wheel pinned by sunday-robotics/learning#1310. Base branch pr3534-base is a2f424e itself, so the diff is exactly these commits.

  • LayerNormMLP recompute: restore the delayed-scaling fp8 meta after recompute (+ test): the recompute path lost the delayed-scaling state swap, so the fc1/fc2 scales stopped updating under activation checkpointing.

  • ops.Linear: do not inject a None bias key when the op has no bias (+ strict round-trip test).

  • Load cuDNN/NVRTC/cuRAND from the python packages before the system toolkit: no CUDNN_HOME / NVRTC_HOME needed.

  • cast: reduce the dbias partial sums across blocks instead of one serial block: the stock reduction is one 256-thread block looping over all row-chunks (23.5 ms/step at our shapes); two-stage parallel reduction, bit-identical, 4.6 ms.

  • ops/fuser: run outside dynamo under delayed scaling (+ test): the fuser's delayed-scaling check inspects fp8 metas that do not exist before the first forward, so a module whose first call is compiled traced its eager path into a quantizer call on a fake tensor; a graph break is not enough (dynamo keeps tracing the resumed frame), so the call runs through torch._dynamo.disable when the active recipe is delayed scaling.

  • cast: pair the gelu/dgelu math on the f32x2 path in the fp8 1D/2D cast kernels: the gelu / dgelu+dbias fp8 casts are issue-bound on the tanh math (2 SFU ops per element); computing two elements per thread on sm100 f32x2 instructions takes them from 3.0 / 3.1 to 3.9 / 4.0 TB/s at 114688x4096, fp8 bytes / amax / dbias bit-identical (all 65536 bf16 inputs checked against the scalar path). tanhf_x2 repeats libdevice tanhf (CUDA 13.0.88) op for op.

  • cast: pair the silu/dsilu math on the f32x2 path in the fp8 gated cast kernel: swiglu fwd 4.67 -> 5.76 TB/s, bwd 5.83 -> 6.61, bit-identical over all 2^32 fp32 inputs.

  • activation: exact (erf) GELU, gelu_erf / dgelu_erf / geglu_erf / dbias_dgelu_erf: nvte_*_erf C API, cast-transpose table entries, tex.*_erf bindings, LayerNormMLP(activation="gelu_erf" / "geglu_erf"), te.ops.GELUErf; forward/backward bit-equal to torch's exact GELU (F.gelu, gelu_backward(approximate="none")) over all 65536 bf16 inputs, f32x2-paired; ~75% of the tanh casts' throughput (erff cost).

  • dynamo: fused attention under dynamic shapes (+ test): torch.compile(dynamic=True) hands FusedAttention symbolic scalars and the custom-op packer baked every scalar into the OpaqueValueBundle, which only takes constants (SymNodeVariable() is not a constant). max_seqlen_q/kv follow the input shape, so they get a SymInt schema slot (SymIntLike); every other scalar is configuration and is specialized through guard_scalar under tracing. The softcap check is spelled as comparisons so a symbolic cap guards instead of graph-breaking on math.isfinite.

Validated on B200 with TE's own suites (test_numerics dbias / recompute, test_fusible_ops, test_torch_compile: 420 + 224 passed; test_te_ops_bias_nvfp4_layout[rht,2d] fail on stock too).

🤖 Generated with Claude Code

Yajvan Ravan and others added 24 commits November 2, 2026 19:25
…compute

_layernorm_mlp_recompute decided whether to restore the quantizers only after putting the
backward's autocast state back, so it saw the default recipe and skipped the restore: a
LayerNormMLP that recomputes internally kept its init fp8 scale forever. Evaluate the check
once, before the state swap. Test: a checkpointed LayerNormMLP ends with the same scale as an
uncheckpointed one (fails on main: scale stuck at 1.0).
_load_from_state_dict added '{prefix}bias = None' for a bias-free Linear op; load_state_dict(strict=True)
then reports it as an unexpected key (None-valued registered params are excluded from the module's own
local-state check). Only alias the bias when the op has one. Test: strict round trip with bias on and off.
…olkit

The framework wheel (torch) is built against and ships the nvidia-* python packages; loading the system
toolkit's copy first makes TE and torch run two different cuDNN versions in one process (cuDNN SDPA then
fails with CUDNN_STATUS_BAD_PARAM after 'import transformer_engine'), and the fix was to export CUDNN_HOME
and NVRTC_HOME by hand. Prefer the python packages when they are installed; the system search is unchanged
as the fallback.
…al block

reduce_dbias_kernel ran as a single 256-thread block (one thread per 4 columns, cols=1024 -> one block)
looping serially over the [rows/128 x cols] fp32 partials: 0.116 ms per call at 900 x 1024, ~30 GB/s, on
one SM, 200+ calls per training step of a ViT-L with biased linears. Add an in-place first stage that
spreads the rows over up to 128 blocks per column slab (each block writes its slab sum into the first
row of its own range), then run the existing one-block pass over those sums with a row stride.
The fuser raised on delayed scaling under torch.compile by inspecting the ops'
fp8 metas, which do not exist before the first forward: a module whose first
call is compiled (te.autocast outside the compiled region) passed the check
vacuously and dynamo traced the eager path into a quantizer call on a fake
tensor. A graph break is not enough either, dynamo keeps tracing the resumed
frame and the fuser's autograd function. Decide from the active recipe and run
the whole call through torch._dynamo.disable when it is delayed scaling; other
recipes keep the custom-op path and the eager fallback.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…t kernels

The gelu+cast and dgelu(+dbias)+cast kernels were issue-bound on the
tanhf math, not on bandwidth. Evaluate two elements per thread with
fma/mul/add.f32x2, mirroring libdevice's tanhf op for op, so the fp8
outputs, amax and dbias stay bit-identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t kernel

The SwiGLU forward and dSwiGLU backward casts were issue-bound on the
sigmoid math. Evaluate two rows per thread with fma/mul/add.f32x2,
repeating libdevice's expf and rcp.rn (one Newton step from rcp.approx,
scalar __frcp_rn past 2^126) op for op; outputs and amax stay
bit-identical for every fp32 input.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s_dgelu_erf

gelu_erf and dgelu_erf are PyTorch's CUDA GeLU expressions and constants
(F.gelu(approximate="none") and its backward), exposed as nvte_gelu_erf,
nvte_dgelu_erf, nvte_geglu_erf, nvte_dgeglu_erf and
nvte_quantize_dbias_dgelu_erf, the matching tex bindings, LayerNormMLP
activation="gelu_erf"/"geglu_erf" and a te.ops GELUErf op (with the
dbias fusion). The fp8 1D/2D cast kernels evaluate them on the paired
f32x2 path, repeating libdevice's erff and expf op for op.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
torch.compile(dynamic=True) hands FusedAttention symbolic scalars, and the
custom-op packer baked every scalar into the OpaqueValueBundle, which only
takes Python constants: dynamo died with
'AsPythonConstantNotImplementedError: SymNodeVariable() is not a constant'.

max_seqlen_q / max_seqlen_kv follow the input shape, so they get a SymInt
schema slot (SymIntLike) and the fake impl sees them symbolically. Every
other scalar is configuration: pack() specializes it through guard_scalar
under tracing, so the graph guards on the value instead of failing on it.
The softcap check is spelled as comparisons so a symbolic cap guards rather
than graph-breaks on math.isfinite.

Test: test_dpa_torch_compile_dynamic (flash / fused, dynamic=True).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Three places made dynamo guard a shared frame on state that differs per module
instance or per caller, so a model with N layers or two input shapes recompiled
the frame N times and fell past the recompile limit into eager:

- A recipe with no compile path (delayed scaling: its quantizers update
  per-module scale and amax tensors in place and are not value-opaque) used to
  graph-break inside te.Linear / LayerNormLinear / LayerNormMLP after
  prepare_forward had been traced, guarding the traced prefix on each instance.
  recipe_compile_fallback_reason() decides this from the recipe alone and each
  forward returns forward_eagerly(self, ...) (torch.compiler.disable) as its first
  statement, so nothing inside the module is traced; the te.ops fuser reuses the
  helper for its existing outside-dynamo path.
- DotProductAttention compared the call's AttentionParams against a
  process-global cache and dynamo guarded the graph on the cache, so callers
  with different batch sizes recompiled each other on every switch.
  _select_attention_backends selects directly under compile; the cache is an
  eager-only memo.
- Backend selection is a constant of the graph (assume_constant_result), which
  cannot take a symbolic argument. AttentionParams.__post_init__ now specializes
  every scalar field on construction (batch size, sequence lengths, head count
  and size, ...), replacing the three ad-hoc guard_scalar calls in forward that
  covered only batch and sequence lengths; a fused-attention selection under
  dynamic shapes no longer graph-breaks.

Tests: test_dpa_torch_compile_dynamic runs with fullgraph=True for both
backends; test_dpa_compile_alternating_batch_sizes (2 graphs for 6 alternating
calls); test_te_modules_delayed_scaling_run_eagerly (3 instances share one
compiled prefix, prepare_forward is never traced, numerics match eager).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Only BasicLinear had a custom op, so a Linear -> SwiGLU -> Linear fuser could
never take the compiled path: it traced its eager implementation, graph-broke at
every extension call and, under current scaling, crashed dynamo on the quantizer
it called while traced. SwiGLU now has forward and backward custom ops in the
BasicLinear pattern (SwiGLUFwdArgs / SwiGLUBwdArgs, forward_compute and
backward_compute with their fake twins, pack_*_args, forward_setup_context); the
eager path runs the same forward_compute. The input quantized for backward
(cache_quantized_input) comes from a quantizer the op builds in its constructor,
so the op args carry a value-opaque constant instead of constructing one in a
traced region. SiTUGLU keeps its kernels through the same hooks, with its soft-cap
parameters in subclass arg dataclasses. Outputs are described with
fake_activation; the GLU interleaving helpers replace the three copies of the
reshape.

A te.ops Linear-SwiGLU-Linear stack compiles as one graph (fullgraph=True) under
bf16, current scaling and MXFP8, with the saved input kept or quantized:
test_te_ops_swiglu_compiles_fullgraph. The fusible-ops SwiGLU family tests cover
the eager path.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… stored ones

Under current scaling and MXFP8 a module class compiled per instance (as a
transformer compiles its blocks: one code object, N instances) recompiled once
per instance and fell past the recompile limit, or raised under fullgraph. Two
guards on the stored quantizers caused it, both because a forward mutated them:

- te.Linear / LayerNormLinear / LayerNormMLP configured usage, internal and
  optimize_for_gemm on self.quantizers[...] inside forward, and the backward
  flipped the usage again. Dynamo guards an opaque constant by equality, and the
  guard was recorded against an object whose value kept changing, so the next
  instance (in its post-backward state) never matched.
- fake_norm passed that same stored quantizer through assume_constant_result,
  which guards an object argument read from a module attribute by identity.

The stored quantizers now hold recipe state only (scale and amax buffers,
reduction group, recipe parameters) and are configured at recipe reset. Every
call configures its own copies: the modules take them from
_call_quantizer(meta_key, index), so the in-forward usage, layout and
amax-reduction-group changes (set_quantizer_amax_reduction_group no longer
clobbers the stored one under compile) all land on per-call objects; in te.ops
the call's quantizers live on its OperationContext, where the backward already
kept them: pre_fuser_forward(ctx) publishes them (BasicLinear and GroupedLinear
as configured copies, other ops as their recipe quantizers), the fuser reads the
neighbours' from the contexts and the fused linear ops and the grouped MLP read
their basic ops' from the same place. SwiGLU builds its saved-input quantizer in
the constructor instead of on the first forward. The equality guard on a stored
quantizer then holds across instances and across the forward/backward usage
flips, and the fake_norm argument is a trace-local copy with no source.

Tests: test_te_modules_compile_once_across_instances and
test_te_ops_compile_once_across_instances (three instances, two training steps
each, one graph) under current scaling and MXFP8.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
bgrad_quantize took the fused dbias + cast kernel only for delayed scaling and
MXFP8; current scaling summed the bias gradient with ATen, then ran its amax
pass and its cast: three reads of every grad-output with a bias, 48 per step on
a 12-layer ViT, and the single largest kernel of the step. The fused kernel
only needs the output's scale, which current scaling knows once its amax pass
has run, so Float8CurrentScalingQuantizer::quantize_with_dbias runs the amax
(or takes a given one), the reduction and the scale as quantize() does and then
nvte_quantize_dbias in place of nvte_quantize_v2: two reads instead of three.
The same holds for LayerNormMLP's fc1 grad: the dact kernel already produced the
amax for current scaling (Impl::FUSED_DACT_AMAX_FP8) but then summed the bias
and cast separately; it now casts with the fused dbias, and the Python side
lists the dbias + dact kernels for current scaling as it does for delayed
scaling and MXFP8, so fc1 goes from four passes to two. The SM 9.0 condition of
the fused kernel (rowwise and columnwise usage) is shared with delayed scaling.

Verification: both sources syntax-check against torch 2.14.1 / CUDA 13.0
headers; the runtime comparison against the unfused path needs the rebuilt
wheel (test_fusible_ops bias and activation cases, bench_recipes current).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…r's arguments

Every TE op under torch.compile went through two torch.library dispatches per
call: the wrapper op (the one a quantized-tensor subclass can enter) flattened
each subclass argument into the base op's slots and forwarded, and the base op
rebuilt the dataclass from a {slot: value} dict, rebuilt the subclass from its
slots, ran the kernel and packed the result under a torch.device context after
a pytree walk over every argument to find that device. The flattening built an
OpaqueValueBundle per subclass argument on each call, validating, freezing and
hashing its contents for Dynamo guards that only ever see trace-time bundles.
On a ViT-sized te.Linear that framing costs more host time than the op's
kernels take on the GPU (1.6 ms against 0.65 ms of GPU work per fwd+bwd), so a
compiled current-scaling model runs host-bound at 60 % GPU busy while the
delayed-scaling recipe, which cannot compile and runs its modules eagerly,
pays none of it.

The two tiers are a tracing need: a torch.library op cannot take a tensor
subclass, so while a FakeTensorMode is active the wrapper still flattens and
forwards to the base op, whose fake kernel sees the flat schema. A real call
now runs the real kernel directly on the wrapper's own arguments
(_make_slot_forwarder): the kernel takes the subclass whole, so nothing is
flattened, rebuilt, bundled or dispatched a second time. _ArgPlan.unpack reads
the flat arguments by slot position (slot_index, built once at registration);
result packing takes the op device from the first non-CPU tensor argument and
hands it to the None sentinels instead of entering a device context. The real
and fake kernels are built once by _make_kernels; _register_base_op only
registers them.

Verification (fork tests on H100, overlay of this tree): test_torch_compile
553 passed, test_fusible_ops bias/activation/linear 830 passed,
test_float8_current_scaling_exact 5 passed, test_numerics accuracy unchanged
(the 72 fp32 "small" failures are identical on the stock wheel). 12-layer
TEViTLayer stack, bs16, compiled: current scaling 60.3 -> 49.2 ms/step wall,
level with delayed scaling (48.8; GPU 10.1 vs 9.8 ms).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Inductor's generated wrapper spells every opaque constant out as its
__fx_repr__ constructor expression inline in the op call (val_to_arg_str ->
get_opaque_obj_repr, no hoisting), so each call of a compiled TE op rebuilds
its 3-5 bundles and 3-5 quantizers: 432 bundles and 384 quantizers per step
on a 12-layer stack. A bundle validated, ordered and hashed its contents in
__init__ for the guards Dynamo evaluates on the trace-time instance; on the
per-call rebuilds, which are only read, that was the largest Python cost of
the op path (17.8 of 102 ms of profiled Python time per step).

The bundle now freezes lazily: validation, ordering and the hash happen on the
first __hash__, __eq__ or __fx_repr__ (what the guard and the codegen do with
the trace-time bundle), so a rebuilt copy pays a dict literal and an object.

Verification (fork tests on H100, overlay of this tree): test_torch_compile
553 passed, test_fusible_ops bias/activation/linear 830 passed,
test_float8_current_scaling_exact 5 passed, test_numerics accuracy unchanged.
12-layer TEViTLayer stack, bs16, compiled under current scaling: cProfile loses
OpaqueValueBundle.__init__ and its hashing chain (profiled Python time
102 -> 87 ms/step); wall 49.2 -> 40.0 ms/step, now below the graph-broken
delayed-scaling stack (45.2). te.Linear fwd+bwd at 256 tokens: compiled
1,581 -> 1,323 us against 739 eager.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
It read the active recipe only, so a caller deciding how to compile a module
before entering its autocast (fullgraph or not) had to re-derive TE's verdict
from the recipe class itself. The recipe is now a parameter and the active one
is the default, so the knowledge of which recipes have a compile path stays in
one place.

Verification: tests/pytorch/test_torch_compile.py 553 passed, 341 skipped on
H100 (overlay of this tree); the no-argument callers are unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A forward configures a copy of each stored quantizer (f7bef038), which assumed
every quantizer implements copy(). The built-in ones do; a custom recipe's need
not, and the reference quantizers shipped with TE do not, so every custom-recipe
forward failed at the first call:

  base.py:1417: AttributeError: 'CurrentScalingQuantizerRef' object has no attribute 'copy'
  basic_linear.py:447: AttributeError: 'NVFP4QuantizerRef' object has no attribute 'copy'

The base class now copies shallowly: every attribute is shared, so recipe state
stays common to the copies and only the usage and layout flags a call sets
diverge, which is what the built-in copy() methods do by hand. Not copy.copy, as
a quantizer's __getstate__ may drop its reduction group for pickling.

Verification: tests/pytorch/test_custom_recipe.py went from 4 failed / 33 passed
(the four forwards above) to all passing on H100 with the overlay build; the new
test_custom_quantizer_default_copy pins the contract.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Linear, LayerNormLinear, LayerNormMLP and OperationFuser each carried the same
five lines at the top of their forward: under tracing, ask the recipe for its
fallback reason, warn, run eagerly. compiled_call_runs_eagerly() in
quantization.py is that check, next to the recipe_compile_fallback_reason it is
built on, with the reasoning (a recipe with no compile path has no traceable
eager path either, so a graph break would not do) in one docstring instead of a
comment block per module. It reads the active recipe itself rather than taking
one: the sites run before prepare_forward, so the active recipe is the only one
they could pass, and reading it in one place keeps the compiled prefix identical
across modules (test_te_modules_delayed_scaling_run_eagerly).

The fuser's per-op delayed-scaling-state check, which only a CustomRecipe with
a DelayedScalingRequest reaches now, loses the enumerate and the "initialization
before forward" comment left behind when f7bef038 moved pre_fuser_forward out
of that loop.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
19a30496 deferred the bundle's validation and hash to the first use, with a
None placeholder for the key and 0 for the hash that _freeze() filled in. The
same deferral as two functools.cached_property members, _key (the validated,
ordered, hashable contents) and _hash: no placeholders, and no method every
accessor has to remember to call first. __fx_repr__ validates on its own, since
it formats the raw values and only runs at codegen.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
__post_init__ walked every field through _concrete_scalars on each eager
construction too, where every field is concrete already (a SymInt only exists
inside a trace) and the walk is a no-op. It now returns at once outside a
trace, as it did for ONNX export, which also says what the specialization is
for.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
e7eb8213 copied bgrad_quantize's workspace sequence (launch once to learn the
workspace shape, allocate it, launch again) into the cast lambda of
Float8CurrentScalingQuantizer::quantize_with_dbias; dact_dbias carried a third
copy. launch_with_workspace in common.h is that sequence once, for any kernel
that reports its workspace this way, and the three sites hand it their launch.

TestFP8CurrentScalingFusedDbias compares the fused paths against the unfused
reduction and cast: bgrad_quantize and dbias_dgelu, rowwise-only and
rowwise + columnwise, which on SM 9.0 are the unfused fallback and the fused
cast + transpose kernel. Bit for bit on the quantized data and scale, up to
reduction order on the bias gradient.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
backend="eager" runs the fx graph dynamo traced as it is, with no AOT autograd
or inductor in between, so each custom op's real kernel is called through
torch.ops with whatever its neighbour returned: a quantized tensor when the
SwiGLU caches its input quantized. That is the direct real-argument path
514330fe added, covered now with a subclass argument.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- layernorm_mlp: e7eb8213 added current scaling to the recipes with the dbias +
  dact fusion list, so the later branch that still named it with the unfused
  recipes was unreachable; the branch and its comment now name only the recipes
  it serves.
- dot_product_attention: the guard_scalar import 7f86e721 stopped using.
- BasicLinear.pre_fuser_forward: the super() call published the stored
  quantizers on the context only for the method to overwrite them with the
  call's copies (f7bef038); it sets every field itself.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…dent bidimensional cast

The bidimensional cast kernel keeps the whole 32-row tile in registers, so the activation runs on
the registers it already holds: FP32 math (two lanes per f32x2 instruction, shared with the FP8
cast in util/activation_x2.cuh), one BF16 rounding, then the existing quantization. The bias
gradient is reduced per CTA from the same registers. Scales are written in the GEMM-swizzled
layout when asked.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Yajvan Ravan and others added 5 commits November 13, 2026 17:33
…l cast

The gated cast keeps the 32-row tile of each half in registers and runs the activation, gate and
gradient math on them, then both quantizations. One CTA covers 32 rows by 256 columns; the backward
pass quantizes the activation and gate gradients as two planes of the same tile. Used for BF16 to
E4M3 SiLU gating with rows % 32 == 0, cols % 256 == 0 and 32-byte aligned operands, otherwise the
TMA kernel runs as before.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The backward reads ln_out and act_out only column-wise (weight-gradient GEMMs),
but the forward saved both usages, keeping a dead row-wise fp8 copy of each per
layer (Linear already trims it). At 86016x1024/4096 that is ~440 MB per layer.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…select, fma.sat in dgelu, swizzle index once per tile, packed dbias sums)

Per 86016x4096 call on B200: gelu_erf fwd 0.451 -> 0.342 ms, dgelu+dbias 0.651 -> 0.554 ms.
Outputs hash-identical to the previous kernel in all 29 harness configurations.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…per plane/tile

17600x5120 on B200: fwd 0.122 -> 0.115 ms, bwd 0.172 -> 0.156 ms; byte differences vs the generic kernel unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…s the narrow tile

The fused activations are issue-bound (ncu at boost clocks, 86016x4096: 68% issue slots busy,
4.2 TB/s), so the 4-wide cluster that helps the cast-only kernel costs them 5-8%; the
dbias+dgelu variant spilled 170 B/thread in the wide tile (two inputs plus 16 column sums in
64 registers) and runs spill-free in the narrow one.

Per call at 86016x4096 on B200 (harness, cluster/tile now chosen by the dispatch):
gelu_erf fwd 0.342 -> 0.327 ms, dbias+dgelu 0.554 -> 0.509 ms, dgelu 0.515 -> 0.488 ms.
Outputs and scales hash-identical to the generic kernel on randn; the all-bf16 input differs
only in the pre-existing NaN/Inf rowwise-scale bytes.

Two persistent-CTA variants that prefetch the next tile (cp.async into shared memory;
a second register buffer at 2 CTAs/SM) hid the load latency (long_scoreboard 2.0 -> 0.4
cycles per issue) but lost more elsewhere: 0.369 ms either way, and the plain cast fell
from 0.210 to 0.266-0.282 ms, so the launch-per-tile structure stays.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Yajvan Ravan and others added 4 commits November 13, 2026 17:33
… cast

LayerNorm with an MXFP8 quantizer ran as two kernels: ln_fwd_tuned_kernel into a
BF16 tensor, then quantize_mxfp8_kernel_cast_only (row-wise + column-wise, GEMM-swizzled
scales). ln_fwd_mxfp8.cu does both in one persistent kernel: tiles of 32 rows arrive
in shared memory by bulk copy while the previous tile is processed; each warp replays the
tuned kernel's one-warp-per-row arithmetic (same reductions, FP32 ops and BF16 rounding),
quantizes its rows from registers with the cast's instructions, parks the BF16 values
in place, and the CTA quantizes the tile column-wise from shared memory. Output bytes,
scales, mu and rsigma are identical to the two-kernel path for BF16 input and weights,
rows a multiple of 128 and 768 or 1024 columns; everything else keeps the previous path.

nvte_layernorm_fwd takes the fused kernel for a GEMM-swizzled MXFP8 output (previously
rejected) or any supported MXFP8 output unless cuDNN is requested; the new
nvte_layernorm_fwd_mxfp8_fused tells the PyTorch extension when to hand it the quantizer's
tensor instead of normalizing to BF16 and quantizing. is_aligned_to moves to
normalization/common.h, shared with the RMSNorm kernel.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…per lane)

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…cp.approx)

Forward is bit-identical to the scalar kernel (0 / 49.9M elements). The backward is not: the scalar kernel
used __frcp_rn, and with rcp.approx 0.012% of dact outputs differ by one rounding step (max 2.9e-11 on a
1.1e-8 gradient). Isolated kernel times at 60x448 rows x 4096: fwd 168 -> 117 us, bwd 216 -> 198 us.
With __frcp_rn per lane it is bit-exact in both directions at 144 / 204 us.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…SE_FAST_MATH=1

Default (__frcp_rn) is bit-identical to the scalar kernel in forward, input gradient and weight gradients
(0 / 49.9M elements; fwd 168 -> 142 us, bwd 218 -> 200 us at 60x448 rows x 4096). NVTE_USE_FAST_MATH=1
selects rcp.approx: fwd 116 us, bwd 199 us, 0.012% of backward elements differ by one rounding step.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant