Repository navigation
Conversation
…compute _layernorm_mlp_recompute decided whether to restore the quantizers only after putting the backward's autocast state back, so it saw the default recipe and skipped the restore: a LayerNormMLP that recomputes internally kept its init fp8 scale forever. Evaluate the check once, before the state swap. Test: a checkpointed LayerNormMLP ends with the same scale as an uncheckpointed one (fails on main: scale stuck at 1.0).
_load_from_state_dict added '{prefix}bias = None' for a bias-free Linear op; load_state_dict(strict=True)
then reports it as an unexpected key (None-valued registered params are excluded from the module's own
local-state check). Only alias the bias when the op has one. Test: strict round trip with bias on and off.
…olkit The framework wheel (torch) is built against and ships the nvidia-* python packages; loading the system toolkit's copy first makes TE and torch run two different cuDNN versions in one process (cuDNN SDPA then fails with CUDNN_STATUS_BAD_PARAM after 'import transformer_engine'), and the fix was to export CUDNN_HOME and NVRTC_HOME by hand. Prefer the python packages when they are installed; the system search is unchanged as the fallback.
…al block reduce_dbias_kernel ran as a single 256-thread block (one thread per 4 columns, cols=1024 -> one block) looping serially over the [rows/128 x cols] fp32 partials: 0.116 ms per call at 900 x 1024, ~30 GB/s, on one SM, 200+ calls per training step of a ViT-L with biased linears. Add an in-place first stage that spreads the rows over up to 128 blocks per column slab (each block writes its slab sum into the first row of its own range), then run the existing one-block pass over those sums with a row stride.
The fuser raised on delayed scaling under torch.compile by inspecting the ops' fp8 metas, which do not exist before the first forward: a module whose first call is compiled (te.autocast outside the compiled region) passed the check vacuously and dynamo traced the eager path into a quantizer call on a fake tensor. A graph break is not enough either, dynamo keeps tracing the resumed frame and the fuser's autograd function. Decide from the active recipe and run the whole call through torch._dynamo.disable when it is delayed scaling; other recipes keep the custom-op path and the eager fallback. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…t kernels The gelu+cast and dgelu(+dbias)+cast kernels were issue-bound on the tanhf math, not on bandwidth. Evaluate two elements per thread with fma/mul/add.f32x2, mirroring libdevice's tanhf op for op, so the fp8 outputs, amax and dbias stay bit-identical. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t kernel The SwiGLU forward and dSwiGLU backward casts were issue-bound on the sigmoid math. Evaluate two rows per thread with fma/mul/add.f32x2, repeating libdevice's expf and rcp.rn (one Newton step from rcp.approx, scalar __frcp_rn past 2^126) op for op; outputs and amax stay bit-identical for every fp32 input. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s_dgelu_erf gelu_erf and dgelu_erf are PyTorch's CUDA GeLU expressions and constants (F.gelu(approximate="none") and its backward), exposed as nvte_gelu_erf, nvte_dgelu_erf, nvte_geglu_erf, nvte_dgeglu_erf and nvte_quantize_dbias_dgelu_erf, the matching tex bindings, LayerNormMLP activation="gelu_erf"/"geglu_erf" and a te.ops GELUErf op (with the dbias fusion). The fp8 1D/2D cast kernels evaluate them on the paired f32x2 path, repeating libdevice's erff and expf op for op. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
torch.compile(dynamic=True) hands FusedAttention symbolic scalars, and the custom-op packer baked every scalar into the OpaqueValueBundle, which only takes Python constants: dynamo died with 'AsPythonConstantNotImplementedError: SymNodeVariable() is not a constant'. max_seqlen_q / max_seqlen_kv follow the input shape, so they get a SymInt schema slot (SymIntLike) and the fake impl sees them symbolically. Every other scalar is configuration: pack() specializes it through guard_scalar under tracing, so the graph guards on the value instead of failing on it. The softcap check is spelled as comparisons so a symbolic cap guards rather than graph-breaks on math.isfinite. Test: test_dpa_torch_compile_dynamic (flash / fused, dynamic=True). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Three places made dynamo guard a shared frame on state that differs per module instance or per caller, so a model with N layers or two input shapes recompiled the frame N times and fell past the recompile limit into eager: - A recipe with no compile path (delayed scaling: its quantizers update per-module scale and amax tensors in place and are not value-opaque) used to graph-break inside te.Linear / LayerNormLinear / LayerNormMLP after prepare_forward had been traced, guarding the traced prefix on each instance. recipe_compile_fallback_reason() decides this from the recipe alone and each forward returns forward_eagerly(self, ...) (torch.compiler.disable) as its first statement, so nothing inside the module is traced; the te.ops fuser reuses the helper for its existing outside-dynamo path. - DotProductAttention compared the call's AttentionParams against a process-global cache and dynamo guarded the graph on the cache, so callers with different batch sizes recompiled each other on every switch. _select_attention_backends selects directly under compile; the cache is an eager-only memo. - Backend selection is a constant of the graph (assume_constant_result), which cannot take a symbolic argument. AttentionParams.__post_init__ now specializes every scalar field on construction (batch size, sequence lengths, head count and size, ...), replacing the three ad-hoc guard_scalar calls in forward that covered only batch and sequence lengths; a fused-attention selection under dynamic shapes no longer graph-breaks. Tests: test_dpa_torch_compile_dynamic runs with fullgraph=True for both backends; test_dpa_compile_alternating_batch_sizes (2 graphs for 6 alternating calls); test_te_modules_delayed_scaling_run_eagerly (3 instances share one compiled prefix, prepare_forward is never traced, numerics match eager). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Only BasicLinear had a custom op, so a Linear -> SwiGLU -> Linear fuser could never take the compiled path: it traced its eager implementation, graph-broke at every extension call and, under current scaling, crashed dynamo on the quantizer it called while traced. SwiGLU now has forward and backward custom ops in the BasicLinear pattern (SwiGLUFwdArgs / SwiGLUBwdArgs, forward_compute and backward_compute with their fake twins, pack_*_args, forward_setup_context); the eager path runs the same forward_compute. The input quantized for backward (cache_quantized_input) comes from a quantizer the op builds in its constructor, so the op args carry a value-opaque constant instead of constructing one in a traced region. SiTUGLU keeps its kernels through the same hooks, with its soft-cap parameters in subclass arg dataclasses. Outputs are described with fake_activation; the GLU interleaving helpers replace the three copies of the reshape. A te.ops Linear-SwiGLU-Linear stack compiles as one graph (fullgraph=True) under bf16, current scaling and MXFP8, with the saved input kept or quantized: test_te_ops_swiglu_compiles_fullgraph. The fusible-ops SwiGLU family tests cover the eager path. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… stored ones Under current scaling and MXFP8 a module class compiled per instance (as a transformer compiles its blocks: one code object, N instances) recompiled once per instance and fell past the recompile limit, or raised under fullgraph. Two guards on the stored quantizers caused it, both because a forward mutated them: - te.Linear / LayerNormLinear / LayerNormMLP configured usage, internal and optimize_for_gemm on self.quantizers[...] inside forward, and the backward flipped the usage again. Dynamo guards an opaque constant by equality, and the guard was recorded against an object whose value kept changing, so the next instance (in its post-backward state) never matched. - fake_norm passed that same stored quantizer through assume_constant_result, which guards an object argument read from a module attribute by identity. The stored quantizers now hold recipe state only (scale and amax buffers, reduction group, recipe parameters) and are configured at recipe reset. Every call configures its own copies: the modules take them from _call_quantizer(meta_key, index), so the in-forward usage, layout and amax-reduction-group changes (set_quantizer_amax_reduction_group no longer clobbers the stored one under compile) all land on per-call objects; in te.ops the call's quantizers live on its OperationContext, where the backward already kept them: pre_fuser_forward(ctx) publishes them (BasicLinear and GroupedLinear as configured copies, other ops as their recipe quantizers), the fuser reads the neighbours' from the contexts and the fused linear ops and the grouped MLP read their basic ops' from the same place. SwiGLU builds its saved-input quantizer in the constructor instead of on the first forward. The equality guard on a stored quantizer then holds across instances and across the forward/backward usage flips, and the fake_norm argument is a trace-local copy with no source. Tests: test_te_modules_compile_once_across_instances and test_te_ops_compile_once_across_instances (three instances, two training steps each, one graph) under current scaling and MXFP8. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
bgrad_quantize took the fused dbias + cast kernel only for delayed scaling and MXFP8; current scaling summed the bias gradient with ATen, then ran its amax pass and its cast: three reads of every grad-output with a bias, 48 per step on a 12-layer ViT, and the single largest kernel of the step. The fused kernel only needs the output's scale, which current scaling knows once its amax pass has run, so Float8CurrentScalingQuantizer::quantize_with_dbias runs the amax (or takes a given one), the reduction and the scale as quantize() does and then nvte_quantize_dbias in place of nvte_quantize_v2: two reads instead of three. The same holds for LayerNormMLP's fc1 grad: the dact kernel already produced the amax for current scaling (Impl::FUSED_DACT_AMAX_FP8) but then summed the bias and cast separately; it now casts with the fused dbias, and the Python side lists the dbias + dact kernels for current scaling as it does for delayed scaling and MXFP8, so fc1 goes from four passes to two. The SM 9.0 condition of the fused kernel (rowwise and columnwise usage) is shared with delayed scaling. Verification: both sources syntax-check against torch 2.14.1 / CUDA 13.0 headers; the runtime comparison against the unfused path needs the rebuilt wheel (test_fusible_ops bias and activation cases, bench_recipes current). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…r's arguments
Every TE op under torch.compile went through two torch.library dispatches per
call: the wrapper op (the one a quantized-tensor subclass can enter) flattened
each subclass argument into the base op's slots and forwarded, and the base op
rebuilt the dataclass from a {slot: value} dict, rebuilt the subclass from its
slots, ran the kernel and packed the result under a torch.device context after
a pytree walk over every argument to find that device. The flattening built an
OpaqueValueBundle per subclass argument on each call, validating, freezing and
hashing its contents for Dynamo guards that only ever see trace-time bundles.
On a ViT-sized te.Linear that framing costs more host time than the op's
kernels take on the GPU (1.6 ms against 0.65 ms of GPU work per fwd+bwd), so a
compiled current-scaling model runs host-bound at 60 % GPU busy while the
delayed-scaling recipe, which cannot compile and runs its modules eagerly,
pays none of it.
The two tiers are a tracing need: a torch.library op cannot take a tensor
subclass, so while a FakeTensorMode is active the wrapper still flattens and
forwards to the base op, whose fake kernel sees the flat schema. A real call
now runs the real kernel directly on the wrapper's own arguments
(_make_slot_forwarder): the kernel takes the subclass whole, so nothing is
flattened, rebuilt, bundled or dispatched a second time. _ArgPlan.unpack reads
the flat arguments by slot position (slot_index, built once at registration);
result packing takes the op device from the first non-CPU tensor argument and
hands it to the None sentinels instead of entering a device context. The real
and fake kernels are built once by _make_kernels; _register_base_op only
registers them.
Verification (fork tests on H100, overlay of this tree): test_torch_compile
553 passed, test_fusible_ops bias/activation/linear 830 passed,
test_float8_current_scaling_exact 5 passed, test_numerics accuracy unchanged
(the 72 fp32 "small" failures are identical on the stock wheel). 12-layer
TEViTLayer stack, bs16, compiled: current scaling 60.3 -> 49.2 ms/step wall,
level with delayed scaling (48.8; GPU 10.1 vs 9.8 ms).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Inductor's generated wrapper spells every opaque constant out as its __fx_repr__ constructor expression inline in the op call (val_to_arg_str -> get_opaque_obj_repr, no hoisting), so each call of a compiled TE op rebuilds its 3-5 bundles and 3-5 quantizers: 432 bundles and 384 quantizers per step on a 12-layer stack. A bundle validated, ordered and hashed its contents in __init__ for the guards Dynamo evaluates on the trace-time instance; on the per-call rebuilds, which are only read, that was the largest Python cost of the op path (17.8 of 102 ms of profiled Python time per step). The bundle now freezes lazily: validation, ordering and the hash happen on the first __hash__, __eq__ or __fx_repr__ (what the guard and the codegen do with the trace-time bundle), so a rebuilt copy pays a dict literal and an object. Verification (fork tests on H100, overlay of this tree): test_torch_compile 553 passed, test_fusible_ops bias/activation/linear 830 passed, test_float8_current_scaling_exact 5 passed, test_numerics accuracy unchanged. 12-layer TEViTLayer stack, bs16, compiled under current scaling: cProfile loses OpaqueValueBundle.__init__ and its hashing chain (profiled Python time 102 -> 87 ms/step); wall 49.2 -> 40.0 ms/step, now below the graph-broken delayed-scaling stack (45.2). te.Linear fwd+bwd at 256 tokens: compiled 1,581 -> 1,323 us against 739 eager. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
It read the active recipe only, so a caller deciding how to compile a module before entering its autocast (fullgraph or not) had to re-derive TE's verdict from the recipe class itself. The recipe is now a parameter and the active one is the default, so the knowledge of which recipes have a compile path stays in one place. Verification: tests/pytorch/test_torch_compile.py 553 passed, 341 skipped on H100 (overlay of this tree); the no-argument callers are unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A forward configures a copy of each stored quantizer (f7bef038), which assumed every quantizer implements copy(). The built-in ones do; a custom recipe's need not, and the reference quantizers shipped with TE do not, so every custom-recipe forward failed at the first call: base.py:1417: AttributeError: 'CurrentScalingQuantizerRef' object has no attribute 'copy' basic_linear.py:447: AttributeError: 'NVFP4QuantizerRef' object has no attribute 'copy' The base class now copies shallowly: every attribute is shared, so recipe state stays common to the copies and only the usage and layout flags a call sets diverge, which is what the built-in copy() methods do by hand. Not copy.copy, as a quantizer's __getstate__ may drop its reduction group for pickling. Verification: tests/pytorch/test_custom_recipe.py went from 4 failed / 33 passed (the four forwards above) to all passing on H100 with the overlay build; the new test_custom_quantizer_default_copy pins the contract. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Linear, LayerNormLinear, LayerNormMLP and OperationFuser each carried the same five lines at the top of their forward: under tracing, ask the recipe for its fallback reason, warn, run eagerly. compiled_call_runs_eagerly() in quantization.py is that check, next to the recipe_compile_fallback_reason it is built on, with the reasoning (a recipe with no compile path has no traceable eager path either, so a graph break would not do) in one docstring instead of a comment block per module. It reads the active recipe itself rather than taking one: the sites run before prepare_forward, so the active recipe is the only one they could pass, and reading it in one place keeps the compiled prefix identical across modules (test_te_modules_delayed_scaling_run_eagerly). The fuser's per-op delayed-scaling-state check, which only a CustomRecipe with a DelayedScalingRequest reaches now, loses the enumerate and the "initialization before forward" comment left behind when f7bef038 moved pre_fuser_forward out of that loop. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
19a30496 deferred the bundle's validation and hash to the first use, with a None placeholder for the key and 0 for the hash that _freeze() filled in. The same deferral as two functools.cached_property members, _key (the validated, ordered, hashable contents) and _hash: no placeholders, and no method every accessor has to remember to call first. __fx_repr__ validates on its own, since it formats the raw values and only runs at codegen. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
__post_init__ walked every field through _concrete_scalars on each eager construction too, where every field is concrete already (a SymInt only exists inside a trace) and the walk is a no-op. It now returns at once outside a trace, as it did for ONNX export, which also says what the specialization is for. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
e7eb8213 copied bgrad_quantize's workspace sequence (launch once to learn the workspace shape, allocate it, launch again) into the cast lambda of Float8CurrentScalingQuantizer::quantize_with_dbias; dact_dbias carried a third copy. launch_with_workspace in common.h is that sequence once, for any kernel that reports its workspace this way, and the three sites hand it their launch. TestFP8CurrentScalingFusedDbias compares the fused paths against the unfused reduction and cast: bgrad_quantize and dbias_dgelu, rowwise-only and rowwise + columnwise, which on SM 9.0 are the unfused fallback and the fused cast + transpose kernel. Bit for bit on the quantized data and scale, up to reduction order on the bias gradient. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
backend="eager" runs the fx graph dynamo traced as it is, with no AOT autograd or inductor in between, so each custom op's real kernel is called through torch.ops with whatever its neighbour returned: a quantized tensor when the SwiGLU caches its input quantized. That is the direct real-argument path 514330fe added, covered now with a subclass argument. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- layernorm_mlp: e7eb8213 added current scaling to the recipes with the dbias + dact fusion list, so the later branch that still named it with the unfused recipes was unreachable; the branch and its comment now name only the recipes it serves. - dot_product_attention: the guard_scalar import 7f86e721 stopped using. - BasicLinear.pre_fuser_forward: the super() call published the stored quantizers on the context only for the method to overwrite them with the call's copies (f7bef038); it sets every field itself. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…dent bidimensional cast The bidimensional cast kernel keeps the whole 32-row tile in registers, so the activation runs on the registers it already holds: FP32 math (two lanes per f32x2 instruction, shared with the FP8 cast in util/activation_x2.cuh), one BF16 rounding, then the existing quantization. The bias gradient is reduced per CTA from the same registers. Scales are written in the GEMM-swizzled layout when asked. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…l cast The gated cast keeps the 32-row tile of each half in registers and runs the activation, gate and gradient math on them, then both quantizations. One CTA covers 32 rows by 256 columns; the backward pass quantizes the activation and gate gradients as two planes of the same tile. Used for BF16 to E4M3 SiLU gating with rows % 32 == 0, cols % 256 == 0 and 32-byte aligned operands, otherwise the TMA kernel runs as before. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The backward reads ln_out and act_out only column-wise (weight-gradient GEMMs), but the forward saved both usages, keeping a dead row-wise fp8 copy of each per layer (Linear already trims it). At 86016x1024/4096 that is ~440 MB per layer. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…select, fma.sat in dgelu, swizzle index once per tile, packed dbias sums) Per 86016x4096 call on B200: gelu_erf fwd 0.451 -> 0.342 ms, dgelu+dbias 0.651 -> 0.554 ms. Outputs hash-identical to the previous kernel in all 29 harness configurations. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…per plane/tile 17600x5120 on B200: fwd 0.122 -> 0.115 ms, bwd 0.172 -> 0.156 ms; byte differences vs the generic kernel unchanged. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…s the narrow tile The fused activations are issue-bound (ncu at boost clocks, 86016x4096: 68% issue slots busy, 4.2 TB/s), so the 4-wide cluster that helps the cast-only kernel costs them 5-8%; the dbias+dgelu variant spilled 170 B/thread in the wide tile (two inputs plus 16 column sums in 64 registers) and runs spill-free in the narrow one. Per call at 86016x4096 on B200 (harness, cluster/tile now chosen by the dispatch): gelu_erf fwd 0.342 -> 0.327 ms, dbias+dgelu 0.554 -> 0.509 ms, dgelu 0.515 -> 0.488 ms. Outputs and scales hash-identical to the generic kernel on randn; the all-bf16 input differs only in the pre-existing NaN/Inf rowwise-scale bytes. Two persistent-CTA variants that prefetch the next tile (cp.async into shared memory; a second register buffer at 2 CTAs/SM) hid the load latency (long_scoreboard 2.0 -> 0.4 cycles per issue) but lost more elsewhere: 0.369 ms either way, and the plain cast fell from 0.210 to 0.266-0.282 ms, so the launch-per-tile structure stays. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… cast LayerNorm with an MXFP8 quantizer ran as two kernels: ln_fwd_tuned_kernel into a BF16 tensor, then quantize_mxfp8_kernel_cast_only (row-wise + column-wise, GEMM-swizzled scales). ln_fwd_mxfp8.cu does both in one persistent kernel: tiles of 32 rows arrive in shared memory by bulk copy while the previous tile is processed; each warp replays the tuned kernel's one-warp-per-row arithmetic (same reductions, FP32 ops and BF16 rounding), quantizes its rows from registers with the cast's instructions, parks the BF16 values in place, and the CTA quantizes the tile column-wise from shared memory. Output bytes, scales, mu and rsigma are identical to the two-kernel path for BF16 input and weights, rows a multiple of 128 and 768 or 1024 columns; everything else keeps the previous path. nvte_layernorm_fwd takes the fused kernel for a GEMM-swizzled MXFP8 output (previously rejected) or any supported MXFP8 output unless cuDNN is requested; the new nvte_layernorm_fwd_mxfp8_fused tells the PyTorch extension when to hand it the quantizer's tensor instead of normalizing to BF16 and quantizing. is_aligned_to moves to normalization/common.h, shared with the RMSNorm kernel. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…per lane) Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…cp.approx) Forward is bit-identical to the scalar kernel (0 / 49.9M elements). The backward is not: the scalar kernel used __frcp_rn, and with rcp.approx 0.012% of dact outputs differ by one rounding step (max 2.9e-11 on a 1.1e-8 gradient). Isolated kernel times at 60x448 rows x 4096: fwd 168 -> 117 us, bwd 216 -> 198 us. With __frcp_rn per lane it is bit-exact in both directions at 144 / 204 us. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…SE_FAST_MATH=1 Default (__frcp_rn) is bit-identical to the scalar kernel in forward, input gradient and weight gradients (0 / 49.9M elements; fwd 168 -> 142 us, bwd 218 -> 200 us at 60x448 rows x 4096). NVTE_USE_FAST_MATH=1 selects rcp.approx: fwd 116 us, bwd 199 us, 0.012% of backward elements differ by one rounding step. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Nine commits on a2f424e, the merge-base of NVIDIA#3534, as shipped in the
2.22.0.dev0+dd19e95dwheel pinned by sunday-robotics/learning#1310. Base branchpr3534-baseis a2f424e itself, so the diff is exactly these commits.LayerNormMLP recompute: restore the delayed-scaling fp8 meta after recompute(+ test): the recompute path lost the delayed-scaling state swap, so the fc1/fc2 scales stopped updating under activation checkpointing.ops.Linear: do not inject a None bias key when the op has no bias(+ strict round-trip test).Load cuDNN/NVRTC/cuRAND from the python packages before the system toolkit: noCUDNN_HOME/NVRTC_HOMEneeded.cast: reduce the dbias partial sums across blocks instead of one serial block: the stock reduction is one 256-thread block looping over all row-chunks (23.5 ms/step at our shapes); two-stage parallel reduction, bit-identical, 4.6 ms.ops/fuser: run outside dynamo under delayed scaling(+ test): the fuser's delayed-scaling check inspects fp8 metas that do not exist before the first forward, so a module whose first call is compiled traced its eager path into a quantizer call on a fake tensor; a graph break is not enough (dynamo keeps tracing the resumed frame), so the call runs throughtorch._dynamo.disablewhen the active recipe is delayed scaling.cast: pair the gelu/dgelu math on the f32x2 path in the fp8 1D/2D cast kernels: the gelu / dgelu+dbias fp8 casts are issue-bound on the tanh math (2 SFU ops per element); computing two elements per thread on sm100 f32x2 instructions takes them from 3.0 / 3.1 to 3.9 / 4.0 TB/s at 114688x4096, fp8 bytes / amax / dbias bit-identical (all 65536 bf16 inputs checked against the scalar path).tanhf_x2repeats libdevice tanhf (CUDA 13.0.88) op for op.cast: pair the silu/dsilu math on the f32x2 path in the fp8 gated cast kernel: swiglu fwd 4.67 -> 5.76 TB/s, bwd 5.83 -> 6.61, bit-identical over all 2^32 fp32 inputs.activation: exact (erf) GELU, gelu_erf / dgelu_erf / geglu_erf / dbias_dgelu_erf:nvte_*_erfC API, cast-transpose table entries,tex.*_erfbindings,LayerNormMLP(activation="gelu_erf" / "geglu_erf"),te.ops.GELUErf; forward/backward bit-equal to torch's exact GELU (F.gelu,gelu_backward(approximate="none")) over all 65536 bf16 inputs, f32x2-paired; ~75% of the tanh casts' throughput (erff cost).dynamo: fused attention under dynamic shapes(+ test):torch.compile(dynamic=True)hands FusedAttention symbolic scalars and the custom-op packer baked every scalar into theOpaqueValueBundle, which only takes constants (SymNodeVariable() is not a constant).max_seqlen_q/kvfollow the input shape, so they get aSymIntschema slot (SymIntLike); every other scalar is configuration and is specialized throughguard_scalarunder tracing. The softcap check is spelled as comparisons so a symbolic cap guards instead of graph-breaking onmath.isfinite.Validated on B200 with TE's own suites (
test_numericsdbias / recompute,test_fusible_ops,test_torch_compile: 420 + 224 passed;test_te_ops_bias_nvfp4_layout[rht,2d]fail on stock too).🤖 Generated with Claude Code