Repository navigation
Conversation
…olkit The framework wheel (torch) is built against and ships the nvidia-* python packages; loading the system toolkit's copy first makes TE and torch run two different cuDNN versions in one process. Prefer the python packages when they are installed; the system search is unchanged as the fallback. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…s, SwiGLU custom ops - dynamo: sequence lengths reach the fused-attention custom op as SymInts (SymIntLike); every other scalar op argument is specialized with a guard under tracing, so torch.compile(dynamic=True) no longer fails on a symbolic scalar in the opaque constant bundle. - attention: a compiled DotProductAttention selects its backend directly instead of reading the process-global selection cache, and AttentionParams specializes its scalar fields on construction under compile, so callers with different shapes do not recompile each other. - ops: SwiGLU and SiTUGLU get forward and backward custom ops; the quantizer for the saved input (cache_quantized_input) is built in the constructor. - A call configures copies of the recipe quantizers, never the stored ones (TransformerEngineBaseModule._call_quantizer, Quantizer.copy; in te.ops the call's quantizers live on its OperationContext, published by pre_fuser_forward(ctx)), so one compiled forward serves every instance of a module class. - A real custom-op call runs the kernel directly on the wrapper's arguments; only a traced call flattens quantized tensors into the base op's slots. - OpaqueValueBundle validates and hashes its contents on first hash or comparison, not on construction. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…y saved MLP inputs - LayerNormMLP activation="gelu_erf" / "geglu_erf" and a te.ops GELUErf op (with the dbias fusion): the exact GELU, F.gelu(approximate="none"). - LayerNormLinear(qkv_rope=(cos, sin, head_dim)): the caller applies RoPE to the q and k parts of the packed output; the backward rotates the q and k output gradient back before the grad-output quantize. - LayerNormLinear / LayerNormMLP forward(return_input_grad_sink=True) also returns a tensor shaped like the input whose gradient the norm backward adds to the input gradient, for a residual connection around the module. - LayerNormMLP drops the row-wise MXFP8 copies of the saved FC1/FC2 inputs; the backward reads them column-wise only. The operations above that have no compiled kernel are in transformer_engine/pytorch/fused_kernels.py as unfused references built from the existing bindings and torch ops: gelu_erf, dgelu_erf, geglu_erf, dgeglu_erf, dbias_dgelu_erf, layernorm_bwd_add, bgrad_quantize_rope. Call sites look them up through the module at call time, so a caller can assign a faster implementation over the attribute. module/_common.py reads layernorm_fwd / layernorm_bwd from transformer_engine_torch at call time instead of from the import-time copy in cpp_extensions, like the other modules. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…U, input-grad sink, saved MLP inputs - test_torch_compile: fused attention under dynamic shapes and alternating batch sizes, te.ops SwiGLU fullgraph, one graph across module / op instances, the input-grad sink against the autograd residual add. - test_numerics: gelu_erf / geglu_erf in the LayerNormMLP accuracy tests; fused_kernels.gelu_erf / dgelu_erf against torch's exact GELU, bit for bit unquantized and, quantized, against the FP8 cast of torch's BF16 result. - test_fusible_ops: Linear op state dict strict round trip. - test_custom_recipe: a custom quantizer gets the default Quantizer.copy. - test_layernorm_mlp_saved_memory: one FP8 copy per saved MLP input. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Removed from this branch, with their tests: the exact (erf) GELU, the qkv RoPE backward, the input-gradient sink, the SwiGLU / SiTUGLU custom ops, the per-call quantizer copies and the lazy OpaqueValueBundle. The learning repo applies what it needs of them by name, from lemi/learning/model/te/. Kept, each inside code that TE captures at import or runs in the middle of a function: - common/__init__.py: cuDNN / NVRTC / cuRAND are loaded from the python packages before the system toolkit. - dynamo/custom_op.py and attention: sequence lengths cross a custom op as SymInt slots, the other scalars are specialized. Without it fused attention under symbolic shapes fails in dynamo with "AsPythonConstantNotImplementedError: SymNodeVariable() is not a constant". - attention: a compiled DotProductAttention picks its backend without the process-global cache. - dynamo/custom_op.py: a real custom-op call runs the kernel directly on the wrapper's arguments. - module/layernorm_mlp.py: the saved MLP inputs keep their column-wise copy only. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The library load order (2d394c6), the compiled DotProductAttention backend choice (1296e41) and the column-wise-only saved LayerNormMLP inputs (bd3c328) are back to the base's code: the learning repo applies each from lemi/learning/model/te/. Left on the base: SymInt sequence lengths and specialized scalars in custom ops (5a612f3) and the direct custom-op call (480dad5). Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…er's arguments Only a traced call (FakeTensorMode active) goes through the base op; a real call skips the second dispatcher round trip and the flatten / rebuild of quantized tensors.
…-call fast path
… the learning repo
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The TransformerEngine build the learning repo's DoTv4Policy runs on: upstream NVIDIA#3534 (torch.compile support for LayerNormLinear / LayerNormMLP, head a285cec) with no changes of our own. This branch's tree equals a285cec. It exists to pin and build that commit until NVIDIA#3534 merges upstream.
Where our fixes live
Every fix the model needs is applied at import from the learning repo's
lemi/learning/model/te/patches/, one file per TE layer or mechanism, asserted against this build. Each one is a defect still present on upstream main (checked at db1aff6). The open upstream PR NVIDIA#3856 would retire the SwiGLU one.What used to be here:
patches/custom_op.pyin the learning repo. Each wrapper op registers its_baseop's kernel as its CUDA kernel.SymIntsequence lengths, scalar specialization and the softcap comparisons. The model compiles withdynamic=False.device-through-every-pack-function refactor was dropped. It gave the same result as upstream'swith device:block.The branch history is kept: the tip is a merge plus a revert, with no force-push.
Wheel
Release lemi-te-2.22.0.dev0+cf0fc7a0:
NVTE_CUDA_ARCHS="90;100"(H100 + B200);cuda-toolkit13.2.2 and cuDNN frontend 1.31 headers;2e244ec70f79d0bc061c575c80b55937204d81eb51f0babb6ccb71c230adcf69.The learning repo pins it by URL. Its TE suite passes on this wheel (45/45, B200).
Build time on one node: 11m45s with
NVTE_BUILD_THREADS_PER_JOB=4 MAX_JOBS=32, against 17m33s with the defaults.Updating
When NVIDIA#3534 moves:
pr3534-baseto its head, and merge it here.patches/common.py.🤖 Generated with Claude Code