Skip to content

Pin upstream #3534 (a285cec8) for the learning repo's TE wheel - #6

Draft
yravan wants to merge 9 commits into
pr3534-basefrom
lemi/python-only
Draft

yravan wants to merge 9 commits into
pr3534-basefrom
lemi/python-only

Conversation

@yravan

@yravan yravan commented Nov 16, 2026 •

Copy link
Copy Markdown
Owner

The TransformerEngine build the learning repo's DoTv4Policy runs on: upstream NVIDIA#3534 (torch.compile support for LayerNormLinear / LayerNormMLP, head a285cec) with no changes of our own. This branch's tree equals a285cec. It exists to pin and build that commit until NVIDIA#3534 merges upstream.

Where our fixes live

Every fix the model needs is applied at import from the learning repo's lemi/learning/model/te/patches/, one file per TE layer or mechanism, asserted against this build. Each one is a defect still present on upstream main (checked at db1aff6). The open upstream PR NVIDIA#3856 would retire the SwiGLU one.

What used to be here:

  • The direct real-call path for custom ops is now patches/custom_op.py in the learning repo. Each wrapper op registers its _base op's kernel as its CUDA kernel.
  • Dynamic-shape support was dropped: SymInt sequence lengths, scalar specialization and the softcap comparisons. The model compiles with dynamic=False.
  • The device-through-every-pack-function refactor was dropped. It gave the same result as upstream's with device: block.

The branch history is kept: the tip is a merge plus a revert, with no force-push.

Wheel

Release lemi-te-2.22.0.dev0+cf0fc7a0:

  • built for NVTE_CUDA_ARCHS="90;100" (H100 + B200);
  • against torch 2.14.1+cu132, pip cuda-toolkit 13.2.2 and cuDNN frontend 1.31 headers;
  • sha256 2e244ec70f79d0bc061c575c80b55937204d81eb51f0babb6ccb71c230adcf69.

The learning repo pins it by URL. Its TE suite passes on this wheel (45/45, B200).

Build time on one node: 11m45s with NVTE_BUILD_THREADS_PER_JOB=4 MAX_JOBS=32, against 17m33s with the defaults.

Updating

When NVIDIA#3534 moves:

  1. Fast-forward pr3534-base to its head, and merge it here.
  2. Rebuild the wheel and publish a new release.
  3. Bump the pin and the version assert in the learning repo's patches/common.py.

🤖 Generated with Claude Code

yravan and others added 5 commits November 16, 2026 23:04
…olkit

The framework wheel (torch) is built against and ships the nvidia-* python
packages; loading the system toolkit's copy first makes TE and torch run two
different cuDNN versions in one process. Prefer the python packages when they
are installed; the system search is unchanged as the fallback.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…s, SwiGLU custom ops

- dynamo: sequence lengths reach the fused-attention custom op as SymInts
  (SymIntLike); every other scalar op argument is specialized with a guard
  under tracing, so torch.compile(dynamic=True) no longer fails on a symbolic
  scalar in the opaque constant bundle.
- attention: a compiled DotProductAttention selects its backend directly
  instead of reading the process-global selection cache, and AttentionParams
  specializes its scalar fields on construction under compile, so callers with
  different shapes do not recompile each other.
- ops: SwiGLU and SiTUGLU get forward and backward custom ops; the quantizer
  for the saved input (cache_quantized_input) is built in the constructor.
- A call configures copies of the recipe quantizers, never the stored ones
  (TransformerEngineBaseModule._call_quantizer, Quantizer.copy; in te.ops the
  call's quantizers live on its OperationContext, published by
  pre_fuser_forward(ctx)), so one compiled forward serves every instance of a
  module class.
- A real custom-op call runs the kernel directly on the wrapper's arguments;
  only a traced call flattens quantized tensors into the base op's slots.
- OpaqueValueBundle validates and hashes its contents on first hash or
  comparison, not on construction.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…y saved MLP inputs

- LayerNormMLP activation="gelu_erf" / "geglu_erf" and a te.ops GELUErf op
  (with the dbias fusion): the exact GELU, F.gelu(approximate="none").
- LayerNormLinear(qkv_rope=(cos, sin, head_dim)): the caller applies RoPE to
  the q and k parts of the packed output; the backward rotates the q and k
  output gradient back before the grad-output quantize.
- LayerNormLinear / LayerNormMLP forward(return_input_grad_sink=True) also
  returns a tensor shaped like the input whose gradient the norm backward adds
  to the input gradient, for a residual connection around the module.
- LayerNormMLP drops the row-wise MXFP8 copies of the saved FC1/FC2 inputs;
  the backward reads them column-wise only.

The operations above that have no compiled kernel are in
transformer_engine/pytorch/fused_kernels.py as unfused references built from
the existing bindings and torch ops: gelu_erf, dgelu_erf, geglu_erf,
dgeglu_erf, dbias_dgelu_erf, layernorm_bwd_add, bgrad_quantize_rope. Call
sites look them up through the module at call time, so a caller can assign a
faster implementation over the attribute.

module/_common.py reads layernorm_fwd / layernorm_bwd from
transformer_engine_torch at call time instead of from the import-time copy in
cpp_extensions, like the other modules.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…U, input-grad sink, saved MLP inputs

- test_torch_compile: fused attention under dynamic shapes and alternating
  batch sizes, te.ops SwiGLU fullgraph, one graph across module / op
  instances, the input-grad sink against the autograd residual add.
- test_numerics: gelu_erf / geglu_erf in the LayerNormMLP accuracy tests;
  fused_kernels.gelu_erf / dgelu_erf against torch's exact GELU, bit for bit
  unquantized and, quantized, against the FP8 cast of torch's BF16 result.
- test_fusible_ops: Linear op state dict strict round trip.
- test_custom_recipe: a custom quantizer gets the default Quantizer.copy.
- test_layernorm_mlp_saved_memory: one FP8 copy per saved MLP input.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Removed from this branch, with their tests: the exact (erf) GELU, the qkv RoPE backward, the input-gradient
sink, the SwiGLU / SiTUGLU custom ops, the per-call quantizer copies and the lazy OpaqueValueBundle. The
learning repo applies what it needs of them by name, from lemi/learning/model/te/.

Kept, each inside code that TE captures at import or runs in the middle of a function:
- common/__init__.py: cuDNN / NVRTC / cuRAND are loaded from the python packages before the system toolkit.
- dynamo/custom_op.py and attention: sequence lengths cross a custom op as SymInt slots, the other scalars are
  specialized. Without it fused attention under symbolic shapes fails in dynamo with
  "AsPythonConstantNotImplementedError: SymNodeVariable() is not a constant".
- attention: a compiled DotProductAttention picks its backend without the process-global cache.
- dynamo/custom_op.py: a real custom-op call runs the kernel directly on the wrapper's arguments.
- module/layernorm_mlp.py: the saved MLP inputs keep their column-wise copy only.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@yravan yravan changed the title Python-only subset of #2-#5 on top of #3534: compile fixes and layer hooks, no compiled changes What the learning repo cannot apply from outside TE, on top of #3534: Python only, 8 files Nov 17, 2026
yravan and others added 3 commits November 17, 2026 03:30
The library load order (2d394c6), the compiled DotProductAttention backend choice
(1296e41) and the column-wise-only saved LayerNormMLP inputs (bd3c328) are back to
the base's code: the learning repo applies each from lemi/learning/model/te/.

Left on the base: SymInt sequence lengths and specialized scalars in custom ops
(5a612f3) and the direct custom-op call (480dad5).

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…er's arguments

Only a traced call (FakeTensorMode active) goes through the base op; a real call
skips the second dispatcher round trip and the flatten / rebuild of quantized
tensors.
@yravan yravan changed the title What the learning repo cannot apply from outside TE, on top of #3534: Python only, 8 files #3534 plus a direct real-call path for custom ops Nov 19, 2026
@yravan yravan changed the title #3534 plus a direct real-call path for custom ops Pin upstream #3534 (a285cec8) for the learning repo's TE wheel Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant