Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
74 commits
Select commit Hold shift + click to select a range
943a183
[PyTorch] Compile Linear and Bias operations with forward fusion
pggPL Sep 17, 2026
086fe5e
[PyTorch] Preserve quantized tensors for dequantized backward
pggPL Sep 17, 2026
92a7552
[PyTorch] Share Linear and Bias compute between eager and compile
pggPL Sep 17, 2026
1a2362a
GDP attention support (#3539)
ksivaman Sep 18, 2026
36eaed4
[PyTorch] Enable compiled BasicLinear tensor parallelism and address …
pggPL Sep 18, 2026
6b50542
[PyTorch] Simplify ops backward bookkeeping and clarify compile tests
pggPL Sep 18, 2026
ac860c1
[PyTorch] Pass custom op context directly to operation compute
pggPL Sep 18, 2026
9233c73
[Common] Add non-TMA MXFP8 cast-only kernels for specialized rowwise-…
tdophung Sep 18, 2026
699ed6e
Fix distributed Newton-Schulz correctness (#3541)
vcherepanov-nv Sep 19, 2026
2b20496
fix: add missing NVTE_BHSD labels to fused_attn to_string (#3538)
andrewwhitecdw Sep 21, 2026
adc96c3
fix: duplicate JUnit XML filename overwrites encoder test results (#3…
andrewwhitecdw Sep 21, 2026
f0b6e71
fix: simplify qkv_format checks using a membership tuple (#3365)
andrewwhitecdw Sep 21, 2026
92d9241
[JAX] Support single-process-multi-device EP via XLA-borrowed communi…
phu0ngng Sep 21, 2026
02839d4
[All] Bump minimum supported cuDNN version to 9.12 (#3236)
cyanguwa Sep 21, 2026
9693205
Revert VERSION.txt to non-semver value to unblock CI
fheinecke Sep 21, 2026
ac544de
Remove the view() calls from Linear/LNLinear/LNMLP modules (#3523)
ptrendx Sep 22, 2026
f5f4e83
[PyTorch][CP] Reduce P2P forward peak memory: O(C) _ O(1) (#2916)
sudhakarsingh27 Sep 22, 2026
52eafa0
[JAX] Align fused attn backward output gradient sharding and other te…
KshitijLakhani Sep 23, 2026
f7273d7
[Common][PyTorch] EP Fused prepare and dispatch for non-eager mode (#…
phu0ngng Sep 23, 2026
5537dd1
Fix FusedSGD zero_grad argument forwarding (#3549)
wujingyue Sep 23, 2026
965b5dc
docs: add datatype and hardware support matrix (#3195)
CodersAcademy006 Sep 23, 2026
0b87832
[PyTorch] Retain NVFP4 RNG tensors through quantization (#3533)
Connor-XY Sep 24, 2026
ec39dcf
NVRTC NVFP4 quantization kernels (#3301)
CarlosGomes98 Sep 24, 2026
21a066d
Add AGENTS.md to Transformer Engine (#3559)
ptrendx Sep 24, 2026
45de18a
[PyTorch] Support distributed weights in GroupedLinear's grouped-tens…
fanshiqing Sep 25, 2026
9c33764
[PyTorch] Disable grouped MLP op fusion on Rubin with unsupported act…
timmoon10 Sep 25, 2026
b86c660
Fix hybrid extra state size mismatch (#3499)
negvet Sep 28, 2026
612d71d
[PyTorch] skip `test_fusion_requires_supported_grad_output_format` on…
kainzhong Sep 28, 2026
febebc4
Drop the unsupported window_size argument from attention backend quer…
Anai-Guo Sep 28, 2026
63b14c2
[Core] Warn when calling cuBLAS versions with wgrad GGEMM data corrup…
timmoon10 Sep 29, 2026
14f85fa
Bump NCCL EP submodule commit (#3561)
phu0ngng Sep 29, 2026
eb56ba2
[Common] Experimental CuTeDSL MXFP8 backends in C++ via TVM-FFI (#3137)
kainzhong Sep 29, 2026
d2fdccb
[PyTorch] Fix ops compile regressions and clarify test scenarios
pggPL Sep 29, 2026
88c26a5
Merge remote-tracking branch 'upstream/main' into ops_linear_bias_review
pggPL Sep 29, 2026
42bb9c4
[PyTorch] Match saved FP8 fake tensor dtypes to sources
pggPL Sep 29, 2026
1770259
Add NVFP4 RHT Support for SM120 and SM121 (#3300)
new-TonyWang Sep 29, 2026
607a35b
Fix race conditions in Triton permutation kernel (#3581)
fheinecke Sep 29, 2026
b8f1303
Reuse the host split sizes GroupedLinear's forward already computed (…
0z5a Sep 29, 2026
863b3b5
Update intialize_ub docstring (#3563)
Prachi-kushwaha Sep 29, 2026
dfece58
[Docs] Refresh README (#3535)
sbhavani Sep 29, 2026
fb0fed6
[PyTorch] Enable fused activation recompute for ScaledTanhSReLU (#3473)
wanyingw Sep 29, 2026
0cc62ec
[Pytorch] Wgrad kernel bug fix for cudnn 1.30 (#3579)
vthumbe1503 Sep 30, 2026
d0fcbb4
[PyTorch] Specialize BasicLinear compile ops by TP size
pggPL Sep 30, 2026
1f24081
Merge upstream main into ops_linear_bias_compile
pggPL Sep 30, 2026
fde54ce
[PyTorch] Borrow NCCL comm in C++ to support the ProcessGroupNCCL2 ba…
phu0ngng Sep 30, 2026
b57646c
[Common] row-scaled nvfp4 path: add single-launch group fused amax (…
cael-ling Sep 30, 2026
f6d1ecb
Isolate FlashAttention 2/3 imports so a broken install cannot break T…
0z5a Sep 30, 2026
08ba459
[common] Reject mismatched bias/out dtype for non-FP8/FP4 GEMM (#3565)
dundysm Sep 30, 2026
ef2bf75
[Docs] Fix broken links (#3597)
sbhavani Sep 30, 2026
5e8a899
MOE Sequential Block with Dispatch and Combine as Basic Ops (#3503)
vthumbe1503 Oct 1, 2026
a16bce3
[Common] row-scaled nvfp4 path: fuse row/col amax into a single TMA-t…
cael-ling Oct 1, 2026
dabf33e
[PyT] Prefer Flash Attn 2 instead of Fused Attn for THD dropout on Bl…
bzantium Oct 1, 2026
b0f0ecd
Fix `test_context_parallel_allgather_attn` test failure (#3598)
fheinecke Oct 1, 2026
0838ecb
Reduce the memory usage of the permutation reference implementation (…
fheinecke Oct 1, 2026
f38a1fc
[PyTorch] Support FA3/FA4 with pad_between_seqs=True (#3205)
xiaoyao0115 Oct 2, 2026
ae56bbc
[Common] Fix CUDA 12.8 build: use an L2 policy in the 128-bit evict_f…
ravimajeti Oct 2, 2026
db759b2
Fix JAX distributed unit test failure (#3611)
fheinecke Oct 2, 2026
4fcb7ca
Produce BOLT-compatible libraries (#3383)
fheinecke Oct 2, 2026
8920bbe
[JAX] Add MoE checkpoint names (#3589)
jberchtold-nvidia Oct 2, 2026
5759fa0
[JAX] Initialize softmax custom calls before execution (#3612)
KshitijLakhani Oct 2, 2026
d0b4b32
Prototype NVFP4 with FP8 UE5M3 block scales (#3325)
timmoon10 Oct 4, 2026
8cd5e20
Bump NCCL EP submodule commit (#3619)
phu0ngng Oct 5, 2026
f68a64a
Fix AArch64 NCCL EP linking with BOLT enabled (#3627)
aybchan Oct 5, 2026
2cd6893
[JAX] Add Triton dependency to JAX extension (#3610)
jberchtold-nvidia Oct 5, 2026
b47a356
[Common] Add launch bounds to multi-tensor swizzle kernels (#3622)
ravimajeti Oct 6, 2026
9d4bd38
Fix MXFP8 ONNX block quantization attributes (#3592)
pggPL Oct 6, 2026
20a3997
[PyTorch] Move compiled bias coverage into Linear tests
pggPL Oct 6, 2026
6d6a500
Skip FA4 for deterministic backward on SM12x (#3595)
janbernloehr Oct 6, 2026
ce29d24
[Common] Remove CuTeDSL first launch protection (#3620)
kainzhong Oct 6, 2026
bba2d2b
Support building docs for publishing via GHA (#3515)
fheinecke Oct 7, 2026
6eb41bd
[PyTorch] Enable LayerNormLinear UB AG overlap under no_grad (#3546)
ravimajeti Oct 7, 2026
5e99464
Preserve tiny sigmoid scores in biased fused routing (#3637)
philipcmonk Oct 7, 2026
14be43f
Build FP8 recipe quantizer params per instance (#3605)
Varun-51 Oct 8, 2026
3eba5e6
Merge upstream main into ops_linear_bias_compile
pggPL Oct 8, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion .github/workflows/docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,13 @@ name: 'Documentation'
on:
pull_request:
workflow_dispatch:
inputs:
enable_public_features:
# Disabling this blocks the cookie banner from appearing on localhost and other domains
description: 'Enable public features (required for public publishing)'
default: false
required: true
type: boolean
workflow_call:
concurrency:
# Group by workflow name + PR number (for PRs) or ref (for branch/tag pushes)
Expand All @@ -27,7 +34,7 @@ jobs:
export GIT_SHA=$(git show-ref --hash HEAD)
- name: 'Build docs'
env:
NVTE_DOCS_DISABLE_PUBLIC_FEATURES: '1'
NVTE_DOCS_DISABLE_PUBLIC_FEATURES: ${{ inputs.enable_public_features && '0' || '1' }}
NVTE_DOCS_SWITCHER_JSON_URL: 'versions1.json'
run: | # SPHINXOPTS="-W" errors out on warnings
doxygen docs/Doxyfile
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/minimum_versions.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ jobs:
apt-get install -y --allow-change-held-packages git python3.10 python3-pip cudnn9-cuda-12 libnccl-dev libnccl2
python3.10 --version
python3.10 -m pip install torch==${MIN_TORCH_VERSION}
python3.10 -m pip install cmake pybind11[global] ninja pydantic "importlib-metadata>=1.0" packaging einops onnxscript "nvidia-cudnn-frontend>=1.25.0"
python3.10 -m pip install cmake pybind11[global] ninja pydantic "importlib-metadata>=1.0" packaging einops onnxscript "nvidia-cudnn-frontend>=1.25.0" "apache-tvm-ffi>=0.1.12"
- name: 'Checkout'
uses: actions/checkout@v3
with:
Expand Down Expand Up @@ -65,7 +65,7 @@ jobs:
apt-get install -y git python3.10 python3-pip cudnn9-cuda-12
python3.10 --version
python3.10 -m pip install jax==${MIN_JAX_VERSION} "flax>=0.7.1"
python3.10 -m pip install cmake pybind11[global] ninja packaging pydantic "nvidia-cudnn-frontend>=1.25.0"
python3.10 -m pip install cmake pybind11[global] ninja packaging pydantic "nvidia-cudnn-frontend>=1.25.0" "apache-tvm-ffi>=0.1.12"
- name: 'Checkout'
uses: actions/checkout@v3
with:
Expand Down
2 changes: 1 addition & 1 deletion 3rdparty/nccl-extensions
Submodule nccl-extensions updated 118 files
407 changes: 407 additions & 0 deletions AGENTS.md

Large diffs are not rendered by default.

3 changes: 3 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Transformer Engine Claude Instructions

@AGENTS.md
479 changes: 55 additions & 424 deletions README.rst

Large diffs are not rendered by default.

1 change: 0 additions & 1 deletion benchmarks/attention/benchmark_attention.py
Original file line number Diff line number Diff line change
Expand Up @@ -195,7 +195,6 @@ def main():
config,
qkv_dtype=dtype,
qkv_layout=qkv_layout,
window_size=config.window_size,
pad_between_seqs=pad_between_seqs,
)
flash_attn_supported, fused_attn_supported, unfused_attn_supported = available_backends
Expand Down
2 changes: 1 addition & 1 deletion build_tools/VERSION.txt
Original file line number Diff line number Diff line change
@@ -1 +1 @@
2.21.0-dev0
2.21.0.dev0
7 changes: 6 additions & 1 deletion build_tools/jax.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
all_files_in_dir,
cudnn_frontend_include_path,
debug_build_enabled,
get_bolt_build_flags,
setup_mpi_flags,
nccl_include_path,
nccl_lib_path,
Expand All @@ -26,7 +27,7 @@

def install_requirements() -> List[str]:
"""Install dependencies for TE/JAX extensions."""
return ["jax", "flax>=0.7.1", "nvidia-cudnn-frontend>=1.25.0"]
return ["jax", "flax>=0.7.1", "nvidia-cudnn-frontend>=1.25.0", "triton"]


def test_requirements() -> List[str]:
Expand Down Expand Up @@ -122,6 +123,9 @@ def setup_jax_extension(
else:
cxx_flags.append("-g0")

bolt_cxx_flags, linker_flags = get_bolt_build_flags()
cxx_flags.extend(bolt_cxx_flags)

setup_mpi_flags(include_dirs, cxx_flags)

if bool(int(os.getenv("NVTE_WITH_CUBLASMP", 0))):
Expand All @@ -144,5 +148,6 @@ def setup_jax_extension(
sources=[str(path) for path in sources],
include_dirs=[str(path) for path in include_dirs],
extra_compile_args=cxx_flags,
extra_link_args=linker_flags,
**kwargs,
)
10 changes: 10 additions & 0 deletions build_tools/pytorch.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
cuda_version,
get_cuda_include_dirs,
debug_build_enabled,
get_bolt_build_flags,
nccl_ep_enabled,
setup_mpi_flags,
)
Expand Down Expand Up @@ -76,6 +77,9 @@ def setup_pytorch_extension(
else:
cxx_flags.append("-g0")

bolt_cxx_flags, linker_flags = get_bolt_build_flags()
cxx_flags.extend(bolt_cxx_flags)

# Version-dependent CUDA options
try:
version = cuda_version()
Expand All @@ -87,6 +91,11 @@ def setup_pytorch_extension(

setup_mpi_flags(include_dirs, cxx_flags)

# ProcessGroupNCCL class declarations are gated on USE_C10D_NCCL. get_nccl_comm_ptr
# borrows the raw ncclComm_t from that backend and is used by both NCCL EP and the
# cuSolverMp path, so the macro is needed regardless of the NCCL EP gate below.
cxx_flags.append("-DUSE_C10D_NCCL")

# Mirror the NCCL EP gate from setup.py / common CMake. When disabled, the
# ep.cpp source no-ops at the #ifdef boundary; without the define it would
# produce undefined references to nvte_ep_*.
Expand Down Expand Up @@ -122,6 +131,7 @@ def setup_pytorch_extension(
sources=[str(src) for src in sources],
include_dirs=[str(inc) for inc in include_dirs],
extra_compile_args={"cxx": cxx_flags},
extra_link_args=linker_flags,
libraries=[str(lib) for lib in libraries],
library_dirs=[str(lib_dir) for lib_dir in library_dirs],
)
67 changes: 67 additions & 0 deletions build_tools/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
import importlib
import os
import re
import shlex
import shutil
import subprocess
import sys
Expand Down Expand Up @@ -43,6 +44,72 @@ def debug_build_enabled() -> bool:
return bool(int(os.getenv("NVTE_BUILD_DEBUG", "0")))


@functools.lru_cache(maxsize=None)
def _cxx_compiler_predefined_macros() -> Tuple[str, ...]:
"""Predefined macros from the configured C++ compiler."""
cxx = shlex.split(os.getenv("CXX", "c++"))
if not cxx:
return ()
try:
result = subprocess.run(
[*cxx, "-dM", "-E", "-x", "c++", "-"],
input="",
capture_output=True,
text=True,
)
except OSError:
return ()
if result.returncode != 0:
return ()
return tuple(
fields[1]
for line in result.stdout.splitlines()
if len(fields := line.split(maxsplit=2)) >= 2 and fields[0] == "#define"
)


@functools.lru_cache(maxsize=None)
def cxx_compiler_supports_flag(flag: str) -> bool:
"""Whether the configured C++ compiler accepts a command-line flag."""
cxx = shlex.split(os.getenv("CXX", "c++"))
if not cxx:
return False
try:
result = subprocess.run(
[*cxx, flag, "-x", "c++", "-fsyntax-only", "-"],
input="",
capture_output=True,
text=True,
)
except OSError:
return False
return result.returncode == 0


def cxx_compiler_targets_arm64() -> bool:
"""Whether the configured C++ compiler targets Arm64."""
macros = _cxx_compiler_predefined_macros()
return any(macro in macros for macro in ("__aarch64__", "__arm64__", "_M_ARM64"))


def get_bolt_build_flags() -> Tuple[List[str], List[str]]:
"""BOLT-compatible host compiler and linker flags."""
compiler_flags = ["-fno-jump-tables"]
if cxx_compiler_supports_flag("-fno-reorder-blocks-and-partition"):
compiler_flags.append("-fno-reorder-blocks-and-partition")
linker_flags = ["-Wl,--emit-relocs", "-Wl,-z,now"]
if cxx_compiler_targets_arm64():
compiler_flags.extend(
[
"-mno-fix-cortex-a53-835769",
"-mno-fix-cortex-a53-843419",
]
)
# The Cortex-A53 843419 workaround is applied by the linker.
linker_flags.append("-mno-fix-cortex-a53-843419")
return compiler_flags, linker_flags


@functools.lru_cache(maxsize=None)
def get_max_jobs_for_parallel_build() -> int:
"""Number of parallel jobs for Nina build"""
Expand Down
26 changes: 20 additions & 6 deletions build_tools/wheel_utils/build_wheels.sh
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ BUILD_JAX=${5:-true}
CUDA_MAJOR=${6:-12}

export NVTE_RELEASE_BUILD=1
export NVTE_WITH_CUTEDSL=${NVTE_WITH_CUTEDSL:-0}
export PIP_CONSTRAINT=""
export TARGET_BRANCH=${TARGET_BRANCH:-}
mkdir -p /wheelhouse/logs
Expand All @@ -23,7 +24,7 @@ git checkout $TARGET_BRANCH
git submodule update --init --recursive

# Install deps
/opt/python/cp310-cp310/bin/pip install cmake pybind11[global] ninja setuptools wheel 'nvidia-cudnn-frontend>=1.25.0'
/opt/python/cp310-cp310/bin/pip install cmake pybind11[global] ninja setuptools wheel 'nvidia-cudnn-frontend>=1.25.0' 'apache-tvm-ffi>=0.1.12'

if $BUILD_METAPACKAGE ; then
cd /TransformerEngine
Expand All @@ -35,23 +36,36 @@ if $BUILD_COMMON ; then
VERSION=`cat build_tools/VERSION.txt`
WHL_BASE="transformer_engine-${VERSION}"

if [[ "$NVTE_WITH_CUTEDSL" != "0" ]]; then
WHEEL_PYTHON_TAG=cp310
WHEEL_ABI_TAG=abi3
else
WHEEL_PYTHON_TAG=py3
WHEEL_ABI_TAG=none
fi

# Create the wheel.
/opt/python/cp310-cp310/bin/python setup.py bdist_wheel --verbose --python-tag=py3 --plat-name=$PLATFORM 2>&1 | tee /wheelhouse/logs/common.txt
/opt/python/cp310-cp310/bin/python setup.py bdist_wheel --verbose --python-tag=$WHEEL_PYTHON_TAG --plat-name=$PLATFORM 2>&1 | tee /wheelhouse/logs/common.txt

# Repack the wheel for specific cuda version.
/opt/python/cp310-cp310/bin/wheel unpack dist/*
# From python 3.10 to 3.11, the package name delimiter in metadata got changed from - (hyphen) to _ (underscore).
sed -i "s/Name: transformer-engine/Name: transformer-engine-cu${CUDA_MAJOR}/g" "transformer_engine-${VERSION}/transformer_engine-${VERSION}.dist-info/METADATA"
sed -i "s/Name: transformer_engine/Name: transformer_engine_cu${CUDA_MAJOR}/g" "transformer_engine-${VERSION}/transformer_engine-${VERSION}.dist-info/METADATA"
mv "${WHL_BASE}/${WHL_BASE}.dist-info" "${WHL_BASE}/transformer_engine_cu${CUDA_MAJOR}-${VERSION}.dist-info"
# Set WHEEL Tag to match the py3-none filename written by line 54.
sed -i "s/Tag: cp310-cp310/Tag: py3-none/g" "${WHL_BASE}/transformer_engine_cu${CUDA_MAJOR}-${VERSION}.dist-info/WHEEL"
if [[ "$NVTE_WITH_CUTEDSL" != "0" ]]; then
# Build with CuTeDSL support: uses the CPython 3.10+ Stable ABI.
sed -i "s/Tag: cp310-cp310/Tag: cp310-abi3/g" "${WHL_BASE}/transformer_engine_cu${CUDA_MAJOR}-${VERSION}.dist-info/WHEEL"
else
# Build without CuTeDSL support: remain python version agnostic.
sed -i "s/Tag: cp310-cp310/Tag: py3-none/g" "${WHL_BASE}/transformer_engine_cu${CUDA_MAJOR}-${VERSION}.dist-info/WHEEL"
fi
/opt/python/cp310-cp310/bin/wheel pack ${WHL_BASE}

# Rename the wheel to make it python version agnostic.
# Rename the wheel to match the tag written above.
whl_name=$(basename dist/*)
IFS='-' read -ra whl_parts <<< "$whl_name"
whl_name_target="${whl_parts[0]}_cu${CUDA_MAJOR}-${whl_parts[1]}-py3-none-${whl_parts[4]}"
whl_name_target="${whl_parts[0]}_cu${CUDA_MAJOR}-${whl_parts[1]}-${WHEEL_PYTHON_TAG}-${WHEEL_ABI_TAG}-${whl_parts[4]}"
rm -rf $WHL_BASE dist
mv *.whl /wheelhouse/"$whl_name_target"
fi
Expand Down
3 changes: 3 additions & 0 deletions docs/api/pytorch.rst
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,9 @@ PyTorch
.. autoapiclass:: transformer_engine.pytorch.GatedDeltaNet2Attention(num_attention_heads, kv_channels, **kwargs)
:members: forward

.. autoapiclass:: transformer_engine.pytorch.GatedDeltaProductAttention(num_attention_heads, kv_channels, num_householder=1, **kwargs)
:members: forward

.. autoapiclass:: transformer_engine.pytorch.MultiheadAttention(hidden_size, num_attention_heads, **kwargs)
:members: forward, set_context_parallel_group, set_tensor_parallel_group

Expand Down
28 changes: 28 additions & 0 deletions docs/ecosystem.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
..
Copyright (c) 2022-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

See LICENSE for license information.

Ecosystem and historical integrations
=======================================

Transformer Engine has also been adopted by community projects and integrated
through past collaborations. The projects below are included as ecosystem and
historical references; their presence here does not imply that the Transformer
Engine project actively maintains or validates each integration.

* `DeepSpeed <https://github.com/deepspeedai/DeepSpeed>`_ — NVIDIA and the
DeepSpeed project have collaborated on Transformer Engine support in the past.
This integration is not actively maintained by the Transformer Engine project.
* `MosaicML Composer <https://github.com/mosaicml/composer>`_ — includes
community support for training with Transformer Engine precision.
* `GPT-NeoX <https://github.com/EleutherAI/gpt-neox>`_ — includes community
integration code for Transformer Engine layers.
* `Hugging Face Nanotron <https://github.com/huggingface/nanotron>`_ — includes
community FP8 integration code using Transformer Engine.
* `Levanter <https://github.com/marin-community/levanter>`_ — includes community
integration with the Transformer Engine JAX API.

For currently highlighted integrations, see the `Integrations
<https://github.com/NVIDIA/TransformerEngine#integrations>`_ section in the
project README.
33 changes: 31 additions & 2 deletions docs/envvars.rst
Original file line number Diff line number Diff line change
Expand Up @@ -93,12 +93,27 @@ Optional Dependencies
:Default: ``0``
:Description: Enable NVSHMEM support. When set to ``1``, requires ``NVSHMEM_HOME`` to be set to the NVSHMEM installation directory.

.. envvar:: NVTE_WITH_CUTEDSL

:Type: ``int`` (0 or 1)
:Default: ``0``
:Description: Enable CuTeDSL support when building ``libtransformer_engine.so``. When set to ``1``, requires
CPython Stable ABIs from Python 3.10 and above, and ``apache-tvm-ffi`` installed.
Note: when linking ``libtransformer_engine.so`` built with ``NVTE_WITH_CUTEDSL=1`` to a standalone C++ executable,
you must provide the definition of these CPython symbols. See ``tests/cpp/CMakeLists.txt`` for an example.

.. envvar:: NVTE_BUILD_ACTIVATION_WITH_FAST_MATH

:Type: CMake option
:Default: ``OFF``
:Description: Compile activation kernels (GELU, ReLU, SwiGLU) with the ``--use_fast_math`` CUDA compiler flag for improved performance at the cost of some precision.

.. envvar:: NVTE_BUILD_LEGACY_STATIC_NVFP4

:Type: CMake option
:Default: ``OFF``
:Description: Additionally compile the legacy static NVFP4 4over6 quantization kernel. Enable this option to provide a static fallback when runtime compilation is disabled with :envvar:`NVTE_DISABLE_NVRTC`. Enabling it increases build time because the NVFP4 4over6 template specializations are instantiated at build time.

CUDA Configuration
^^^^^^^^^^^^^^^^^^

Expand Down Expand Up @@ -130,6 +145,20 @@ General
value if an application legitimately creates more tensor handles than
the default pool can hold.

.. envvar:: NVTE_ENABLE_CUTEDSL_BACKEND

:Type: ``int`` (0 or 1)
:Default: ``0``
:Description: Use CuTeDSL kernel implementations when available. This requires that Transformer Engine was built with ``NVTE_WITH_CUTEDSL=1``.
It also requires ``apache-tvm-ffi``, ``nvidia-cutlass-dsl`` to be installed and you are using CPython 3.10 or later.
If these requirements are not met, Transformer Engine will fall back to using CUDA C++ kernels if possible.

.. envvar:: NVTE_WARN_IF_CUTEDSL_BACKEND_NOT_CHOSEN

:Type: ``int`` (0 or 1)
:Default: ``0``
:Description: Warn when TE falls back to the CUDA C++ kernels instead of dispatching to available CuTeDSL kernels. Useful to verify if the CuTeDSL path is taken.

.. envvar:: NVTE_GROUPED_TENSOR_HANDLE_POOL_SIZE_MB

:Type: ``int`` (positive integer)
Expand Down Expand Up @@ -305,13 +334,13 @@ Kernel Configuration

:Type: ``int`` (0 or 1)
:Default: ``0``
:Description: Enable fast math optimizations in runtime-compiled (NVRTC) kernels. This trades numerical accuracy for performance. These optimizations are experimental and inconsistently implemented.
:Description: Enable fast math optimizations in runtime-compiled (NVRTC / CuTeDSL) kernels. This trades numerical accuracy for performance. These optimizations are experimental and inconsistently implemented.

.. envvar:: NVTE_DISABLE_NVRTC

:Type: ``int`` (0 or 1)
:Default: ``0``
:Description: Disable NVRTC (CUDA Runtime Compilation) support. When set to ``1``, runtime kernel compilation is disabled. Existing transpose operations select their static fallback automatically. Fused softmax and normalization paths require their corresponding ``NVTE_BUILD_LEGACY_STATIC_FUSED_SOFTMAX`` or ``NVTE_BUILD_LEGACY_STATIC_NORM`` CMake option to have been enabled when the library was built; otherwise no static fallback is available.
:Description: Disable NVRTC (CUDA Runtime Compilation) support. When set to ``1``, runtime kernel compilation is disabled. Existing transpose operations select their static fallback automatically. NVFP4 4over6 quantization, fused softmax, and normalization paths require their corresponding ``NVTE_BUILD_LEGACY_STATIC_NVFP4``, ``NVTE_BUILD_LEGACY_STATIC_FUSED_SOFTMAX``, or ``NVTE_BUILD_LEGACY_STATIC_NORM`` CMake option to have been enabled when the library was built; otherwise no static fallback is available.

.. envvar:: NVTE_USE_CUTLASS_GROUPED_GEMM

Expand Down
6 changes: 3 additions & 3 deletions docs/examples/attention/attention.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@
" <tr>\n",
" <td rowspan=\"3\">PyTorch</td>\n",
" <td>cuDNN attention (`FusedAttention`)</td>\n",
" <td rowspan=\"3\"> [transformer_engine.pytorch.attention](https://github.com/NVIDIA/TransformerEngine/blob/main/transformer_engine/pytorch/attention.py)</td>\n",
" <td rowspan=\"3\"> [transformer_engine.pytorch.attention](https://github.com/NVIDIA/TransformerEngine/tree/main/transformer_engine/pytorch/attention)</td>\n",
" </tr>\n",
" <tr>\n",
" <td> flash-attention (`FlashAttention`)</td>\n",
Expand Down Expand Up @@ -476,12 +476,12 @@
" </tr>\n",
"</table>\n",
"\n",
"Some example usage of the different layouts can be found at [test_dpa_qkv_layout](https://github.com/NVIDIA/TransformerEngine/blob/main/tests/pytorch/attention/test_attention.py) and [test_dpa_qkv_layout_thd](https://github.com/NVIDIA/TransformerEngine/blob/main/tests/pytorch/attention/test_attention.py). Transformer Engine also provides a utility function [transformer_engine.pytorch.attention.dot_product_attention.utils.get_qkv_layout](https://github.com/NVIDIA/TransformerEngine/blob/main/transformer_engine/pytorch/attention.py) to help determine which layout a set of `q`, `k`, `v` tensors have (PyTorch only).\n",
"Some example usage of the different layouts can be found at [test_dpa_qkv_layout](https://github.com/NVIDIA/TransformerEngine/blob/main/tests/pytorch/attention/test_attention.py) and [test_dpa_qkv_layout_thd](https://github.com/NVIDIA/TransformerEngine/blob/main/tests/pytorch/attention/test_attention.py). Transformer Engine also provides a utility function [transformer_engine.pytorch.attention.dot_product_attention.utils.get_qkv_layout](https://github.com/NVIDIA/TransformerEngine/blob/main/transformer_engine/pytorch/attention/dot_product_attention/utils.py) to help determine which layout a set of `q`, `k`, `v` tensors have (PyTorch only).\n",
"\n",
"<div class=\"alert alert-info\">\n",
"<b>Note</b>\n",
" \n",
"When RoPE is employed, the <code>qkv_layout</code> may change in Transformer Engine PyTorch through [get_qkv_layout](https://github.com/NVIDIA/TransformerEngine/blob/main/transformer_engine/pytorch/attention.py). This is due to the in-place nature of our RoPE implementations. We convert `q`, `k`, `v` tensors from their initial layout to the corresponding <code>hd_hd_hd</code> layout. For example, from <code>sbh3d</code> in <code>pytorch.MultiHeadAttention</code> before RoPE, to <code>sbhd_sbhd_sbhd</code> in <code>pytorch.DotProductAttention</code> after RoPE.\n",
"When RoPE is employed, the <code>qkv_layout</code> may change in Transformer Engine PyTorch through [get_qkv_layout](https://github.com/NVIDIA/TransformerEngine/blob/main/transformer_engine/pytorch/attention/dot_product_attention/utils.py). This is due to the in-place nature of our RoPE implementations. We convert `q`, `k`, `v` tensors from their initial layout to the corresponding <code>hd_hd_hd</code> layout. For example, from <code>sbh3d</code> in <code>pytorch.MultiHeadAttention</code> before RoPE, to <code>sbhd_sbhd_sbhd</code> in <code>pytorch.DotProductAttention</code> after RoPE.\n",
"</div>\n"
]
},
Expand Down
Loading
Loading