Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
1414c9d
Add torch stable ABI foundation: torch_compat layer and stable Tensor…
pggPL Aug 10, 2026
ec32413
Keep torch>=2.1 support: default build no longer touches stable headers
pggPL Aug 10, 2026
783f0b3
Rename torch_compat to torch_stable
pggPL Aug 10, 2026
98c7864
Use torch::stable::is_tensor_pyobject in the stable tensor caster
pggPL Aug 10, 2026
cb3933b
Reject too-old runtime torch in stable-ABI builds
pggPL Aug 10, 2026
9e1809e
Add module docstring to generated _build_config.py (pylint C0114)
pggPL Aug 10, 2026
f9d6424
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 10, 2026
1a2362a
GDP attention support (#3539)
ksivaman Sep 18, 2026
9233c73
[Common] Add non-TMA MXFP8 cast-only kernels for specialized rowwise-…
tdophung Sep 18, 2026
699ed6e
Fix distributed Newton-Schulz correctness (#3541)
vcherepanov-nv Sep 19, 2026
2b20496
fix: add missing NVTE_BHSD labels to fused_attn to_string (#3538)
andrewwhitecdw Sep 21, 2026
adc96c3
fix: duplicate JUnit XML filename overwrites encoder test results (#3…
andrewwhitecdw Sep 21, 2026
f0b6e71
fix: simplify qkv_format checks using a membership tuple (#3365)
andrewwhitecdw Sep 21, 2026
92d9241
[JAX] Support single-process-multi-device EP via XLA-borrowed communi…
phu0ngng Sep 21, 2026
02839d4
[All] Bump minimum supported cuDNN version to 9.12 (#3236)
cyanguwa Sep 21, 2026
9693205
Revert VERSION.txt to non-semver value to unblock CI
fheinecke Sep 21, 2026
ac544de
Remove the view() calls from Linear/LNLinear/LNMLP modules (#3523)
ptrendx Sep 22, 2026
f5f4e83
[PyTorch][CP] Reduce P2P forward peak memory: O(C) _ O(1) (#2916)
sudhakarsingh27 Sep 22, 2026
52eafa0
[JAX] Align fused attn backward output gradient sharding and other te…
KshitijLakhani Sep 23, 2026
f7273d7
[Common][PyTorch] EP Fused prepare and dispatch for non-eager mode (#…
phu0ngng Sep 23, 2026
5537dd1
Fix FusedSGD zero_grad argument forwarding (#3549)
wujingyue Sep 23, 2026
965b5dc
docs: add datatype and hardware support matrix (#3195)
CodersAcademy006 Sep 23, 2026
0b87832
[PyTorch] Retain NVFP4 RNG tensors through quantization (#3533)
Connor-XY Sep 24, 2026
ec39dcf
NVRTC NVFP4 quantization kernels (#3301)
CarlosGomes98 Sep 24, 2026
21a066d
Add AGENTS.md to Transformer Engine (#3559)
ptrendx Sep 24, 2026
45de18a
[PyTorch] Support distributed weights in GroupedLinear's grouped-tens…
fanshiqing Sep 25, 2026
9c33764
[PyTorch] Disable grouped MLP op fusion on Rubin with unsupported act…
timmoon10 Sep 25, 2026
b86c660
Fix hybrid extra state size mismatch (#3499)
negvet Sep 28, 2026
612d71d
[PyTorch] skip `test_fusion_requires_supported_grad_output_format` on…
kainzhong Sep 28, 2026
febebc4
Drop the unsupported window_size argument from attention backend quer…
Anai-Guo Sep 28, 2026
63b14c2
[Core] Warn when calling cuBLAS versions with wgrad GGEMM data corrup…
timmoon10 Sep 29, 2026
14f85fa
Bump NCCL EP submodule commit (#3561)
phu0ngng Sep 29, 2026
eb56ba2
[Common] Experimental CuTeDSL MXFP8 backends in C++ via TVM-FFI (#3137)
kainzhong Sep 29, 2026
1770259
Add NVFP4 RHT Support for SM120 and SM121 (#3300)
new-TonyWang Sep 29, 2026
607a35b
Fix race conditions in Triton permutation kernel (#3581)
fheinecke Sep 29, 2026
b8f1303
Reuse the host split sizes GroupedLinear's forward already computed (…
0z5a Sep 29, 2026
863b3b5
Update intialize_ub docstring (#3563)
Prachi-kushwaha Sep 29, 2026
dfece58
[Docs] Refresh README (#3535)
sbhavani Sep 29, 2026
fb0fed6
[PyTorch] Enable fused activation recompute for ScaledTanhSReLU (#3473)
wanyingw Sep 29, 2026
0cc62ec
[Pytorch] Wgrad kernel bug fix for cudnn 1.30 (#3579)
vthumbe1503 Sep 30, 2026
fde54ce
[PyTorch] Borrow NCCL comm in C++ to support the ProcessGroupNCCL2 ba…
phu0ngng Sep 30, 2026
b57646c
[Common] row-scaled nvfp4 path: add single-launch group fused amax (…
cael-ling Sep 30, 2026
f6d1ecb
Isolate FlashAttention 2/3 imports so a broken install cannot break T…
0z5a Sep 30, 2026
08ba459
[common] Reject mismatched bias/out dtype for non-FP8/FP4 GEMM (#3565)
dundysm Sep 30, 2026
ef2bf75
[Docs] Fix broken links (#3597)
sbhavani Sep 30, 2026
5e8a899
MOE Sequential Block with Dispatch and Combine as Basic Ops (#3503)
vthumbe1503 Oct 1, 2026
a16bce3
[Common] row-scaled nvfp4 path: fuse row/col amax into a single TMA-t…
cael-ling Oct 1, 2026
dabf33e
[PyT] Prefer Flash Attn 2 instead of Fused Attn for THD dropout on Bl…
bzantium Oct 1, 2026
b0f0ecd
Fix `test_context_parallel_allgather_attn` test failure (#3598)
fheinecke Oct 1, 2026
0838ecb
Reduce the memory usage of the permutation reference implementation (…
fheinecke Oct 1, 2026
f38a1fc
[PyTorch] Support FA3/FA4 with pad_between_seqs=True (#3205)
xiaoyao0115 Oct 2, 2026
ae56bbc
[Common] Fix CUDA 12.8 build: use an L2 policy in the 128-bit evict_f…
ravimajeti Oct 2, 2026
db759b2
Fix JAX distributed unit test failure (#3611)
fheinecke Oct 2, 2026
4fcb7ca
Produce BOLT-compatible libraries (#3383)
fheinecke Oct 2, 2026
8920bbe
[JAX] Add MoE checkpoint names (#3589)
jberchtold-nvidia Oct 2, 2026
5759fa0
[JAX] Initialize softmax custom calls before execution (#3612)
KshitijLakhani Oct 2, 2026
d0b4b32
Prototype NVFP4 with FP8 UE5M3 block scales (#3325)
timmoon10 Oct 4, 2026
8cd5e20
Bump NCCL EP submodule commit (#3619)
phu0ngng Oct 5, 2026
f68a64a
Fix AArch64 NCCL EP linking with BOLT enabled (#3627)
aybchan Oct 5, 2026
2cd6893
[JAX] Add Triton dependency to JAX extension (#3610)
jberchtold-nvidia Oct 5, 2026
b47a356
[Common] Add launch bounds to multi-tensor swizzle kernels (#3622)
ravimajeti Oct 6, 2026
9d4bd38
Fix MXFP8 ONNX block quantization attributes (#3592)
pggPL Oct 6, 2026
6d6a500
Skip FA4 for deterministic backward on SM12x (#3595)
janbernloehr Oct 6, 2026
ce29d24
[Common] Remove CuTeDSL first launch protection (#3620)
kainzhong Oct 6, 2026
bba2d2b
Support building docs for publishing via GHA (#3515)
fheinecke Oct 7, 2026
6eb41bd
[PyTorch] Enable LayerNormLinear UB AG overlap under no_grad (#3546)
ravimajeti Oct 7, 2026
5e99464
Preserve tiny sigmoid scores in biased fused routing (#3637)
philipcmonk Oct 7, 2026
14be43f
Build FP8 recipe quantizer params per instance (#3605)
Varun-51 Oct 8, 2026
ff092e6
Merge upstream main into te_stable_tensor_caster
pggPL Oct 8, 2026
d06801d
Require PyTorch 2.15 for stable ABI builds
pggPL Oct 8, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion .github/workflows/docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,13 @@ name: 'Documentation'
on:
pull_request:
workflow_dispatch:
inputs:
enable_public_features:
# Disabling this blocks the cookie banner from appearing on localhost and other domains
description: 'Enable public features (required for public publishing)'
default: false
required: true
type: boolean
workflow_call:
concurrency:
# Group by workflow name + PR number (for PRs) or ref (for branch/tag pushes)
Expand All @@ -27,7 +34,7 @@ jobs:
export GIT_SHA=$(git show-ref --hash HEAD)
- name: 'Build docs'
env:
NVTE_DOCS_DISABLE_PUBLIC_FEATURES: '1'
NVTE_DOCS_DISABLE_PUBLIC_FEATURES: ${{ inputs.enable_public_features && '0' || '1' }}
NVTE_DOCS_SWITCHER_JSON_URL: 'versions1.json'
run: | # SPHINXOPTS="-W" errors out on warnings
doxygen docs/Doxyfile
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/minimum_versions.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ jobs:
apt-get install -y --allow-change-held-packages git python3.10 python3-pip cudnn9-cuda-12 libnccl-dev libnccl2
python3.10 --version
python3.10 -m pip install torch==${MIN_TORCH_VERSION}
python3.10 -m pip install cmake pybind11[global] ninja pydantic "importlib-metadata>=1.0" packaging einops onnxscript "nvidia-cudnn-frontend>=1.25.0"
python3.10 -m pip install cmake pybind11[global] ninja pydantic "importlib-metadata>=1.0" packaging einops onnxscript "nvidia-cudnn-frontend>=1.25.0" "apache-tvm-ffi>=0.1.12"
- name: 'Checkout'
uses: actions/checkout@v3
with:
Expand Down Expand Up @@ -65,7 +65,7 @@ jobs:
apt-get install -y git python3.10 python3-pip cudnn9-cuda-12
python3.10 --version
python3.10 -m pip install jax==${MIN_JAX_VERSION} "flax>=0.7.1"
python3.10 -m pip install cmake pybind11[global] ninja packaging pydantic "nvidia-cudnn-frontend>=1.25.0"
python3.10 -m pip install cmake pybind11[global] ninja packaging pydantic "nvidia-cudnn-frontend>=1.25.0" "apache-tvm-ffi>=0.1.12"
- name: 'Checkout'
uses: actions/checkout@v3
with:
Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -43,3 +43,4 @@ tensor_dumps/
artifacts/
.DS_Store
.claude/
transformer_engine/pytorch/_build_config.py
2 changes: 1 addition & 1 deletion 3rdparty/nccl-extensions
Submodule nccl-extensions updated 118 files
407 changes: 407 additions & 0 deletions AGENTS.md

Large diffs are not rendered by default.

3 changes: 3 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Transformer Engine Claude Instructions

@AGENTS.md
479 changes: 55 additions & 424 deletions README.rst

Large diffs are not rendered by default.

1 change: 0 additions & 1 deletion benchmarks/attention/benchmark_attention.py
Original file line number Diff line number Diff line change
Expand Up @@ -195,7 +195,6 @@ def main():
config,
qkv_dtype=dtype,
qkv_layout=qkv_layout,
window_size=config.window_size,
pad_between_seqs=pad_between_seqs,
)
flash_attn_supported, fused_attn_supported, unfused_attn_supported = available_backends
Expand Down
2 changes: 1 addition & 1 deletion build_tools/VERSION.txt
Original file line number Diff line number Diff line change
@@ -1 +1 @@
2.21.0-dev0
2.21.0.dev0
7 changes: 6 additions & 1 deletion build_tools/jax.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
all_files_in_dir,
cudnn_frontend_include_path,
debug_build_enabled,
get_bolt_build_flags,
setup_mpi_flags,
nccl_include_path,
nccl_lib_path,
Expand All @@ -26,7 +27,7 @@

def install_requirements() -> List[str]:
"""Install dependencies for TE/JAX extensions."""
return ["jax", "flax>=0.7.1", "nvidia-cudnn-frontend>=1.25.0"]
return ["jax", "flax>=0.7.1", "nvidia-cudnn-frontend>=1.25.0", "triton"]


def test_requirements() -> List[str]:
Expand Down Expand Up @@ -122,6 +123,9 @@ def setup_jax_extension(
else:
cxx_flags.append("-g0")

bolt_cxx_flags, linker_flags = get_bolt_build_flags()
cxx_flags.extend(bolt_cxx_flags)

setup_mpi_flags(include_dirs, cxx_flags)

if bool(int(os.getenv("NVTE_WITH_CUBLASMP", 0))):
Expand All @@ -144,5 +148,6 @@ def setup_jax_extension(
sources=[str(path) for path in sources],
include_dirs=[str(path) for path in include_dirs],
extra_compile_args=cxx_flags,
extra_link_args=linker_flags,
**kwargs,
)
28 changes: 28 additions & 0 deletions build_tools/pytorch.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
cuda_version,
get_cuda_include_dirs,
debug_build_enabled,
get_bolt_build_flags,
nccl_ep_enabled,
setup_mpi_flags,
)
Expand Down Expand Up @@ -76,6 +77,9 @@ def setup_pytorch_extension(
else:
cxx_flags.append("-g0")

bolt_cxx_flags, linker_flags = get_bolt_build_flags()
cxx_flags.extend(bolt_cxx_flags)

# Version-dependent CUDA options
try:
version = cuda_version()
Expand All @@ -87,6 +91,11 @@ def setup_pytorch_extension(

setup_mpi_flags(include_dirs, cxx_flags)

# ProcessGroupNCCL class declarations are gated on USE_C10D_NCCL. get_nccl_comm_ptr
# borrows the raw ncclComm_t from that backend and is used by both NCCL EP and the
# cuSolverMp path, so the macro is needed regardless of the NCCL EP gate below.
cxx_flags.append("-DUSE_C10D_NCCL")

# Mirror the NCCL EP gate from setup.py / common CMake. When disabled, the
# ep.cpp source no-ops at the #ifdef boundary; without the define it would
# produce undefined references to nvte_ep_*.
Expand All @@ -112,6 +121,24 @@ def setup_pytorch_extension(
if bool(int(os.getenv("NVTE_WITH_CUBLASMP", 0))):
cxx_flags.append("-DNVTE_WITH_CUBLASMP")

# Experimental: build the torch_stable layer against the torch stable ABI
# (requires torch >= 2.15). Without the flag the same code compiles against
# the full torch ABI. See csrc/torch_stable.h.
torch_stable_abi = bool(int(os.getenv("NVTE_TORCH_STABLE_ABI", "0")))
if torch_stable_abi:
cxx_flags.append("-DNVTE_WITH_TORCH_STABLE")
cxx_flags.append("-DTORCH_TARGET_VERSION=0x020f000000000000")

# Record build configuration for runtime checks (stable-ABI builds link
# shims that only exist in libtorch >= TORCH_TARGET_VERSION, so the Python
# package must reject older runtime torch before loading the extension).
build_config = Path(csrc_header_files).parent / "_build_config.py"
build_config.write_text(
'"""Build configuration. Generated by build_tools/pytorch.py, do not edit."""\n'
f"TORCH_STABLE_ABI = {torch_stable_abi}\n"
"TORCH_STABLE_ABI_MIN_TORCH = (2, 15)\n"
)

# Construct PyTorch CUDA extension
sources = [str(path) for path in sources]
include_dirs = [str(path) for path in include_dirs]
Expand All @@ -122,6 +149,7 @@ def setup_pytorch_extension(
sources=[str(src) for src in sources],
include_dirs=[str(inc) for inc in include_dirs],
extra_compile_args={"cxx": cxx_flags},
extra_link_args=linker_flags,
libraries=[str(lib) for lib in libraries],
library_dirs=[str(lib_dir) for lib_dir in library_dirs],
)
67 changes: 67 additions & 0 deletions build_tools/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
import importlib
import os
import re
import shlex
import shutil
import subprocess
import sys
Expand Down Expand Up @@ -43,6 +44,72 @@ def debug_build_enabled() -> bool:
return bool(int(os.getenv("NVTE_BUILD_DEBUG", "0")))


@functools.lru_cache(maxsize=None)
def _cxx_compiler_predefined_macros() -> Tuple[str, ...]:
"""Predefined macros from the configured C++ compiler."""
cxx = shlex.split(os.getenv("CXX", "c++"))
if not cxx:
return ()
try:
result = subprocess.run(
[*cxx, "-dM", "-E", "-x", "c++", "-"],
input="",
capture_output=True,
text=True,
)
except OSError:
return ()
if result.returncode != 0:
return ()
return tuple(
fields[1]
for line in result.stdout.splitlines()
if len(fields := line.split(maxsplit=2)) >= 2 and fields[0] == "#define"
)


@functools.lru_cache(maxsize=None)
def cxx_compiler_supports_flag(flag: str) -> bool:
"""Whether the configured C++ compiler accepts a command-line flag."""
cxx = shlex.split(os.getenv("CXX", "c++"))
if not cxx:
return False
try:
result = subprocess.run(
[*cxx, flag, "-x", "c++", "-fsyntax-only", "-"],
input="",
capture_output=True,
text=True,
)
except OSError:
return False
return result.returncode == 0


def cxx_compiler_targets_arm64() -> bool:
"""Whether the configured C++ compiler targets Arm64."""
macros = _cxx_compiler_predefined_macros()
return any(macro in macros for macro in ("__aarch64__", "__arm64__", "_M_ARM64"))


def get_bolt_build_flags() -> Tuple[List[str], List[str]]:
"""BOLT-compatible host compiler and linker flags."""
compiler_flags = ["-fno-jump-tables"]
if cxx_compiler_supports_flag("-fno-reorder-blocks-and-partition"):
compiler_flags.append("-fno-reorder-blocks-and-partition")
linker_flags = ["-Wl,--emit-relocs", "-Wl,-z,now"]
if cxx_compiler_targets_arm64():
compiler_flags.extend(
[
"-mno-fix-cortex-a53-835769",
"-mno-fix-cortex-a53-843419",
]
)
# The Cortex-A53 843419 workaround is applied by the linker.
linker_flags.append("-mno-fix-cortex-a53-843419")
return compiler_flags, linker_flags


@functools.lru_cache(maxsize=None)
def get_max_jobs_for_parallel_build() -> int:
"""Number of parallel jobs for Nina build"""
Expand Down
26 changes: 20 additions & 6 deletions build_tools/wheel_utils/build_wheels.sh
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ BUILD_JAX=${5:-true}
CUDA_MAJOR=${6:-12}

export NVTE_RELEASE_BUILD=1
export NVTE_WITH_CUTEDSL=${NVTE_WITH_CUTEDSL:-0}
export PIP_CONSTRAINT=""
export TARGET_BRANCH=${TARGET_BRANCH:-}
mkdir -p /wheelhouse/logs
Expand All @@ -23,7 +24,7 @@ git checkout $TARGET_BRANCH
git submodule update --init --recursive

# Install deps
/opt/python/cp310-cp310/bin/pip install cmake pybind11[global] ninja setuptools wheel 'nvidia-cudnn-frontend>=1.25.0'
/opt/python/cp310-cp310/bin/pip install cmake pybind11[global] ninja setuptools wheel 'nvidia-cudnn-frontend>=1.25.0' 'apache-tvm-ffi>=0.1.12'

if $BUILD_METAPACKAGE ; then
cd /TransformerEngine
Expand All @@ -35,23 +36,36 @@ if $BUILD_COMMON ; then
VERSION=`cat build_tools/VERSION.txt`
WHL_BASE="transformer_engine-${VERSION}"

if [[ "$NVTE_WITH_CUTEDSL" != "0" ]]; then
WHEEL_PYTHON_TAG=cp310
WHEEL_ABI_TAG=abi3
else
WHEEL_PYTHON_TAG=py3
WHEEL_ABI_TAG=none
fi

# Create the wheel.
/opt/python/cp310-cp310/bin/python setup.py bdist_wheel --verbose --python-tag=py3 --plat-name=$PLATFORM 2>&1 | tee /wheelhouse/logs/common.txt
/opt/python/cp310-cp310/bin/python setup.py bdist_wheel --verbose --python-tag=$WHEEL_PYTHON_TAG --plat-name=$PLATFORM 2>&1 | tee /wheelhouse/logs/common.txt

# Repack the wheel for specific cuda version.
/opt/python/cp310-cp310/bin/wheel unpack dist/*
# From python 3.10 to 3.11, the package name delimiter in metadata got changed from - (hyphen) to _ (underscore).
sed -i "s/Name: transformer-engine/Name: transformer-engine-cu${CUDA_MAJOR}/g" "transformer_engine-${VERSION}/transformer_engine-${VERSION}.dist-info/METADATA"
sed -i "s/Name: transformer_engine/Name: transformer_engine_cu${CUDA_MAJOR}/g" "transformer_engine-${VERSION}/transformer_engine-${VERSION}.dist-info/METADATA"
mv "${WHL_BASE}/${WHL_BASE}.dist-info" "${WHL_BASE}/transformer_engine_cu${CUDA_MAJOR}-${VERSION}.dist-info"
# Set WHEEL Tag to match the py3-none filename written by line 54.
sed -i "s/Tag: cp310-cp310/Tag: py3-none/g" "${WHL_BASE}/transformer_engine_cu${CUDA_MAJOR}-${VERSION}.dist-info/WHEEL"
if [[ "$NVTE_WITH_CUTEDSL" != "0" ]]; then
# Build with CuTeDSL support: uses the CPython 3.10+ Stable ABI.
sed -i "s/Tag: cp310-cp310/Tag: cp310-abi3/g" "${WHL_BASE}/transformer_engine_cu${CUDA_MAJOR}-${VERSION}.dist-info/WHEEL"
else
# Build without CuTeDSL support: remain python version agnostic.
sed -i "s/Tag: cp310-cp310/Tag: py3-none/g" "${WHL_BASE}/transformer_engine_cu${CUDA_MAJOR}-${VERSION}.dist-info/WHEEL"
fi
/opt/python/cp310-cp310/bin/wheel pack ${WHL_BASE}

# Rename the wheel to make it python version agnostic.
# Rename the wheel to match the tag written above.
whl_name=$(basename dist/*)
IFS='-' read -ra whl_parts <<< "$whl_name"
whl_name_target="${whl_parts[0]}_cu${CUDA_MAJOR}-${whl_parts[1]}-py3-none-${whl_parts[4]}"
whl_name_target="${whl_parts[0]}_cu${CUDA_MAJOR}-${whl_parts[1]}-${WHEEL_PYTHON_TAG}-${WHEEL_ABI_TAG}-${whl_parts[4]}"
rm -rf $WHL_BASE dist
mv *.whl /wheelhouse/"$whl_name_target"
fi
Expand Down
3 changes: 3 additions & 0 deletions docs/api/pytorch.rst
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,9 @@ PyTorch
.. autoapiclass:: transformer_engine.pytorch.GatedDeltaNet2Attention(num_attention_heads, kv_channels, **kwargs)
:members: forward

.. autoapiclass:: transformer_engine.pytorch.GatedDeltaProductAttention(num_attention_heads, kv_channels, num_householder=1, **kwargs)
:members: forward

.. autoapiclass:: transformer_engine.pytorch.MultiheadAttention(hidden_size, num_attention_heads, **kwargs)
:members: forward, set_context_parallel_group, set_tensor_parallel_group

Expand Down
28 changes: 28 additions & 0 deletions docs/ecosystem.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
..
Copyright (c) 2022-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

See LICENSE for license information.

Ecosystem and historical integrations
=======================================

Transformer Engine has also been adopted by community projects and integrated
through past collaborations. The projects below are included as ecosystem and
historical references; their presence here does not imply that the Transformer
Engine project actively maintains or validates each integration.

* `DeepSpeed <https://github.com/deepspeedai/DeepSpeed>`_ — NVIDIA and the
DeepSpeed project have collaborated on Transformer Engine support in the past.
This integration is not actively maintained by the Transformer Engine project.
* `MosaicML Composer <https://github.com/mosaicml/composer>`_ — includes
community support for training with Transformer Engine precision.
* `GPT-NeoX <https://github.com/EleutherAI/gpt-neox>`_ — includes community
integration code for Transformer Engine layers.
* `Hugging Face Nanotron <https://github.com/huggingface/nanotron>`_ — includes
community FP8 integration code using Transformer Engine.
* `Levanter <https://github.com/marin-community/levanter>`_ — includes community
integration with the Transformer Engine JAX API.

For currently highlighted integrations, see the `Integrations
<https://github.com/NVIDIA/TransformerEngine#integrations>`_ section in the
project README.
Loading
Loading