Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion 3rdparty/nccl-extensions
11 changes: 9 additions & 2 deletions docs/api/pytorch.rst
Original file line number Diff line number Diff line change
Expand Up @@ -129,13 +129,20 @@ NCCL-based dispatch and combine for experts sharded across ranks. See

``MoeDispatch`` and ``MoeCombine`` can wrap the local expert operations in a
``transformer_engine.pytorch.ops.Sequential``. They share an ``EpConfig`` and
an ``EpBuffer``; the guide shows how to connect the per-expert counts and
routing weights through the sequence's extra-tensor channels.
an ``EpBuffer``. A buffer holds the routing state of one call until its backward has run, so
several calls in flight (for example pipeline microbatches) each need their own ``EpBuffer`` and
op instance. The guide shows how to connect the per-expert counts and routing weights through the
sequence's extra-tensor channels. ``moe_overlap_supported`` tells whether such a sequence runs as
one fused op.

.. autoapiclass:: transformer_engine.pytorch.ops.MoeDispatch

.. autoapiclass:: transformer_engine.pytorch.ops.MoeCombine

.. autoapifunction:: transformer_engine.pytorch.ops.moe_overlap_supported

.. autoapifunction:: transformer_engine.pytorch.ops.moe_overlap_unsupported_reason

.. autoapiclass:: transformer_engine.pytorch.ep.EpConfig

.. autoapifunction:: transformer_engine.pytorch.ep.ep_bootstrap
Expand Down
21 changes: 21 additions & 0 deletions docs/envvars.rst
Original file line number Diff line number Diff line change
Expand Up @@ -161,6 +161,27 @@ General
``WARN`` emits a warning explaining why CuTeDSL kernels are unavailable and falls back to CUDA kernels. ``ERROR`` raises an error and fails instead.
Only effective when the CuTeDSL backend is enabled (see :envvar:`NVTE_ENABLE_CUTEDSL_BACKEND`).

.. envvar:: NVTE_MOE_OVERLAP

:Type: ``str`` (directory)
:Default: unset (overlap off)
:Description: Directory of the CuTeDSL grouped GEMMs that overlap the NCCL EP dispatch and
combine with the expert GEMMs (``ep_gemm.h``, used by the fused
:class:`transformer_engine.pytorch.ops.MoeDispatch` expert MLP). The overlap is
off unless this is set together with ``NVTE_ENABLE_CUTEDSL_BACKEND=1``; then
``ep_bootstrap`` defaults to 24 communication SMs unless ``max_num_sms`` is
given. Otherwise the unfused ops run. The directory is the ``gemm/`` tree of
the EPCompOverlap kernel sources (with ``blackwell/``, ``rubin/`` and
``utils/``); its modules are imported by name and their version is not checked.

.. envvar:: NVTE_EP_OVERLAP_COMBINE_CHUNK_COLS

:Type: ``int``
:Default: 1024 when it divides the hidden size, else the whole row
:Description: Hidden columns of the overlapped combine per readiness flag (see
:envvar:`NVTE_MOE_OVERLAP`). The hidden size or a multiple of 1024 dividing it.
Smaller values let the combine start earlier but cost more flags to set and poll.

.. envvar:: NVTE_GROUPED_TENSOR_HANDLE_POOL_SIZE_MB

:Type: ``int`` (positive integer)
Expand Down
Loading
Loading