Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 67 additions & 0 deletions docs/api/jax.rst
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ Modules
.. autoapifunction:: transformer_engine.jax.fp8_autocast
.. autoapifunction:: transformer_engine.jax.autocast
.. autoapifunction:: transformer_engine.jax.update_collections
.. autoapifunction:: transformer_engine.jax.grouped_situglu


.. autoapiclass:: transformer_engine.jax.flax.LayerNorm(epsilon=1e-6, layernorm_type='layernorm', **kwargs)
Expand Down Expand Up @@ -68,14 +69,80 @@ Router, routing kernels, and grouped dense for MoE layers. See

.. autoapifunction:: transformer_engine.jax.router.fused_topk_with_score_function

.. autoapifunction:: transformer_engine.jax.router.fused_topk_with_score_function_qb

.. autoapifunction:: transformer_engine.jax.router.fused_moe_aux_loss

.. autoapifunction:: transformer_engine.jax.permutation.sort_chunks_by_index

.. autoapifunction:: transformer_engine.jax.dense.grouped_dense

.. autoapiclass:: transformer_engine.jax.moe.SiTUActivation

``SiTUActivation`` uses native grouped activation kernels between the expert
GEMMs, including their backward pass. Device group counts bound active rows;
unused receive capacity is zero-filled without evaluating the activation there.
FP32, FP16 and BF16 activation outputs are supported. MXFP8 expert execution
uses grouped quantization after the activation. The standalone
``grouped_situglu(x, group_sizes, beta1=4.0, beta2=25.0)`` function has the same
math and automatic differentiation support. It expects contiguous gate/up
halves in each row of ``x[capacity, 2H]``. Counts must be nonnegative device
int32 values with a sum no greater than capacity. Distributed expert callers
should pass local arrays and counts inside ``shard_map``.

.. autoapifunction:: transformer_engine.jax.moe.moe

.. autoapiclass:: transformer_engine.jax.moe.RouterComputationInfo

Latent MoE models can pass full-width hidden states through
``RouterComputationInfo(router_input=hidden_states, ...)`` while calling
``moe(latent_states, router_info, ...)`` with narrower expert activations.
Batch and sequence dimensions must agree. Router parameters use the router
width; expert parameters and EP bootstrap buffers use the latent width.
The custom VJP returns gradients to both inputs, including when the latent
is a differentiable projection of the router input. The default
``router_input=None`` preserves routing on the expert input.

Quantile Balancing statistics
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Set ``RouterComputationInfo.qb_bin_bounds`` to an FP32 ``[lower, upper]`` array
to collect QB statistics during MoE routing. Sigmoid routing, an FP32 expert
bias, no routing groups and zero auxiliary loss are required. The existing CUDA
QB kernels provide ``fused_atomic`` and ``two_kernel`` modes, selected by
``qb_histogram_mode``. The number of bins defaults to 1000.

The bounds must be finite and ordered; their values are read on device without
host validation. Kimi uses ``[min(bias)-1, max(bias)+1]``. Keep both bias and
bounds unchanged throughout every microbatch in a training step. With no mask, all token rows must be valid. Optional ``qb_token_mask``
(``[batch, sequence]``) excludes padding: CUDA computes scores and cutoff,
while a JAX histogram counts only valid rows. This masked path does not use
CUDA histogram accumulation. The standalone router accepts ``token_mask``
with shape ``[tokens]`` for the same behavior.

With QB enabled, ``moe`` returns a fifth result, an int32 histogram array of
shape ``[token_shards, experts, bins]``. Its leading axis is sharded across the
compound DP/FSDP/EP token axes. The forward does not reduce these histograms
or update biases. Accumulate per-shard counts across microbatches and only then
sum the leading axis to obtain each expert's global-batch distribution. The
training framework derives and mean-centers the new bias from that distribution,
applying it on the next step. Default calls keep the four-result API.

The standalone QB router returns ``(probs, routing_map, histogram)`` and accepts
an optional initial histogram for functional accumulation. Its FFI declares
input/output aliasing for the CUDA kernel's in-place histogram writes. For SPMD,
place the standalone operation inside ``shard_map`` with the expert axis
replicated and expose local histograms as a sharded leading axis, as MoE does.
Neither histogram collection nor bias/bounds inputs have a gradient. The router
reuses the ordinary sigmoid routing backward for gradients through its weights.
Use fresh forward statistics or explicit scan carry state; an external mutable
histogram would be counted twice when activation checkpointing recomputes it.

The Flax wrapper accepts ``qb_bin_bounds`` on each call and exposes
``qb_num_bins`` and ``qb_histogram_mode`` as module configuration. It returns
the same optional fifth result. Step-level bias updates, checkpointing that
bias state, and TPU implementations belong to the training framework.

.. autoapifunction:: transformer_engine.jax.ep.ep_bootstrap

.. autoapifunction:: transformer_engine.jax.ep.ep_finalize
Expand Down
124 changes: 124 additions & 0 deletions tests/cpp/operator/test_act.cu
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,11 @@
* See LICENSE for license information.
************************************************************************/

#include <algorithm>
#include <cmath>
#include <cstring>
#include <memory>
#include <limits>
#include <iomanip>
#include <iostream>
#include <random>
Expand Down Expand Up @@ -424,3 +426,125 @@ INSTANTIATE_TEST_SUITE_P(
::testing::ValuesIn(test::all_fp_types),
::testing::Values(std::pair<size_t, size_t>{768, 2816})),
test_name_generator);

namespace {

template <typename T>
void test_grouped_situglu(const size_t hidden, const int alignment_mode, const int count_mode,
const std::pair<float, float> caps) {
constexpr size_t rows = 13;
const size_t input_offset = alignment_mode == 1 ? 1 : 0;
const size_t output_offset = alignment_mode == 0 ? 0 : 1;
const size_t active = count_mode == 0 ? 8 : (count_mode == 1 ? rows : 0);
const DType dtype = test::TypeInfo<T>::dtype;
const std::vector<size_t> input_shape{rows, 2 * hidden}, output_shape{rows, hidden};
test::Tensor input("group_situ_input", std::vector<size_t>{rows * 2 * hidden + input_offset + 1},
dtype);
test::Tensor grad("group_situ_grad", std::vector<size_t>{rows * hidden + input_offset + 1},
dtype);
test::Tensor output("group_situ_output", std::vector<size_t>{rows * hidden + output_offset + 1},
dtype);
test::Tensor dx("group_situ_dx", std::vector<size_t>{rows * 2 * hidden + output_offset + 1},
dtype);
test::Tensor counts("group_situ_counts", std::vector<size_t>{3}, DType::kInt64);
for (test::Tensor *tensor : {&input, &grad, &output, &dx}) {
std::fill_n(tensor->rowwise_cpu_dptr<T>(), test::product(tensor->shape()),
static_cast<T>(std::numeric_limits<float>::quiet_NaN()));
}
auto *x = input.rowwise_cpu_dptr<T>() + input_offset;
auto *dy = grad.rowwise_cpu_dptr<T>() + input_offset;
for (size_t row = 0; row < active; ++row) {
for (size_t col = 0; col < hidden; ++col) {
x[row * 2 * hidden + col] = static_cast<T>(static_cast<float>(col % 11) - 5.25f);
x[row * 2 * hidden + hidden + col] = static_cast<T>(static_cast<float>(row) - 4.5f);
dy[row * hidden + col] = static_cast<T>(0.125f * static_cast<float>(col % 7) - 0.25f);
}
}
auto *sizes = counts.rowwise_cpu_dptr<int64_t>();
sizes[0] = count_mode == 2 ? 0 : 3;
sizes[1] = 0;
sizes[2] = count_mode == 2 ? 0 : 5;
for (test::Tensor *tensor : {&input, &grad, &output, &dx, &counts}) tensor->from_cpu();

GroupedTensorWrapper in_group(3, input_shape), grad_group(3, output_shape);
GroupedTensorWrapper out_group(3, output_shape), dx_group(3, input_shape);
in_group.set_rowwise_data(static_cast<T *>(input.rowwise_dptr()) + input_offset, dtype,
input_shape);
grad_group.set_rowwise_data(static_cast<T *>(grad.rowwise_dptr()) + input_offset, dtype,
output_shape);
out_group.set_rowwise_data(static_cast<T *>(output.rowwise_dptr()) + output_offset, dtype,
output_shape);
dx_group.set_rowwise_data(static_cast<T *>(dx.rowwise_dptr()) + output_offset, dtype,
input_shape);
if (count_mode != 1) {
in_group.set_first_dims(counts.rowwise_dptr(), DType::kInt64, std::vector<size_t>{3});
}
nvte_group_situglu(in_group.data(), out_group.data(), caps.first, caps.second, nullptr);
nvte_group_dsituglu(grad_group.data(), in_group.data(), dx_group.data(), caps.first, caps.second,
nullptr);
ASSERT_EQ(cudaDeviceSynchronize(), cudaSuccess);
output.to_cpu();
dx.to_cpu();
const auto *y = output.rowwise_cpu_dptr<T>() + output_offset;
const auto *actual_dx = dx.rowwise_cpu_dptr<T>() + output_offset;
const double tolerance =
dtype == DType::kFloat32 ? 2e-5 : (dtype == DType::kFloat16 ? .003 : .02);
for (size_t row = 0; row < rows; ++row) {
for (size_t col = 0; col < hidden; ++col) {
double expected_y = 0, expected_gate = 0, expected_up = 0;
if (row < active) {
const double gate = static_cast<float>(x[row * 2 * hidden + col]);
const double up = static_cast<float>(x[row * 2 * hidden + hidden + col]);
const double upstream = static_cast<float>(dy[row * hidden + col]);
const double sigmoid = 1 / (1 + std::exp(-gate));
const double tg = std::tanh(gate / caps.first), tu = std::tanh(up / caps.second);
expected_y = caps.first * tg * sigmoid * caps.second * tu;
expected_gate = upstream *
((1 - tg * tg) * sigmoid + caps.first * tg * sigmoid * (1 - sigmoid)) *
caps.second * tu;
expected_up = upstream * caps.first * tg * sigmoid * (1 - tu * tu);
}
EXPECT_NEAR(static_cast<float>(y[row * hidden + col]), expected_y,
tolerance * (1 + std::abs(expected_y)));
EXPECT_NEAR(static_cast<float>(actual_dx[row * 2 * hidden + col]), expected_gate,
tolerance * (1 + std::abs(expected_gate)));
EXPECT_NEAR(static_cast<float>(actual_dx[row * 2 * hidden + hidden + col]), expected_up,
tolerance * (1 + std::abs(expected_up)));
}
}
for (test::Tensor *tensor : {&output, &dx}) {
const auto *data = tensor->rowwise_cpu_dptr<T>();
if (output_offset) EXPECT_TRUE(std::isnan(static_cast<float>(data[0])));
EXPECT_TRUE(std::isnan(static_cast<float>(data[test::product(tensor->shape()) - 1])));
}
}

class GroupedSiTUGLUTest : public ::testing::TestWithParam<
std::tuple<DType, size_t, int, int, std::pair<float, float>>> {};

TEST_P(GroupedSiTUGLUTest, AlignmentAndInactiveRows) {
const auto [dtype, hidden, alignment_mode, count_mode, caps] = GetParam();
switch (dtype) {
case DType::kFloat32:
test_grouped_situglu<float>(hidden, alignment_mode, count_mode, caps);
break;
case DType::kFloat16:
test_grouped_situglu<test::fp16>(hidden, alignment_mode, count_mode, caps);
break;
case DType::kBFloat16:
test_grouped_situglu<test::bf16>(hidden, alignment_mode, count_mode, caps);
break;
default:
FAIL() << "Unexpected test dtype";
}
}

INSTANTIATE_TEST_SUITE_P(Operator, GroupedSiTUGLUTest,
::testing::Combine(::testing::Values(DType::kFloat32, DType::kFloat16,
DType::kBFloat16),
::testing::Values(size_t{3}, size_t{16}, size_t{32}),
::testing::Values(0, 1, 2), ::testing::Values(0, 1, 2),
::testing::Values(std::pair<float, float>{4, 25},
std::pair<float, float>{2, 8})));

} // namespace
130 changes: 128 additions & 2 deletions tests/jax/moe/test_moe_resource_api.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
#
# See LICENSE for license information.

"""MoE resource resolution and MeshResource-only API without EP kernels."""
"""MoE resource and activation API contracts without EP kernels."""

import importlib
import inspect
Expand All @@ -13,15 +13,68 @@
import pytest
from jax.sharding import Mesh

from transformer_engine.jax import grouped_situglu
from transformer_engine.jax.flax import _MoEBlock
from transformer_engine.jax.moe import (
RouterComputationInfo,
SiTUActivation,
_apply_ffn_activation,
_moe_mesh_axes,
_resolve_moe_mesh_resource,
)
from transformer_engine.jax.sharding import MeshResource, global_mesh_resource, global_shard_guard


@pytest.mark.parametrize("betas", [(4.0, 25.0), (2.0, 7.0)])
def test_situ_value_and_both_branch_derivatives(betas):
gate_beta, up_beta = betas
gate = jnp.asarray([-10000.0, -12.0, -0.5, 0.0, 0.5, 12.0, 10000.0])
up = jnp.asarray([10000.0, 40.0, -3.0, 2.0, 3.0, -40.0, -10000.0])
activation = SiTUActivation(gate_beta, up_beta)
counts = jnp.asarray([1], jnp.int32)

def activate(g, u):
pair = jnp.concatenate([g, u])[None, :]
return grouped_situglu(pair, counts, gate_beta, up_beta)[0]

output, pullback = jax.vjp(activate, gate, up)
dg, du = pullback(jnp.ones_like(output))
np.testing.assert_array_equal(
output, _apply_ffn_activation(activation, gate[None, :], up[None, :], counts)[0]
)

g, u = np.asarray(gate, np.float64), np.asarray(up, np.float64)
sigmoid = 0.5 + 0.5 * np.tanh(g / 2)
tg, tu = np.tanh(g / gate_beta), np.tanh(u / up_beta)
capped_gate, capped_up = gate_beta * tg * sigmoid, up_beta * tu
derivative_gate = (1 - tg**2) * sigmoid + gate_beta * tg * sigmoid * (1 - sigmoid)
for actual, expected in (
(output, capped_gate * capped_up),
(dg, derivative_gate * capped_up),
(du, capped_gate * (1 - tu**2)),
):
np.testing.assert_allclose(actual, expected, atol=2e-5, rtol=2e-5)


def test_situ_bf16_uses_fp32_before_final_cast():
gate = jnp.asarray([0.25, 1.5, -3.0, 12.0, 64.0], dtype=jnp.bfloat16)
up = jnp.asarray([0.75, -2.0, 32.0, 64.0, -128.0], dtype=jnp.bfloat16)
actual = _apply_ffn_activation(
SiTUActivation(), gate[None, :], up[None, :], jnp.asarray([1], jnp.int32)
)[0]
g, u = np.asarray(gate, np.float64), np.asarray(up, np.float64)
expected = jnp.asarray(100 * np.tanh(g / 4) / (1 + np.exp(-g)) * np.tanh(u / 25), jnp.bfloat16)
assert actual.dtype == jnp.bfloat16
np.testing.assert_array_equal(actual, expected)


@pytest.mark.parametrize("value", [0, -1, float("inf"), float("nan")])
@pytest.mark.parametrize("field", ["gate_beta", "up_beta"])
def test_situ_rejects_invalid_soft_caps(field, value):
with pytest.raises(ValueError, match=field):
SiTUActivation(**{field: value})


@pytest.fixture(autouse=True)
def no_global_resource():
with global_shard_guard(None):
Expand All @@ -36,6 +89,73 @@ def test_resource_required_for_function_and_block():
_MoEBlock().init(jax.random.PRNGKey(0), jnp.ones((1, 1, 4)))


@pytest.mark.parametrize("shape", [(1, 4), (2, 1, 8), (1, 2, 8)])
def test_router_input_shape_is_validated(shape):
module = importlib.import_module("transformer_engine.jax.moe")
router = RouterComputationInfo(
gate_kernel=jnp.ones((8, 2)), num_experts_per_tok=1, router_input=jnp.ones(shape)
)
with pytest.raises(ValueError, match="router_input must have shape"):
module.moe(
jnp.ones((1, 1, 4)),
router,
None,
None,
num_experts=2,
mesh_resource=MeshResource(ep_resource="ep"),
)


def test_gate_kernel_uses_router_width():
module = importlib.import_module("transformer_engine.jax.moe")
router = RouterComputationInfo(
gate_kernel=jnp.ones((4, 2)), num_experts_per_tok=1, router_input=jnp.ones((1, 1, 8))
)
with pytest.raises(ValueError, match="gate_kernel must have shape"):
module.moe(
jnp.ones((1, 1, 4)),
router,
None,
None,
num_experts=2,
mesh_resource=MeshResource(ep_resource="ep"),
)


@pytest.mark.parametrize(
"overrides,match",
[
({"score_function": "softmax"}, "requires sigmoid"),
({"num_groups": 2, "group_topk": 1}, "ungrouped"),
({"aux_loss_coeff": 0.1}, "aux_loss_coeff"),
({"aux_loss_coeff": -0.1}, "aux_loss_coeff"),
({"use_pre_softmax": True}, "use_pre_softmax"),
({"qb_num_bins": 0}, "num_bins"),
({"expert_bias": None}, "expert_bias"),
({"qb_histogram_mode": "invalid"}, "histogram_mode"),
],
)
def test_qb_configuration_rejected_before_ep(overrides, match):
module = importlib.import_module("transformer_engine.jax.moe")
kwargs = dict(
gate_kernel=jnp.ones((4, 2)),
num_experts_per_tok=1,
score_function="sigmoid",
expert_bias=jnp.zeros(2, jnp.float32),
qb_bin_bounds=jnp.asarray([-1.0, 1.0], jnp.float32),
)
kwargs.update(overrides)
with pytest.raises(ValueError, match=match):
module.moe(
jnp.ones((1, 1, 4)),
RouterComputationInfo(**kwargs),
None,
None,
num_experts=2,
mesh_resource=MeshResource(ep_resource="ep"),
)


def test_explicit_resource_overrides_global_and_is_snapshotted():
explicit = MeshResource(dp_resource="dp", fsdp_resource="fsdp", ep_resource="ep")
enclosing = MeshResource(ep_resource="other")
Expand Down Expand Up @@ -131,7 +251,13 @@ def test_public_api_uses_selected_resource(monkeypatch, quantize):
def fake_vjp(*args):
captured.update(signature.bind(*args).arguments)
assert global_mesh_resource().ep_resource == "ep"
return args[0], None, jnp.zeros((1,), jnp.int32), jnp.zeros((2,), jnp.int32)
return (
args[0],
None,
jnp.zeros((1,), jnp.int32),
jnp.zeros((2,), jnp.int32),
jnp.zeros((0, 0, 0), jnp.int32),
)

monkeypatch.setattr(module, "_moe", fake_vjp)
mesh = Mesh(np.asarray(jax.devices()[:1]).reshape(1, 1, 1), ("dp", "fsdp", "ep"))
Expand Down
Loading