Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
151f2b2
[PyTorch] Add torch.compile custom ops for GroupedLinear
pggPL Sep 10, 2026
5dbaa37
Merge grouped_linear_refactor into grouped_linear_compile
pggPL Sep 18, 2026
30548a2
Merge refreshed GroupedLinear refactor into compile support
pggPL Oct 1, 2026
e552947
[PyTorch] Fix GroupedLinear compile metadata and extend fused input s…
pggPL Oct 1, 2026
a6d62a5
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 1, 2026
7a85879
[PyTorch] Compile fused GroupedLinear with MXFP8 and NVFP4
pggPL Oct 2, 2026
524ea34
[PyTorch] Compile single grouped weights and biases
pggPL Oct 2, 2026
d4b9c2d
[PyTorch] Simplify GroupedLinear saved tensor state
pggPL Oct 2, 2026
3bfcf5c
[PyTorch] Preserve gradient structure in custom-op returns
pggPL Oct 2, 2026
5e99464
Preserve tiny sigmoid scores in biased fused routing (#3637)
philipcmonk Oct 7, 2026
14be43f
Build FP8 recipe quantizer params per instance (#3605)
Varun-51 Oct 8, 2026
7f0c9d3
Merge grouped_linear_refactor into grouped_linear_compile
pggPL Oct 8, 2026
6f3ba5f
Merge upstream main into grouped_linear_compile
pggPL Oct 8, 2026
c25fa27
[PyTorch] Allow deterministic dprob for the GLU fused grouped MLP (#3…
ZhiyuLi-Nvidia Oct 8, 2026
62b1389
[Common] Fall back from gated TMA kernels when shared memory does not…
ravimajeti Oct 8, 2026
45223d7
[PyTorch] Compile Linear and Bias operations with forward fusion (#3547)
pggPL Oct 8, 2026
7e7898f
[PyTorch] torch.compile support for FusedAttention (#3472)
pggPL Oct 8, 2026
39c30c5
[Common] Fix TMA synchronization in quantization kernels (#3417)
Oleg-Goncharov Oct 9, 2026
d9beab1
[Common] Fuse RMSNorm forward with MXFP8 quantization (#3657)
sraman-rgb Oct 9, 2026
074eb5d
[Common] Link python to C++ dist tests with CuTeDSL enabled (#3648)
kainzhong Oct 9, 2026
f57b8b8
[Common] Enable all activations for MXFP8 C++ Unittest (#3616)
kainzhong Oct 9, 2026
16580f9
[PyTorch][CP][CI] Reset all CP process pools before retrying a failed…
sudhakarsingh27 Oct 9, 2026
15dcc84
[Common] Requantize grouped MXFP8 tensors with 128x128 tiles (#3656)
sraman-rgb Oct 9, 2026
76ae4b0
[Common] Fix grouped MXFP8 quantization with varying last dim & swizz…
kainzhong Oct 9, 2026
33e7ff3
[Common] Swizzle MXFP8 scaling factors with one CTA per scale tile (#…
sraman-rgb Oct 11, 2026
2e959f3
[PyTorch] Fix NVFP4 weight views of the single-group fused grouped ML…
sraman-rgb Oct 12, 2026
81775f2
[PyTorch] Interpolate the class name in six op error messages (#3667)
v0ropaev Oct 12, 2026
1c9f6f8
[PyTorch][CP] Support D=256 THD metadata with FA4 b33 (#3540)
sudhakarsingh27 Oct 12, 2026
459dedc
[Common][PyTorch] Fuse the MLA key/value RoPE with MXFP8 quantization…
sraman-rgb Oct 12, 2026
20e97da
Bump nccl ep (#3677)
phu0ngng Oct 13, 2026
e394b12
[PyTorch] Distributed tests set device in NCCL init and manually shut…
timmoon10 Oct 13, 2026
fc48751
[PyTorch] Allow ZeroTensor dummy weight gradients (#3670)
sraman-rgb Oct 13, 2026
39a8b87
[PyTorch] Return the argmax from parallel_cross_entropy (#3671)
sraman-rgb Oct 13, 2026
0c4d961
Change to enable CuteDSL DenseGEMM for eval (#3666)
vasunvidia Oct 13, 2026
e6afa9d
[PyTorch] Refactor GroupedLinear forward and backward (#3606)
pggPL Oct 13, 2026
18b63ad
Merge upstream main into grouped_linear_compile
pggPL Oct 13, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion 3rdparty/nccl-extensions
Submodule nccl-extensions updated 57 files
+5 −1 nccl_ep/CMakeLists.txt
+6 −3 nccl_ep/Makefile
+18 −2 nccl_ep/README.md
+10 −0 nccl_ep/ci/run_nccl_ep_ci_tests.sh
+432 −0 nccl_ep/device/cft.cuh
+3 −3 nccl_ep/device/count_kernel.cuh
+17 −0 nccl_ep/device/device_primitives.cuh
+653 −269 nccl_ep/device/ht_ep.cuh
+425 −24 nccl_ep/device/ht_ep_adapter.cu
+80 −10 nccl_ep/device/ht_ep_adapter.cuh
+2 −0 nccl_ep/device/ht_ep_configs.cuh
+63 −9 nccl_ep/device/jit/ht_combine_jit.cuh
+3 −1 nccl_ep/device/jit/ht_dispatch_jit.cuh
+41 −19 nccl_ep/device/jit/jit_runtime.cc
+4 −0 nccl_ep/device/jit/jit_runtime.hpp
+3 −1 nccl_ep/device/jit/ll_clean_jit.cuh
+17 −5 nccl_ep/device/jit/ll_combine_jit.cuh
+145 −30 nccl_ep/device/jit/ll_dispatch_jit.cuh
+20 −25 nccl_ep/device/ll/ll_combine_lsa.cuh
+562 −78 nccl_ep/device/ll/ll_dispatch_lsa.cuh
+131 −0 nccl_ep/device/ll/ll_dispatch_recipe.cuh
+216 −0 nccl_ep/device/ll/ll_ep_smem.cuh
+6 −2 nccl_ep/device/ll/ll_logfmt.cuh
+21 −0 nccl_ep/device/ll/ll_lsa_primitives.cuh
+469 −283 nccl_ep/device/ll_ep.cuh
+69 −8 nccl_ep/device/ll_ep_adapter.cu
+84 −13 nccl_ep/device/ll_ep_adapter.cuh
+474 −0 nccl_ep/device/ll_ep_cft.cuh
+0 −114 nccl_ep/device/ll_ep_smem.cuh
+25 −5 nccl_ep/device/macros.cuh
+38 −1 nccl_ep/docs/documentation/api_reference.md
+6 −0 nccl_ep/docs/documentation/quantization.md
+12 −5 nccl_ep/docs/documentation/zero_copy.md
+230 −52 nccl_ep/ep_bench.cu
+56 −0 nccl_ep/ht_compact_prob.h
+2 −6 nccl_ep/include/common.hpp
+53 −0 nccl_ep/include/nccl_ep.h
+875 −294 nccl_ep/nccl_ep.cc
+687 −0 nccl_ep/nccl_ep_cft.cc
+165 −0 nccl_ep/nccl_ep_cft.h
+73 −0 nccl_ep/nccl_ep_env.cc
+26 −0 nccl_ep/nccl_ep_env.h
+6 −0 nccl_ep/nccl_ep_test_internal.h
+14 −1 nccl_ep/tests/Makefile
+24 −2 nccl_ep/tests/run_tests.sh
+169 −0 nccl_ep/tests/test_cft_driver_api.cu
+347 −6 nccl_ep/tests/test_ht_bwd.cu
+115 −0 nccl_ep/tests/test_ht_chunk_config.cu
+215 −0 nccl_ep/tests/test_ht_chunk_handle.cu
+550 −0 nccl_ep/tests/test_ht_compact_prob.cu
+680 −0 nccl_ep/tests/test_ht_dispatch_async.cu
+274 −0 nccl_ep/tests/test_lifecycle.cu
+71 −0 nccl_ep/tests/test_ll_kernel_selection.cu
+12 −0 nccl_ep/tests/test_public_struct_abi.cu
+234 −0 nccl_ep/tests/test_quantization_recipe.cu
+30 −0 python/nccl/m2n/README.md
+379 −0 python/nccl/m2n/examples/reshard_jax.py
6 changes: 6 additions & 0 deletions docs/envvars.rst
Original file line number Diff line number Diff line change
Expand Up @@ -342,6 +342,12 @@ Kernel Configuration
:Default: ``0``
:Description: Disable NVRTC (CUDA Runtime Compilation) support. When set to ``1``, runtime kernel compilation is disabled. Existing transpose operations select their static fallback automatically. NVFP4 4over6 quantization, fused softmax, and normalization paths require their corresponding ``NVTE_BUILD_LEGACY_STATIC_NVFP4``, ``NVTE_BUILD_LEGACY_STATIC_FUSED_SOFTMAX``, or ``NVTE_BUILD_LEGACY_STATIC_NORM`` CMake option to have been enabled when the library was built; otherwise no static fallback is available.

.. envvar:: NVTE_MXFP8_PER_TILE_SWIZZLE

:Type: ``int`` (0 or 1)
:Default: ``0``
:Description: Swizzle MXFP8 scaling factors for GEMM with kernels that give each 128-row tile of scaling factors (row-wise) or each 128-column by 32-row block (column-wise) its own thread block, instead of batching 32 tiles in one block. This lowers the latency of swizzling the scaling factors of tensors with few tiles, such as activations.

.. envvar:: NVTE_USE_CUTLASS_GROUPED_GEMM

:Type: ``int`` (0 or 1)
Expand Down
2 changes: 1 addition & 1 deletion qa/L0_pytorch_unittest/test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ NVTE_GDN2_TEST_REQUIRED=1 python3 -m pytest --tb=auto --junitxml=$XML_LOG_DIR/py
NVTE_GDP_TEST_REQUIRED=1 python3 -m pytest --tb=auto --junitxml=$XML_LOG_DIR/pytest_test_gdp_attention.xml $TE_PATH/tests/pytorch/attention/test_gdp_attention.py || test_fail "test_gdp_attention.py"
NVTE_ALLOW_UNSAFE_PICKLE_EXTRA_STATE=1 NVTE_ALLOW_NONDETERMINISTIC_ALGO=0 python3 -m pytest --tb=auto --junitxml=$XML_LOG_DIR/pytest_test_attention_deterministic.xml $TE_PATH/tests/pytorch/attention/test_attention.py || test_fail "NVTE_ALLOW_NONDETERMINISTIC_ALGO=0 test_attention.py"
python3 -m pytest --tb=auto --junitxml=$XML_LOG_DIR/pytest_test_linear_mxfp8_attention.xml $TE_PATH/tests/pytorch/attention/test_linear_mxfp8_attention.py || test_fail "test_linear_mxfp8_attention.py"
python3 -m pytest --tb=auto --junitxml=$XML_LOG_DIR/pytest_test_fused_mla_q_uproj.xml $TE_PATH/tests/pytorch/attention/test_fused_mla_q_uproj.py || test_fail "test_fused_mla_q_uproj.py"
python3 -m pytest --tb=auto --junitxml=$XML_LOG_DIR/pytest_test_fused_mla.xml $TE_PATH/tests/pytorch/attention/test_fused_mla.py || test_fail "test_fused_mla.py"
python3 -m pytest --tb=auto --junitxml=$XML_LOG_DIR/pytest_test_kv_cache.xml $TE_PATH/tests/pytorch/attention/test_kv_cache.py || test_fail "test_kv_cache.py"
python3 -m pytest --tb=auto --junitxml=$XML_LOG_DIR/pytest_test_cu_seqlens_cache.xml $TE_PATH/tests/pytorch/attention/test_cu_seqlens_cache.py || test_fail "test_cu_seqlens_cache.py"
python3 -m pytest --tb=auto --junitxml=$XML_LOG_DIR/pytest_test_mixed_thd_attention.xml $TE_PATH/tests/pytorch/attention/test_mixed_thd_attention.py || test_fail "test_mixed_thd_attention.py"
Expand Down
17 changes: 12 additions & 5 deletions qa/L3_pytorch_FA_versions_test/test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,8 @@ export FLASH_ATTN_CUDA_ARCHS=$sm_arch
CP_FA_VERSION=""
if [ $sm_arch -gt 90 ]
then
FA_versions=(4.0.0b11)
CP_FA_VERSION="4.0.0b11"
FA_versions=(4.0.0b33)
CP_FA_VERSION="4.0.0b33"
elif [ $sm_arch -eq 90 ]
then
FA_versions=(3.0.0b1)
Expand Down Expand Up @@ -66,13 +66,13 @@ do
elif [[ "${fa_version}" == 4.* ]]
then
export NVTE_FLASH_ATTN_V4=1
# FA4 is intentionally last in every version array. Its b11 test pin needs
# CUTLASS DSL 4.4.2, so replace the image-matched stack only for this final
# FA4 is intentionally last in every version array. CUTLASS DSL 4.8.0 supports
# SM107, so replace the image-matched stack only for this final
# iteration; later iterations would otherwise need that stack restored.
pip3 uninstall -y nvidia-cutlass-dsl nvidia-cutlass-dsl-libs-base \
nvidia-cutlass-dsl-libs-cu12 nvidia-cutlass-dsl-libs-cu13 \
|| error_exit "Failed to isolate CUTLASS DSL for Flash Attention $fa_version"
pip3 install flash-attn-4==${fa_version} nvidia-cutlass-dsl[cu13]==4.4.2 \
pip3 install flash-attn-4==${fa_version} nvidia-cutlass-dsl[cu13]==4.8.0 \
--no-build-isolation || error_exit "Failed to install Flash Attention $fa_version"
else
export NVTE_FLASH_ATTN_V3=1
Expand Down Expand Up @@ -135,6 +135,13 @@ do
fi
NVTE_TORCH_COMPILE=0 NVTE_ALLOW_UNSAFE_PICKLE_EXTRA_STATE=1 python3 -m pytest -v -s --junitxml=$XML_ATTN $TE_PATH/tests/pytorch/attention/test_attention.py || test_fail "test_attention.py (FA $fa_version)"
fi

if [[ "${fa_version}" == 4.* ]]; then
XML_BACKEND="$XML_LOG_DIR/pytest_test_attention_backend_selection_fa${fa_tag}.xml"
NVTE_TORCH_COMPILE=0 python3 -m pytest -v -s --junitxml="$XML_BACKEND" \
"$TE_PATH/tests/pytorch/attention/test_attention_backend_selection.py" \
|| test_fail "FA4 backend-selection tests (FA $fa_version)"
fi
done

if [ "$RET" -ne 0 ]; then
Expand Down
9 changes: 4 additions & 5 deletions tests/cpp/operator/test_cast_mxfp8.cu
Original file line number Diff line number Diff line change
Expand Up @@ -746,14 +746,13 @@ std::vector<MXFP82DScalingDirection> scaling_directions_2d_quantize = {
MXFP82DScalingDirection::Bidirectional,
};

// Only GeLU activation tests are supported
std::vector<ActivationType> Activation_types = {
ActivationType::Identity,
ActivationType::GeLU,
// ActivationType::SiLU,
// ActivationType::ReLU,
// ActivationType::QGeLU,
// ActivationType::SReLU,
ActivationType::SiLU,
ActivationType::ReLU,
ActivationType::QGeLU,
ActivationType::SReLU,
};

class FusedCastMXFP8TestSuite : public ::testing::TestWithParam
Expand Down
10 changes: 5 additions & 5 deletions tests/cpp/operator/test_cast_mxfp8_grouped.cu
Original file line number Diff line number Diff line change
Expand Up @@ -657,10 +657,10 @@ std::vector<ProcessingMethod> processing_methods = {
std::vector<ActivationKind> activation_kinds = {
ActivationKind::Identity,
ActivationKind::GeLU,
// ActivationKind::SiLU,
// ActivationKind::ReLU,
// ActivationKind::QGeLU,
// ActivationKind::SReLU,
ActivationKind::SiLU,
ActivationKind::ReLU,
ActivationKind::QGeLU,
ActivationKind::SReLU,
};

enum ScalingDirection {
Expand Down Expand Up @@ -1012,7 +1012,7 @@ INSTANTIATE_TEST_SUITE_P(
GroupedFusedCastMXFP8TestSuite,
::testing::Combine(
::testing::Values(ProcessingMethod::CAST_DBIAS_DACT),
::testing::Values(ActivationKind::GeLU),
::testing::ValuesIn(activation_kinds),
::testing::ValuesIn(scaling_directions),
::testing::ValuesIn(input_config_same_both_multichunk),
::testing::Values(DType::kBFloat16),
Expand Down
38 changes: 28 additions & 10 deletions tests/cpp/operator/test_multi_swizzle.cu
Original file line number Diff line number Diff line change
Expand Up @@ -300,14 +300,22 @@ void performTestMultiTensorRoundtrip(const int num_tensors, const size_t M, cons
// Test suites
// ===================================================================

// Sets NVTE_MXFP8_PER_TILE_SWIZZLE while it lives, so that MXFP8 swizzles use the per-tile
// kernels (enable) or the default ones.
class PerTileSwizzle {
public:
explicit PerTileSwizzle(bool enable) {
setenv("NVTE_MXFP8_PER_TILE_SWIZZLE", enable ? "1" : "0", 1);
}
~PerTileSwizzle() { unsetenv("NVTE_MXFP8_PER_TILE_SWIZZLE"); }
};

class MultiTensorSwizzleTestSuite
: public ::testing::TestWithParam<std::tuple<int, size_t, size_t, bool>> {};
: public ::testing::TestWithParam<std::tuple<std::tuple<int, size_t, size_t, bool>, bool>> {};

TEST_P(MultiTensorSwizzleTestSuite, TestMultiTensorSwizzle) {
const auto num_tensors = std::get<0>(GetParam());
const auto M = std::get<1>(GetParam());
const auto K = std::get<2>(GetParam());
const auto rowwise = std::get<3>(GetParam());
const auto& [num_tensors, M, K, rowwise] = std::get<0>(GetParam());
const PerTileSwizzle per_tile_swizzle(std::get<1>(GetParam()));
performTestMultiTensorSwizzle(num_tensors, M, K, rowwise);
}

Expand Down Expand Up @@ -381,19 +389,29 @@ std::vector<std::tuple<int, size_t, size_t, bool>> multi_tensor_test_cases = {
{2, 128, 4352, true},
// Colwise num_tiles_k = 512 / 32 / 4 = 4 selects vec_load_size = 4 (int4 kernel)
{2, 512, 4096, false},
// MXFP8 swizzle: more tensors than one launch takes, several row-wise tiles per CTA,
// and 128-row column-wise chunks
{80, 128, 128, true},
{80, 128, 128, false},
{20, 16384, 128, true},
{20, 128, 8192, false},
// A full 128-row column-wise chunk and a clipped last one (132 scale rows)
{20, 4224, 2048, false},
};

} // namespace

INSTANTIATE_TEST_SUITE_P(
OperatorTest,
MultiTensorSwizzleTestSuite,
::testing::ValuesIn(multi_tensor_test_cases),
::testing::Combine(::testing::ValuesIn(multi_tensor_test_cases), ::testing::Values(false, true)),
[](const testing::TestParamInfo<MultiTensorSwizzleTestSuite::ParamType>& info) {
return "n" + std::to_string(std::get<0>(info.param)) +
"_M" + std::to_string(std::get<1>(info.param)) +
"_K" + std::to_string(std::get<2>(info.param)) +
(std::get<3>(info.param) ? "_row" : "_col");
const auto& test_case = std::get<0>(info.param);
return "n" + std::to_string(std::get<0>(test_case)) +
"_M" + std::to_string(std::get<1>(test_case)) +
"_K" + std::to_string(std::get<2>(test_case)) +
(std::get<3>(test_case) ? "_row" : "_col") +
(std::get<1>(info.param) ? "_per_tile" : "");
});

INSTANTIATE_TEST_SUITE_P(
Expand Down
150 changes: 144 additions & 6 deletions tests/cpp/operator/test_normalization_mxfp8.cu
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,8 @@ void dequantize_2x(Tensor& input, Tensor& output, bool is_training)
}

template <typename InputType, typename OutputType>
void performTest(const size_t N, const size_t H, const bool zero_centered_gamma, NormType norm_type, bool is_training, const bool zero_centered_gamma_in_weight_dtype) {
void performTest(const size_t N, const size_t H, const bool zero_centered_gamma, NormType norm_type, bool is_training, const bool zero_centered_gamma_in_weight_dtype,
const bool use_cudnn) {

cudaDeviceProp prop;
cudaGetDeviceProperties(&prop, 0);
Expand Down Expand Up @@ -134,6 +135,9 @@ void performTest(const size_t N, const size_t H, const bool zero_centered_gamma,
} else {
nvte_enable_zero_centered_gamma_in_weight_dtype(false);
}
// RMSNorm shapes that are multiples of 128 use Transformer Engine's fused MXFP8 kernel
// unless cuDNN is requested.
nvte_enable_cudnn_norm_fwd(use_cudnn);

// Forward kernel
float epsilon = 1e-5;
Expand Down Expand Up @@ -163,6 +167,7 @@ void performTest(const size_t N, const size_t H, const bool zero_centered_gamma,
if (zero_centered_gamma_in_weight_dtype) {
nvte_enable_zero_centered_gamma_in_weight_dtype(false);
}
nvte_enable_cudnn_norm_fwd(false);

Tensor dequantized_output("dequantized_output", std::vector<size_t>{ N, H }, DType::kFloat32, true, true);

Expand Down Expand Up @@ -197,7 +202,7 @@ void performTest(const size_t N, const size_t H, const bool zero_centered_gamma,
nullptr, // amax
1.f, // scale
zero_centered_gamma,
true, // CuDNN is the only MXFP8 backend currently
true, // The MXFP8 backends add 1 to a zero-centered gamma as cuDNN does
zero_centered_gamma_in_weight_dtype);

cudaDeviceSynchronize();
Expand Down Expand Up @@ -235,6 +240,95 @@ std::vector<std::pair<size_t, size_t>> test_cases = {
{2048, 12288},
};

// Index of compact scaling factor (i, j) in the GEMM-swizzled layout: 128x4 tiles of i x j,
// num_tiles_j tiles along j.
size_t swizzled_scale_idx(size_t i, size_t j, size_t num_tiles_j) {
return ((i / 128) * num_tiles_j + j / 4) * 512 + (i % 32) * 16 + ((i % 128) / 32) * 4 + j % 4;
}

// RMSNorm with MXFP8 output written with GEMM-swizzled scaling factors must give the same
// data as with compact scaling factors, and the compact scaling factors in swizzled order.
// The swizzled output is computed on a single SM, as with an SM margin, which launches the
// kernels over chunks of rows.
template <typename InputType, typename OutputType>
void performSwizzledScalesTest(const size_t N, const size_t H, const bool is_training) {
if (getDeviceComputeCapability() < blackwellComputeCapability) {
GTEST_SKIP();
}
cudaDeviceProp prop;
cudaGetDeviceProperties(&prop, 0);

const DType itype = TypeInfo<InputType>::dtype;
const DType otype = TypeInfo<OutputType>::dtype;
Tensor input("input", std::vector<size_t>{ N, H }, itype);
Tensor gamma("gamma", std::vector<size_t>{ H }, itype);
Tensor rsigma("rsigma", std::vector<size_t>{ N }, DType::kFloat32);
Tensor rsigma_swizzled("rsigma_swizzled", std::vector<size_t>{ N }, DType::kFloat32);
Tensor z("z", std::vector<size_t>{ N, H }, otype, true, is_training, NVTE_MXFP8_1D_SCALING);
Tensor z_swizzled("z_swizzled", std::vector<size_t>{ N, H }, otype, true, is_training,
NVTE_MXFP8_1D_SCALING);
z_swizzled.set_with_gemm_swizzled_scales(true);
fillUniform(&input);
fillUniform(&gamma);

const auto run = [&](Tensor &out, Tensor &rs, const int sm_count) {
Tensor workspace;
nvte_rmsnorm_fwd(input.data(), gamma.data(), 1e-5f, out.data(), rs.data(),
workspace.data(), sm_count, false, 0);
workspace = Tensor("workspace", workspace.rowwise_shape(), workspace.dtype());
nvte_rmsnorm_fwd(input.data(), gamma.data(), 1e-5f, out.data(), rs.data(),
workspace.data(), sm_count, false, 0);
};
run(z, rsigma, prop.multiProcessorCount);
run(z_swizzled, rsigma_swizzled, 1);
cudaDeviceSynchronize();
auto err = cudaGetLastError();
ASSERT_EQ(err, cudaSuccess) << cudaGetErrorString(err);
z.to_cpu();
z_swizzled.to_cpu();
rsigma.to_cpu();
compareResults("rsigma", rsigma_swizzled, rsigma.rowwise_cpu_dptr<float>(), true, 0, 0);

const auto bytes = [](const OutputType *p) { return reinterpret_cast<const uint8_t *>(p); };
compareResults("rowwise_data", bytes(z_swizzled.rowwise_cpu_dptr<OutputType>()),
bytes(z.rowwise_cpu_dptr<OutputType>()), N * H);
const NVTEShape row_shape = z.rowwise_scale_inv_shape();
const size_t row_dim0 = row_shape.data[0], row_dim1 = row_shape.data[1];
std::vector<uint8_t> ref(row_dim0 * row_dim1);
const uint8_t *compact = z.rowwise_cpu_scale_inv_ptr<uint8_t>();
for (size_t i = 0; i < row_dim0; ++i) {
for (size_t j = 0; j < row_dim1; ++j) {
ref[swizzled_scale_idx(i, j, row_dim1 / 4)] = compact[i * row_dim1 + j];
}
}
compareResults("rowwise_scales", z_swizzled.rowwise_cpu_scale_inv_ptr<uint8_t>(), ref.data(),
ref.size());

if (is_training) {
compareResults("colwise_data", bytes(z_swizzled.columnwise_cpu_dptr<OutputType>()),
bytes(z.columnwise_cpu_dptr<OutputType>()), N * H);
// Compact column-wise scaling factors are [rows / 32, cols]; the swizzled layout tiles
// columns by 128 and scale rows by 4.
const NVTEShape col_shape = z.columnwise_scale_inv_shape();
const size_t col_dim0 = col_shape.data[0], col_dim1 = col_shape.data[1];
ref.assign(col_dim0 * col_dim1, 0);
compact = z.columnwise_cpu_scale_inv_ptr<uint8_t>();
for (size_t i = 0; i < col_dim0; ++i) {
for (size_t j = 0; j < col_dim1; ++j) {
ref[swizzled_scale_idx(j, i, col_dim0 / 4)] = compact[i * col_dim1 + j];
}
}
compareResults("colwise_scales", z_swizzled.columnwise_cpu_scale_inv_ptr<uint8_t>(),
ref.data(), ref.size());
}
}

std::vector<std::pair<size_t, size_t>> swizzled_scales_test_cases = {
{128, 128},
{768, 2304},
{256, 7168},
};

std::vector<NormType> norms = {
NormType::LayerNorm,
NormType::RMSNorm
Expand All @@ -246,7 +340,7 @@ class MxNormTestSuite : public ::testing::TestWithParam< std::tuple<NormType,
transformer_engine::DType,
transformer_engine::DType,
std::pair<size_t, size_t>,
bool, bool, bool>> {};
bool, bool, bool, bool>> {};

TEST_P(MxNormTestSuite, TestMxNorm) {
using namespace transformer_engine;
Expand All @@ -259,10 +353,15 @@ TEST_P(MxNormTestSuite, TestMxNorm) {
const bool zero_centered_gamma = std::get<4>(GetParam());
const bool is_training = std::get<5>(GetParam());
const bool zero_centered_gamma_in_weight_dtype = std::get<6>(GetParam());
const bool use_cudnn = std::get<7>(GetParam());
if (norm_type == NormType::LayerNorm && use_cudnn) {
GTEST_SKIP() << "LayerNorm with MXFP8 output always uses cuDNN";
}

TRANSFORMER_ENGINE_TYPE_SWITCH_FP16_FP32_ONLY(input_type, InputType,
TRANSFORMER_ENGINE_TYPE_SWITCH_FP8_ONLY(output_type, OutputType,
performTest<InputType, OutputType>(size.first, size.second, zero_centered_gamma, norm_type, is_training, zero_centered_gamma_in_weight_dtype);
performTest<InputType, OutputType>(size.first, size.second, zero_centered_gamma, norm_type, is_training, zero_centered_gamma_in_weight_dtype,
use_cudnn);
);
);
}
Expand All @@ -277,7 +376,8 @@ INSTANTIATE_TEST_SUITE_P(
::testing::ValuesIn(test_cases),
::testing::Values(true, false),
::testing::Values(true, false),
::testing::Values(true, false)),
::testing::Values(true, false),
::testing::Values(false, true)),
[](const testing::TestParamInfo<MxNormTestSuite::ParamType>& info) {
std::string name = normToString.at(std::get<0>(info.param)) + "_" +
test::typeName(std::get<1>(info.param)) + "X" +
Expand All @@ -286,6 +386,44 @@ INSTANTIATE_TEST_SUITE_P(
std::to_string(std::get<3>(info.param).second) + "X" +
std::to_string(std::get<4>(info.param)) + "out" +
std::to_string(int(std::get<5>(info.param)) + 1) + "x" +
std::to_string(std::get<6>(info.param));
std::to_string(std::get<6>(info.param)) +
(std::get<7>(info.param) ? "Xcudnn" : "");
return name;
});

class MxNormSwizzledScalesTestSuite
: public ::testing::TestWithParam<std::tuple<transformer_engine::DType,
transformer_engine::DType,
std::pair<size_t, size_t>, bool>> {};

TEST_P(MxNormSwizzledScalesTestSuite, TestMxRMSNormSwizzledScales) {
using namespace transformer_engine;
using namespace test;

const DType input_type = std::get<0>(GetParam());
const DType output_type = std::get<1>(GetParam());
const auto size = std::get<2>(GetParam());
const bool is_training = std::get<3>(GetParam());

TRANSFORMER_ENGINE_TYPE_SWITCH_FP16_FP32_ONLY(input_type, InputType,
TRANSFORMER_ENGINE_TYPE_SWITCH_FP8_ONLY(output_type, OutputType,
performSwizzledScalesTest<InputType, OutputType>(size.first, size.second, is_training);
);
);
}

INSTANTIATE_TEST_SUITE_P(
OperatorTest,
MxNormSwizzledScalesTestSuite,
::testing::Combine(
::testing::Values(DType::kFloat32, DType::kBFloat16, DType::kFloat16),
::testing::Values(DType::kFloat8E5M2, DType::kFloat8E4M3),
::testing::ValuesIn(swizzled_scales_test_cases),
::testing::Values(true, false)),
[](const testing::TestParamInfo<MxNormSwizzledScalesTestSuite::ParamType>& info) {
return test::typeName(std::get<0>(info.param)) + "X" +
test::typeName(std::get<1>(info.param)) + "X" +
std::to_string(std::get<2>(info.param).first) + "X" +
std::to_string(std::get<2>(info.param).second) + "X" +
std::to_string(int(std::get<3>(info.param)) + 1) + "x";
});
Loading