Skip to content

[JAX] Enable fused SiTU-GLU in MoEBlock - #76

Open
pggPL wants to merge 1 commit into
jax_moe_qbfrom
jax_moe_fused_situ
Open

pggPL wants to merge 1 commit into
jax_moe_qbfrom
jax_moe_fused_situ

Conversation

@pggPL

@pggPL pggPL commented Nov 10, 2026 •

Copy link
Copy Markdown
Owner

Description

Build on #75 (stack: #73 → #74 → #75, with shared vectorized GLU templates in #74) to run SiTU-GLU through cuDNN's existing fused grouped GEMM/MXFP8 forward and backward kernels. Device expert offsets bound active receive rows, including overallocated buffers. Requires the cuDNN Frontend JAX bridge change merged into develop on 2026-11-14: NVIDIA/cudnn-frontend#2540 SiTU fusion needs cuDNN Frontend 1.33.0 or a develop build at or after 30201756 (first nightly: 1.33.0.dev82169718); the general fusion floor stays 1.32.0. The eligibility check inspects the installed JAX entry signatures for situ_beta1/situ_beta2, so older 1.33.0 dev builds fall back to the unfused path instead of failing.

Validated the rebuilt stack on four GB200 GPUs: 54 standalone grouped SiTU tests, 36 resource/API tests, 40 FSDP/layout cases, and 26 QB router tests. The four-rank MoE run passed 38 cases per rank, including 12 complete fused forward/backward configurations covering SwiGLU controls, SiTU 4/25 and 2/8, standard/native weight layouts, and early/late routing weights. Twelve cases were skipped because the unfused fixture intentionally disables fusion. The native grouped fallback now reuses #74's shared GLU templates; its common regression sweep passed 212 C++ cases, plus 22 PyTorch and 22 existing JAX activation cases. Existing numerical tolerances are unchanged. Pre-commit, source license checks, C++ lint and the full JAX Python lint suite passed (10.00/10). No performance improvement is claimed; SM103 runtime and full Kimi-K3 training remain unverified.

Type of change

  • Documentation change (change only to the documentation, either a fix or a new content)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Infra/Build change
  • Code refactoring

Changes

  • Detect SiTU-capable shared cuDNN JAX APIs and select fusion on SM100/SM103; use [JAX] Add JAX SiTU fallback to MoEBlock #74's native grouped SiTU forward/backward with separate GEMMs for older frontends, BF16, or unsupported configurations.
  • Forward both soft-cap parameters through the existing grouped GLU/dGLU adapter and MoE custom VJP without changing Kimi's late routing-weight policy.
  • Extend capability, fallback, and independent-reference gradient tests; document the optional fused path.

Checklist:

  • I have read and followed the contributing guidelines
  • The functionality is complete
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

Select the shared cuDNN grouped GEMM SiTU forward/backward APIs when their JAX bridge exposes both soft-cap parameters. Preserve fallback for incompatible frontends and devices, and pass device expert offsets through the existing fused path.

[skip ci]

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
@pggPL
pggPL force-pushed the jax_moe_fused_situ branch from 4c7ac44 to ab2ae5e Compare November 12, 2026 13:49
@pggPL
pggPL marked this pull request as ready for review November 14, 2026 14:08

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant