Skip to content

[JAX] Add JAX SiTU fallback to MoEBlock - #74

Draft
pggPL wants to merge 3 commits into
jax_moe_fused_situ_upstreamfrom
jax_moe_situ
Draft

pggPL wants to merge 3 commits into
jax_moe_fused_situ_upstreamfrom
jax_moe_situ

Conversation

@pggPL

@pggPL pggPL commented Nov 5, 2026 •

Copy link
Copy Markdown
Owner

Description

Stacked directly on NVIDIA/TransformerEngine#3873. Add a JAX/VJP fallback to its existing SiTU MoE support. The public API and shared parameter container come from that PR: activation_type="situglu", optional activation_params={"gate_beta": 4.0, "up_beta": 25.0}, and ActivationParams.

Eligible MXFP8 configurations continue using cuDNN SiTU-GLU/dGLU fusion. With use_cudnn_fusion=False, or when fusion is unavailable, evaluate both soft caps in FP32 using JAX between the existing grouped GEMMs and compute both branch gradients with jax.vjp. Device expert counts mask inactive values and gradients, including NaN-filled receive-buffer tails. Activation output keeps the projection dtype, and fallback MXFP8 quantization remains separate. Ineligible fusion requests warn and fall back using the existing MoE policy.

Validated 59 focused cases on RTX 5880 Ada, using the actual shared parameter and tensor classes, existing activation/VJP routines and complete per-shard forward/backward source. This includes existing default/custom parameter-forwarding tests with mocked cuDNN, fusion selection for SM89/100/103/107, FP32/FP16/BF16 activation and gradient checks, inactive/empty capacity, dynamic counts, vmap/rematerialization, and forward/backward composition with early/late routing weights, biases and SwiGLU controls. Native grouped GEMM/quantization are JAX test doubles in the backward harness; this is not a full TE/EP or fused-cuDNN GPU integration result. Also passed 48 existing adapter compatibility cases on CPU, full JAX/common Python lint (10.00/10), pre-commit, source license and diff checks. Distributed SiTU cases are prepared in the existing MoE suite; full Blackwell/NCCL/EP and Kimi-K3 training were not rerun.

Type of change

  • Documentation change (change only to the documentation, either a fix or a new content)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Infra/Build change
  • Code refactoring

Changes

  • Add SiTU gate/up operations to the shared activation converter and reuse _jax_act_lu / _jax_quantize_dact_dbias in MoE, with capacity masking at the call sites.
  • Reuse the activation string and shared ActivationParams plumbing from upstream [JAX] Enable cuDNN SiTU-GLU fusion in MoEBlock NVIDIA/TransformerEngine#3873.
  • Allow SiTU with disabled or unsupported cuDNN fusion, retaining existing eligible fusion selection.
  • Preserve projection dtypes, early routing-weight gradients, grouped GEMMs and grouped quantization.
  • Cover fallback behavior and update the existing API, distributed numerical tests and documentation.

Checklist:

  • I have read and followed the contributing guidelines
  • The functionality is complete
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

@pggPL
pggPL changed the base branch from kimi-k3-support-base-20261105 to jax_moe_router_input November 6, 2026 11:59
@pggPL pggPL changed the title [JAX] Add two-branch SiTU activation to MoEBlock [JAX] Add native grouped SiTU-GLU to MoEBlock Nov 12, 2026
@pggPL pggPL changed the title [JAX] Add native grouped SiTU-GLU to MoEBlock [JAX] Add SiTU activation fallback to MoEBlock Nov 18, 2026
Reuse the upstream SiTU activation string and ActivationParams container. Keep eligible cuDNN fusion and use masked JAX activation operations with a VJP for unfused execution.

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
@pggPL pggPL changed the title [JAX] Add SiTU activation fallback to MoEBlock [JAX] Add JAX SiTU fallback to MoEBlock Nov 18, 2026
@pggPL
pggPL changed the base branch from jax_moe_router_input to jax_moe_fused_situ_upstream November 18, 2026 13:50
Define SiTU gate and up operations in the existing parameter-aware activation converter. Use one masked gated-activation helper and VJP for SiTU and SwiGLU, and move fallback documentation from the API index to the moe parameters.

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Remove _apply_ffn_activation and call the existing JAX activation and VJP routines. Keep receive-capacity masking at the MoE call sites and exercise the actual forward/backward composition.

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant