Repository navigation
Conversation
pggPL
changed the base branch from
kimi-k3-support-base-20261105
to
jax_moe_router_input
November 6, 2026 11:59
This was referenced Nov 6, 2026
Reuse the upstream SiTU activation string and ActivationParams container. Keep eligible cuDNN fusion and use masked JAX activation operations with a VJP for unfused execution. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
pggPL
changed the base branch from
jax_moe_router_input
to
jax_moe_fused_situ_upstream
November 18, 2026 13:50
Define SiTU gate and up operations in the existing parameter-aware activation converter. Use one masked gated-activation helper and VJP for SiTU and SwiGLU, and move fallback documentation from the API index to the moe parameters. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Remove _apply_ffn_activation and call the existing JAX activation and VJP routines. Keep receive-capacity masking at the MoE call sites and exercise the actual forward/backward composition. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Stacked directly on NVIDIA/TransformerEngine#3873. Add a JAX/VJP fallback to its existing SiTU MoE support. The public API and shared parameter container come from that PR:
activation_type="situglu", optionalactivation_params={"gate_beta": 4.0, "up_beta": 25.0}, andActivationParams.Eligible MXFP8 configurations continue using cuDNN SiTU-GLU/dGLU fusion. With
use_cudnn_fusion=False, or when fusion is unavailable, evaluate both soft caps in FP32 using JAX between the existing grouped GEMMs and compute both branch gradients withjax.vjp. Device expert counts mask inactive values and gradients, including NaN-filled receive-buffer tails. Activation output keeps the projection dtype, and fallback MXFP8 quantization remains separate. Ineligible fusion requests warn and fall back using the existing MoE policy.Validated 59 focused cases on RTX 5880 Ada, using the actual shared parameter and tensor classes, existing activation/VJP routines and complete per-shard forward/backward source. This includes existing default/custom parameter-forwarding tests with mocked cuDNN, fusion selection for SM89/100/103/107, FP32/FP16/BF16 activation and gradient checks, inactive/empty capacity, dynamic counts, vmap/rematerialization, and forward/backward composition with early/late routing weights, biases and SwiGLU controls. Native grouped GEMM/quantization are JAX test doubles in the backward harness; this is not a full TE/EP or fused-cuDNN GPU integration result. Also passed 48 existing adapter compatibility cases on CPU, full JAX/common Python lint (10.00/10), pre-commit, source license and diff checks. Distributed SiTU cases are prepared in the existing MoE suite; full Blackwell/NCCL/EP and Kimi-K3 training were not rerun.
Type of change
Changes
_jax_act_lu/_jax_quantize_dact_dbiasin MoE, with capacity masking at the call sites.ActivationParamsplumbing from upstream [JAX] Enable cuDNN SiTU-GLU fusion in MoEBlock NVIDIA/TransformerEngine#3873.Checklist: