Repository navigation
Conversation
Keep integrated TE routing on full-width hidden states while dispatching narrower expert activations. Return independent input cotangents through RouterComputationInfo and expose the input in the Flax wrapper. Runtime validation is deferred at the author request. [skip ci] Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Evaluate configurable gate and up soft caps in FP32 with gradients for both branches. Keep SiTU on the unfused grouped-GEMM path for BF16 and MXFP8 and preserve existing activations. Runtime validation is deferred at the author request. [skip ci] Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Stack the SiTU PR on the router-input PR so the tip provides both capabilities to MaxText. Runtime validation remains deferred. [skip ci] Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
This was referenced Nov 10, 2026
Expose grouped SiTU C APIs and an XLA FFI custom VJP for FP32, FP16 and BF16. Keep expert counts on device, skip inactive inputs and zero inactive outputs and gradients. Replace the JAX MoE SiTU formulas with the native calls; MXFP8 quantization remains a separate grouped operation. Validate values and FP64 analytic gradients, dynamic counts, empty experts, NaN tails, vmap and checkpoint replay, plus four-GPU BF16/MXFP8 MoE execution. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
9 of 13 tasks
Instantiate the existing vectorized gated activation and derivative kernels with SiTUGLUParam and optional device group counts. Preserve their alignment dispatch and skip inactive input and gradient loads while zeroing inactive output rows. Keep ordinary GLU instantiations on the compile-time ungrouped path. Remove the separate grouped SiTU CUDA kernel and translation unit; retain the C and JAX APIs. Cover aligned, uniformly unaligned and differently aligned storage, counts, caps, FP64 derivatives and NaN tails in common C++ tests. Extend PyTorch SiTU coverage to a width outside the TMA path. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Bind the existing CUDA QB router with functional histogram aliasing and expose per-token-shard histogram contributions from MoEBlock. Leave global-step bias updates to the training framework. Runtime validation is deferred at the author request. [skip ci] Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Keep CUDA routing and cutoff generation while building masked histograms from saved scores in JAX. Preserve the existing CUDA histogram fast path without a mask. Runtime validation remains deferred. [skip ci] Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
The internal MoE custom VJP always returns a fifth histogram result, even when QB is disabled. Keep the API test stub consistent while retaining the public four-result contract without QB. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Kimi-K3 Quantile Balancing needs per-expert score distributions, while the JAX router currently returns only routing weights/maps and MoE assignment counts. Expose the existing TE CUDA QB router through JAX FFI and return histogram contributions from MoEBlock for downstream global-batch balancing.
This draft is stacked on #74 (
jax_moe_situ), which includes #73. It adds the TE statistics path; the training framework owns global-step accumulation, quantile recovery and bias updates. No common CUDA or PyTorch code changes are needed.Rebased on #74's native grouped SiTU implementation. Validated on GB200: native build/import, 26 QB router tests, and eight four-rank QB MoE cases covering both histogram modes, masked/unmasked tokens and rematerialized gradients. All 36 resource/API tests passed; the API stub now matches the internal five-result QB contract while the public API without QB still returns four results. The complete #76 stack passed 38 distributed MoE cases per rank. Pre-commit, source license checks, changed C++ lint and full JAX Python lint passed. Full Kimi-K3 training and performance remain unverified. This PR remains a draft for human review.
Type of change
Changes
nvte_fused_topk_with_score_function_forward_qb_v2with bothfused_atomicandtwo_kernelmodes. The latter calls the existingnvte_qb_histogram_accumulateinternally. FFI histogram input/output aliasing preserves JAX functional semantics while reusing the in-place kernel.fused_topk_with_score_function_qb: FP32 sigmoid routing, bytemap/bitmap outputs, optional initial int32 histogram, and gradients through routing probabilities using the existing backward. Histogram collection, bias and bounds have no gradient.qb_bin_bounds,qb_num_binsandqb_histogram_modeto functional/Flax MoE. QB calls return a fifth[token_shards, experts, bins]result, sharded over the existing DP/FSDP/EP token axes. Forward performs no histogram collective; default calls retain four results.token_mask/qb_token_maskfor sequence padding. With a mask, CUDA computes routing/cutoff and JAX accumulates masked histogram counts from saved scores. Without a mask, the existing CUDA histogram fast path remains unchanged. Prepare mixed/all-valid/all-padding mask and gradient/remat checks.Checklist: