Skip to content

[JAX] Expose Quantile Balancing router statistics - #75

Draft
pggPL wants to merge 8 commits into
jax_moe_situfrom
jax_moe_qb
Draft

pggPL wants to merge 8 commits into
jax_moe_situfrom
jax_moe_qb

Conversation

@pggPL

@pggPL pggPL commented Nov 9, 2026 •

Copy link
Copy Markdown
Owner

Description

Kimi-K3 Quantile Balancing needs per-expert score distributions, while the JAX router currently returns only routing weights/maps and MoE assignment counts. Expose the existing TE CUDA QB router through JAX FFI and return histogram contributions from MoEBlock for downstream global-batch balancing.

This draft is stacked on #74 (jax_moe_situ), which includes #73. It adds the TE statistics path; the training framework owns global-step accumulation, quantile recovery and bias updates. No common CUDA or PyTorch code changes are needed.

Rebased on #74's native grouped SiTU implementation. Validated on GB200: native build/import, 26 QB router tests, and eight four-rank QB MoE cases covering both histogram modes, masked/unmasked tokens and rematerialized gradients. All 36 resource/API tests passed; the API stub now matches the internal five-result QB contract while the public API without QB still returns four results. The complete #76 stack passed 38 distributed MoE cases per rank. Pre-commit, source license checks, changed C++ lint and full JAX Python lint passed. Full Kimi-K3 training and performance remain unverified. This PR remains a draft for human review.

Type of change

  • Documentation change (change only to the documentation, either a fix or a new content)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Infra/Build change
  • Code refactoring

Changes

  • Bind nvte_fused_topk_with_score_function_forward_qb_v2 with both fused_atomic and two_kernel modes. The latter calls the existing nvte_qb_histogram_accumulate internally. FFI histogram input/output aliasing preserves JAX functional semantics while reusing the in-place kernel.
  • Add fused_topk_with_score_function_qb: FP32 sigmoid routing, bytemap/bitmap outputs, optional initial int32 histogram, and gradients through routing probabilities using the existing backward. Histogram collection, bias and bounds have no gradient.
  • Add opt-in qb_bin_bounds, qb_num_bins and qb_histogram_mode to functional/Flax MoE. QB calls return a fifth [token_shards, experts, bins] result, sharded over the existing DP/FSDP/EP token axes. Forward performs no histogram collective; default calls retain four results.
  • Prepare numerical/gradient, scan accumulation, mutable-bounds, clamp/tie, distributed-histogram, MoE/remat and invalid-configuration tests. Document lifecycle, sharding and ownership.
  • Support optional token_mask / qb_token_mask for sequence padding. With a mask, CUDA computes routing/cutoff and JAX accumulates masked histogram counts from saved scores. Without a mask, the existing CUDA histogram fast path remains unchanged. Prepare mixed/all-valid/all-padding mask and gradient/remat checks.
  • Bounds must be finite and ordered; their values are not validated through a host synchronization. No TPU implementation, histogram-to-bias finalizer, MaxText integration or performance claim is included in this TE PR. The downstream training framework owns those steps.

Checklist:

  • I have read and followed the contributing guidelines
  • The functionality is complete
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

pggPL added 3 commits November 5, 2026 12:20
Keep integrated TE routing on full-width hidden states while dispatching narrower expert activations. Return independent input cotangents through RouterComputationInfo and expose the input in the Flax wrapper.

Runtime validation is deferred at the author request. [skip ci]

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Evaluate configurable gate and up soft caps in FP32 with gradients for both branches. Keep SiTU on the unfused grouped-GEMM path for BF16 and MXFP8 and preserve existing activations.

Runtime validation is deferred at the author request. [skip ci]

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Stack the SiTU PR on the router-input PR so the tip provides both capabilities to MaxText. Runtime validation remains deferred. [skip ci]

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Expose grouped SiTU C APIs and an XLA FFI custom VJP for FP32, FP16 and BF16. Keep expert counts on device, skip inactive inputs and zero inactive outputs and gradients. Replace the JAX MoE SiTU formulas with the native calls; MXFP8 quantization remains a separate grouped operation.

Validate values and FP64 analytic gradients, dynamic counts, empty experts, NaN tails, vmap and checkpoint replay, plus four-GPU BF16/MXFP8 MoE execution.

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Instantiate the existing vectorized gated activation and derivative kernels with SiTUGLUParam and optional device group counts. Preserve their alignment dispatch and skip inactive input and gradient loads while zeroing inactive output rows. Keep ordinary GLU instantiations on the compile-time ungrouped path.

Remove the separate grouped SiTU CUDA kernel and translation unit; retain the C and JAX APIs. Cover aligned, uniformly unaligned and differently aligned storage, counts, caps, FP64 derivatives and NaN tails in common C++ tests. Extend PyTorch SiTU coverage to a width outside the TMA path.

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Bind the existing CUDA QB router with functional histogram aliasing and expose per-token-shard histogram contributions from MoEBlock. Leave global-step bias updates to the training framework.

Runtime validation is deferred at the author request. [skip ci]

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Keep CUDA routing and cutoff generation while building masked histograms from saved scores in JAX. Preserve the existing CUDA histogram fast path without a mask.

Runtime validation remains deferred. [skip ci]

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
The internal MoE custom VJP always returns a fifth histogram result, even when QB is disabled. Keep the API test stub consistent while retaining the public four-result contract without QB.

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant