Repository navigation
Conversation
Use separate full-width FP32 router inputs and two-branch SiTU from the stacked TE changes. Preserve the Google decoder and split expert parameter tree, with expert-only TE quantization and dropless dispatch capacity. Base: Google Kimi-K3 stack through PR 6178. Numerical/model/checkpoint validation is deferred at the user request. [skip ci] Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Accumulate native or TE router histograms across microbatches, then update persistent routing bias after accepted optimizer steps. Preserve padding, eval, skip-step and checkpoint contracts. Pin the complete TE dependency stack and retain the Google full model defaults. Runtime validation remains deferred at the author request. [skip ci] Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Map CUDA-visible devices to local ordinals, enter the TE mesh-resource guard in the parity oracle, expose Latent MoE receive-capacity intermediates and stop sibling launcher ranks after failure. Record the small-scale GB200 validation in the guide. Validated with focused GPU/CPU tests, four-GPU BF16 output and gradient parity, proxy training, QB timing/profile runs and a scanned GA2 masked-QB smoke. Full-model parity and checkpoint continuation remain pending. [skip ci] Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
9 of 13 tasks
The full-hidden router and QB statistics now run in MaxText's native router. TE receives the top-k indices and weights via its upstream RoutingMapInfo, which removes the dependency on the TE router_input and QB statistics changes; only the two-branch SiTU activation remains a non-upstream TE requirement. The unused moe_qb_histogram_mode option is dropped and the QB histogram template is the same on both paths. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Integrate Google's Kimi-K3 text-training model with the complete TE JAX MoE stack: separate full-hidden router inputs, two-branch SiTU and Quantile Balancing. Unlike the earlier integration draft, QB statistics now feed a global-step router-bias update shared by the native JAX and TE paths.
This draft is based on
kimi-k3-google-stack-20261110(402fad78f7f902533dd5ae8d000b4b3284f657a2), a merge of the pinned Google heads for #5550 (decoder), #6178 (scan), #6014 (KDA), #5542 (MXFP4 checkpoint conversion) and #5551 (tokenizer). The exact Google revisions are indocs/guides/kimi_k3_te.md. The base separates Google's changes from this PR's integration diff; the previous fork PR #1 remains unchanged.Requires the stacked TE drafts #73 → #74 → #75, pinned to
ee66c66ec2e2205f59b4a0d7f1f1dbd89dce0130intests/end_to_end/gpu/kimi/te_revision.txt. This includes padding-mask support added to AI-Hypercomputer#75 for this integration.wi_0/wi_1checkpoint parameters, KDA/MLA and Attention Residuals. Keep global quantization disabled while selecting routed-expert BF16 or MXFP8 through TE.GPUSis mandatory.This draft has small-scale BF16 validation and does not claim full original Kimi training support. It does not add the vision encoder/projector, the original Per-Head Muon recipe, autoregressive KDA cache, pipeline/context parallelism, a dedicated optimized GPU KDA kernel or trainable native MXFP4 grouped GEMMs. MXFP4 checkpoints must be dequantized for the TE dense-weight path. Native distributed histogram performance, full-model memory and HF logit parity, broader distributed coverage, checkpoint continuation and MXFP8 training still require validation. Human review is required before marking this draft ready.
Tests
Validated on 2026-11-10 with four GB200 GPUs connected by NVLink, one GPU per process, EP2/FSDP2; driver 580.196, CUDA 13.5, cuDNN 9.27, JAX
0.12.0.dev20261014+0e5b59d3dd, Flax 0.12.10 and TEee66c66e. Runtime fixes are in MaxTextb43fb62f234aaab2e22ff3f9850efa2bb43f6679.tests/jax/test_fused_router.py -k qb: 26 passed on GPU, including histogram modes, padding, gradients and remat.JAX_PLATFORMS=cpu,quantile_balancing_test.py: 6 passed; QB cases inkimi_k3_latent_moe_test.pyandkimi_k3_scan_test.py: 3 passed; GPU-initialization cases inmax_utils_test.py: 6 passed. Initial GPU invocations of these MaxText tests were skipped by the repository's CPU-only policy and are not counted as passes.GPUS=4 EP=2 MOE_PARITY=1 bash tests/end_to_end/gpu/kimi/run_te_exact.sh: passed on the final runtime code. BF16 output relative RMS error 0.67%, maximum parameter/input-gradient relative RMS error 1.17%, within the existing 3% tolerance.scan_layers=true, gradient accumulation of two and masked QB. This validates the distributed plumbing at toy dimensions, not full-model parity.Unprofiled timing used the last 10 steps per run and the slowest rank's mean, averaged over two runs per setting: QB off 90.24 ms/step, QB on 89.83 ms/step. The -0.45% difference is within run-to-run variation and does not establish a speedup. QB changes routing, so this E2E comparison is not an isolated kernel-overhead measurement. A separate E=896/K=16/T=8192 forward-router profile measured about 106 us without QB, 136 us with fused-atomic QB, 140 us with two-kernel QB and 176 us with masked QB; these are sums of GPU kernel durations over five captured calls.
Static source/formatting, launcher syntax and diff checks passed. CI was not triggered. Full-model HF parity, checkpoint continuation, larger distributed configurations and MXFP8 training remain pending. Human review is required before marking the draft ready. Instructions and limitations are in
docs/guides/kimi_k3_te.md.Checklist