Skip to content

Integrate Kimi-K3 training with TE MoE and Quantile Balancing - #2

Draft
pggPL wants to merge 4 commits into
kimi-k3-google-stack-20261110from
kimi_k3_te_qb
Draft

pggPL wants to merge 4 commits into
kimi-k3-google-stack-20261110from
kimi_k3_te_qb

Conversation

@pggPL

@pggPL pggPL commented Nov 10, 2026 •

Copy link
Copy Markdown
Owner

Description

Integrate Google's Kimi-K3 text-training model with the complete TE JAX MoE stack: separate full-hidden router inputs, two-branch SiTU and Quantile Balancing. Unlike the earlier integration draft, QB statistics now feed a global-step router-bias update shared by the native JAX and TE paths.

This draft is based on kimi-k3-google-stack-20261110 (402fad78f7f902533dd5ae8d000b4b3284f657a2), a merge of the pinned Google heads for #5550 (decoder), #6178 (scan), #6014 (KDA), #5542 (MXFP4 checkpoint conversion) and #5551 (tokenizer). The exact Google revisions are in docs/guides/kimi_k3_te.md. The base separates Google's changes from this PR's integration diff; the previous fork PR #1 remains unchanged.

Requires the stacked TE drafts #73 → #74 → #75, pinned to ee66c66ec2e2205f59b4a0d7f1f1dbd89dce0130 in tests/end_to_end/gpu/kimi/te_revision.txt. This includes padding-mask support added to AI-Hypercomputer#75 for this integration.

  • Preserve the Google text architecture: full FP32 router, latent experts, SiTU caps 4/25, late routing weights, separate wi_0/wi_1 checkpoint parameters, KDA/MLA and Attention Residuals. Keep global quantization disabled while selecting routed-expert BF16 or MXFP8 through TE.
  • Collect per-expert QB histograms with padding excluded. TE returns token-shard contributions; native JAX uses a chunked LAX baseline. Accumulate histograms in the gradient-accumulation carry, recover and center the pooled quantile once after an accepted optimizer step, and preserve the bias on empty/rejected steps. Bias stays fixed across microbatches. Evaluation disables collection; checkpoints persist bias without histogram state.
  • Enable QB for Kimi-K3 without auxiliary balancing losses or sign-based updates. Add configuration guards, fail clearly for missing TE capabilities, and prepare math/state/GA/scan checks. The launcher defaults to 93 layers and 896 experts; smaller workloads require explicit dimension overrides and GPUS is mandatory.

This draft has small-scale BF16 validation and does not claim full original Kimi training support. It does not add the vision encoder/projector, the original Per-Head Muon recipe, autoregressive KDA cache, pipeline/context parallelism, a dedicated optimized GPU KDA kernel or trainable native MXFP4 grouped GEMMs. MXFP4 checkpoints must be dequantized for the TE dense-weight path. Native distributed histogram performance, full-model memory and HF logit parity, broader distributed coverage, checkpoint continuation and MXFP8 training still require validation. Human review is required before marking this draft ready.

Tests

Validated on 2026-11-10 with four GB200 GPUs connected by NVLink, one GPU per process, EP2/FSDP2; driver 580.196, CUDA 13.5, cuDNN 9.27, JAX 0.12.0.dev20261014+0e5b59d3dd, Flax 0.12.10 and TE ee66c66e. Runtime fixes are in MaxText b43fb62f234aaab2e22ff3f9850efa2bb43f6679.

  • TE JAX source build and imports passed. tests/jax/test_fused_router.py -k qb: 26 passed on GPU, including histogram modes, padding, gradients and remat.
  • With JAX_PLATFORMS=cpu, quantile_balancing_test.py: 6 passed; QB cases in kimi_k3_latent_moe_test.py and kimi_k3_scan_test.py: 3 passed; GPU-initialization cases in max_utils_test.py: 6 passed. Initial GPU invocations of these MaxText tests were skipped by the repository's CPU-only policy and are not counted as passes.
  • GPUS=4 EP=2 MOE_PARITY=1 bash tests/end_to_end/gpu/kimi/run_te_exact.sh: passed on the final runtime code. BF16 output relative RMS error 0.67%, maximum parameter/input-gradient relative RMS error 1.17%, within the existing 3% tolerance.
  • Synthetic training with four layers, 32 experts/top16, full hidden/latent widths, sequence length 256, batch 1/GPU: QB-on smoke passed 8 steps. Four alternating QB-off/on/on/off runs passed 30 steps each, followed by a separate 22-step run profiling three warmed steps. All ranks had finite loss/gradients and zero capacity overflow.
  • A small 13-layer/8-expert model passed 6 steps on four GPUs with scan_layers=true, gradient accumulation of two and masked QB. This validates the distributed plumbing at toy dimensions, not full-model parity.

Unprofiled timing used the last 10 steps per run and the slowest rank's mean, averaged over two runs per setting: QB off 90.24 ms/step, QB on 89.83 ms/step. The -0.45% difference is within run-to-run variation and does not establish a speedup. QB changes routing, so this E2E comparison is not an isolated kernel-overhead measurement. A separate E=896/K=16/T=8192 forward-router profile measured about 106 us without QB, 136 us with fused-atomic QB, 140 us with two-kernel QB and 176 us with masked QB; these are sums of GPU kernel durations over five captured calls.

Static source/formatting, launcher syntax and diff checks passed. CI was not triggered. Full-model HF parity, checkpoint continuation, larger distributed configurations and MXFP8 training remain pending. Human review is required before marking the draft ready. Instructions and limitations are in docs/guides/kimi_k3_te.md.

Checklist

  • I have performed a self-review of my code.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests and provided workload links above if applicable.
  • I have made corresponding documentation changes and included the guide in the documentation table of contents.

Use separate full-width FP32 router inputs and two-branch SiTU from the stacked TE changes. Preserve the Google decoder and split expert parameter tree, with expert-only TE quantization and dropless dispatch capacity.

Base: Google Kimi-K3 stack through PR 6178. Numerical/model/checkpoint validation is deferred at the user request. [skip ci]
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Accumulate native or TE router histograms across microbatches, then update persistent routing bias after accepted optimizer steps. Preserve padding, eval, skip-step and checkpoint contracts. Pin the complete TE dependency stack and retain the Google full model defaults.

Runtime validation remains deferred at the author request. [skip ci]

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Map CUDA-visible devices to local ordinals, enter the TE mesh-resource guard in the parity oracle, expose Latent MoE receive-capacity intermediates and stop sibling launcher ranks after failure. Record the small-scale GB200 validation in the guide.

Validated with focused GPU/CPU tests, four-GPU BF16 output and gradient parity, proxy training, QB timing/profile runs and a scanned GA2 masked-QB smoke. Full-model parity and checkpoint continuation remain pending. [skip ci]

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
The full-hidden router and QB statistics now run in MaxText's native
router. TE receives the top-k indices and weights via its upstream
RoutingMapInfo, which removes the dependency on the TE router_input and
QB statistics changes; only the two-branch SiTU activation remains a
non-upstream TE requirement. The unused moe_qb_histogram_mode option is
dropped and the QB histogram template is the same on both paths.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant