Skip to content

Integrate TE MoEBlock with the Google Kimi-K3 architecture - #1

Draft
pggPL wants to merge 1 commit into
kimi-k3-google-base-20261106from
kimi_k3_te_exact
Draft

pggPL wants to merge 1 commit into
kimi-k3-google-base-20261106from
kimi_k3_te_exact

Conversation

@pggPL

@pggPL pggPL commented Nov 6, 2026

Copy link
Copy Markdown
Owner

Description

Integrate TE MoEBlock with the Google Kimi-K3 architecture without replacing full-width routing or two-branch SiTU. This PR is based on Google #6178, commit f9cfba05aa09885417e6b107fd3a90d74d339956, which contains AI-Hypercomputer#5550 and adds scanned layers. The snapshot base branch keeps this fork diff limited to the TE integration.

Requires the head of pggPL/TransformerEngine#74, which is stacked on #73. TE owns routing, EP dispatch/combine, expert grouped GEMMs, and backward. The router consumes the original hidden state in FP32; experts consume the latent and apply both SiTU soft caps. Google's decoder, KDA/MLA, Attention Residuals, shared/dense SiTU and latent projections/norm remain unchanged.

Keep prefuse_moe_weights=false to retain the Google split expert parameter names and shapes. Values are concatenated only at the TE call boundary. Global quantization stays empty; routed experts select BF16 or MXFP8 through te_gmm_quantization. Require worst-case receive capacity because the Google latent block does not propagate overflow metadata to the trainer.

Here, exact refers to the architecture and forward/gradient contract of the Google baseline, within the chosen numerical precision. Numerical execution and checkpoint continuation are not yet validated. This does not add features missing from the baseline, including quantile-balancing bias updates or autoregressive KDA state caching.

Tests

Model execution and numerical tests are intentionally deferred at the user's request. Source/shell syntax and formatting were checked. The prepared BF16 component check compares the complete latent MoE output and all parameter/input gradients against an independent oracle with nonzero selection bias and routing scale:

MOE_PARITY=1 bash tests/end_to_end/gpu/kimi/run_te_exact.sh

The same launcher prepares a four-layer training slice by default, with optional scanned layers and expert MXFP8. These runs have not been executed. The existing up-activation guard test now distinguishes supported TE SiTU from unsupported fused activations. Details and limitations are in docs/guides/kimi_k3_te.md.

Checklist

  • I have performed a self-review of my code.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests and provided workload links above if applicable.
  • I have made corresponding changes to the documentation and its table of contents.

Use separate full-width FP32 router inputs and two-branch SiTU from the stacked TE changes. Preserve the Google decoder and split expert parameter tree, with expert-only TE quantization and dropless dispatch capacity.

Base: Google Kimi-K3 stack through PR 6178. Numerical/model/checkpoint validation is deferred at the user request. [skip ci]
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant