Repository navigation
Conversation
Use separate full-width FP32 router inputs and two-branch SiTU from the stacked TE changes. Preserve the Google decoder and split expert parameter tree, with expert-only TE quantization and dropless dispatch capacity. Base: Google Kimi-K3 stack through PR 6178. Numerical/model/checkpoint validation is deferred at the user request. [skip ci] Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
This was referenced Nov 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Integrate TE MoEBlock with the Google Kimi-K3 architecture without replacing full-width routing or two-branch SiTU. This PR is based on Google #6178, commit
f9cfba05aa09885417e6b107fd3a90d74d339956, which contains AI-Hypercomputer#5550 and adds scanned layers. The snapshot base branch keeps this fork diff limited to the TE integration.Requires the head of pggPL/TransformerEngine#74, which is stacked on #73. TE owns routing, EP dispatch/combine, expert grouped GEMMs, and backward. The router consumes the original hidden state in FP32; experts consume the latent and apply both SiTU soft caps. Google's decoder, KDA/MLA, Attention Residuals, shared/dense SiTU and latent projections/norm remain unchanged.
Keep
prefuse_moe_weights=falseto retain the Google split expert parameter names and shapes. Values are concatenated only at the TE call boundary. Global quantization stays empty; routed experts select BF16 or MXFP8 throughte_gmm_quantization. Require worst-case receive capacity because the Google latent block does not propagate overflow metadata to the trainer.Here, exact refers to the architecture and forward/gradient contract of the Google baseline, within the chosen numerical precision. Numerical execution and checkpoint continuation are not yet validated. This does not add features missing from the baseline, including quantile-balancing bias updates or autoregressive KDA state caching.
Tests
Model execution and numerical tests are intentionally deferred at the user's request. Source/shell syntax and formatting were checked. The prepared BF16 component check compares the complete latent MoE output and all parameter/input gradients against an independent oracle with nonzero selection bias and routing scale:
The same launcher prepares a four-layer training slice by default, with optional scanned layers and expert MXFP8. These runs have not been executed. The existing up-activation guard test now distinguishes supported TE SiTU from unsupported fused activations. Details and limitations are in
docs/guides/kimi_k3_te.md.Checklist