Skip to content

[JAX] Support separate router inputs in MoEBlock - #73

Closed
pggPL wants to merge 2 commits into
kimi-k3-support-base-20261105from
jax_moe_router_input
Closed

pggPL wants to merge 2 commits into
kimi-k3-support-base-20261105from
jax_moe_router_input

Conversation

@pggPL

@pggPL pggPL commented Nov 5, 2026 •

Copy link
Copy Markdown
Owner

Description

Kimi-K3 computes routing from full-width hidden states while its routed experts process a down-projected latent. Add an optional, independently shaped RouterComputationInfo.router_input and expose it through the Flax MoEBlock wrapper. Existing calls continue routing on the expert input.

This is a draft for code review; runtime validation is deferred. The base branch is a snapshot of NVIDIA main at e29c5d691dc0e9b05f285dc19ddbf44abe26c33a, so this fork PR contains only the feature change.

This is the first PR in the stack: #73 → #74. pggPL/maxtext#1 uses the head of #74.

Type of change

  • Documentation change (change only to the documentation, either a fix or a new content)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Infra/Build change
  • Code refactoring

Changes

  • Keep integrated TE routing, EP dispatch/combine, and the expert FFN while allowing separate router activations.
  • Return independent router-input and expert-input cotangents, preserving their dtypes and sharding.
  • Register the Flax gate kernel at router width and retain latent-width expert parameters and EP buffers.
  • Add shape guards, documentation, and prepared BF16/MXFP8 gradient tests with a latent projection, mixed router dtypes, auxiliary loss, and rematerialization.

Checklist:

  • I have read and followed the contributing guidelines
  • The functionality is complete
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

Keep integrated TE routing on full-width hidden states while dispatching narrower expert activations. Return independent input cotangents through RouterComputationInfo and expose the input in the Flax wrapper.

Runtime validation is deferred at the author request. [skip ci]

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Drop the prose paragraph duplicating the router_input docstring.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
@pggPL pggPL closed this Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant