Skip to content

[Common][PyTorch] EP Gemm Overlap - #3906

Open
phu0ngng wants to merge 1 commit into
NVIDIA:mainfrom
phu0ngng:feat/moe-overlap-ops-main
Open

phu0ngng wants to merge 1 commit into
NVIDIA:mainfrom
phu0ngng:feat/moe-overlap-ops-main

Conversation

@phu0ngng

Copy link
Copy Markdown
Collaborator

Description

Please include a brief summary of the changes, relevant motivation and context.

Fixes # (issue)

Type of change

  • Documentation change (change only to the documentation, either a fix or a new content)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Infra/Build change
  • Code refactoring

Changes

Please list the changes introduced in this PR:

  • Change A
  • Change B

Checklist:

  • I have read and followed the contributing guidelines
  • The functionality is complete
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

@greptile-apps

greptile-apps Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge, with a non-blocking gap in MXFP8 CUDA graph test coverage.

Findings

  1. P2 MXFP8 replay lacks coverage ▶

Summary

Adds expert-parallel communication overlapped with grouped GEMMs for the PyTorch MoE operation sequence.

  • Supported MoE sequences overlap communication with expert GEMMs.
  • Distributed tests cover the fused MoE overlap path.

No new actionable issues were established.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Input tokens and routing] --> B[Dispatch overlaps FC1 and SwiGLU]
  B --> C[FC2 overlaps combine]
  C --> D[Output tokens]
  D --> E[Backward dispatch overlaps FC2 gradient]
  E --> F[FC1 gradient overlaps backward combine]
  F --> G[Input and routing gradients]
  E --> H[FC2 weight gradient]
  F --> I[FC1 weight gradient]
Loading

Reviews (2) · Last reviewed commit: "[Common][PyTorch] Add MoE EP communicati..." · Reviewed by Greptile

Comment on lines +1323 to +1325
config = self._overlap_config()
ops = self._moe_ops(config, self._make_buffer_from_config(config))
ref_ops = self._moe_ops(config, self._make_buffer_from_config(config))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 MXFP8 replay lacks coverage

test_moe_overlap_cuda_graph builds both models with _overlap_config() and never enables MXFP8 autocast. It therefore misses capture and replay of the new shared dispatch window, requantization, and MXFP8 weight gradients. The tests could pass while that training path breaks.

Run this test with both BF16 and MXFP8BlockScaling, using MXFP8 transport for the latter. Compare replayed outputs and gradients after changing the inputs and routing.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

…) with in-call weight gradients for BF16 and MXFP8

Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com>
@phu0ngng
phu0ngng force-pushed the feat/moe-overlap-ops-main branch from 82d9201 to 7c92f4a Compare November 20, 2026 17:15
@phu0ngng phu0ngng added the 3.0 label Nov 20, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant