Skip to content

[PyTorch] Test sequence-parallel Linear with 3D inputs and small micro-batches - #3791

Open
0z5a wants to merge 1 commit into
NVIDIA:mainfrom
0z5a:test-sp-3d-input-3720
Open

0z5a wants to merge 1 commit into
NVIDIA:mainfrom
0z5a:test-sp-3d-input-3720

Conversation

@0z5a

@0z5a 0z5a commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

Description

The distributed Linear tests in run_numerics.py only use 2D inputs, so the sequence-parallel all-gathers never see the [seq, micro-batch, hidden] layout. #3720 shows what that misses. With the NVFP4 recipe, the 2.21.0 pre-release d77ef88c crashed in the backward all-gather of a row-parallel Linear with sequence parallelism when the micro-batch was 1, and the existing tests passed on it. #3628 later fixed the shape bookkeeping on main and on release_v2.21.

This PR adds a test for that layout. It is test-only.

Related: #3720, #3628

Type of change

  • Documentation change (change only to the documentation, either a fix or a new content)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Infra/Build change
  • Code refactoring

Changes

  • tests/pytorch/distributed/run_numerics.py: test_linear_3d_input runs a sequence-parallel te.Linear, column- and row-parallel, on [seq, micro-batch, hidden] inputs with micro-batch 1 and 3. The sequence has WORLD_SIZE * SEQ_LEN tokens, so every rank holds SEQ_LEN of them in the sequence-parallel layout. The test compares the output and the input gradient with a single-GPU Linear, using the existing helpers and tolerances. It runs for every recipe that test_numerics.py parametrizes.

Testing

Two B300 GPUs (sm_103), TE built for 103a, torch 2.14.1+cu130, cuDNN 9.24:

TE test_linear_3d_input, NVFP4
d77ef88c (2.21.0 pre-release from #3720) fails: row-parallel, micro-batch 1 in swap_first_dims ("Input is not allocated"); row-parallel, micro-batch 3 in CheckScaleTensorShape (columnwise_scale_inv.shape == expected). Both column-parallel cases pass, and so does the existing test_linear.
7457e47d (v2.21.0, includes #3628) passes
main 63451d6b passes

On main with this PR, tests/pytorch/distributed/test_numerics.py passes for all six parametrizations (None, fp8, mxfp8, fp8_cs, fp8_block_scaling, nvfp4): 6 passed. Black 24.4.2 with the repository's pre-commit options leaves the file unchanged.

Checklist:

  • I have read and followed the contributing guidelines
  • The functionality is complete
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

…o-batches

The distributed Linear tests use 2D inputs, so the sequence-parallel
all-gathers never see the [seq, micro-batch, hidden] layout. With the
NVFP4 recipe, a 2.21.0 pre-release (d77ef88) crashed in the backward
all-gather of a row-parallel Linear at micro-batch 1 and 3 (NVIDIA#3720),
while these tests passed. NVIDIA#3628 fixed the shape bookkeeping on main and
release_v2.21. Add column- and row-parallel cases with 3D inputs and
micro-batches of 1 and 3.

Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
@github-actions github-actions Bot added the community-contribution PRs from external contributor outside the core maintainers, representing community-driven work. label Nov 1, 2026
@0z5a
0z5a marked this pull request as ready for review November 1, 2026 21:01
@greptile-apps

greptile-apps Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Low impact] The PR appears safe to merge.

Summary

Adds sequence-parallel Linear tests for three-dimensional inputs.

  • Distributed Linear tests now cover 3D sequence-parallel inputs.

No actionable issues found. Tests were not run during this review.

Reviews (1) · Last reviewed commit: "[PyTorch] Test sequence-parallel Linear ..." · Reviewed by Greptile

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-contribution PRs from external contributor outside the core maintainers, representing community-driven work.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant