Skip to content

[Common] Swizzle + padding fusion for Unpadded Input Scales - #3768

Merged
Oleg-Goncharov merged 14 commits into
NVIDIA:mainfrom
vthumbe1503:swizzle-compact-padding-fusion
Nov 20, 2026
Merged

Oleg-Goncharov merged 14 commits into
NVIDIA:mainfrom
vthumbe1503:swizzle-compact-padding-fusion

Conversation

@vthumbe1503

@vthumbe1503 vthumbe1503 commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Allow MXFP8 scale swizzling to consume unswizzled scale buffers that omit outer-dimension padding and produce fully padded GEMM-swizzled output. This enables consumers such as FSDP to avoid restoring that padding in a separate operation after allgather when the scales will immediately be swizzled.

Primary comparison: separate re-padding versus fused padding/swizzle

All times are µs per API call; multi-tensor timings cover the whole batch, not one tensor. Speedup = separate time / fused time, so larger is better.

Fusion is faster in all 35 configurations: 1.84–13.39× overall; 1.84–2.63× for single-tensor cases and 4.86–13.39× for same-shape multi-tensor batches.

MXFP8 rowwise

Original #3682 shape Tested logical shape Tensors Pad → swizzle (µs) Fused (µs) Speedup
4096×128 4064×128 1 3.583 1.408 2.55×
4096×256 4064×256 1 3.584 1.409 2.54×
4096×512 4064×512 1 3.712 1.410 2.63×
4096×768 4064×768 1 3.845 1.665 2.31×
4096×1024 4064×1024 1 4.299 1.665 2.58×
4096×1280 4064×1280 1 4.608 1.920 2.40×
4096×1536 4064×1536 1 4.613 1.920 2.40×
4096×1664 4064×1664 1 4.736 1.922 2.46×
4096×1792 4064×1792 1 4.613 1.922 2.40×
4096×1920 4064×1920 1 4.741 1.920 2.47×
4096×2048 4064×2048 1 4.224 1.920 2.20×
4096×2560 4064×2560 1 4.485 2.048 2.19×
4096×4096 4064×4096 1 4.996 2.432 2.05×
2048×7168 2016×7168 1 6.368 3.331 1.91×
4096×7168 4064×7168 1 5.893 3.203 1.84×

MXFP8 columnwise

Original #3682 shape Tested logical shape Tensors Pad → swizzle (µs) Fused (µs) Speedup
4096×1536 4064×1536 1 4.357 1.665 2.62×
4096×512 4064×512 1 3.972 1.584 2.51×
4096×2048 4064×2048 1 4.228 1.664 2.54×
4224×2048 4192×2048 1 4.228 1.664 2.54×
4096×7168 4064×7168 1 4.613 1.920 2.40×
7168×4096 7136×4096 1 4.613 1.921 2.40×
2048×7168 2016×7168 1 4.356 1.666 2.61×

MXFP8 multi-tensor rowwise

Original #3682 shape Tested logical shape Tensors Pad → swizzle (µs) Fused (µs) Speedup
7168×4096 7136×4096 8 29.392 4.696 6.26×
4096×2048 4064×2048 8 22.313 2.310 9.66×
4096×128 4064×128 4 10.134 1.665 6.09×
16384×128 16352×128 4 10.753 1.667 6.45×
256×256 224×256 4 8.843 1.467 6.03×
2048×7168 2016×7168 15 51.296 6.421 7.99×
4096×7168 4064×7168 8 30.795 6.332 4.86×

MXFP8 multi-tensor columnwise

Original #3682 shape Tested logical shape Tensors Pad → swizzle (µs) Fused (µs) Speedup
4096×2048 4064×2048 8 22.323 2.691 8.30×
4224×2048 4192×2048 20 64.823 5.504 11.78×
1024×7168 992×7168 32 102.812 7.677 13.39×
4224×2048 4192×2048 13 42.940 5.075 8.46×
4096×7168 4064×7168 8 30.438 6.094 4.99×
7168×4096 7136×4096 8 30.910 5.785 5.34×

Secondary comparison: cost of reading unpadded scales

This answers the separate question of whether the fused swizzle itself is slower than swizzling an already-padded input. Neither side below includes a preceding padding operation. Both run on the PR head, with the same logical values and output dimensions.

This was a separate paired measurement pass from the primary comparison, so its fused timings can differ slightly. Percent change = unpadded time / already-padded time − 1; positive means slower. Do not derive the separate-padding cost by subtracting values across the two passes.

All 35 already-padded versus unpadded comparisons

MXFP8 rowwise

Tested logical shape Tensors Already-padded input (µs) Unpadded input (µs) Change
4064×128 1 1.281 1.408 +9.9%
4064×256 1 1.409 1.409 +0.0%
4064×512 1 1.409 1.410 +0.1%
4064×768 1 1.538 1.665 +8.2%
4064×1024 1 1.664 1.666 +0.1%
4064×1280 1 1.794 1.920 +7.0%
4064×1536 1 1.794 1.920 +7.0%
4064×1664 1 1.920 1.920 +0.0%
4064×1792 1 1.920 1.922 +0.1%
4064×1920 1 1.920 1.922 +0.1%
4064×2048 1 1.665 1.920 +15.4%
4064×2560 1 1.920 2.050 +6.8%
4064×4096 1 2.050 2.432 +18.6%
2016×7168 1 2.947 3.331 +13.0%
4064×7168 1 3.203 3.203 +0.0%

MXFP8 columnwise

Tested logical shape Tensors Already-padded input (µs) Unpadded input (µs) Change
4064×1536 1 1.793 1.665 -7.1%
4064×512 1 1.666 1.562 -6.3%
4064×2048 1 1.793 1.665 -7.1%
4192×2048 1 1.793 1.666 -7.0%
4064×7168 1 1.922 1.794 -6.7%
7136×4096 1 1.923 1.920 -0.1%
2016×7168 1 1.792 1.666 -7.0%

MXFP8 multi-tensor rowwise

Tested logical shape Tensors Already-padded input (µs) Unpadded input (µs) Change
7136×4096 8 4.605 4.707 +2.2%
4064×2048 8 2.177 2.467 +13.3%
4064×128 4 1.664 1.665 +0.0%
16352×128 4 1.664 1.666 +0.1%
224×256 4 1.461 1.486 +1.7%
2016×7168 15 6.326 6.411 +1.3%
4064×7168 8 6.263 6.327 +1.0%

MXFP8 multi-tensor columnwise

Tested logical shape Tensors Already-padded input (µs) Unpadded input (µs) Change
4064×2048 8 2.817 2.690 -4.5%
4192×2048 20 5.005 5.499 +9.9%
992×7168 32 7.648 7.624 -0.3%
4192×2048 13 4.613 5.128 +11.2%
4064×7168 8 5.386 6.061 +12.5%
7136×4096 8 6.020 5.764 -4.2%

The largest uniform-case overhead versus already-padded input is 18.6% rowwise / 12.5% columnwise; some cases are unchanged or faster. The primary comparison shows that eliminating explicit re-padding more than compensates for this overhead in every tested supported configuration.

Type of change

  • Documentation change (change only to the documentation, either a fix or a new content)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Infra/Build change
  • Code refactoring

Changes

  • Accept MXFP8 scale buffers that omit only outer-dimension padding.
  • Produce fully padded swizzled output without reading beyond the unpadded input.
  • Add compact-scale swizzle correctness coverage.

Checklist:

  • I have read and followed the contributing guidelines
  • The functionality is complete
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

vthumbe1503 and others added 5 commits October 25, 2026 12:55
Use per-tile swizzle kernels for small MXFP8 scale tensors and multi-tensor cases. Require 16-byte alignment for input and output scale buffers selected by the swizzle.

Signed-off-by: Varun Thumbe <vthumbe@nvidia.com>
Signed-off-by: Varun Thumbe <vthumbe@nvidia.com>
Signed-off-by: Varun Thumbe <vthumbe@nvidia.com>
Signed-off-by: Varun Thumbe <vthumbe@nvidia.com>
Signed-off-by: Varun Thumbe <vthumbe@nvidia.com>
Signed-off-by: Varun Thumbe <vthumbe@nvidia.com>
@vthumbe1503
vthumbe1503 marked this pull request as ready for review November 18, 2026 06:56
@vthumbe1503

Copy link
Copy Markdown
Collaborator Author

/te-ci L0

@greptile-apps

greptile-apps Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no actionable issues were found.

Summary

This PR lets MXFP8 and NVFP4 scale swizzling fill missing outer-dimension padding while producing GEMM-ready scales.

  • Scale swizzling accepts compact inputs and writes padded GEMM scales.

No new actionable issues found. GPU tests were not run.

Reviews (6) · Last reviewed commit: "Merge branch 'main' into swizzle-compact..." · Reviewed by Greptile

@vthumbe1503 vthumbe1503 changed the title Swizzle compact padding fusion Swizzle + padding fusion for Unpadded Input Scales Nov 18, 2026
Signed-off-by: Varun Thumbe <vthumbe@nvidia.com>
@vthumbe1503

Copy link
Copy Markdown
Collaborator Author

/te-ci L0

@vthumbe1503

Copy link
Copy Markdown
Collaborator Author

/te-ci L0

@Oleg-Goncharov Oleg-Goncharov left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just two minor comments. The performance gains look impressive!

As a future improvement, we could implement a grouped kernel to reduce the overhead of multiple kernel launches.

@vthumbe1503

Copy link
Copy Markdown
Collaborator Author

/te-ci L0

@Oleg-Goncharov
Oleg-Goncharov self-requested a review November 20, 2026 11:24

@Oleg-Goncharov Oleg-Goncharov left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Oleg-Goncharov Oleg-Goncharov changed the title Swizzle + padding fusion for Unpadded Input Scales [Common] Swizzle + padding fusion for Unpadded Input Scales Nov 20, 2026
@Oleg-Goncharov
Oleg-Goncharov merged commit 6af366a into NVIDIA:main Nov 20, 2026
35 of 38 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants