Repository navigation
[Common] Swizzle + padding fusion for Unpadded Input Scales - #3768
Merged
Oleg-Goncharov merged 14 commits intoNov 20, 2026
Merged
Conversation
Use per-tile swizzle kernels for small MXFP8 scale tensors and multi-tensor cases. Require 16-byte alignment for input and output scale buffers selected by the swizzle. Signed-off-by: Varun Thumbe <vthumbe@nvidia.com>
Signed-off-by: Varun Thumbe <vthumbe@nvidia.com>
Signed-off-by: Varun Thumbe <vthumbe@nvidia.com>
for more information, see https://pre-commit.ci
Signed-off-by: Varun Thumbe <vthumbe@nvidia.com>
Signed-off-by: Varun Thumbe <vthumbe@nvidia.com>
Signed-off-by: Varun Thumbe <vthumbe@nvidia.com>
vthumbe1503
marked this pull request as ready for review
November 18, 2026 06:56
Collaborator
Author
|
/te-ci L0 |
Contributor
[Medium impact] The PR appears safe to merge; no actionable issues were found. SummaryThis PR lets MXFP8 and NVFP4 scale swizzling fill missing outer-dimension padding while producing GEMM-ready scales.
No new actionable issues found. GPU tests were not run. Reviews (6) · Last reviewed commit: "Merge branch 'main' into swizzle-compact..." · Reviewed by Greptile |
Signed-off-by: Varun Thumbe <vthumbe@nvidia.com>
Collaborator
Author
|
/te-ci L0 |
Signed-off-by: vthumbe1503 <vthumbe@nvidia.com>
Collaborator
Author
|
/te-ci L0 |
Oleg-Goncharov
left a comment
Collaborator
There was a problem hiding this comment.
LGTM, just two minor comments. The performance gains look impressive!
As a future improvement, we could implement a grouped kernel to reduce the overhead of multiple kernel launches.
Signed-off-by: Varun Thumbe <vthumbe@nvidia.com>
Collaborator
Author
|
/te-ci L0 |
Oleg-Goncharov
self-requested a review
November 20, 2026 11:24
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Allow MXFP8 scale swizzling to consume unswizzled scale buffers that omit outer-dimension padding and produce fully padded GEMM-swizzled output. This enables consumers such as FSDP to avoid restoring that padding in a separate operation after allgather when the scales will immediately be swizzled.
Primary comparison: separate re-padding versus fused padding/swizzle
All times are µs per API call; multi-tensor timings cover the whole batch, not one tensor. Speedup = separate time / fused time, so larger is better.
Fusion is faster in all 35 configurations: 1.84–13.39× overall; 1.84–2.63× for single-tensor cases and 4.86–13.39× for same-shape multi-tensor batches.
MXFP8 rowwise
MXFP8 columnwise
MXFP8 multi-tensor rowwise
MXFP8 multi-tensor columnwise
Secondary comparison: cost of reading unpadded scales
This answers the separate question of whether the fused swizzle itself is slower than swizzling an already-padded input. Neither side below includes a preceding padding operation. Both run on the PR head, with the same logical values and output dimensions.
This was a separate paired measurement pass from the primary comparison, so its fused timings can differ slightly. Percent change = unpadded time / already-padded time − 1; positive means slower. Do not derive the separate-padding cost by subtracting values across the two passes.
All 35 already-padded versus unpadded comparisons
MXFP8 rowwise
MXFP8 columnwise
MXFP8 multi-tensor rowwise
MXFP8 multi-tensor columnwise
The largest uniform-case overhead versus already-padded input is 18.6% rowwise / 12.5% columnwise; some cases are unchanged or faster. The primary comparison shows that eliminating explicit re-padding more than compensates for this overhead in every tested supported configuration.
Type of change
Changes
Checklist: