Repository navigation
Conversation
Add a grouped BF16-to-FP8 kernel for compatible matrices with a common K and variable M, producing both rowwise and columnwise GEMM-ready outputs. Specialize output modes and use packed BF16x2 amax operations. Select the optimized path automatically through existing quantization APIs, preserving fallback behavior for unsupported configurations. Document the dispatch control and add coverage for boundary shapes, empty tensors, fallbacks, and CUDA Graph replay.
Compile the grouped kernel body for lower PTX targets while retaining runtime dispatch on Hopper. Clarify that CUDA Graph replay uses captured shapes and buffer addresses. Add special-value equivalence tests for packed BF16 amax and vary matrix width independently of power-of-two scaling.
anaruse
requested review from
Oleg-Goncharov,
ksivaman and
ptrendx
as code owners
November 20, 2026 06:31
Contributor
[Medium impact] The PR appears safe to merge; no actionable issues were found. SummaryAdds automatic grouped BF16-to-FP8 blockwise quantization on Hopper for compatible split and multi-tensor calls.
No actionable issues found. GPU tests were not run during this review. Reviews (1) · Last reviewed commit: "Fix grouped FP8 quantization PTX compati..." · Reviewed by Greptile |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Optimize BF16-to-FP8 1D blockwise quantization on Hopper by processing multiple matrices in a single kernel launch. Eligible calls through the existing split and multi-tensor APIs use this path automatically, without public API changes.
Type of change
Changes
Batch matrices with a common K and variable M into grouped launches.
Specialize output modes at compile time and use packed BF16x2 operations for local amax.
Support E4M3/E5M2, empty tensors, partial tiles, and separate or bulk allocations.
The optimization requires both row-wise and column-wise GEMM-ready outputs. Other configurations retain their existing paths. Set
NVTE_FP8_BLOCK_QUANTIZATION_GROUPED=0to disable it.CUDA Graph replay requires fixed shapes and buffer addresses; input values may change in place.
Validation
Tested on H100:
Tests cover exact FP8 payload and valid inverse-scale comparisons, special values, scaling boundaries, fallback behavior, and CUDA Graph replay.
Lower-target PTX JIT execution on Hopper has not yet been validated. Test runs report an existing FlashAttention 4 version warning.
Performance
Measured on a H100 80GB HBM3 with CUDA Graph enabled. Inputs are BF16, outputs are FP8(E4M3), and each call produces both row-wise and column-wise outputs with a quantization block size of 128.
Times are mean GPU elapsed times for all 16 experts, from the start of the first quantization kernel to completion of the last.
Checklist: