Skip to content

Optimize grouped FP8 blockwise quantization on Hopper - #3905

Open
anaruse wants to merge 2 commits into
NVIDIA:mainfrom
anaruse:feature/hopper-grouped-fp8-quantization
Open

anaruse wants to merge 2 commits into
NVIDIA:mainfrom
anaruse:feature/hopper-grouped-fp8-quantization

Conversation

@anaruse

@anaruse anaruse commented Nov 20, 2026

Copy link
Copy Markdown

Description

Optimize BF16-to-FP8 1D blockwise quantization on Hopper by processing multiple matrices in a single kernel launch. Eligible calls through the existing split and multi-tensor APIs use this path automatically, without public API changes.

Type of change

  • New feature (non-breaking change which adds functionality)

Changes

  • Batch matrices with a common K and variable M into grouped launches.

  • Specialize output modes at compile time and use packed BF16x2 operations for local amax.

  • Support E4M3/E5M2, empty tensors, partial tiles, and separate or bulk allocations.

The optimization requires both row-wise and column-wise GEMM-ready outputs. Other configurations retain their existing paths. Set NVTE_FP8_BLOCK_QUANTIZATION_GROUPED=0 to disable it.

CUDA Graph replay requires fixed shapes and buffer addresses; input values may change in place.

Validation

Tested on H100:

python -m pytest -q -x -p no:cacheprovider \
  tests/pytorch/test_float8blockwisetensor.py -k grouped
# 129 passed, 208 deselected
 
python -m pytest -q -x -p no:cacheprovider \
  tests/pytorch/test_float8blockwisetensor.py
# 305 passed, 32 xpassed

Tests cover exact FP8 payload and valid inverse-scale comparisons, special values, scaling boundaries, fallback behavior, and CUDA Graph replay.

Lower-target PTX JIT execution on Hopper has not yet been validated. Test runs report an existing FlashAttention 4 version warning.

Performance

Measured on a H100 80GB HBM3 with CUDA Graph enabled. Inputs are BF16, outputs are FP8(E4M3), and each call produces both row-wise and column-wise outputs with a quantization block size of 128.

Experts Matrix shape per expert (M × K) Existing implementation [µs] Optimized implementation [µs] Latency reduction
16 2048 × 2048 176.9 96.2 45.6%
16 2048 × 768 115.4 40.4 65.0%
16 2048 × 1536 159.2 71.4 55.2%

Times are mean GPU elapsed times for all 16 experts, from the start of the first quantization kernel to completion of the last.

Checklist:

  • I have read and followed the contributing guidelines
  • The functionality is complete
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

Add a grouped BF16-to-FP8 kernel for compatible matrices with a common K
and variable M, producing both rowwise and columnwise GEMM-ready outputs.
Specialize output modes and use packed BF16x2 amax operations.

Select the optimized path automatically through existing quantization APIs,
preserving fallback behavior for unsupported configurations. Document the
dispatch control and add coverage for boundary shapes, empty tensors,
fallbacks, and CUDA Graph replay.
Compile the grouped kernel body for lower PTX targets while retaining
runtime dispatch on Hopper.

Clarify that CUDA Graph replay uses captured shapes and buffer addresses.
Add special-value equivalence tests for packed BF16 amax and vary matrix
width independently of power-of-two scaling.
@github-actions github-actions Bot added the community-contribution PRs from external contributor outside the core maintainers, representing community-driven work. label Nov 20, 2026
@greptile-apps

greptile-apps Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no actionable issues were found.

Summary

Adds automatic grouped BF16-to-FP8 blockwise quantization on Hopper for compatible split and multi-tensor calls.

  • Hopper groups compatible BF16 matrices into one FP8 quantization launch.

No actionable issues found. GPU tests were not run during this review.

Reviews (1) · Last reviewed commit: "Fix grouped FP8 quantization PTX compati..." · Reviewed by Greptile

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-contribution PRs from external contributor outside the core maintainers, representing community-driven work.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant