Skip to content

[Common] Use VR TMA override instruction for quantization kernels - #3879

Open
Oleg-Goncharov wants to merge 9 commits into
NVIDIA:mainfrom
Oleg-Goncharov:pr_rubin_tma_override
Open

Oleg-Goncharov wants to merge 9 commits into
NVIDIA:mainfrom
Oleg-Goncharov:pr_rubin_tma_override

Conversation

@Oleg-Goncharov

@Oleg-Goncharov Oleg-Goncharov commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Use Rubin (SM107) instruction-local TMA overrides in native grouped MXFP8 quantization, dequantization, and requantization. For groups with varying last or both dimensions, pass the stage address, dimensions, and stride directly to the TMA load or store. This removes the descriptor-update kernel and per-group descriptor publication and acquire on that path.

Type of change

  • Documentation change (change only to the documentation, either a fix or a new content)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Infra/Build change
  • Code refactoring

Changes

  • Add Rubin TMA load and store wrappers and enable the grouped override path on SM107 with CUDA 13.4 or later.
  • Share override operand construction and load/store dispatch across the three grouped operations. Bypass the legacy 64-entry descriptor table on the Rubin override path.
  • Use small template tensor maps to avoid the CUDA 13.5 large-allocation class. Bound each transfer to the stage and handle out-of-bounds loads and stores.
  • Generate sm_107a for architecture-specific kernels when CMake is configured with 107, and deduplicate architecture-specific targets.

Checklist:

  • I have read and followed the contributing guidelines
  • The functionality is complete
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

Oleg-Goncharov and others added 5 commits November 17, 2026 17:35
Signed-off-by: Oleg Goncharov <ogoncharov@nvidia.com>
Signed-off-by: Oleg Goncharov <ogoncharov@nvidia.com>
Signed-off-by: Oleg Goncharov <ogoncharov@nvidia.com>
Signed-off-by: Oleg Goncharov <ogoncharov@nvidia.com>
pre-commit-ci Bot and others added 4 commits November 20, 2026 14:50
Signed-off-by: Oleg Goncharov <ogoncharov@nvidia.com>
Signed-off-by: Oleg Goncharov <ogoncharov@nvidia.com>
@Oleg-Goncharov
Oleg-Goncharov marked this pull request as ready for review November 20, 2026 16:22
@greptile-apps

greptile-apps Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 4/5

[Medium impact] The new transfer path reaches three grouped MXFP8 operations on Rubin.

The PR appears safe to merge, with a non-blocking gap in tests for groups above 64 members.

Findings

  1. P2 Larger groups lack test coverage ▶

Summary

Adds Rubin instruction-local TMA overrides to grouped MXFP8 quantization, dequantization, and requantization.

  • Rubin grouped transfers pass tensor details directly to TMA instructions.
  • Grouped MXFP8 quantization uses direct TMA overrides on Rubin.
  • Grouped MXFP8 dequantization uses direct TMA overrides on Rubin.
  • Grouped MXFP8 requantization uses direct TMA overrides on Rubin.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Grouped MXFP8 operation] --> B{Rubin and CUDA 13.4 or later?}
  B -->|Yes, varying width| C[Create small template maps]
  C --> D[Read group dimensions and offsets]
  D --> E[Transfer each stage with local overrides]
  B -->|No, varying width| F[Update per-group descriptor table]
  F --> G[Acquire descriptors and transfer stages]
  B -->|Common width| H[Use existing single-map path]
Loading

Reviews (1) · Last reviewed commit: "[pre-commit.ci] auto fixes from pre-comm..." · Reviewed by Greptile

Comment on lines +523 to +525
if (test_case.shapes.size() > 64 && test::getDeviceComputeCapability() != 107) {
GTEST_SKIP() << "More than 64 grouped TMA descriptors requires Rubin overrides.";
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Larger groups lack test coverage

The new skip never runs: kGroupShapeCases has at most 40 members, and both varying-width cases have only three. Add cases with at least 65 members for VARYING_LAST_DIM and VARYING_BOTH_DIMS, including empty members near the old limit. Otherwise, this suite cannot catch regressions in the newly allowed path that bypasses the 64-entry table.

Knowledge Base Used: Verification and CI matrix

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant