Skip to content

feat(nvfp4): support a 32-element scale block (MXFP4 E8M0) - #3

Merged
NolanHo merged 2 commits into
mainfrom
feat/mxfp4-32-element-scales-final
Nov 18, 2026
Merged

NolanHo merged 2 commits into
mainfrom
feat/mxfp4-32-element-scales-final

Conversation

@NolanHo

@NolanHo NolanHo commented Nov 18, 2026

Copy link
Copy Markdown

Adds support for a scale block wider than one thread's payload, which is what MXFP4 needs: it places one
power-of-two scale per 32 elements, where NVFP4 places one E4M3 scale per 16.

Only the 16-element case was representable. Not for licensing reasons — the block size is baked into three
successive dtype guards sitting on top of three hard-coded 16-element geometries, so lifting one guard only
exposed the next.

Commit 1 — the hidden coupling

tid_Y_colwise was computed with a bare % 4, which is BUFF_IN_DIM_Y / SCALE_DIM written as a literal. It is
correct today only because that quotient is 4 at SCALE_DIM = 16. At a 32-element block the window count is 2, so
the index still reaches 3 and the derived offsets walk past the buffers: thread_offset_Y_colwise reaches 96
against a 64-row sIn, and out_tr_thread_offset_X reaches 48 against a BUFF_OUT_TR_DIM_X of 32. The 8-byte
shared store crosses the row and runs off the end of sOut_tr into whatever is allocated next — measured, that is
the rowwise scale array, whose rows 0/1 and 4/5 are overwritten while 2/3 stay intact, matching the two overflow
addresses. compute-sanitizer --tool memcheck reports no error, because the store stays inside the dynamic
shared allocation as a whole.

Behaviour is unchanged at SCALE_DIM = 16; this commit only replaces the literal with what it always meant and
pins the divisibility it depends on. It is first because the change that follows is what makes the literal a real
out-of-bounds write.

Commit 2 — the block size as a parameter

SCALE_DIM meant two things at once: how many elements one thread packs (which fixes the payload layout,
rIn[2][SCALE_DIM], rOut[SCALE_DIM/8]) and how many elements one scale covers (SCALES_PER_CHUNK_*,
SCALES_PER_TILE_*, THREADS_PER_SCALE_ROWWISE, the transpose window count). NVFP4 uses 16 for both, so the
conflation was invisible. They are now separate:

  • ELTS_PER_THREAD (fixed 16) keeps every payload-side use;
  • ScaleGeom<SD> carries the scale geometry for a block size SD, with the divisibility the derivation depends
    on pinned by static_assert rather than assumed;
  • ScalesType2D / ScalesTypeTr2D and the device helpers take SD, so both block sizes live in one translation
    unit rather than a duplicated one.

The transpose thread layout follows the block size: THREADS_X_TR = THREADS_NUM / SCALES_PER_TILE_Y makes
THREADS_Y_TR equal the number of scale rows per tile, so ITERATIONS_TR is 1 at both sizes — TILE_DIM_X / 2
(= 32) at 16, which is the historical layout exactly, and TILE_DIM_X (= 64) at 32.

When one thread does not cover a whole scale, the threads sharing it must agree on one scale: each reads its
partner's half-window out of shared memory and folds the magnitudes into the same amax. Shuffles are avoided
deliberately — lanes can return early, and __shfl_xor_sync across a divergent warp is undefined.

The block size is a compile-time property because it shapes the tile arithmetic, while the scale dtype is only
known at runtime at the call site, so both instantiations are emitted and the 32-element one is selected for
kFloat8E8M0 scales.

The historical SCALE_DIM / SCALES_PER_CHUNK_* names and the 16 defaults on the aliases and device templates
remain for the grouped quantise path, which borrows this namespace and only ever uses the 16-element block.

Verification

Bit-for-bit against the artifact that is already deployed and already accepted, across both block sizes and two
shapes — a stronger statement than numerical agreement:

                              this branch                        previously accepted
(256,256)    nvfp4  deq  f919a517dcf803e4  scl 4c0f9434dcd7cb65   identical    relerr 0.0948
(256,256)    mxfp4  deq  4442b39f50a9cbd8  scl 3b09022e57af0adf   identical    relerr 0.1145
(2304,2560)  nvfp4  deq  de1807175af2b884  scl 16da91513c020544   identical    relerr 0.0952
(2304,2560)  mxfp4  deq  1ac77c1236d0d301  scl 20f31bd7d5fd0583   identical    relerr 0.1153  scale (2304, 80)

The scale tensor for (2304, 2560) at a 32-element block is (2304, 80), i.e. 2304 x 2560/32.

Note

custom_recipes/reference_nvfp4.py:1022 raises UnboundLocalError: row_input on this branch point when
pow_2_scales=True, so the reference comparator could not be used; the comparison above is against a prior
build of this kernel instead. That defect is unrelated to this change and pre-exists it.

`tid_Y_colwise` is computed with a bare `% 4`, which is `BUFF_IN_DIM_Y / SCALE_DIM` -- the number
of SCALE_DIM-tall windows in the input buffer -- written out as a literal. It is correct today only
because that quotient is 4 at the current SCALE_DIM = 16.

At a 32-element block the window count is 2, so tid_Y_colwise still reaches 3 and the derived
offsets walk past the buffers: thread_offset_Y_colwise reaches 96 against a 64-row sIn, and
out_tr_thread_offset_X reaches 48 against a BUFF_OUT_TR_DIM_X of 32. The 8-byte shared store then
crosses the row and runs off the end of sOut_tr into whatever is allocated next -- measured, that is
the rowwise scale array, whose rows 0/1 and 4/5 are overwritten while 2/3 stay intact, matching the
two overflow addresses. compute-sanitizer reports no error, because the store stays inside the
dynamic shared allocation as a whole.

Behaviour is unchanged at SCALE_DIM = 16 (the derived value is 4), so this is a
correctness-of-expression change with no numerical effect. The static_assert pins the divisibility
the derivation depends on.

This is the first commit of the block-size change that follows.
MXFP4 places one power-of-two scale per 32 elements; NVFP4 places one E4M3 scale per 16. Only the
second was representable, and not for licensing reasons: the block size is baked into three
successive dtype guards sitting on top of three hard-coded 16-element geometries, so lifting one
guard only exposed the next.

The kernel carried a single SCALE_DIM that meant two things at once -- how many elements one thread
packs (the payload layout, rIn[2][SCALE_DIM], rOut[SCALE_DIM/8]) and how many elements one scale
covers (SCALES_PER_CHUNK_*, SCALES_PER_TILE_*, THREADS_PER_SCALE_ROWWISE, the transpose window
count). NVFP4 uses 16 for both, so the conflation was invisible. They are now separate:
ELTS_PER_THREAD (fixed 16) keeps the payload-side uses, and ScaleGeom<SD> carries the scale geometry
for a block size SD with its divisibility pinned by static_assert.

The transpose thread layout follows the block size: THREADS_X_TR = THREADS_NUM / SCALES_PER_TILE_Y
makes THREADS_Y_TR equal the scale rows per tile, so ITERATIONS_TR is 1 at both sizes -- TILE_DIM_X
/ 2 at 16, the historical layout, and TILE_DIM_X at 32.

When one thread does not cover a whole scale, the threads sharing it must agree: each reads its
partner's half-window from shared memory and folds the magnitudes into the same amax. Shuffles are
avoided deliberately -- lanes can return early and __shfl_xor_sync across a divergent warp is
undefined.

The block size is compile-time because it shapes the tile arithmetic, while the scale dtype is only
known at runtime, so both instantiations are emitted and the call site selects on the output scale
dtype. The historical SCALE_DIM / SCALES_PER_CHUNK_* names and the 16 defaults remain for the
grouped quantise path, which borrows this namespace and only uses the 16-element block.

Verified bit-identical to the previously accepted artifact in all four configurations: NVFP4 and
MXFP4 at 256x256 and 2304x2560, dequantised payload and scales matching exactly, NVFP4 relerr
unchanged at 0.0948/0.0952, MXFP4 at 0.1145/0.1153 with scale shape (2304, 80).
@NolanHo
NolanHo merged commit 06703e7 into main Nov 18, 2026
9 checks passed
@NolanHo
NolanHo deleted the feat/mxfp4-32-element-scales-final branch November 18, 2026 06:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant