Repository navigation
feat(nvfp4): support a 32-element scale block (MXFP4 E8M0) - #3
Merged
Merged
Conversation
`tid_Y_colwise` is computed with a bare `% 4`, which is `BUFF_IN_DIM_Y / SCALE_DIM` -- the number of SCALE_DIM-tall windows in the input buffer -- written out as a literal. It is correct today only because that quotient is 4 at the current SCALE_DIM = 16. At a 32-element block the window count is 2, so tid_Y_colwise still reaches 3 and the derived offsets walk past the buffers: thread_offset_Y_colwise reaches 96 against a 64-row sIn, and out_tr_thread_offset_X reaches 48 against a BUFF_OUT_TR_DIM_X of 32. The 8-byte shared store then crosses the row and runs off the end of sOut_tr into whatever is allocated next -- measured, that is the rowwise scale array, whose rows 0/1 and 4/5 are overwritten while 2/3 stay intact, matching the two overflow addresses. compute-sanitizer reports no error, because the store stays inside the dynamic shared allocation as a whole. Behaviour is unchanged at SCALE_DIM = 16 (the derived value is 4), so this is a correctness-of-expression change with no numerical effect. The static_assert pins the divisibility the derivation depends on. This is the first commit of the block-size change that follows.
MXFP4 places one power-of-two scale per 32 elements; NVFP4 places one E4M3 scale per 16. Only the second was representable, and not for licensing reasons: the block size is baked into three successive dtype guards sitting on top of three hard-coded 16-element geometries, so lifting one guard only exposed the next. The kernel carried a single SCALE_DIM that meant two things at once -- how many elements one thread packs (the payload layout, rIn[2][SCALE_DIM], rOut[SCALE_DIM/8]) and how many elements one scale covers (SCALES_PER_CHUNK_*, SCALES_PER_TILE_*, THREADS_PER_SCALE_ROWWISE, the transpose window count). NVFP4 uses 16 for both, so the conflation was invisible. They are now separate: ELTS_PER_THREAD (fixed 16) keeps the payload-side uses, and ScaleGeom<SD> carries the scale geometry for a block size SD with its divisibility pinned by static_assert. The transpose thread layout follows the block size: THREADS_X_TR = THREADS_NUM / SCALES_PER_TILE_Y makes THREADS_Y_TR equal the scale rows per tile, so ITERATIONS_TR is 1 at both sizes -- TILE_DIM_X / 2 at 16, the historical layout, and TILE_DIM_X at 32. When one thread does not cover a whole scale, the threads sharing it must agree: each reads its partner's half-window from shared memory and folds the magnitudes into the same amax. Shuffles are avoided deliberately -- lanes can return early and __shfl_xor_sync across a divergent warp is undefined. The block size is compile-time because it shapes the tile arithmetic, while the scale dtype is only known at runtime, so both instantiations are emitted and the call site selects on the output scale dtype. The historical SCALE_DIM / SCALES_PER_CHUNK_* names and the 16 defaults remain for the grouped quantise path, which borrows this namespace and only uses the 16-element block. Verified bit-identical to the previously accepted artifact in all four configurations: NVFP4 and MXFP4 at 256x256 and 2304x2560, dequantised payload and scales matching exactly, NVFP4 relerr unchanged at 0.0948/0.0952, MXFP4 at 0.1145/0.1153 with scale shape (2304, 80).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds support for a scale block wider than one thread's payload, which is what MXFP4 needs: it places one
power-of-two scale per 32 elements, where NVFP4 places one E4M3 scale per 16.
Only the 16-element case was representable. Not for licensing reasons — the block size is baked into three
successive dtype guards sitting on top of three hard-coded 16-element geometries, so lifting one guard only
exposed the next.
Commit 1 — the hidden coupling
tid_Y_colwisewas computed with a bare% 4, which isBUFF_IN_DIM_Y / SCALE_DIMwritten as a literal. It iscorrect today only because that quotient is 4 at
SCALE_DIM = 16. At a 32-element block the window count is 2, sothe index still reaches 3 and the derived offsets walk past the buffers:
thread_offset_Y_colwisereaches 96against a 64-row
sIn, andout_tr_thread_offset_Xreaches 48 against aBUFF_OUT_TR_DIM_Xof 32. The 8-byteshared store crosses the row and runs off the end of
sOut_trinto whatever is allocated next — measured, that isthe rowwise scale array, whose rows 0/1 and 4/5 are overwritten while 2/3 stay intact, matching the two overflow
addresses.
compute-sanitizer --tool memcheckreports no error, because the store stays inside the dynamicshared allocation as a whole.
Behaviour is unchanged at
SCALE_DIM = 16; this commit only replaces the literal with what it always meant andpins the divisibility it depends on. It is first because the change that follows is what makes the literal a real
out-of-bounds write.
Commit 2 — the block size as a parameter
SCALE_DIMmeant two things at once: how many elements one thread packs (which fixes the payload layout,rIn[2][SCALE_DIM],rOut[SCALE_DIM/8]) and how many elements one scale covers (SCALES_PER_CHUNK_*,SCALES_PER_TILE_*,THREADS_PER_SCALE_ROWWISE, the transpose window count). NVFP4 uses 16 for both, so theconflation was invisible. They are now separate:
ELTS_PER_THREAD(fixed 16) keeps every payload-side use;ScaleGeom<SD>carries the scale geometry for a block sizeSD, with the divisibility the derivation dependson pinned by
static_assertrather than assumed;ScalesType2D/ScalesTypeTr2Dand the device helpers takeSD, so both block sizes live in one translationunit rather than a duplicated one.
The transpose thread layout follows the block size:
THREADS_X_TR = THREADS_NUM / SCALES_PER_TILE_YmakesTHREADS_Y_TRequal the number of scale rows per tile, soITERATIONS_TRis 1 at both sizes —TILE_DIM_X / 2(= 32) at 16, which is the historical layout exactly, and
TILE_DIM_X(= 64) at 32.When one thread does not cover a whole scale, the threads sharing it must agree on one scale: each reads its
partner's half-window out of shared memory and folds the magnitudes into the same amax. Shuffles are avoided
deliberately — lanes can return early, and
__shfl_xor_syncacross a divergent warp is undefined.The block size is a compile-time property because it shapes the tile arithmetic, while the scale dtype is only
known at runtime at the call site, so both instantiations are emitted and the 32-element one is selected for
kFloat8E8M0scales.The historical
SCALE_DIM/SCALES_PER_CHUNK_*names and the 16 defaults on the aliases and device templatesremain for the grouped quantise path, which borrows this namespace and only ever uses the 16-element block.
Verification
Bit-for-bit against the artifact that is already deployed and already accepted, across both block sizes and two
shapes — a stronger statement than numerical agreement:
The scale tensor for
(2304, 2560)at a 32-element block is(2304, 80), i.e.2304 x 2560/32.Note
custom_recipes/reference_nvfp4.py:1022raisesUnboundLocalError: row_inputon this branch point whenpow_2_scales=True, so the reference comparator could not be used; the comparison above is against a priorbuild of this kernel instead. That defect is unrelated to this change and pre-exists it.