Repository navigation
feat(module): allow parameters to be stored in 4 bits - #4
Merged
Merged
Conversation
There was no way to store a model parameter in 4 bits. `fp8_model_init` covers fp8 storage only, and a CustomRecipe quantizer factory is a per-GEMM-slot *compute* quantiser: it quantises activations and weights for a GEMM and never allocates a parameter. So a model could compute in FP4 while every weight in memory stayed bf16. Parameter quantisation has to happen in `reset_parameters()`, where the high-precision weight has just been built, which is why a `params_dtype` passed at construction does not prevent the storage swap -- the parameter is built in `params_dtype` and then replaced here, exactly as the fp8 branch below already does. With `primary_weights_in_fp4 = "nvfp4"`, every 2-D parameter is quantised to NVFP4Tensor. Biases and any 1-D parameter stay in `params_dtype` on purpose. NVFP4Tensor is the 4-bit type that survives being wrapped as an `nn.Parameter` (measured); the reference implementation's tensor raises on `detach` and cannot be one, which is why this offers NVFP4 and not an E8M0 block-32 format. Measured on DeepSeek-V4.1-Flash, whose 384 routed experts dominate the footprint: ``` expert_routed 129.4 GiB bf16 -> 35.6 GiB bytes_per_param 2.001 -> 1.013 attention stays bf16 embed_out stays bf16 ``` Without this branch the same model settles 56,738 MiB higher on device and a 128K sequence no longer fits, which is how the absence was found: the switch is set by the caller and silently ignored here.
…e used NVFP4 pairs E2M1 elements with an E4M3 scale per 16 elements. MXFP4 uses the same elements with a power-of-two E8M0 scale per 32, and nothing in this codebase accepted that scale format: the Python constructor, the C++ constructor, the output-tensor checks and the dispatch guards each refused it, so each one had to be lifted before the next was even reached. This adds E8M0 support across that surface. Four things are worth calling out because none of them is obvious from the diff: **The scale shape follows the scale dtype**, at every site that computes it. A 32-element block has half as many scales per row as a 16-element one, and that has to be derived from the dtype rather than assumed. **E8M0's largest finite value is 2^127, which an int cannot hold**, and C++17 forbids the constexpr float-to-int conversion the existing extrema path performed. The max is therefore taken from the type's own definition rather than through that conversion. **Scale rounding goes through the bit pattern, not through `exp2f`/`ceilf`/`log2f`.** In this build those three compile under fast-math in a way that stores byte 0 for every block's scale -- measured, and the reason the first attempt produced an unusable tensor rather than a slightly wrong one. E8M0 is a biased exponent, so the scale is written as `_byte = _e + 127` directly. **The runtime scale-max equality check is vacuous for E8M0** (2^127 has no integer form to compare against) and is skipped for that case rather than silently comparing garbage. Dequantisation for E8M0 has no global factor, and the kernel's 16-element unit is deliberately unchanged: elements per thread and elements per scale are separate quantities, which the block-size change earlier in this series made explicit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
There was no way to store a model parameter in 4 bits.
fp8_model_initcovers fp8 storage only, and a CustomRecipe quantizer factory is a per-GEMM-slot computequantiser: it quantises activations and weights for a GEMM and never allocates a parameter. So a model could
compute in FP4 while every weight in memory stayed bf16.
Parameter quantisation has to happen in
reset_parameters(), where the high-precision weight has just beenbuilt — which is why a
params_dtypepassed at construction does not prevent the storage swap. The parameteris built in
params_dtypeand then replaced here, exactly as the fp8 branch below already does it.With
primary_weights_in_fp4 = "nvfp4", every 2-D parameter is quantised toNVFP4Tensor. Biases and any 1-Dparameter stay in
params_dtypeon purpose.NVFP4Tensoris the 4-bit type that survives being wrapped as annn.Parameter(measured); the reference implementation's tensor raises ondetachand cannot be one, which iswhy this offers NVFP4 and not an E8M0 block-32 format.
Measured
On DeepSeek-V4.1-Flash, whose 384 routed experts dominate the footprint — same model, same recipe, two builds
of this branch point, one without this commit:
The absence of this branch is silent: the switch is set by the caller and
getattris simply never consulted,with no error and no warning. It was found by comparing the resident-dtype census between two builds, not by
anything failing loudly.