Skip to content

feat(module): allow parameters to be stored in 4 bits - #4

Merged
NolanHo merged 2 commits into
mainfrom
feat/fp4-resident-params
Nov 18, 2026
Merged

NolanHo merged 2 commits into
mainfrom
feat/fp4-resident-params

Conversation

@NolanHo

@NolanHo NolanHo commented Nov 18, 2026

Copy link
Copy Markdown

There was no way to store a model parameter in 4 bits.

fp8_model_init covers fp8 storage only, and a CustomRecipe quantizer factory is a per-GEMM-slot compute
quantiser: it quantises activations and weights for a GEMM and never allocates a parameter. So a model could
compute in FP4 while every weight in memory stayed bf16.

Parameter quantisation has to happen in reset_parameters(), where the high-precision weight has just been
built — which is why a params_dtype passed at construction does not prevent the storage swap. The parameter
is built in params_dtype and then replaced here, exactly as the fp8 branch below already does it.

With primary_weights_in_fp4 = "nvfp4", every 2-D parameter is quantised to NVFP4Tensor. Biases and any 1-D
parameter stay in params_dtype on purpose. NVFP4Tensor is the 4-bit type that survives being wrapped as an
nn.Parameter (measured); the reference implementation's tensor raises on detach and cannot be one, which is
why this offers NVFP4 and not an E8M0 block-32 format.

Measured

On DeepSeek-V4.1-Flash, whose 384 routed experts dominate the footprint — same model, same recipe, two builds
of this branch point, one without this commit:

                          without        with
GPU settled               148,976 MiB   92,238 MiB
bytes_per_param               1.976        1.013
  NVFP4Tensor                  0.91        36.51
  bfloat16                   185.62        59.06
128K sequence            does not fit   41.1 s median (41.0-41.1 over 3 runs)

The absence of this branch is silent: the switch is set by the caller and getattr is simply never consulted,
with no error and no warning. It was found by comparing the resident-dtype census between two builds, not by
anything failing loudly.

There was no way to store a model parameter in 4 bits. `fp8_model_init` covers fp8 storage only,
and a CustomRecipe quantizer factory is a per-GEMM-slot *compute* quantiser: it quantises activations
and weights for a GEMM and never allocates a parameter. So a model could compute in FP4 while every
weight in memory stayed bf16.

Parameter quantisation has to happen in `reset_parameters()`, where the high-precision weight has
just been built, which is why a `params_dtype` passed at construction does not prevent the storage
swap -- the parameter is built in `params_dtype` and then replaced here, exactly as the fp8 branch
below already does.

With `primary_weights_in_fp4 = "nvfp4"`, every 2-D parameter is quantised to NVFP4Tensor. Biases and
any 1-D parameter stay in `params_dtype` on purpose. NVFP4Tensor is the 4-bit type that survives being
wrapped as an `nn.Parameter` (measured); the reference implementation's tensor raises on `detach` and
cannot be one, which is why this offers NVFP4 and not an E8M0 block-32 format.

Measured on DeepSeek-V4.1-Flash, whose 384 routed experts dominate the footprint:

```
expert_routed      129.4 GiB bf16  ->  35.6 GiB
bytes_per_param    2.001  ->  1.013
attention          stays bf16       embed_out stays bf16
```

Without this branch the same model settles 56,738 MiB higher on device and a 128K sequence no longer
fits, which is how the absence was found: the switch is set by the caller and silently ignored here.
…e used

NVFP4 pairs E2M1 elements with an E4M3 scale per 16 elements. MXFP4 uses the same elements with a
power-of-two E8M0 scale per 32, and nothing in this codebase accepted that scale format: the Python
constructor, the C++ constructor, the output-tensor checks and the dispatch guards each refused it,
so each one had to be lifted before the next was even reached.

This adds E8M0 support across that surface. Four things are worth calling out because none of them
is obvious from the diff:

**The scale shape follows the scale dtype**, at every site that computes it. A 32-element block has
half as many scales per row as a 16-element one, and that has to be derived from the dtype rather
than assumed.

**E8M0's largest finite value is 2^127, which an int cannot hold**, and C++17 forbids the constexpr
float-to-int conversion the existing extrema path performed. The max is therefore taken from the
type's own definition rather than through that conversion.

**Scale rounding goes through the bit pattern, not through `exp2f`/`ceilf`/`log2f`.** In this build
those three compile under fast-math in a way that stores byte 0 for every block's scale -- measured,
and the reason the first attempt produced an unusable tensor rather than a slightly wrong one. E8M0
is a biased exponent, so the scale is written as `_byte = _e + 127` directly.

**The runtime scale-max equality check is vacuous for E8M0** (2^127 has no integer form to compare
against) and is skipped for that case rather than silently comparing garbage.

Dequantisation for E8M0 has no global factor, and the kernel's 16-element unit is deliberately
unchanged: elements per thread and elements per scale are separate quantities, which the block-size
change earlier in this series made explicit.
@NolanHo
NolanHo merged commit 8d7fac1 into main Nov 18, 2026
7 of 9 checks passed
@NolanHo
NolanHo deleted the feat/fp4-resident-params branch November 18, 2026 16:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant