NF12 is a lossless, fixed-length 12-bit encoding of BF16 weight matrices,
based on "NyoomFloat12: Accelerating LLM Inference via Lossless 12-bit Weight
Compression" (Liberman, Fang, Zhang, Dao, Fu, Together AI, 2026). This library
implements this data type efficiently on NVIDIA GPUs, from the Ampere generation
up until Blackwell, on AMD RDNA4 (gfx1200 / gfx1201) via HIP, and on Apple
Silicon GPUs through Metal for PyTorch MPS and MLX (see docs/METAL.md).
This library's bit and memory layout (tiled) are not compatible with the one described in the paper. You can read the spec for more info on the format.
- NVIDIA GPU with SM80 or newer (Ampere and later), an AMD RDNA4 GPU
(
gfx1200/gfx1201), or an Apple Silicon Mac (macOS 14+). - CUDA toolkit 13.0+ for NVIDIA, ROCm with
amdclang++for AMD, or Xcode with its Metal Toolchain (xcodebuild -downloadComponent MetalToolchain) for Apple. - A C++ compiler, CMake 3.26+, and Ninja for builds.
- uv for Python environment management and builds.
- Python 3.10+ and PyTorch 2.10+ with CUDA, ROCm or MPS support for the Python package; optionally MLX 0.32+ on a Mac.
- Weight shapes:
N(out_features) a multiple of 16,K(in_features) a multiple of 64.
uv pip install nf12Linux x86-64 and ARM64 wheels include kernels for SM80, 86, 89, 90, 100, 103, 110, and 120. They bundle the CUDA 13 runtime and require a compatible NVIDIA driver. There are no macOS wheels yet; build the Metal backend from source.
From source:
uv sync --locked # all supported architectures
NF12_CUDA_ARCHS=120 uv sync --locked # one architecture for development
uv sync --locked --extra mlx # macOS: Metal for PyTorch MPS and MLXTo defer compilation until the first native operation:
NF12_BUILD_NATIVE=OFF uv sync --lockedYou can build the C library alone with CMake:
cmake -B build/c -G Ninja -DNF12_BUILD_TORCH=OFF -DNF12_CUDA_ARCHS=120
cmake --build build/cThe HIP backend supports gfx1200 and gfx1201 with a
C API, and the full python/nf12 /
torch.ops.nf12 API: the torch extension compiles the same
csrc/torch/ops.cpp as the CUDA build and backs it with the HIP kernels, so
there is one op layer rather than a second ROCm implementation. That covers
encode/decode, linear (including the bf16t twin), pack_bf16t, embedding
and the whole MoE op set (moe_align, moe_linear, moe_combine,
moe_silu_mul, moe_decode, _fused_moe). _fused_moe at T == 1 takes a
native two-kernel fused route instead of the composed one; see
csrc/hip/README.md.
Validated on ROCm 10.1.0 with HIP 7.16.26385 on Radeon AI PRO R9700
(gfx1201). See the validated toolchain and test instructions.
The package follows the CUDA build's NF12_BUILD_NATIVE switch:
| Setting | ROCm wheel contents | Compilation |
|---|---|---|
ON (default) |
HIP libraries and sources | During the package build |
OFF |
Sources only | JIT on first native use, cached per user |
NF12_ENABLE_HIP enables the HIP backend; it does not control when compilation
happens. With ROCm PyTorch installed, the top-level build enables HIP by default.
For a native ROCm wheel, build in the environment containing the ROCm PyTorch
wheel and the build tools:
python -m pip install 'scikit-build-core>=0.10' ninja
python -m pip wheel --no-build-isolation --no-deps .An isolated build resolves its own PyTorch wheel and may select CUDA or CPU instead. The torch extension must be built against the ROCm wheel that loads it.
Build from source with the ROCm toolkit, a C++ compiler, CMake 3.26+ and Ninja.
cmake -S . -B build/hip -G Ninja \
-DNF12_ENABLE_HIP=ON -DNF12_BUILD_TORCH=ON \
-DNF12_HIP_ARCHS=gfx1201 \
-DCMAKE_HIP_COMPILER=/opt/rocm/bin/amdclang++
cmake --build build/hip --target nf12_torchUse -DNF12_HIP_ARCHS=gfx1200 for a gfx1200 GPU. To build only the C library,
set -DNF12_BUILD_TORCH=OFF and build --target nf12_hip. When an installed
wheel has no libraries, nf12._jit builds both from its packaged sources.
Fused linear kernels support batches of 1–256 tokens. For larger batches, the torch API decodes bounded slices and runs BF16 GEMMs; the raw C API uses native M chunks with caller-owned workspace. See the ROCm validation report for measured performance.
See the shared ABI design for the CUDA/HIP boundary.
See the HIP documentation for R9700 benchmark results and kernel implementation details.
import torch
import nf12
w = torch.randn(4096, 4096, dtype=torch.bfloat16, device="cuda")
nw = nf12.encode(w)
assert torch.equal(nf12.decode(nw).view(torch.int16), w.view(torch.int16))
x = torch.randn(2, 7, 4096, dtype=torch.bfloat16, device="cuda")
y = nf12.linear(x, nw) # preserves leading dimensionsOn a Mac, use device="mps": the same API runs on the Metal backend. MLX arrays
work too, through the same functions or nf12.mlx (which does not import torch):
import mlx.core as mx
import nf12.mlx as nm
w = mx.random.normal((4096, 4096)).astype(mx.bfloat16)
nw = nm.encode(w) # nf12.encode(w) routes MLX arrays here as well
y = nm.linear(mx.random.normal((7, 4096)).astype(mx.bfloat16), nw)Build with the mlx extra (uv sync --locked --extra mlx) for MLX.
Use nf12.NF12Linear.from_linear(layer) to convert a BF16 CUDA or MPS nn.Linear.
Decoding recovers the original weights. GEMM outputs can differ from cuBLAS
due to accumulation order.
w13 = nf12.encode_moe(gate_up) # BF16 [E, 2I, K], gate rows first
w2 = nf12.encode_moe(down) # BF16 [E, K, I]
# x: [T, K], router weights and IDs: [T, topk]
out = nf12.fused_moe(x, w13, w2, topk_weights, topk_ids)nw = nf12.encode(model.embed_tokens.weight) # BF16 [V, H]
x = nf12.embedding(token_ids, nw) # [*token_ids.shape, H]
embed = nf12.NF12Embedding.from_weight(nw)
lm_head = nf12.NF12Linear.from_weight(nw) # shares the encoded weightsShow benchmark tables
NF12 / BF16 output tokens per second. Each cell shows concurrency 1 / 8 / 32. Above 1× favors NF12. HTTP generation, 128 input / 256 output tokens, 68 server runs with alternating BF16/NF12 order. PyTorch 2.14, CUDA 13.0.
| GPU | Qwen3-8B | Granite 1B MoE | Qwen3-30B-A3B |
|---|---|---|---|
| CMP 170HX (SM80) | 1.24× / 1.16× / 1.07× | 0.93× / 1.08× / 1.01× | NF12 only* |
| RTX A6000 (SM86) | 1.30× / 1.24× / 1.15× | 1.04× / 1.16× / 1.18× | — |
| RTX 6000 Ada (SM89) | 1.07× / 1.14× / 1.23× | 1.03× / 1.17× / 1.16× | — |
| H200 (SM90) | 1.02× / 1.01× / 0.86× | Unstable | 1.09× / 1.10× / 0.93× |
| B200 (SM100) | 1.07× / 0.95× / 0.81× | 0.95× / 0.80× / 0.78× | 0.90× / 0.83× / 0.87× |
| Jetson Thor (SM110) | 1.32× / 1.34× / 1.28× | 1.10× / 1.14× / 1.14× | NF12 only* |
| RTX PRO 6000 Blackwell (SM120) | 1.25× / 1.18× / 1.21× | 1.22× / 1.14× / 1.06× | 1.27× / 1.23× / 1.26× |
*Qwen3-30B fit only with NF12 under the tested memory caps: 75% on CMP 170HX, 40% on Thor. BF16 was not attempted there. —: excluded on the 48 GB GPUs. H200 Granite timings were unstable, so no pooled speedup is reported.
The serving harness used a benchmark-only nf12.fused_moe adapter.
The baseline used native BF16 backends, including FlashInfer TRT-LLM on B200.
Prefill results are mixed. Greedy outputs can differ; these are not accuracy results.
| Model | BF16 GiB | NF12 GiB |
|---|---|---|
| Qwen3-8B | 15.27 | 11.49 |
| Granite 1B MoE | 2.50 | 1.89 |
| Qwen3-30B-A3B | 56.88 | 42.80 |
NF12 layer latency in µs (speedup over the previous composed NF12 path, not BF16). Real weights, uniform routing, cold CUDA graphs rotating at least 2 GiB. Median of two run medians. PyTorch 2.15, CUDA 13.2.
| GPU | Qwen3-30B-A3B | DeepSeek-V2-Lite |
|---|---|---|
| H200 | 23.2 (1.37×) | 30.5 (1.24×) |
| B200 | 18.9 (1.48×) | 22.1 (1.49×) |
| RTX PRO 6000 Blackwell | 38.9 (1.20×) | 59.6 (1.10×) |
Apache-2.0.