Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
f6286f5
[KV] FP8 e4m3 KV cache with per-vector scales
Oct 15, 2026
ee375cb
[Docs] FP8 KV cache usage, semantics and limits
Oct 15, 2026
0e530b2
[KV] Fused FP8 quantize-and-store kernel for Triton
0z5a Oct 16, 2026
f36f2f9
[Docs] FP8 KV: fused Triton writer, current token is read in FP8
0z5a Oct 16, 2026
6087e67
[Fix] Merge main into FP8 KV cache support
0z5a Oct 21, 2026
b2803a2
Merge remote-tracking branch 'upstream/main' into kv/fp8-kv-cache
0z5a Nov 1, 2026
fbfeaad
Run the merged FP8 tests on GPUs with less shared memory
0z5a Nov 1, 2026
8baa95f
fix(attention): reject FP8 per-vector KV on FlashAttention with guidance
0z5a Nov 6, 2026
d2043f5
feat(attention): support FP8 per-vector KV on FA2-4
0z5a Nov 6, 2026
529151c
feat(attention): advertise FP8 KV on FA2-4
0z5a Nov 6, 2026
65163d4
fix(engine): reject CUDA graphs for FP8 KV on FlashAttention
0z5a Nov 6, 2026
9c81c31
fix(attention): keep FA3 paged FP8 disabled until validated
0z5a Nov 6, 2026
9fed631
perf(attention): fuse FP8 KV staging into one Triton launch
0z5a Nov 6, 2026
260404a
perf(attention): cache FP8 staging buffers and page-table map
0z5a Nov 6, 2026
c03f75e
Merge main and read FP8 KV scales in the windowed Torch reference
0z5a Nov 13, 2026
0f31f60
Cover FP8 scales in windowed Torch attention
0z5a Nov 13, 2026
7f001e1
Merge main and read FP8 KV scales in Triton and graph window paths
0z5a Nov 16, 2026
03bb389
Cover FP8 scales in Triton and graph-replayed window attention
0z5a Nov 16, 2026
9ed813e
Merge main and preserve FP8 scales in packed window attention
0z5a Nov 17, 2026
f915e9d
Use native FlashAttention-4 FP8 for paged KV
bjf-frz Nov 18, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,11 @@ Packed prefill uses `ExecutionConfig.packed_prefill` and the
`--packed-prefill` / `--no-packed-prefill` flags. Input transfer is controlled
independently by `use_uva`; see the [selection rules](docs/runtime.md#packed-prefill).

FP8 KV defaults to per-vector scales on Torch/Triton. Experimental native FA4 FP8,
including Dory sliding attention, uses `flash-attn-4==4.0.0b39` and calibrates
a fixed scale at startup. See the
[hardware and calibration requirements](docs/runtime.md#experimental-fa4-static-scale-adapter).

## Model Support


Expand Down
74 changes: 74 additions & 0 deletions docs/runtime.md
Original file line number Diff line number Diff line change
Expand Up @@ -325,6 +325,80 @@ zero watermark in non-incremental mode, see [scheduler admission](scheduler.md#a
Pool sizing establishes physical capacity; scheduler admission decides how much
of that capacity can safely be promised to new work.

## FP8 KV cache

`CacheConfig(kv_cache_dtype="fp8_e4m3")` (CLI `--kv-cache-dtype fp8_e4m3`) stores K and V as
float8 e4m3 codes with one float32 scale per (block, layer, token, KV head) vector
(`scale = amax / 448`). Values are quantized when written and dequantized by the attention
backend when read, so a block costs `head_dim + 4` bytes per vector instead of
`head_dim * element_size`: 51.6% of BF16 at `head_dim=128`, and automatic sizing or
`--kv-cache-memory-bytes` admits about 1.94x as many tokens. The default `"auto"` stores the
model dtype and leaves every existing path unchanged.

- Supported: Torch (reference) and Triton attention; synchronous and asynchronous scheduling;
CUDA graphs (quantized writes are captured); LAST_EXITED finalize copies, including the
asynchronous finalize kernel, copy codes and scales; preemption snapshots carry the scales;
`KVCacheManager.read` returns dequantized values.
- Triton's split-KV decode and packed paged-prefill kernels both apply the scales on load
(prefill dequantizes to the query dtype before `tl.dot`). Device-masked writes (speculative
verification, bucketed CUDA graphs) use the same fused writer with the row mask applied.
- Triton writes with one fused quantize-and-store kernel per layer. The Torch backend quantizes
with Torch ops and is the reference; both writers produce the same codes and scales.
- With Triton FP8 KV, PD transfer sends K/V scales with the corresponding cache
rows. P and D must use the same KV dtype.
- Each layer writes the current token's K/V before attending, so attention reads the current
token in FP8 as well. Accuracy should be checked per model and workload; adaptive exits can
change where a token exits because the gate sees slightly different attention.

### Experimental FA4 static-scale adapter

Use `--kv-cache-dtype fp8_e4m3 --attention-backend flash_attn_4`
(Python: `CacheConfig(kv_cache_dtype="fp8_e4m3")` with `attention_backend="flash_attn_4"`).
Backend `auto` also selects this path when FA4 meets the execution requirements. The prototype
targets FA4 4.0.0b39 on SM10x with BF16 models and head dimensions 64, 128 or 256.
Install the pinned dependency with `pip install -e '.[flash-attn4]'`. Both BF16
and FP8 use FA4's public API; FP8 passes `TensorwiseScales`, without a private
forward entry point or temporary BF16 KV staging.
At startup the engine runs a short BF16 prefill with deterministic vocabulary
IDs and one decode input when context length permits. It measures the maximum
absolute Q/K/V value across all layers and loops, adds 10% headroom, divides by 448,
and rounds up to a power of two. The result stays fixed for the lifetime of the
KV pool and is recorded in the startup log. Values outside `±448 * scale`
saturate. FA4 quantizes Q with this scale and receives `[batch, kv_head]`
Q/K/V descales in both decode and packed prefill. This changes compute precision
in addition to KV storage precision. Validate model accuracy on representative
requests because the startup probe only samples a short synthetic sequence.

The existing per-vector scale pools and finalize/snapshot paths are retained
for a small reviewable prototype; byte accounting still includes them. Snapshots
with different scale policies cannot be restored. Torch/Triton can read the
fixed-scale cache for reference checks. FA4 packed-vs-decode bitwise matching is
not advertised for FP8. In PD, workers calibrate from the same startup inputs
and verify that their fixed scale matches before exchanging KV pages.

Dory uses the same adapter: sliding attention passes `(window - 1, 0)` to FA4,
because the runtime window includes the current token. Full-attention layers
keep their causal mask. Packed prompt chunks retain bottom-right alignment
against an existing prefix. Both `per_loop` and `last_loop` KV modes, using
their required layouts, support asynchronous scheduling and CUDA Graphs in
the input, recurrent and output stages. Dory's existing rejection of mixed
scheduling and boundary-KV prefix caching remains in effect.

Validation on B300 with FA4 4.0.0b39 covers 69 GPU cases across kernel,
end-to-end and regression checks: head dimensions 64/128/256, packed sliding
windows with existing prefixes, snapshot/finalize paths, and Dory 2/4-loop
eager versus async single/multi-stream Graph execution without fallback.
The real Dory 4B checkpoint also completed a 1030-token prompt across its
1024-token window and matched FP8 eager output under asynchronous Graphs.

An unprofiled eager comparison on that checkpoint (2 loops, 128 input tokens,
16 output tokens, one request, three warmed repetitions) measured median
output throughput of 20.45 tok/s with BF16 and 18.21 tok/s with FP8. Allocated
KV pool storage fell from 576 MiB to 292.5 MiB at equal block counts. This
small case does not demonstrate a speedup or establish model accuracy. Its
fixed scale of 16 came from one separate calibration prompt and is not a
recommended default; validate calibration and task quality for deployment.

## Replay and validation commands

Trace mode expects `depths_by_request`, with one depth per output: full prefill depth first, then each decode input's exit depth. Admission validates coverage and loop bounds. Request IDs must match the trace. The public engine replay fixes depth decisions; token generation still follows the configured sampler.
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ text = []
pd = ["nixl==1.4.1"]
serve = []
flash-attn = ["flash-attn>=2.8.3"]
flash-attn4 = ["flash-attn-4==4.0.0b30"]
flash-attn4 = ["flash-attn-4==4.0.0b39"]
triton = []
dev = ["pytest>=8", "ruff>=0.9"]

Expand Down
115 changes: 111 additions & 4 deletions tests/test_backend_selection.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,13 @@
def packages(monkeypatch):
from vllm_rlt.kernels import flash_attention

installed = {"flash_attn", "flash_attn_3.flash_attn_interface", "flash_attn.cute", "triton"}
installed = {
"flash_attn",
"flash_attn_3.flash_attn_interface",
"flash_attn.cute",
"TensorwiseScales",
"triton",
}
original_import = flash_attention.importlib.import_module

def import_module(name):
Expand All @@ -28,14 +34,19 @@ def import_module(name):
if name in ("flash_attn", "flash_attn_3.flash_attn_interface", "flash_attn.cute"):
if name not in installed:
raise ImportError(f"{name} is not installed")
return SimpleNamespace(
result = SimpleNamespace(
flash_attn_with_kvcache=lambda *args, **kwargs: None,
flash_attn_varlen_func=lambda *args, **kwargs: None,
)
if name == "flash_attn.cute" and "TensorwiseScales" in installed:
result.TensorwiseScales = SimpleNamespace
return result
return original_import(name)

monkeypatch.setattr(flash_attention.importlib, "import_module", import_module)
monkeypatch.setattr(flash_attention, "version", lambda name: "test")
monkeypatch.setattr(
flash_attention, "version", lambda name: "4.0.0b39" if name == "flash-attn-4" else "test"
)
monkeypatch.setattr(torch.version, "hip", None)
return installed

Expand All @@ -53,6 +64,94 @@ def test_cpu_auto_reports_torch_and_runs_reference_attention(caplog):
torch.testing.assert_close(selected(q, k, v, table, lengths), explicit(q, k, v, table, lengths))


@pytest.mark.parametrize(
"sm,dtype,head_dim,supported",
[
((10, 3), torch.bfloat16, 128, True),
((9, 0), torch.bfloat16, 128, False),
((10, 3), torch.float16, 128, False),
((10, 3), torch.bfloat16, 256, True),
],
)
def test_fa4_static_fp8_selection(packages, monkeypatch, sm, dtype, head_dim, supported):
monkeypatch.setattr(torch.cuda, "get_device_capability", lambda device: sm)
args = ("flash_attn_4", torch.device("cuda"), dtype, head_dim, 16)
requirement = BackendCapabilities(fp8_kv_static=True)
if not supported:
with pytest.raises(ValueError, match="fp8_kv_static"):
create_backend(*args, required_capabilities=requirement)
return
selected = create_backend(*args, required_capabilities=requirement)
assert not selected.capabilities.packed_matches_decode
with pytest.raises(ValueError, match="fp8_kv"):
create_backend(*args, required_capabilities=BackendCapabilities(fp8_kv=True))
selected = create_backend("auto", *args[1:], required_capabilities=requirement)
assert selected.info["selected_backend"] == "flash_attn_4"


def test_fa4_static_fp8_requires_public_scale_api(packages, monkeypatch):
monkeypatch.setattr(torch.cuda, "get_device_capability", lambda device: (10, 3))
packages.remove("TensorwiseScales")
args = ("flash_attn_4", torch.device("cuda"), torch.bfloat16, 128, 16)
assert create_backend(*args).info["selected_backend"] == "flash_attn_4"
with pytest.raises(ValueError, match="fp8_kv_static"):
create_backend(*args, required_capabilities=BackendCapabilities(fp8_kv_static=True))


@pytest.mark.parametrize(
"name,sm,block_size",
[("flash_attn_2", (8, 9), 256), ("flash_attn_3", (9, 0), 16), ("flash_attn_4", (10, 3), 16)],
)
def test_flash_rejects_per_vector_fp8_without_staging(packages, monkeypatch, name, sm, block_size):
monkeypatch.setattr(torch.cuda, "get_device_capability", lambda device: sm)
with pytest.raises(ValueError, match="fp8_kv"):
create_backend(
name,
torch.device("cuda"),
torch.bfloat16,
128,
block_size,
required_capabilities=BackendCapabilities(fp8_kv=True),
)


@pytest.mark.parametrize("calibrate", [False, True])
def test_auto_static_fp8_keeps_graph_support_and_falls_back_if_scale_api_missing(
packages, monkeypatch, calibrate
):
monkeypatch.setattr(torch.cuda, "get_device_capability", lambda device: (10, 3))
args = ("auto", torch.device("cuda"), torch.bfloat16, 128, 16)
required = BackendCapabilities(fp8_kv=calibrate, fp8_kv_static=not calibrate, cuda_graphs=True)
options = dict(required_capabilities=required, allow_fp8_calibration=calibrate)
selected = create_backend(*args, **options)
assert selected.info["selected_backend"] == "flash_attn_4"
assert selected.capabilities.fp8_kv_static and selected.capabilities.cuda_graphs
assert not selected.capabilities.packed_matches_decode

packages.remove("TensorwiseScales")
selected = create_backend(*args, **options)
assert selected.info["selected_backend"] == "triton"
assert selected.capabilities.fp8_kv_static and selected.capabilities.cuda_graphs


def test_dory_requirements_select_native_fa4_and_reject_old_hd256(packages, monkeypatch):
monkeypatch.setattr(torch.cuda, "get_device_capability", lambda device: (10, 3))
args = (torch.device("cuda"), torch.bfloat16, 256, 16)
required = BackendCapabilities(
sliding_window=True, fp8_kv_static=True, cuda_graphs=True, async_scheduling=True
)
selected = create_backend("auto", *args, required_capabilities=required)
assert selected.info["selected_backend"] == "flash_attn_4"
assert selected.capabilities.sliding_window
packages.remove("TensorwiseScales")
with pytest.raises(ValueError, match="sliding_window"):
create_backend("flash_attn_4", *args, required_capabilities=required)
assert (
create_backend("auto", *args, required_capabilities=required).info["selected_backend"]
== "triton"
)


@pytest.mark.parametrize(
"sm,dtype,head_dim,block_size,missing,expected",
[
Expand Down Expand Up @@ -270,7 +369,15 @@ class ProfileReached(Exception):
pass

def plan_cache(
model, cache, scheduler, execution, selected, *, speculative=None, staged_prefill=None
model,
cache,
scheduler,
execution,
selected,
*,
speculative=None,
staged_prefill=None,
fp8_scale=None,
):
assert selected == "triton" # FA2 cannot use the configured 16-token pages.
assert staged_prefill is False
Expand Down
3 changes: 2 additions & 1 deletion tests/test_cuda_graph.py
Original file line number Diff line number Diff line change
Expand Up @@ -455,7 +455,8 @@ def test_bucketed_kv_graph_combines_active_rows_and_pipeline_write_mask():

keys = torch.full((4, 1, 2, 1, 4), -1.0, device="cuda")
values = torch.full_like(keys, -2.0)
cache = SimpleNamespace(key_cache=keys, value_cache=values)
# A BF16 pool has no FP8 scales, as KVCacheManager reports them.
cache = SimpleNamespace(key_cache=keys, value_cache=values, key_scale=None, value_scale=None)
batch = SimpleNamespace(
active_rows=torch.zeros((), dtype=torch.int32, device="cuda"),
write_mask=torch.ones(5, dtype=torch.bool, device="cuda"),
Expand Down
10 changes: 8 additions & 2 deletions tests/test_dory_gpu.py
Original file line number Diff line number Diff line change
Expand Up @@ -449,9 +449,10 @@ def compile_tile(*args, window, **kwargs):
assert probe(index, 256, 16, PackedWindowShape(16, 4, 36, 1024)) is None


@pytest.mark.parametrize("fp8_kv", [False, True])
@pytest.mark.parametrize(("window", "expected_windows"), [(None, [0]), (128, [0, 128])])
def test_prefill_tile_probe_compiles_only_the_default_and_caches_result(
monkeypatch, window, expected_windows
monkeypatch, window, expected_windows, fp8_kv
):
from types import SimpleNamespace

Expand All @@ -470,12 +471,17 @@ def test_prefill_tile_probe_compiles_only_the_default_and_caches_result(
def compile_tile(*args, **kwargs):
assert (kwargs["BLOCK_M"], kwargs["BLOCK_N"]) == (32, 32)
assert kwargs["COMPENSATE"]
assert kwargs["HAS_SCALE"] is fp8_kv
assert args[1].dtype == (torch.float8_e4m3fn if fp8_kv else torch.bfloat16)
if fp8_kv:
assert args[7].dtype == args[8].dtype == torch.float32
assert args[7].shape == args[1].shape[:-1]
assert all(arg.device.type == "meta" for arg in args if isinstance(arg, torch.Tensor))
windows.append(kwargs["WINDOW"])
return SimpleNamespace(metadata=SimpleNamespace(shared=limit))

monkeypatch.setattr(attention._paged_prefill_kernel, "warmup", compile_tile)
args = (device, torch.bfloat16, 64, 16, PackedWindowShape(4, 2, 2, window))
args = (device, torch.bfloat16, 64, 16, PackedWindowShape(4, 2, 2, window, fp8_kv))

attention._select_packed_window_tile.cache_clear()
try:
Expand Down
Loading
Loading