Skip to content

feat(memory): size KV capacity from the post-warmup measurement - #200

Draft
kunkunblueberry wants to merge 3 commits into
InferMatrixLab:mainfrom
kunkunblueberry:kunkun/rfc192-measured-capacity
Draft

kunkunblueberry wants to merge 3 commits into
InferMatrixLab:mainfrom
kunkunblueberry:kunkun/rfc192-measured-capacity

Conversation

@kunkunblueberry

@kunkunblueberry kunkunblueberry commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Implements the capacity-planning part of RFC #192 assigned to me in the issue, on the terms agreed there: the planner consumes the step-1 preparation report, and the logical allocator only ever receives fully backed blocks.

Required human contributor answers

TODO(contributor): the four answers and the PR-necessity decision are reserved for the human contributor; the recorded split in RFC #192 describes the agreed scope but is not a substitute for them.

1. What outcome are you trying to achieve?

TODO(contributor)

2. What is the highest-leverage variable for that outcome?

TODO(contributor)

3. How will this work move that variable rather than create busywork?

TODO(contributor)

4. Who is the customer or user, and what do they actually need?

TODO(contributor)

PR necessity and bugfix evidence

TODO(contributor)

Purpose

Two pieces the measured-sizing path needs, neither of which depends on the unmerged orchestration in #198 or the storage in #194.

measured_kv_capacity() (vllm_rlt/core/memory.py) turns the final drained phase of the preparation report into a block count.

  • The physical budget is the driver's remaining memory. Weights, execution buffers, captured graphs and anything already mapped are excluded from it, so none is subtracted a second time. total - free only bounds the remaining instance budget under gpu_memory_utilization.
  • existing_blocks is the already-mapped bootstrap prefix. It is added to the measured delta and never sized from free memory again, so the returned count is the total the pool may hold; a cumulative commit(total) maps only the new part.
  • reservation_blocks clamps the result to what the virtual reservation can actually back. Truncation is recorded as a coverage gap, and limited_by states which bound applied (budget, useful_blocks, reservation).
  • Transient and safety are explicit inputs and are recorded with the measured free/total bytes, the resident bytes and the gaps the preparation reported. A zero transient allowance is rejected when gaps exist: an unexercised path cannot be sized as if it were measured.

Publish gate (vllm_rlt/core/kv_cache_manager.py): KVCacheManager accepts a capacity source whose tensors span the full reservation and whose backing grows through commit(blocks) -> backed_blocks.

  • Requests only receive published blocks, publish_capacity() re-reads the storage's committed count before growing the free pool, and published <= committed <= max is enforced, so an allocation cannot address unbacked memory.
  • A source that claims more committed blocks than it reserves is rejected.
  • Release stays reference-based, as agreed for feat(kv): add CUDA VMM KV storage with fixed addresses #194: the owner drops references, the manager never unmaps.
  • Without a capacity source the existing behaviour is unchanged (torch.empty, max == committed == published).

Rebased onto bed3400, which merged the FP8 KV work (#129). Mapped storage currently rejects kv_cache_dtype != "auto" because a VMM pool has no place for the per-vector scales yet; that is a declared limitation of this step, not a silent fallback.

The second commit separates the two plane-setup paths (_open_torch_planes / _open_mapped_planes), so the FP8 capability rules and the mapped-pool binding each live in one place. Behaviour is unchanged, and no line from #129 was dropped: the FP8-related lines were diffed against main, and the only differences are the new mapped-path guard and the local storage -> storage_dtype rename that the new parameter required.

Test Plan

vllm-rlt Version / Commit: 59bb592 (based on bed3400)

  • python -m ruff check . and python -m ruff format --check . (the cpu-tests.yml commands).
  • python -m pytest -q in a worktree of this head, after each commit of the branch.
  • Device check with real weights (Ouro-1.4B BF16 snapshot, A100-SXM4-80GB): sample mem_get_info() at a drained boundary, take bytes_per_block and max_useful_blocks from the engine's own plan_cache, and run the planner on those real numbers with existing_blocks set to the pool the estimate already mapped.

Test Result

  • Ruff: All checks passed!, 140 files already formatted.
  • CPU suite: 1213 passed, 32 failed. The same 32 tests fail on a clean checkout of bed3400 on this host (test_async_speculative_sampling_plan.py 12, test_serving.py 18, test_pd.py 1, test_ouro.py 1), so this change adds no failures.
  • New tests: tests/test_measured_capacity.py, 28 cases. Planner expectations are hand-computed from the KV layout and the stated margins (resident 20 GiB, cap 72 GiB, remaining 52 GiB, transient 4 GiB, safety 256 MiB, 3 MiB blocks, 16298 additional blocks). Cases cover the existing-prefix accounting, a bootstrap-only pool, reservation truncation versus the useful-block cap, the zero-transient-with-gaps rule, and the unbacked/oversubscribed storage guards.
  • Real weights: plan_cache maps 21852 blocks (3.0 MiB each). The planner returns 22894 additional blocks, 44746 total, and only the additional 67.07 GiB spends the 67.07 GiB budget. This shows the arithmetic on real numbers; it is not a capacity claim, because the sample is taken after model load rather than after warmup and capture.

Not run: any GPU execution path, because none changes here. The end-to-end A/B belongs to the integration.

Corrections from self-review

An earlier revision of this branch returned the measured delta as if it were the total block count; with the Ouro checkpoint that under-counts the pool by the 21852 already-mapped blocks. The review also closed four smaller holes: no reservation clamp, a bootstrap-only pool failing although it was usable, a zero transient allowance accepted while the preparation reported gaps, and a capacity source allowed to claim more committed blocks than it reserves. Each is covered by a test named after the behaviour.

Known and deliberately not changed here: the storage parameter of KVCacheManager shadows the name main used for the FP8 tensor dtype (hence the local storage_dtype rename), and self.device is re-derived from the bound tensors, which would hide a device mismatch between the backend and the pool rather than reject it.

Limitations and next step

  • The report is not consumed by StageEngine yet: prepare_runtime() arrives with feat(runtime): prepare runners and CUDA Graphs before engine readiness #198 and the VMM storage with feat(kv): add CUDA VMM KV storage with fixed addresses #194. The agreed ordering (prepare_runtime -> plan -> commit -> publish -> replay and representative execution -> ready) and recording the storage kind plus any fallback reason in memory_plan are the next change, stacked on those two.
  • Because of that, measured_kv_capacity() has no caller yet: its evidence is the unit suite plus one real-number check, not an end-to-end run. That is why this PR stays draft until the integration lands.
  • PD/NIXL keeps torch.empty for the first integration and will record that reason rather than silently choosing a path.
  • No measured claim is made for long contexts, random block speculation or the full concurrent memory envelope; those stay estimated and are reported as gaps.

Turn the RFC InferMatrixLab#192 step-1 preparation report into a KV block count, and let
KVCacheManager grow its published range without reallocating tensors.

- measured_kv_capacity() reads the final drained phase of the report: the
  driver's remaining memory is the physical budget, so weights, execution
  buffers, captured graphs and any live mapping are never subtracted twice.
  The utilization cap still bounds the instance, and the transient allowance
  and safety reserve are explicit inputs that are recorded with the result.
  Coverage gaps reported by the preparation (long contexts, random
  speculation, unreported batch/history combinations) travel with the plan,
  so an estimated margin is never presented as a measured one.
- KVCacheManager accepts a capacity source whose tensors span the full
  reservation and whose physical backing grows by commit(). Requests are only
  ever handed published blocks, publishing requires the storage to report the
  blocks as backed, and published <= committed <= max is enforced, so no
  allocation can address unmapped memory. Release stays reference-based, as
  agreed for the storage contract: the owner drops references, the manager
  never unmaps.
- With no capacity source nothing changes: tensors come from torch.empty and
  max == committed == published.

Engine wiring (prepare_runtime -> plan -> commit -> publish -> replay and
representative execution -> ready) needs InferMatrixLab#198 for the report and InferMatrixLab#194 for the
storage, so it is the next step rather than part of this change.

Signed-off-by: kunkunblueberry <kunkunblueberry@gmail.com>
Move the tensor/scale allocation and the mapped-pool binding out of
KVCacheManager.__init__ into _open_torch_planes and _open_mapped_planes.

The FP8 capability rules, the allocation and the per-vector scale lifecycle now
sit in one method, and the mapped path states its own FP8 limitation instead of
being special-cased at the top of __init__. The capacity fields keep the same
meaning: both paths set max_blocks and committed_blocks, and __init__ publishes
only afterwards.

Behaviour is unchanged: the FP8 KV, backend-selection, Dory, prepared-KV and
KV-cache suites pass as before, the full CPU suite keeps the same 32
pre-existing failures, and no line from the FP8 work merged in InferMatrixLab#129 was dropped
(diffed the fp8-related lines against main; the only differences are the new
mapped-path guard and the local `storage` -> `storage_dtype` rename that the
new parameter required).

Signed-off-by: kunkunblueberry <kunkunblueberry@gmail.com>
measured_kv_capacity() reported the measured delta as if it were the total block
count. Driver free memory excludes the bootstrap prefix that is already mapped,
so an integration would have committed only the delta and under-counted the pool
by the bootstrap size: with the Ouro-1.4B configuration, 22894 instead of 44746
blocks, about 64 GiB of KV capacity.

- existing_blocks adds the already-mapped prefix to the measured delta and keeps
  it out of the free-memory sizing, so it is never counted twice.
- reservation_blocks clamps the result to what the pool can actually grow to,
  and the truncation is recorded as a coverage gap.
- limited_by reports which bound applied (budget, useful_blocks, reservation).
- A bootstrap-only pool is now accepted when no additional capacity fits,
  instead of failing although the existing prefix is usable.
- A zero transient allowance is rejected when the preparation reported coverage
  gaps, because an unexercised path cannot be sized as if it were measured.
- KVCacheManager rejects a capacity source that claims more committed blocks
  than it reserves.

Verified on the real checkpoint: existing 21852 blocks, additional 22894, total
44746, and only the additional part spends the 67.07 GiB budget.

Signed-off-by: kunkunblueberry <kunkunblueberry@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant