Repository navigation
feat(memory): size KV capacity from the post-warmup measurement - #200
Draft
kunkunblueberry wants to merge 3 commits into
Draft
kunkunblueberry wants to merge 3 commits into
kunkunblueberry wants to merge 3 commits into
Conversation
Turn the RFC InferMatrixLab#192 step-1 preparation report into a KV block count, and let KVCacheManager grow its published range without reallocating tensors. - measured_kv_capacity() reads the final drained phase of the report: the driver's remaining memory is the physical budget, so weights, execution buffers, captured graphs and any live mapping are never subtracted twice. The utilization cap still bounds the instance, and the transient allowance and safety reserve are explicit inputs that are recorded with the result. Coverage gaps reported by the preparation (long contexts, random speculation, unreported batch/history combinations) travel with the plan, so an estimated margin is never presented as a measured one. - KVCacheManager accepts a capacity source whose tensors span the full reservation and whose physical backing grows by commit(). Requests are only ever handed published blocks, publishing requires the storage to report the blocks as backed, and published <= committed <= max is enforced, so no allocation can address unmapped memory. Release stays reference-based, as agreed for the storage contract: the owner drops references, the manager never unmaps. - With no capacity source nothing changes: tensors come from torch.empty and max == committed == published. Engine wiring (prepare_runtime -> plan -> commit -> publish -> replay and representative execution -> ready) needs InferMatrixLab#198 for the report and InferMatrixLab#194 for the storage, so it is the next step rather than part of this change. Signed-off-by: kunkunblueberry <kunkunblueberry@gmail.com>
kunkunblueberry
force-pushed
the
kunkun/rfc192-measured-capacity
branch
from
November 19, 2026 15:20
5386a34 to
c63df73
Compare
Move the tensor/scale allocation and the mapped-pool binding out of KVCacheManager.__init__ into _open_torch_planes and _open_mapped_planes. The FP8 capability rules, the allocation and the per-vector scale lifecycle now sit in one method, and the mapped path states its own FP8 limitation instead of being special-cased at the top of __init__. The capacity fields keep the same meaning: both paths set max_blocks and committed_blocks, and __init__ publishes only afterwards. Behaviour is unchanged: the FP8 KV, backend-selection, Dory, prepared-KV and KV-cache suites pass as before, the full CPU suite keeps the same 32 pre-existing failures, and no line from the FP8 work merged in InferMatrixLab#129 was dropped (diffed the fp8-related lines against main; the only differences are the new mapped-path guard and the local `storage` -> `storage_dtype` rename that the new parameter required). Signed-off-by: kunkunblueberry <kunkunblueberry@gmail.com>
measured_kv_capacity() reported the measured delta as if it were the total block count. Driver free memory excludes the bootstrap prefix that is already mapped, so an integration would have committed only the delta and under-counted the pool by the bootstrap size: with the Ouro-1.4B configuration, 22894 instead of 44746 blocks, about 64 GiB of KV capacity. - existing_blocks adds the already-mapped prefix to the measured delta and keeps it out of the free-memory sizing, so it is never counted twice. - reservation_blocks clamps the result to what the pool can actually grow to, and the truncation is recorded as a coverage gap. - limited_by reports which bound applied (budget, useful_blocks, reservation). - A bootstrap-only pool is now accepted when no additional capacity fits, instead of failing although the existing prefix is usable. - A zero transient allowance is rejected when the preparation reported coverage gaps, because an unexercised path cannot be sized as if it were measured. - KVCacheManager rejects a capacity source that claims more committed blocks than it reserves. Verified on the real checkpoint: existing 21852 blocks, additional 22894, total 44746, and only the additional part spends the 67.07 GiB budget. Signed-off-by: kunkunblueberry <kunkunblueberry@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the capacity-planning part of RFC #192 assigned to me in the issue, on the terms agreed there: the planner consumes the step-1 preparation report, and the logical allocator only ever receives fully backed blocks.
Required human contributor answers
TODO(contributor): the four answers and the PR-necessity decision are reserved for the human contributor; the recorded split in RFC #192 describes the agreed scope but is not a substitute for them.
1. What outcome are you trying to achieve?
TODO(contributor)
2. What is the highest-leverage variable for that outcome?
TODO(contributor)
3. How will this work move that variable rather than create busywork?
TODO(contributor)
4. Who is the customer or user, and what do they actually need?
TODO(contributor)
PR necessity and bugfix evidence
TODO(contributor)
Purpose
Two pieces the measured-sizing path needs, neither of which depends on the unmerged orchestration in #198 or the storage in #194.
measured_kv_capacity()(vllm_rlt/core/memory.py) turns the final drained phase of the preparation report into a block count.total - freeonly bounds the remaining instance budget undergpu_memory_utilization.existing_blocksis the already-mapped bootstrap prefix. It is added to the measured delta and never sized from free memory again, so the returned count is the total the pool may hold; a cumulativecommit(total)maps only the new part.reservation_blocksclamps the result to what the virtual reservation can actually back. Truncation is recorded as a coverage gap, andlimited_bystates which bound applied (budget,useful_blocks,reservation).Publish gate (
vllm_rlt/core/kv_cache_manager.py):KVCacheManageraccepts a capacity source whosetensorsspan the full reservation and whose backing grows throughcommit(blocks) -> backed_blocks.publish_capacity()re-reads the storage's committed count before growing the free pool, andpublished <= committed <= maxis enforced, so an allocation cannot address unbacked memory.torch.empty,max == committed == published).Rebased onto
bed3400, which merged the FP8 KV work (#129). Mapped storage currently rejectskv_cache_dtype != "auto"because a VMM pool has no place for the per-vector scales yet; that is a declared limitation of this step, not a silent fallback.The second commit separates the two plane-setup paths (
_open_torch_planes/_open_mapped_planes), so the FP8 capability rules and the mapped-pool binding each live in one place. Behaviour is unchanged, and no line from #129 was dropped: the FP8-related lines were diffed againstmain, and the only differences are the new mapped-path guard and the localstorage->storage_dtyperename that the new parameter required.Test Plan
vllm-rlt Version / Commit:
59bb592(based onbed3400)python -m ruff check .andpython -m ruff format --check .(thecpu-tests.ymlcommands).python -m pytest -qin a worktree of this head, after each commit of the branch.mem_get_info()at a drained boundary, takebytes_per_blockandmax_useful_blocksfrom the engine's ownplan_cache, and run the planner on those real numbers withexisting_blocksset to the pool the estimate already mapped.Test Result
All checks passed!,140 files already formatted.1213 passed, 32 failed. The same 32 tests fail on a clean checkout ofbed3400on this host (test_async_speculative_sampling_plan.py12,test_serving.py18,test_pd.py1,test_ouro.py1), so this change adds no failures.tests/test_measured_capacity.py, 28 cases. Planner expectations are hand-computed from the KV layout and the stated margins (resident 20 GiB, cap 72 GiB, remaining 52 GiB, transient 4 GiB, safety 256 MiB, 3 MiB blocks, 16298 additional blocks). Cases cover the existing-prefix accounting, a bootstrap-only pool, reservation truncation versus the useful-block cap, the zero-transient-with-gaps rule, and the unbacked/oversubscribed storage guards.plan_cachemaps 21852 blocks (3.0 MiB each). The planner returns 22894 additional blocks, 44746 total, and only the additional 67.07 GiB spends the 67.07 GiB budget. This shows the arithmetic on real numbers; it is not a capacity claim, because the sample is taken after model load rather than after warmup and capture.Not run: any GPU execution path, because none changes here. The end-to-end A/B belongs to the integration.
Corrections from self-review
An earlier revision of this branch returned the measured delta as if it were the total block count; with the Ouro checkpoint that under-counts the pool by the 21852 already-mapped blocks. The review also closed four smaller holes: no reservation clamp, a bootstrap-only pool failing although it was usable, a zero transient allowance accepted while the preparation reported gaps, and a capacity source allowed to claim more committed blocks than it reserves. Each is covered by a test named after the behaviour.
Known and deliberately not changed here: the
storageparameter ofKVCacheManagershadows the namemainused for the FP8 tensor dtype (hence the localstorage_dtyperename), andself.deviceis re-derived from the bound tensors, which would hide a device mismatch between the backend and the pool rather than reject it.Limitations and next step
StageEngineyet:prepare_runtime()arrives with feat(runtime): prepare runners and CUDA Graphs before engine readiness #198 and the VMM storage with feat(kv): add CUDA VMM KV storage with fixed addresses #194. The agreed ordering (prepare_runtime -> plan -> commit -> publish -> replay and representative execution -> ready) and recording the storage kind plus any fallback reason inmemory_planare the next change, stacked on those two.measured_kv_capacity()has no caller yet: its evidence is the unit suite plus one real-number check, not an end-to-end run. That is why this PR stays draft until the integration lands.torch.emptyfor the first integration and will record that reason rather than silently choosing a path.