Repository navigation
Conversation
Prepare the serving runners after connector initialization and before worker readiness. Direct CUDA engines also prepare during initialization. - Reuse serving execution and current KV allocations for bounded prefill, decode, sampling, and configured speculative warmup - Capture planned Graph shapes and verify representative replays while retaining lazy capture and reserves for uncovered shapes - Drain device work before releasing scratch requests, KV, and async state; preserve serving RNG and business counters - Add idempotent preparation and phased coverage, Graph, and memory reports - Keep estimated KV sizing and make CPU preparation workload-free - Add lifecycle, failure, BF16/FP8, and request-reuse regressions and docs Real-checkpoint validation preserves outputs within each KV dtype. Preparation reduces first-request setup at the cost of slower startup; it does not establish a steady-state throughput improvement or implement the rest of RFC InferMatrixLab#192. Refs InferMatrixLab#192
Clarify that the startup report is a preparation snapshot and does not remeasure or validate subsequent storage commitment. Document full-reservation graph views, backed-block publication, physical VMM memory accounting, post-commit replay validation, and the current ordinary-storage requirement for PD/NIXL.
…imits Record completed recurrent decode batch/history pairs, including concurrent preparation, without adding startup workloads. Keep explicit estimated transient allowances for unmeasured combinations and random block speculation. Add regressions for capacity limits, staged/concurrent execution and the block/pipeline sampling distinction, and document the report contract.
kunkunblueberry
added a commit
to kunkunblueberry/vllm-rlt
that referenced
this pull request
Nov 19, 2026
Turn the RFC InferMatrixLab#192 step-1 preparation report into a KV block count, and let KVCacheManager grow its published range without reallocating tensors. - measured_kv_capacity() reads the final drained phase of the report: the driver's remaining memory is the physical budget, so weights, execution buffers, captured graphs and any live mapping are never subtracted twice. The utilization cap still bounds the instance, and the transient allowance and safety reserve are explicit inputs that are recorded with the result. Coverage gaps reported by the preparation (long contexts, random speculation, unreported batch/history combinations) travel with the plan, so an estimated margin is never presented as a measured one. - KVCacheManager accepts a capacity source whose tensors span the full reservation and whose physical backing grows by commit(). Requests are only ever handed published blocks, publishing requires the storage to report the blocks as backed, and published <= committed <= max is enforced, so no allocation can address unmapped memory. Release stays reference-based, as agreed for the storage contract: the owner drops references, the manager never unmaps. - With no capacity source nothing changes: tensors come from torch.empty and max == committed == published. Engine wiring (prepare_runtime -> plan -> commit -> publish -> replay and representative execution -> ready) needs InferMatrixLab#198 for the report and InferMatrixLab#194 for the storage, so it is the next step rather than part of this change. Signed-off-by: kunkunblueberry <kunkunblueberry@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Required human contributor answers
1. What outcome are you trying to achieve?
Complete step 1 of RFC #192: prepare the runtime and CUDA Graphs before the engine becomes ready, establish a preparation/measurement boundary for later measured KV sizing, and reduce initialization work charged to the first serving request.
2. What is the highest-leverage variable for that outcome?
The ordering of runtime preparation relative to engine readiness, and the memory retained after preparation. Later KV sizing needs to account for actual execution paths, CUDA Graphs, and persistent workspaces.
3. How will this work move that variable rather than create busywork?
Use the actual serving runners for bounded warmup, Graph capture, and representative replay before readiness. Drain device work and clean up temporary state before reporting memory and coverage. Validate the change with fixed-capacity output comparisons, independent timing, and profiles, and disclose the additional startup cost.
4. Who is the customer or user, and what do they actually need?
Users deploying recurrent models with CUDA engines, and developers integrating runtime memory measurement and KV storage for RFC #192. They need predictable readiness, interpretable runtime memory observations, and preparation that preserves serving outputs and state.
PR necessity and bugfix evidence
A PR is needed to introduce a shared preparation entry point, readiness ordering, and phase reports for the subsequent measured-sizing work. Configuration changes alone cannot provide this lifecycle. This is a feature implementing the first step of RFC #192, rather than a standalone bugfix or completion of the whole RFC. First-request effects were measured with a real checkpoint and investigated with independent profiles.
Purpose
Add idempotent
StageEngine.prepare_runtime()and run preparation before engine readiness:ready.This PR keeps estimated KV sizing and existing capacity allocation. It does not implement VMM, measured final sizing, a larger KV pool, or final capacity publication. The integration interfaces with the other RFC owners still need agreement in #192.
Phase driver-free values are boundary samples, not a continuous physical memory high-water mark. The report retains the allocator cache rather than calling
empty_cachebetween stages; current KV byte accounting includes FP8 scales.See the maintainer-confirmed split. This PR has no dependency on the other implementation branches.
Refs #192
Test Plan
vllm-rlt Version / Commit: base
bed34006ea8dc5b6ba1a3abd99a9fd81bd91a31bplus headc8868f03ee04f5858bb7f8be73a02bfb359f319d.Environment: Linux, one RTX 5090 32 GiB without CPU offload, Python 3.11.15, PyTorch 2.11.0+cu128, Triton 3.6.0, NVIDIA driver 580.178.04.
58872a72616c736595b8b7662079c5b12c5a162ec16eae94f21c348dfa9885af.Regression commands:
Test Result
Regression tests
pre-commit run --all-files(lint, format, full CPU hook)git diff --checkOn Windows with PyTorch 2.8.0+cpu and no Triton, the full hook recorded 1210 passed, 927 skipped, and 26 failures. All 26 failing cases reproduce on clean
bed3400: 12 missing-Triton imports, 12 unsupported FP8 CPU operations, one Huginn dense-reference tolerance failure, and one worker-pipe failure. No Windows full-suite pass is claimed.The two GPU skips require two devices. The pinned FA4 paged path rejects SM120; 29 FA4-related cases were excluded from the applicable run, including metadata cases that hardcode FA4. The FA4 failures reproduce on clean main. Three ordinary-KV mock failures also reproduced on main; the missing
key_scale=Nonefixture field was supplied and all three cases passed separately. No backend guard was relaxed.Output and numerical preservation
All comparisons below are within the same KV dtype, across the three arms:
Only valid KV positions were compared, not padding or unused pool storage. This validates behavior preservation rather than task accuracy or BF16-versus-FP8 equivalence. Internal speculative distributions were not inspected. Current-base full-checkpoint qualification covers Ouro; older real-checkpoint results for other models are not counted here.
Independent unprofiled performance
Same pinned checkpoint; 257 input tokens, eight greedy outputs, four loops, one active request, 2048 BF16 KV blocks, 16-token pages, chunk/token budget 128, maximum three requests, Triton with static buffers and Graphs. Three fresh processes per arm/mode, alternating order, with equivalent serving warmup before the steady request. Shared compilation caches were primed by qualification.
Values are medians [min-max]. Startup excludes weight loading. TTFT starts after
add_requestand measures the engine-visible first output, excluding network/tokenization.Preparation reduces setup charged to the first request after readiness, at the cost of slower startup. Startup plus first completion becomes slower: approximately 1.02 -> 2.29 s sync and 1.07 -> 3.42 s async, excluding weight loading.
Steady throughput changes by -0.42% sync and +0.45% async with overlapping ranges. These measurements do not establish sustained throughput improvement or absence of regressions on all workloads. Throughput is eight successful tokens divided by one request's elapsed time, not saturation throughput. Each per-process p95 uses seven output intervals; the reported median of three p95 values is descriptive only.
The prepared ready state retains approximately 24.55 MiB more PyTorch allocated memory and 76 MiB more reserved memory; fixed KV capacity is unchanged.
Profiler observations and scope
Twelve valid profiles (six independent ops-only/full pairs) cover the complete first decode iteration and its startup-preparation counterpart. Every full profile has nonempty shapes, memory timeline, and stacks. An export-stability diagnostic hook was used only in full-profile collection, not production or clean timing.
StageEngine.prepare_runtime -> prepare_runner -> actual runner submissions -> Graph capture/replay moves first-use work earlier; it does not reduce model arithmetic. First-decode capture happens after the first output and cannot alone explain the TTFT change. The profiles establish capture relocation, not a complete causal decomposition of TTFT. Steady Graph keys, capture counts, and fallback counts stayed fixed while replay counts increased.
Author self-review covered ownership, lifecycle/failure cleanup, execution contracts, and shared logic; no further actionable issue was found. This is not an independent review. Multi-GPU/NIXL, unsupported FA4, unreported long contexts, and mixed speculative/prompt overlap remain unqualified. Measured final sizing and post-storage-commit replay validation belong to the later RFC integration.
Startup coverage follow-up (
4a5b198)This reporting-only follow-up adds actual completed recurrent decode
batch_size/history_tokenspairs, including concurrent preparation. It explicitly retains estimated transient allowances for unreported long-context/batch combinations and random block speculation. The default startup workload, sizing policy and serving execution are unchanged. Pipeline speculation remains greedy-only. Regression tests and runtime documentation cover the reporting contract.Validation on the final patch: Linux
pre-commit run --all-filespassed; full regression selection recorded 1242 passed / 35 skipped, with 896 device cases deselected. The selected-device startup suite recorded 87 passed; other applicable regressions recorded 326 passed / 2 skipped / 32 deselected. Unsupported paged FA4 SM120 cases and the previously disclosed fixture paths remain outside the applicable selection; no full-device-suite pass is claimed.At fixed 768 KV blocks and 16-token pages, the pinned Ouro-1.4B BF16 checkpoint was compared against
060203dacross sync, async, staged prefill and async block speculation with BF16/FP8 KV. Three capacity-feasible workloads used prompt lengths 3064, 1016/1016/1016, and 2024/504/504, with eight subsequent positions reserved per request. All 112 matched request pairs / 672 output tokens preserved IDs, depths, finish reasons, observed ordinary-sampler logits and valid prompt KV including FP8 scales. Comparisons are within each execution mode and KV dtype; these are behavior-preservation checks, not task-accuracy or BF16-versus-FP8 equivalence results.A separate controlled serving-runner probe exercised three 1016-token histories in one speculative round (nine verification rows): greedy and seeded-random outputs and accepted counts matched in both KV dtypes. Normal scheduling did not reach three requests in one speculative round for the equal-length workload, so this runner check is separate. Its incremental PyTorch allocated peak was approximately 0.93 MiB greedy / 27.24 MiB random. These tested capacity boundaries do not cover the model's 64K context or every overlap/workspace shape.
Unprofiled reverse-order startup rechecks (three repeats after shape priming) gave prior/follow-up medians of 1.1198/1.1114 s sync, 2.0625/2.0739 s async, and 3.0775/3.0057 s async speculation. Initial and reversed-order effects differed; no stable startup performance gain or regression is established. All new remote experiment files were removed after retrieving the results.