Skip to content

feat(runtime): prepare runners and CUDA Graphs before engine readiness - #198

Open
cecilsd wants to merge 3 commits into
InferMatrixLab:mainfrom
cecilsd:runtime-startup-warmup
Open

cecilsd wants to merge 3 commits into
InferMatrixLab:mainfrom
cecilsd:runtime-startup-warmup

Conversation

@cecilsd

@cecilsd cecilsd commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Required human contributor answers

1. What outcome are you trying to achieve?

Complete step 1 of RFC #192: prepare the runtime and CUDA Graphs before the engine becomes ready, establish a preparation/measurement boundary for later measured KV sizing, and reduce initialization work charged to the first serving request.

2. What is the highest-leverage variable for that outcome?

The ordering of runtime preparation relative to engine readiness, and the memory retained after preparation. Later KV sizing needs to account for actual execution paths, CUDA Graphs, and persistent workspaces.

3. How will this work move that variable rather than create busywork?

Use the actual serving runners for bounded warmup, Graph capture, and representative replay before readiness. Drain device work and clean up temporary state before reporting memory and coverage. Validate the change with fixed-capacity output comparisons, independent timing, and profiles, and disclose the additional startup cost.

4. Who is the customer or user, and what do they actually need?

Users deploying recurrent models with CUDA engines, and developers integrating runtime memory measurement and KV storage for RFC #192. They need predictable readiness, interpretable runtime memory observations, and preparation that preserves serving outputs and state.

PR necessity and bugfix evidence

A PR is needed to introduce a shared preparation entry point, readiness ordering, and phase reports for the subsequent measured-sizing work. Configuration changes alone cannot provide this lifecycle. This is a feature implementing the first step of RFC #192, rather than a standalone bugfix or completion of the whole RFC. First-request effects were measured with a real checkpoint and investigated with independent profiles.

Purpose

Add idempotent StageEngine.prepare_runtime() and run preparation before engine readiness:

  • Direct CUDA engines prepare during initialization. Workers initialize/register the connector, prepare the engine, and only then send ready.
  • Reuse current physically allocated KV and actual serving runners for bounded prefill, decode, greedy/stochastic sampling, and configured speculative work.
  • Capture the planned CUDA Graph shapes and verify representative replays. Keep existing lazy capture and memory reserves for uncovered shapes.
  • Drain participating streams and submissions before releasing scratch requests, KV leases, and async/speculative state. Preserve serving RNG and business counters; do not admit scratch requests to the scheduler or publish prefix entries.
  • Report stage timings, allocator/driver memory observations, Graph activity, and coverage gaps. Failed preparation cleans up partial state and does not produce a successful report or worker readiness.
  • Keep CPU preparation free of synthetic workloads and document the runtime lifecycle.

This PR keeps estimated KV sizing and existing capacity allocation. It does not implement VMM, measured final sizing, a larger KV pool, or final capacity publication. The integration interfaces with the other RFC owners still need agreement in #192.

Phase driver-free values are boundary samples, not a continuous physical memory high-water mark. The report retains the allocator cache rather than calling empty_cache between stages; current KV byte accounting includes FP8 scales.

See the maintainer-confirmed split. This PR has no dependency on the other implementation branches.

Refs #192

Test Plan

vllm-rlt Version / Commit: base bed34006ea8dc5b6ba1a3abd99a9fd81bd91a31b plus head c8868f03ee04f5858bb7f8be73a02bfb359f319d.

Environment: Linux, one RTX 5090 32 GiB without CPU offload, Python 3.11.15, PyTorch 2.11.0+cu128, Triton 3.6.0, NVIDIA driver 580.178.04.

  • Full Linux CPU regression suite and targeted startup/Graph/async/speculation/split-prefill/input/BF16/FP8 GPU regressions.
  • Full Ouro-1.4B BF16 checkpoint, SHA256 58872a72616c736595b8b7662079c5b12c5a162ec16eae94f21c348dfa9885af.
  • Compare clean main, deferred candidate, and startup-prepared candidate: five execution modes (sync Graph, async Graph, staged-prefill + async Graph, sync eager, async speculative) and BF16/FP8 KV, totaling 30 fresh processes. Use 2048 KV blocks, 16-token pages, maximum three requests, chunk/token budget 128, input lengths 31/257/1100, eight output tokens, greedy and fixed-seed temperature 0.8/top-k 20/top-p 0.9 sampling.
  • Measure performance in separate processes with profiling disabled. Collect independent ops-only and full CPU/CUDA profiles; full enables shapes, memory, and stacks, while ops-only disables all three.

Regression commands:

pre-commit run --all-files
python -m pytest -q -m "not gpu"
CUDA_VISIBLE_DEVICES=<selected-device> python -m pytest -q --run-gpu tests/test_runtime_startup.py
CUDA_VISIBLE_DEVICES=<selected-device> python -m pytest -q --run-gpu \
  tests/test_fp8_kv_cache.py tests/test_cuda_graph.py \
  tests/test_async_speculative.py tests/test_async_state.py \
  tests/test_async_speculative_graphs.py tests/test_async_speculative_random.py \
  tests/test_split_prefill.py tests/test_input_preparation.py \
  -k 'not flash_attn and not triton_graph_fuses_unmasked_key_and_value_writes and not packed_metadata_supports_query_lengths'
CUDA_VISIBLE_DEVICES=<selected-device> python -m pytest -q --run-gpu \
  tests/test_cuda_graph.py -k triton_graph_fuses_unmasked_key_and_value_writes

Test Result

Regression tests

Check Result
Full Linux CPU 1236 passed, 35 skipped; 892 GPU cases deselected
RTX 5090 startup and separately retested ordinary-KV fixture cases 80 passed
Other applicable RTX 5090 regressions 307 passed, 2 skipped
Linux pre-commit run --all-files (lint, format, full CPU hook) Passed on a fresh Python 3.10 / PyTorch 2.11.0+cpu / Triton 3.6.0 environment
Ruff lint/format and git diff --check Passed

On Windows with PyTorch 2.8.0+cpu and no Triton, the full hook recorded 1210 passed, 927 skipped, and 26 failures. All 26 failing cases reproduce on clean bed3400: 12 missing-Triton imports, 12 unsupported FP8 CPU operations, one Huginn dense-reference tolerance failure, and one worker-pipe failure. No Windows full-suite pass is claimed.

The two GPU skips require two devices. The pinned FA4 paged path rejects SM120; 29 FA4-related cases were excluded from the applicable run, including metadata cases that hardcode FA4. The FA4 failures reproduce on clean main. Three ordinary-KV mock failures also reproduced on main; the missing key_scale=None fixture field was supplied and all three cases passed separately. No backend guard was relaxed.

Output and numerical preservation

All comparisons below are within the same KV dtype, across the three arms:

Observation Result
150 matched request cases / 1200 matched output tokens Identical token IDs and exit depths
1188 observed ordinary-sampler logit rows Exact match
17280 logical layer/plane KV observations Exact match; FP8 includes code and scale bytes
Fixed KV capacity and corresponding storage bytes Unchanged

Only valid KV positions were compared, not padding or unused pool storage. This validates behavior preservation rather than task accuracy or BF16-versus-FP8 equivalence. Internal speculative distributions were not inspected. Current-base full-checkpoint qualification covers Ouro; older real-checkpoint results for other models are not counted here.

Independent unprofiled performance

Same pinned checkpoint; 257 input tokens, eight greedy outputs, four loops, one active request, 2048 BF16 KV blocks, 16-token pages, chunk/token budget 128, maximum three requests, Triton with static buffers and Graphs. Three fresh processes per arm/mode, alternating order, with equivalent serving warmup before the steady request. Shared compilation caches were primed by qualification.

Values are medians [min-max]. Startup excludes weight loading. TTFT starts after add_request and measures the engine-visible first output, excluding network/tokenization.

Metric Sync main Sync prepared Async main Async prepared
Engine startup (s) 0.016 [0.016-0.016] 2.042 [1.979-2.052] 0.018 [0.017-0.019] 3.181 [2.996-3.244]
First-request TTFT (ms) 728.0 [707.9-734.5] 140.8 [132.4-144.3] 749.9 [702.6-772.2] 141.1 [140.5-148.8]
First-request completion (ms) 1001.5 [989.2-1016.4] 251.1 [242.3-255.7] 1054.4 [953.7-1061.5] 240.7 [240.1-248.6]
Steady request output throughput (tokens/s) 32.840 [32.372-32.975] 32.702 [31.834-33.451] 33.455 [32.610-34.689] 33.606 [32.572-34.422]
Steady p95 output interval (ms) 15.795 [15.749-15.847] 15.823 [15.782-15.944] 14.685 [14.635-15.957] 14.713 [14.464-14.731]

Preparation reduces setup charged to the first request after readiness, at the cost of slower startup. Startup plus first completion becomes slower: approximately 1.02 -> 2.29 s sync and 1.07 -> 3.42 s async, excluding weight loading.

Steady throughput changes by -0.42% sync and +0.45% async with overlapping ranges. These measurements do not establish sustained throughput improvement or absence of regressions on all workloads. Throughput is eight successful tokens divided by one request's elapsed time, not saturation throughput. Each per-process p95 uses seven output intervals; the reported median of three p95 values is descriptive only.

The prepared ready state retains approximately 24.55 MiB more PyTorch allocated memory and 76 MiB more reserved memory; fixed KV capacity is unchanged.

Profiler observations and scope

Twelve valid profiles (six independent ops-only/full pairs) cover the complete first decode iteration and its startup-preparation counterpart. Every full profile has nonempty shapes, memory timeline, and stacks. An export-stability diagnostic hook was used only in full-profile collection, not production or clean timing.

Ops-only observation Main first decode Prepared first decode Prepared startup decode
Sync kernel calls 8194 5473 8193
Async kernel calls 9564 6844 8198
Sync Graph instantiations 1 0 1
Async Graph instantiations 1 0 1

StageEngine.prepare_runtime -> prepare_runner -> actual runner submissions -> Graph capture/replay moves first-use work earlier; it does not reduce model arithmetic. First-decode capture happens after the first output and cannot alone explain the TTFT change. The profiles establish capture relocation, not a complete causal decomposition of TTFT. Steady Graph keys, capture counts, and fallback counts stayed fixed while replay counts increased.

Author self-review covered ownership, lifecycle/failure cleanup, execution contracts, and shared logic; no further actionable issue was found. This is not an independent review. Multi-GPU/NIXL, unsupported FA4, unreported long contexts, and mixed speculative/prompt overlap remain unqualified. Measured final sizing and post-storage-commit replay validation belong to the later RFC integration.

Startup coverage follow-up (4a5b198)

This reporting-only follow-up adds actual completed recurrent decode batch_size/history_tokens pairs, including concurrent preparation. It explicitly retains estimated transient allowances for unreported long-context/batch combinations and random block speculation. The default startup workload, sizing policy and serving execution are unchanged. Pipeline speculation remains greedy-only. Regression tests and runtime documentation cover the reporting contract.

Validation on the final patch: Linux pre-commit run --all-files passed; full regression selection recorded 1242 passed / 35 skipped, with 896 device cases deselected. The selected-device startup suite recorded 87 passed; other applicable regressions recorded 326 passed / 2 skipped / 32 deselected. Unsupported paged FA4 SM120 cases and the previously disclosed fixture paths remain outside the applicable selection; no full-device-suite pass is claimed.

At fixed 768 KV blocks and 16-token pages, the pinned Ouro-1.4B BF16 checkpoint was compared against 060203d across sync, async, staged prefill and async block speculation with BF16/FP8 KV. Three capacity-feasible workloads used prompt lengths 3064, 1016/1016/1016, and 2024/504/504, with eight subsequent positions reserved per request. All 112 matched request pairs / 672 output tokens preserved IDs, depths, finish reasons, observed ordinary-sampler logits and valid prompt KV including FP8 scales. Comparisons are within each execution mode and KV dtype; these are behavior-preservation checks, not task-accuracy or BF16-versus-FP8 equivalence results.

A separate controlled serving-runner probe exercised three 1016-token histories in one speculative round (nine verification rows): greedy and seeded-random outputs and accepted counts matched in both KV dtypes. Normal scheduling did not reach three requests in one speculative round for the equal-length workload, so this runner check is separate. Its incremental PyTorch allocated peak was approximately 0.93 MiB greedy / 27.24 MiB random. These tested capacity boundaries do not cover the model's 64K context or every overlap/workspace shape.

Unprofiled reverse-order startup rechecks (three repeats after shape priming) gave prior/follow-up medians of 1.1198/1.1114 s sync, 2.0625/2.0739 s async, and 3.0775/3.0057 s async speculation. Initial and reversed-order effects differed; no stable startup performance gain or regression is established. All new remote experiment files were removed after retrieving the results.

Prepare the serving runners after connector initialization and before worker
readiness. Direct CUDA engines also prepare during initialization.

- Reuse serving execution and current KV allocations for bounded prefill,
  decode, sampling, and configured speculative warmup
- Capture planned Graph shapes and verify representative replays while
  retaining lazy capture and reserves for uncovered shapes
- Drain device work before releasing scratch requests, KV, and async state;
  preserve serving RNG and business counters
- Add idempotent preparation and phased coverage, Graph, and memory reports
- Keep estimated KV sizing and make CPU preparation workload-free
- Add lifecycle, failure, BF16/FP8, and request-reuse regressions and docs

Real-checkpoint validation preserves outputs within each KV dtype. Preparation
reduces first-request setup at the cost of slower startup; it does not establish
a steady-state throughput improvement or implement the rest of RFC InferMatrixLab#192.

Refs InferMatrixLab#192
Clarify that the startup report is a preparation snapshot and does not
remeasure or validate subsequent storage commitment.

Document full-reservation graph views, backed-block publication,
physical VMM memory accounting, post-commit replay validation, and
the current ordinary-storage requirement for PD/NIXL.
…imits

Record completed recurrent decode batch/history pairs, including concurrent
preparation, without adding startup workloads. Keep explicit estimated
transient allowances for unmeasured combinations and random block speculation.

Add regressions for capacity limits, staged/concurrent execution and the
block/pipeline sampling distinction, and document the report contract.
kunkunblueberry added a commit to kunkunblueberry/vllm-rlt that referenced this pull request Nov 19, 2026
Turn the RFC InferMatrixLab#192 step-1 preparation report into a KV block count, and let
KVCacheManager grow its published range without reallocating tensors.

- measured_kv_capacity() reads the final drained phase of the report: the
  driver's remaining memory is the physical budget, so weights, execution
  buffers, captured graphs and any live mapping are never subtracted twice.
  The utilization cap still bounds the instance, and the transient allowance
  and safety reserve are explicit inputs that are recorded with the result.
  Coverage gaps reported by the preparation (long contexts, random
  speculation, unreported batch/history combinations) travel with the plan,
  so an estimated margin is never presented as a measured one.
- KVCacheManager accepts a capacity source whose tensors span the full
  reservation and whose physical backing grows by commit(). Requests are only
  ever handed published blocks, publishing requires the storage to report the
  blocks as backed, and published <= committed <= max is enforced, so no
  allocation can address unmapped memory. Release stays reference-based, as
  agreed for the storage contract: the owner drops references, the manager
  never unmaps.
- With no capacity source nothing changes: tensors come from torch.empty and
  max == committed == published.

Engine wiring (prepare_runtime -> plan -> commit -> publish -> replay and
representative execution -> ready) needs InferMatrixLab#198 for the report and InferMatrixLab#194 for the
storage, so it is the next step rather than part of this change.

Signed-off-by: kunkunblueberry <kunkunblueberry@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant