Skip to content

test(perf): scale the cache compile guards to the runner and restore the 1.10x cold ratio - #1412

Closed
kvlonge wants to merge 4 commits into
mainfrom
test/fresh-cache-runner-scale
Closed

kvlonge wants to merge 4 commits into
mainfrom
test/fresh-cache-runner-scale

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 11, 2026

Copy link
Copy Markdown
Contributor

Why

The fresh-process cache guards fail on runner speed. Unchanged code missed the 9.0 s and 33.6 s cold budgets at 9.06 s and 33.78 s, each within 6% of its own cache-disabled control. Across about 22 recent jobs per size, cold wall and CPU spread 1.6x to 1.9x between runner classes, while cold over cache-disabled stayed at 0.94 to 1.15 wall and 0.99 to 1.05 CPU.

No absolute ceiling that fits that spread catches a 20% slower compiler, and a slower shared cold path slows the cache-disabled control too. The same-runner base/head jobs are what catch it, but their cold limits of 1.42x and 1.49x, and 2.93x on model analysis CPU, were sized against main before native analysis. Main now has it on both sides: twelve cold readings since then measured 0.95 to 1.03 wall and 0.96 to 1.01 CPU, and six model analysis readings 0.96 to 1.01.

Changes

  • The fresh-process cache guard scales its wall budgets by measure_runner_calibration().factor, as the dense guards do, and prints them. Its ratio and fingerprint checks are unchanged.
  • The dense and fresh base/head cold comparisons and the model analysis CPU gate return to the 1.10x limit. The dense edit limits stay as approved.
  • The 1.42 and 1.49 multipliers on the absolute fresh and dense cold budgets stay. Their comments now say to drop them when the native cold path is optimised, not when the ratio guards return.
  • COMPILE_BENCHMARKS.md says why absolute budgets cannot catch a slower compiler.

Test-only, so no DuckShop case.

Verification

  • Same machine, fresh 3,000 models, cold, three alternating runs: identical builds gave 1.018 wall and 0.979 CPU and passed. Injecting CPU work equal to 20% of cold wall into every head compile gave 1.234 wall and 1.073 CPU and failed the 1.10x limit; the previous 1.42x and 1.71x limits would have passed it.
  • Both 3,000-model cache guards pass locally (factor 1.000).
  • make check-ci passes, except rust-lint: no Rust changed.

…the 1.10x cold ratio

The fresh-process cache guards failed on runner speed alone: cold wall and
CPU spread 1.6x to 1.9x between hosted runner classes, while cold over the
in-job cache-disabled control stayed within 1.05 on CPU. Their wall budgets
now scale by the fixed calibration workload the dense guards already use.

A slower compiler is caught by the same-runner base/head comparisons. Their
cold and model analysis allowances were sized against main before native
analysis; main now has it on both sides, and recent readings sit at 0.95 to
1.03, so they return to 1.10x. An injected slowdown of 20% of cold wall
fails the restored limit at 1.234.
@kvlonge
kvlonge enabled auto-merge (squash) November 11, 2026 09:26
@kvlonge kvlonge added the full-verify Run the full Verify matrix on this pull request label Nov 11, 2026 — with Amp for GitHub

kvlonge commented Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

Folded into #1413 so the fixes share one CI run. The branch is kept.

@kvlonge kvlonge closed this Nov 11, 2026
auto-merge was automatically disabled November 11, 2026 16:50

Pull request was closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-verify Run the full Verify matrix on this pull request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant