Skip to content

test(perf): set the dense guard calibration reference below the EPYC 7763's readings - #1398

Merged
kvlonge merged 1 commit into
mainfrom
test/dense-guard-reference
Nov 10, 2026
Merged

kvlonge merged 1 commit into
mainfrom
test/dense-guard-reference

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

Why

The dense 5,000-model guard scales its budgets by the runner calibration's fastest repetition over a reference. The 0.55 s reference sat inside the EPYC 7763's own readings, so 7763 runs got factor 1.0 or slightly more depending on noise. Their warm (about 9.2 s) and edit (about 9.6 s) medians sit right at the unscaled 9.0 s and 9.5 s budgets, so unchanged code passed or failed at random.

Changes

  • The calibration reference drops from 0.55 s to 0.50 s, below every 7763 reading so far.
  • Budgets, repetitions, the workload and the same-runner ratio jobs are unchanged. Test-only, so there is no DuckShop case.
CPU Calibration (s) Factor at 0.55 Factor at 0.50
EPYC 9V45 0.31-0.39 1.00 1.00
EPYC 9V74 0.46-0.52 1.00 1.00-1.04
Xeon 8573C 0.54 1.00 1.08
EPYC 7763 0.540-0.62 1.00-1.13 1.08-1.24
Xeon 8370C 0.67-0.68 1.22-1.24 1.34-1.36

The reference is below 7763's minimum (0.540 s), so every 7763 run now gets at least 1.08: warm 9.72 s and edit 10.26 s against observed medians of 9.18-9.22 s and 9.57-9.60 s. Two 7763 jobs on unchanged compiler code read 0.540-0.587 s, and one failed at factor 1.000 with a warm median of 9.22 s.

The factor over-reads the compile slowdown between classes, which leaves slack and never undershoots. On 8370C the absolute guard could miss a regression of up to about 45%; the same-runner ratio jobs still catch it.

Verification

  • make check-ci passes.
  • The calibration helper runs locally and reports the 0.50 s reference.
  • The factor ranges above come from recorded calibration readings on hosted runners.

…7763's readings

The 0.55 s reference sat inside the EPYC 7763's own calibration readings
(0.540 s to 0.62 s), so whether a 7763 run got a larger budget depended
on noise, while its warm and edit medians sit right at the unscaled
budgets. At 0.50 s every 7763 reading scales the budgets by at least
1.08. Classes the budgets fit keep a factor of at most 1.04.
@kvlonge kvlonge added the full-verify Run the full Verify matrix on this pull request label Nov 10, 2026
@kvlonge
kvlonge enabled auto-merge (squash) November 10, 2026 00:43
@kvlonge
kvlonge merged commit b7a4f0b into main Nov 10, 2026
42 of 63 checks passed
@kvlonge
kvlonge deleted the test/dense-guard-reference branch November 10, 2026 01:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-verify Run the full Verify matrix on this pull request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant