Skip to content

ci: generate release baseline benchmarks with the baseline environment - #1021

Merged
kvlonge merged 1 commit into
mainfrom
ci/release-perf-baseline-env
Oct 8, 2026
Merged

kvlonge merged 1 commit into
mainfrom
ci/release-perf-baseline-env

Conversation

@kvlonge

@kvlonge kvlonge commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

Why

The release PR's "Compare with the previous release" job failed before measuring: the previous release's benchmark generator ran with the head interpreter and only the baseline source on PYTHONPATH, so it imported the head sqlbuild package. Once head removed a constant the baseline generator still imports, generation failed with ImportError. The compile ratio guard had already been fixed for the same class of bug.

Changes

  • Probe the baseline environment's interpreter for its package directories and put them on PYTHONPATH after the baseline source and ahead of the head packages, so sqlbuild, its native module and its runtime dependencies resolve to the baseline install. The head interpreter still supplies test-only generator dependencies (pytest) that the baseline runtime environment lacks. Applies to both the pristine and dense baseline generators.
  • Fail clearly if the baseline environment uses a different Python minor version.
  • Widen the workflow path filter to the other comparison helpers the job imports.

Verification

  • New tests: a baseline generator importing a sqlbuild symbol absent from this checkout succeeds (pristine and dense); both fail without the fix.
  • Ran the full comparison locally against baseline 0.139.0 with small model counts: baseline generation and every measured command complete; generator imports of sqlbuild, polyglot_sql, duckdb and scripts resolve to the baseline install/source.
  • make check-ci, focused release/compile performance tests, public-repository scan.

The release comparison ran the previous release's benchmark generator with
the head interpreter and only the baseline source on PYTHONPATH, so the
generator imported the head sqlbuild package and failed once head removed a
symbol the baseline still imports. Put the baseline environment's package
directories ahead of the head packages so sqlbuild and its native
dependencies resolve to the baseline install, and require the baseline
environment to use the same Python minor version.
@kvlonge
kvlonge enabled auto-merge (squash) October 8, 2026 04:32
@kvlonge
kvlonge merged commit 99a76e2 into main Oct 8, 2026
34 checks passed
@kvlonge
kvlonge deleted the ci/release-perf-baseline-env branch October 8, 2026 05:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant