Skip to content

feat(native): make native discovery the default compiler engine - #1003

Merged
kvlonge merged 5 commits into
mainfrom
feat/native-discovery-default
Oct 7, 2026
Merged

kvlonge merged 5 commits into
mainfrom
feat/native-discovery-default

Conversation

@kvlonge

@kvlonge kvlonge commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

Why

Native discovery matches Python discovery byte for byte on every corpus checked, and it is faster on cold and edit compiles. It still deferred to Python on Python 3.13 and 3.14, because it carried only the 3.12 Unicode table and the 3.12 inspect.cleandoc. This change makes it exact on every supported Python and makes it the default.

Changes

  • Generate one str.isalnum() table per Unicode version of a supported CPython (15.0.0, 15.1.0, 16.0.0). Select the table by unicodedata.unidata_version on each call. Reproduce inspect.cleandoc per minor version: 3.13 changed it to strip spaces only. An unknown Python or Unicode version still defers to Python.
  • SQLBUILD_COMPILER_ENGINE now defaults to native. python remains available as the oracle, and the CI differential still compares the two engines.
  • Compiler and Rules stores remain namespaced by engine, so the first compile after upgrading is cold once. Semantic fingerprints do not change.
  • Tests and docs that named the Python engine's cache paths now use the active engine's directories.
  • A CI job on Python 3.13 and 3.14 runs the text oracles and a bounded engine differential.
  • A new E2E test shows the unset default runs native discovery with output identical to Python.

Verification

  • make rust-check, make check-ci and make test.
  • Text oracles (every code point; seeded cleandoc bodies) pass on Python 3.12, 3.13 and 3.14.
  • The compiler differential is identical (python vs native) on the per-PR corpus on 3.12, 3.13 and 3.14, on the dense 3,000-model corpus with and without custom rules, and on a private real project.
  • Fresh-process compile timing on a real project (five alternating rounds per engine): discovery takes about 30% less time on cold and edit compiles. Whole compiles are faster or unchanged; no-change compiles reuse the previous compile in both engines.
  • Focused E2E runs pass for engine switching, compile reuse, cache invalidation, the Rules cache, scope and skills.

Native discovery deferred to Python whenever the running Python's Unicode
data differed from the one table it carried (CPython 3.12), so 3.13 and
3.14 never ran natively. Generate a `str.isalnum()` table for the Unicode
version of each supported CPython (15.0.0, 15.1.0, 16.0.0), select it by
`unicodedata.unidata_version` per call, and reproduce `inspect.cleandoc`
per Python minor version (3.13 strips spaces only). Unknown Python or
Unicode versions still defer to Python.
With SQLBUILD_COMPILER_ENGINE unset, compiles now discover project inputs
natively. The differential harness is byte-identical on the per-PR corpus
(on Python 3.12, 3.13 and 3.14), the dense corpus and a private real
project, and cold, edit and no-change compiles are no slower.
SQLBUILD_COMPILER_ENGINE=python keeps the Python discovery as the oracle,
and CI keeps comparing both engines.

Compiler and Rules stores stay namespaced by engine, so the first compile
after upgrading is cold once; semantic fingerprints do not change.
Tests that hardcoded the Python engine's cache paths now resolve the
active engine's directories, and an E2E test proves the unset default
runs native discovery with output identical to the Python engine.
…very on Python 3.13 and 3.14

Docs now name the native engine's compile reuse and Rules cache
directories, the unused Python-engine reuse path constant is removed, and
a CI job runs the native text oracles and a bounded engine differential
on Python 3.13 and 3.14.
@kvlonge
kvlonge enabled auto-merge (squash) October 7, 2026 10:36
@kvlonge
kvlonge merged commit d0c6760 into main Oct 7, 2026
38 checks passed
@kvlonge
kvlonge deleted the feat/native-discovery-default branch October 7, 2026 12:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant