Repository navigation
perf(compile): reuse per-file facts on warm compiles - #886
Merged
Merged
Conversation
Add a content-addressed fact cache under target/cache/compiler/facts-v1. Entries are keyed by every input they depend on plus the producing code identity (sqlbuild, Polyglot and PyYAML versions, Python version, and a source stat fingerprint for editable installs), verified by digest, and restricted to SQLBuild value types on load. Corrupt, foreign or stale entries are misses, and --no-cache, compile_cache = false and SQLBUILD_DISABLE_COMPILE_CACHE disable reads and writes. Warm compiles reuse per-file source YAML and SQL test discovery, per-test native SQL test extraction, and per-test expected-model relationship scans; only edited files and tests are recomputed. Add unit coverage for the store and fresh-process E2E coverage that edits models, sources, tests, macros, enums, constants, vars, targets, local config, Python loaders and environment variables after a warm run and compares the result with a cache-disabled compile.
The MODEL header regex was evaluated separately for header token preparation, header parsing and header column locations. Match it once per file and reuse the match; public helpers keep their behaviour.
Model cache identities and custom-rule model fact identities serialize and hash every model on each run, including fully cached warm runs. Compute them on the bounded native rule pool; results and error order are unchanged.
…tarts Hashing large model SQL and reading project files release the GIL per operation and then stall behind the GIL-bound native rules thread. Compute SQL rule identities and collect project files before starting that thread. Encode custom-rule project facts without key sorting; the encoding stays deterministic and injective, so cache validity is unchanged while the fingerprint no longer sorts millions of small mappings.
Store one fact per slot with the slot as the primary key and verify the expected cache key on read. Publishing becomes one upsert per fact instead of a slot-index delete plus insert, while retention stays bounded to one entry per file or test.
…tain writer errors
kvlonge
enabled auto-merge (squash)
September 30, 2026 08:09
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A warm compile of the dense 5,000-model benchmark took ~20 s even with nothing changed: source YAML parsing, SQL test extraction and expected-model scanning were redone for every file, each MODEL header was matched several times, and native rule identities were computed serially while Python and Rust threads contended for the GIL.
Changes
target/cache/compiler/facts-v1/for parsed source YAML, SQL test files and SQL test facts, keyed by file contents plus code versions (editable installs include source file stats); checksummed; corrupt databases rebuilt; errors never cached; respects--no-cache,SQLBUILD_DISABLE_COMPILE_CACHEandcompile_cache = false. Writes run in a background thread with multi-row inserts sized for SQLite before 3.32; non-corruption SQLite errors skip the write without output.--no-cache.Verification
make test,make check-ci, dense 5000 and cache 3000 guards under 4 GiB; one review plus follow-up.