Repository navigation
perf(compile): close the fresh compile gap with ordered native scheduling and mimalloc - #889
Merged
Merged
Conversation
Hand out each native analysis batch through a shared queue ordered by SQL size, so the widest queries start first instead of becoming a late critical path. Results keep request order. Fresh 3000 native analysis wall falls from about 2.24 s to 1.91 s with identical output.
Native analysis spends about 40% of its time in the system allocator building and dropping syntax trees. mimalloc (local-dynamic TLS for the dlopen-loaded extension) cuts fresh 3000 native analysis from about 6.4 to 4.8 CPU-s and 1.9 to 1.4 s wall with identical output. Fresh 3000 peak memory falls from about 830 to 690 MB; dense 3000 CPU falls about 7% at similar peak memory.
Tests in one project render the same model SQL many times while lifting its CTEs into the comparison query; fresh 3000 splits about 1,150 statements from only 50 distinct SQL strings. Keep a parse of SQL once it repeats within a render batch, bounded to 1 MiB of SQL text, and clone it for each later use; per-test CTE renaming is unchanged. Unrepeated SQL is never retained, so dense projects keep their peak memory. Fresh 3000 native planning and rendering falls from about 455 to 380 ms with identical artifacts.
Fresh 3000 --no-cache is back within 1.10x of 0.119.1 on the same machine (medians of 3 interleaved runs: 8.22 s vs 7.86 s wall, 12.56 vs 12.54 CPU-s), so the temporary 1.30 limit is no longer needed.
…puts Build the same Linux, macOS and Windows wheel matrix as publish.yml, without uploading, whenever crates, Cargo manifests, pyproject.toml or the verify workflow change, and smoke-import sqlbuild._native from each built wheel. The aggregate Verify job treats the job like the other path-selected suites.
kvlonge
enabled auto-merge (squash)
September 30, 2026 16:31
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Fresh 3000-model
--no-cachecompiles were 1.175x wall / 1.16x CPU of 0.119.1. Native query analysis runs in dependency waves whose wall time was set by one or two huge queries that started late, about 40% of native CPU was the system allocator, and SQL-test comparison rendering re-parsed the same model SQL for every test.Changes
wheelsjob builds and smoke-imports every published wheel platform on PRs that change native inputs; it feedsVerify.Verification
--no-cachevs 0.119.1: 1.042x wall, 1.010x CPU (was 1.175x / 1.16x); peak memory 829 -> 682 MB. Dense 3000 20.75 -> 19.78 s.make test,make check-ci; one independent review.