Repository navigation
perf(plan): cut plan and build planning time on large projects - #875
Merged
Merged
Conversation
Rename discovery fingerprinted every unbuilt model under each rename map even when no earlier model could match, and the plan fingerprinted every entry again for storage. Skip matching without candidates, reuse fingerprints across rename maps and phases, attach stored fingerprints only to entries that will be built, and skip them for sqb plan, which never stores them.
Resolving column reads through a UNION chain materialised every branch's full argument tree at each level, so fixture validation grew cubically with fixture rows. Read only the needed arguments. Order keys with a heap instead of re-sorting the ready list, and look up the package version once instead of per logged statement.
Expression sources report column names without types, so every source looked changed and every downstream model was analysed again during planning. Treat inspected columns as changed only when names differ or a reported type differs from the compile-time shape.
Every build fingerprinted every model it built, about 7 ms on average and linear in query size, so a warm build of an unchanged 3,000-model project spent 25 s recomputing identical values. Persist fingerprints in the project-local compile cache, keyed by query, metadata, dialect, visible renames, algorithm, and sqlbuild and polyglot versions, with digest-checked entries that fall back to recomputing when damaged. Also stop walking each query's local names twice.
Plan and build located every model's output columns during discovery, which took about 2.4 s on the 3,000-model benchmark, while compile skips it. The only consumer, contract diagnostics, already computes those locations lazily when they are missing. Skip the eager scan for plan and build as compile does.
Fixture validation re-inferred the columns of every fixture body for every test, although identical bodies under the same inference profile always infer the same columns. Memoise inference per planning context, keyed by fixture SQL and an equal profile, and bypass the memo for profiles that carry a binding catalog.
Bound sqb plan --json on the 3,000-model inspection benchmark relative to a warm compile measured in the same test, cap its peak RSS, and fail when planning CPU grows super-linearly from a 300-model copy. Build a 1,000-model DuckDB benchmark from an empty warehouse with wall and RSS limits, then plan against its stored state after editing models and after renaming one. Raise the lineage ceilings to 3.0 s for CI headroom. Refs: CHI-432
kvlonge
enabled auto-merge (squash)
September 29, 2026 00:09
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
On a 3,000-model benchmark with an empty DuckDB warehouse,
sqb plantook ~139 s and 1.8 GB andsqb buildspent ~138 s planning, against ~6 s forsqb compile. It regressed in 0.118.0 (rename discovery fingerprinted every model with no possible match, then every entry again) and 0.121.0 (every expression source looked changed, so all downstream models were re-analysed), on top of cubic UNION handling in SQL test planning. No guard measured plan or build.Changes
sqb planno longer computes the build-only stored fingerprints; a local content-addressed cache undertarget/cache/lets warm builds reuse them (no warehouse state changes).Verification
sqb plan139 s -> 10.6 s (1.81 -> 0.95 GB); warm build planning 138 s -> 10.7 s.make test,make check-ci, performance target under 4 GiB; one review plus follow-up.