Repository navigation
feat(native): make native model analysis the default with a native analysis cache - #1079
Merged
Merged
Conversation
…lysis parity test
…unts The legacy branch's cast-constant template shifts the template rotation, so the fast CTE-recovery floor is 39. Every seeded CTE-recovery and legacy-analysis floor in the parity cases is now its exact deterministic count.
…on facts Main's finished-session facts read the NativeEnrichment lineage variant, which the legacy branch renamed to NativeFacts. Native legacy analyses now keep session facts like other native analyses, and the Python-analysed facts case uses a model that still defers. The shared-analysis parity test passes inference profiles.
The type system, model analysis, semantic checks, contracts, fast lineage facts, SQL-test glue and project assembly stages move from the preview tier to shipped, so an unset SQLBUILD_COMPILER_ENGINE runs them natively. SQLBUILD_COMPILER_ENGINE=python remains the oracle. Tests that expected only native-preview to answer these seams now expect both native engines to, and the shipped differential requires analysis coverage.
…rf guard make compiler-baselines now records the shipped native engine answering the analysis stages exactly as native-preview does; goldens are unchanged. The same-runner edit guard accepts analysis cache bypasses, which native model analysis reports instead of misses, as proof that an edit was analysed.
Native analysis requests searched every model's SQL with a case-insensitive regex. An upper-cased substring prefilter, which every match must pass, skips the search for most models; a generated test matches Python's search.
Native model analysis now reads and fills one content-addressed store, model-analyses.bin, next to the other native stores. Each key covers the model's whole session request, the session-wide request, and the types, nullability, closed shapes and catalog schema of every relation the model names; native legacy analyses also require the digest of every relation they read. The store environment adds the build identity and Python's analysis cache identity: inference profile, versions, target, vars and known declarations. Python's answers and failed analyses are never stored, and an unreadable store opens empty. Hits replay the catalog changes an uncached batch makes, so output equals --no-cache. Large batches are prepared in chunks while the previous chunk is analysed, and member keys, sharing and diagnostics decode run on the analysis pool.
…hecks once Every source and SQL test carried its whole file's text into the native metadata checks, so a large sources file crossed once per source. Rows now name a file by index into one list of texts, shared natively. The semantic completion counts follow the earlier change that keeps native legacy analyses' session facts.
…quality An E2E test compiles twice with the native analysis store and checks each change misses exactly the models it affects, an unreadable store misses everything, and the incremental output equals --no-cache. The Python cache sabotage case now pins the Python engine, whose cache it breaks. The same-runner performance comparison checks the head's last warm and edit compile against an uncached compile of the same project, and no longer accepts analysis cache bypasses as cached runs.
The native session analysed every shared query again in each wave, while Python's binding catalog remembers exact, fully projected shared results and projects them onto later members with the same complete native input. The session now keeps the same memo, keyed as Python keys it, and remembers no result with binding diagnostics, which are located in their own member's SQL. Cache keys digest each relation's facts once per wave, and the set-operation search is tried only at offsets where a keyword can start. Tests show every relation fact reaches its digest and the reading model's key.
The same-runner phase gates compared python against native-preview; native is now the default engine for these phases, so the gates compare it instead.
… digest Shared-result keys held each member's whole shared-query identity as text, so a 10,000-model wave kept hundreds of megabytes of keys; they are now digests, the query identity hashed once per shared query. Large waves are analysed in chunks of about 512 members, a shared query's members together, so later chunks reuse earlier chunks' exact results and each native response is freed before the next.
The crates guide lists the analysis stages in the shipped tier, and the compile docs describe the native model-analyses.bin cache and its keys.
On main with the M3 close-out, the shipped engine's entries again equal native-preview's, and preview no longer defers legacy analysis or input enrichment to Python.
With analysis shipped, the default engine no longer runs Python's analysis cache, dataflow scheduler, shared-result memo, batched preparation or expression memo. Tests that observe those internals now pin the Python engine through a shared fixture. The target-writer spies and the planning failure helper follow the native SQL-test planner the default engine uses, and the promotion parity test counts calls only after its base compile.
Key hashing helpers are KeyHasher methods, varint limits are named constants, chunk analysis takes one input struct and returns its results, and the stub and source renames are named functions. The analysis schedule joins the mappings helpers and the set-operation search joins the syntax checks, keeping both helper containers within their module limits. Cache tests use case tables, shared helpers and expected outcomes. The phase-gate seam test asserts the planned counts it collects, and comments move into docstrings.
Each dense size now also compiles in process and checks that the native semantic metadata request carries no more file text than the project holds. Copying a sources file's text once per source, as before the file-text dedupe, sends about 188 million characters for a 1,000-model project with 11.8 million, and the guard fails.
kvlonge
enabled auto-merge (squash)
October 10, 2026 12:05
Native analysis serves unchanged models from per-model entries, so warm and edit compiles report entry hits instead of batch hits, and the build no longer writes the Python analysis SQLite cache. Memory budgets rise where native analysis holds more at peak: lineage at 3k to 1 GiB, dag at 3k to 1 GiB, and 10k fresh compiles to 2.5 GiB.
…s in the ratio script test
kvlonge
disabled auto-merge
October 10, 2026 12:56
A model whose compact analysis asked for the legacy fallback carries that request as a logged failure, and the session marked every model with a logged failure uncacheable. Models the native legacy analysis then answered, and analyses that failed outright, were analysed again on every compile. Both are now stored like other native outcomes; only Python's answers stay uncached.
… wave A native legacy analysis is keyed on every relation's types and nullability, and each wave with a stored legacy outcome digested both whole tables again. Shape tables, which only ever gain entries, now keep a digest chained over their insertions, so the whole-table digest costs two hashes. The stored format is bumped, so earlier entries miss once.
kvlonge
enabled auto-merge (squash)
October 10, 2026 13:28
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Make native model analysis the default, with a per-model cache for edits.
Changes
--no-cache. CI gates native phase CPU against Python.Verification
Focused/Rust tests, static checks, differentials (328 identical), hygiene; goldens unchanged. Cache invalidation/uncached equality covered. Fresh 3k cold 1.09x; 10k 1.18x. Latest dense edit: wall 1.138x, CPU 1.650x; unexpected SQL-test replanning remains unresolved. Guard tests cover scoped limits and above-limit failures.
Expectation changes: