Repository navigation
feat(analysis): analyse models with SQLQuack's native parsers and semantics - #1397
Merged
Merged
Conversation
The native dialect trait now has parse and normalize_type, with read_whole and type_match provided on top. read_whole refuses parser gaps, empty input, more than one statement and statements the parser kept unparsed. type_match reports NotCompared when either side has no normalized type, so the contract check makes no finding instead of guessing. Parser gaps carry the parser's message and whether the failure is a syntax error, and the P026 skip appends that message. AnalysisRoute gains an Unavailable arm, which skips a model with P026 instead of analysing it. The contracts type comparer goes through the route's type_match, and normalize_text_type is public so native dialects can reuse the shared text type rules.
A native dialect now names its sqlquack-semantics profile, and the route provides analyse and macro_body on top of read_whole. analyse turns the query's lineage and findings into the facts the analysis session keeps: output columns typed through the dialect's normalize_type, lineage rows over the model's referenced resources, the columns deciding rows, the star flags, and findings in SQLBuild's codes through the same sqlbuild_code the Polyglot path uses, so implicit-conversion findings stay blocking. Unknown functions a dialect declares as extensions are not reported. StandInSchema gives probe queries, such as a macro body's, one relation of known columns. read_whole now reports SQL with no statement, such as a lone comment, as no statement rather than more than one.
The text fallback gave family Timestamp to any type whose text held TIMESTAMP, so an ARRAY, MAP or STRUCT holding a timestamp read as a timestamp where Polyglot reads it as Other. The family now comes from the leading type name, so TIMESTAMP(3), TIMESTAMP WITH TIME ZONE and TIMESTAMP_NTZ stay timestamps and complex types do not.
…p stubs in code type_match now takes the inferred type as optional, and NotCompared keeps whether the declared type alone forbids null. A ClickHouse column declared without Nullable that receives a nullable value of unknown type is flagged again, while no type finding is made. ClickHouse types are compared without whitespace outside quoted strings, so Decimal(10, 2) is Decimal(10,2). Respelling cursor stubs for a native parser now leaves text inside strings and comments, and longer names that end in a stub's text, as written.
…alects The dynamic pivot proof reads Polyglot's parse of the model. A dialect on a native or unavailable analysis route now gets a proof failure naming why, instead of a Polyglot parse. Every dialect is still on the Polyglot route, so no proof changes yet.
…lysed dialects" The pivot proof moves onto SQLQuack's parser for every dialect in its own change, so no route needs to skip it.
…ules require Each analysis_route and semantic_validation entry module now exposes one function, with ClickHouse's spelling helpers and the read_whole gap in _helpers. Doc comments are one line, the route's tests run every case from a named test case table, and report_skip takes the P026 code and message together. The analysis_route module no longer re-exports its entries. normalize_text_type and clickhouse_type_match keep an exception for having no importer yet: their native dialect modules land in their own changes.
sqlquack main 19c51fa2 carries the Postgres parser work and the train that followed a9dde6a0. sqlquack-syntax-postgres now depends on stacker, which re-resolves a few Windows-only windows-sys edges in the lock.
Snowflake, DuckDB and MotherDuck models now take the native analysis route. Each arm parses with its SQLQuack parser, analyses under its sqlquack-semantics profile, and compares types through respellings that keep Polyglot's type names. The SQLite arm is in place but stays on the legacy route until its parser is in SQLQuack's depth probe. SQLQuack crates are pinned to main edfa30b2, which adds the DuckDB and SQLite parsers.
…ive DuckDB parser The native DuckDB parser reads WHERE id = ANY [1, 2], which DuckDB runs, so the seed's sql_analysis false model no longer needed its opt-out and require_sql_analysis rejected it. A NEAREST join is valid DuckDB that the parser keeps as a gap, and the Postgres and Snowflake parsers refuse it too. Only the fixture's SQL text changes in the six seed goldens.
…eference scan On a native analysis route the gap gate reads the model SQL and hooks whole, so assembly no longer reruns Polyglot's syntax validation there: a model the native parser reads gets no Polyglot P001. The gate now runs after reference scanning, so a located scanner error wins over a raw normalisation failure, and a model whose reference calls already failed P012 gets no P026. The Polyglot route keeps its order and validation.
…names A full comparison with Polyglot's normalize_type showed the DuckDB, Snowflake and SQLite tables missing names from other dialects, such as TIMESTAMP_NTZ on DuckDB, and multi-word names such as TIMESTAMP WITHOUT TIME ZONE, which came back unknown. Two spellings Polyglot treats as one type could then compare as different and raise K002. Every bare name now matches Polyglot except Snowflake's BIT, whose Polyglot family no alias can give. Type names may span five words. The deep-nesting fallback-columns case drops to 100 calls: the native DuckDB parser's nesting guard refuses 200, so that model is skipped with P026.
…wn parser A wave whose dialect routes natively now analyses each model in parallel on the analysis pool through the dialect's analyse, with the session's schema and nullability, instead of Polyglot. Placeholder defaults are filled before the parse, SQL macro checks still run on the outcome, and the Polyglot completion and cache attachment steps are skipped for native routes.
…ting rules Finding which cursor stubs and hook parameters sit in code used one quoting policy for every dialect, with backslash escapes in single-quoted strings. A DuckDB or Postgres string ending in a literal backslash, such as 'C:\', then read as unclosed and the model failed with a P001. The scan now uses the dialect's lexical syntax, so E'' escape strings, Snowflake backslash escapes and // comments still hide stubs. The dialect syntax mapping is the same one the rules crate moves to sqlbuild-sqltext.
A native route skipped the analysis cache, so every compile re-analysed every model and a warm run could not answer from stored outcomes. The cache now attaches on every route. A native wave analyses only the models the cache misses, a hit's cleaned SQL is the dialect's normalised SQL, and the Polyglot catalog is only prepared for the Polyglot route.
On a native route a model without lineage facts, such as one with SQL analysis turned off, got no fast lineage, where Polyglot used to parse it. Its lineage is now read by the dialect's own parser and analysis over the fast lineage schema, with its placeholder defaults filled. SQL the parser cannot read is still left out, and the debug event gives P026's reason with the construct, line and column.
The text fallback now takes a type's family from its top-level type, so a STRUCT, ARRAY or MAP holding a timestamp reads as other, not timestamp. Only those 666 dialect entries move; every top-level timestamp is unchanged.
The same project traced after a cold compile, from the analysis cache alone and from compile reuse, gives the compile report and traces of a run without any cache, so a restore keeps every analysed model's lineage facts.
A native route's analysis gate ran before hook expansion, when a named SQL hook is still only its name, so it read inline hooks but never named ones. A named hook the parser cannot read then reached the resource SQL check, whose Polyglot parse reported an unlocated P001, while an inline hook got P026. The gate now runs after hook expansion and reads both kinds, and on a native route the resource SQL check no longer turns a Polyglot rejection of hook SQL into a P001: the gate has given P026 for SQL the dialect's parser cannot read.
A native route analysed every model with no extension functions, so a function the adapter declares a return type for was reported unknown. The session now passes those function names to the dialect's analysis.
…s as fensu requires A native model outcome and a native fast lineage are now entry modules of the analysis route, analysis_route::main::model_outcome and ::fast_lineage, each with one visible function. The session builds its own schema for the outcome, fast lineage passes in the references it scanned, and where a parser gap is and why it is unreadable live in one route helper. The warm-restore lineage test takes its stores from local test cases. Behaviour is unchanged.
Pins SQLQuack at main eba39976 and lists SQLite among the native dialects. The generated-contracts golden records 47 K002 rows that the native type comparison now leaves NotCompared: declared or inferred types it cannot normalise, such as CHAR(3) under DuckDB or malformed spellings.
…ack-io/sqlbuild into wip/native-arms # Conflicts: # crates/sqlbuild-analysis/src/analysis_route/_helpers/mod.rs # crates/sqlbuild-analysis/src/analysis_route/tests.rs # crates/sqlbuild-analysis/src/analysis_route/tests/test_types.rs
…arity link Not a SQLQuack main commit yet; the pin moves to main once the link lands.
…t start The native DuckDB route places the invalid DATE literal's B218 at the statement start, because SQLQuack literals carry no span. Code, line and message are unchanged; the column, end position and the literal's note go.
On a native route, contract type checks asked the dialect to match the declared and inferred types of every column, and the dialect parses both spellings each time. The Polyglot route already normalized each spelling once per project. A dense 3,000-model DuckDB project spent 220 ms in contracts against 57 ms on the Polyglot route. The project's type comparer now keeps each native match by declared type, inferred type and nullability, so a repeated pair is parsed once. Contracts on that project take 61 ms; findings are unchanged.
SQL-test planning read every model template with the native parser's strict parse after the Polyglot token fast path was removed, so dense projects spent their write phase waiting on template parses. simple_analysis_template restores the bounded recognizer for DuckDB and Snowflake on SQLQuack's dialect tokens. It reads the template after markers_as_names, so a relation named for a marker call reads back as that call, as named_marker_calls does. Text outside the tokens must be whitespace and no token may carry a comment; anything else, and every other dialect, keeps the full parse. A differential test compares the recognized facts with parsed_analysis_template over generated templates and their token deletions and insertions. Compiled output is byte-identical on the dense 3000-model project and DuckShop. Dense 3000 write_ms: 670-800 ms before, 96-124 ms after; main is 103 ms.
fensu.toml takes main's control-store read allowlist entry whole. The earlier merge of c650fcd had kept the cutover's shorter list, which dropped eight of main's paths.
The native analysis route makes cold compiles slower than main on the same runner. Kevin accepted a temporary allowance on the three gates it fails; CHI-1177 restores 1.10 and removes these lines. The limits are the ratios measured at b827e88 against main a1b06e6 on an 8-core runner, the higher of two rounds, plus 10%: - dense cold: wall 1.326/1.347 -> 1.49, CPU 1.642/1.677 -> 1.85 - fresh cold: wall 1.285/1.153 -> 1.42, CPU 1.551/1.404 -> 1.71 - dense model_analysis_cpu_ms (--select=*): 2.660/2.554 -> 2.93 compare_compile_performance gains --cold-max-ratio and --cold-max-cpu-ratio, the cold counterparts of the edit overrides, so the warm and edit modes of the same jobs keep 1.10 and their existing edit limits. The contracts and custom-rules phase gates, the absolute budgets and every other ratio are unchanged.
…lk dupes entry The allowance comments name their removal condition instead of a ticket. fensu.toml gets back main's allowlist entry for the three JSON syntax tree walks (type_trees.rs, dict_walk.rs and walk.rs), which an earlier merge of main had replaced.
…te recognizer as fensu requires The native parse store logs through the SQL-test scan store's debug helper, both on the sqlbuild.compile logger, instead of a copy that called into Python on its own. The bounded template recognizer and its tests handle a failed scan or tokenize explicitly, pass the dialect instead of matching on its name, and keep doc comments to one line. Behaviour is unchanged.
SQLQuack now builds each dialect only with its feature, so SQLBuild enables the ones it reads: every dialect's semantics, and fingerprints for the seven dialects SQLQuack fingerprints, Databricks included. The Snowflake-family parse entry points (parse_strict, parse_statements_with, dialect_tokens, tokenize) are reached through sqlquack-parse. SQLClean moves to the commit pinning the same SQLQuack, so Cargo.lock holds one SQLQuack source. ClickHouse's engine type spelling is no longer public in SQLQuack. The ClickHouse arm keeps it as a local helper copied from SQLQuack with its family and alias tables; across 38,603 spellings (5,837 known types) it returns exactly what the previous public function returned. Databricks fingerprints now keep the case of columns named Any, Some and Order, so their nine strict xfails become ordinary cases.
v0.6.4 is the release of the commit already pinned: it changes only SQLClean's version and pins the same SQLQuack f9e4ae6e.
It went unused once the store logged through the shared debug helper.
6 of 7 tasks
kvlonge
added
the
full-verify
Amp for GitHub
kvlonge
enabled auto-merge (squash)
November 9, 2026 22:46
… for the native route The failure corpus already expects P001 for its syntax cases, since the native DuckDB parser rejects them as syntax errors rather than skipping analysis. The goldens still recorded P026. SQLQuack literals carry spans again at f9e4ae6e, so the unsupported DATE literal's error points at the literal (column 21) with its type note, as it did before the native route.
The native parser switch raises cold compile cost by the ratios the same-runner guards already allow. Scale the dense all-rules cold wall budgets by the approved 1.49 dense cold wall ceiling and the fresh-process wall budgets by the approved 1.42 fresh cold wall ceiling, at every size. Warm, edit, CPU-ratio, memory and every other budget are unchanged. The multipliers go when the 1.10x ratio guards return.
The varied shared-graph cache guard hashes the compile JSON and compiled artifacts like the fresh-process guards, and changes for the same two reasons: on 1,000 models, 143 models gain one column-lineage edge each, and three compiled SQL test files list their CTEs in a different order. Nothing else in the compile JSON or the artifacts differs from main. CI and a local run give the same four fingerprints.
SQLClean 0.7.0 lays out Databricks with SQLQuack, where earlier releases left Databricks files as written with a warning. The left-as-written case now uses ClickHouse, which SQLClean does not format. The backtick and layout-only Databricks cases expect the formatted output, and the backtick case also checks that a second format changes nothing.
kvlonge
disabled auto-merge
November 10, 2026 03:00
SQLQuack 56c7c15e brings faster statement analysis. SQLClean v0.8.0 pins the same SQLQuack revision, so the lock keeps one SQLQuack source.
kvlonge
enabled auto-merge (squash)
November 10, 2026 04:22
…shape The native route analysed every model on its own, while the Polyglot route shared one analysis between models whose SQL differs only in the relations they read. Dense projects repeat a few hundred shapes across thousands of models, so the native route did several times the work. A wave now keys each model on its analysis SQL with every referenced relation renamed to its position, the shapes of the relations it reads under that renaming, and the order its reference names sort in, since lineage lists sources by relation name. Models with equal keys take one analysis, whose lineage maps back to each model's own resources. A model whose key cannot be built exactly is analysed on its own: names that are not plain lower-case identifiers, that the parser reads differently from a plain name, or that the SQL also spells outside a reference. A shared analysis with findings, a failure, or a relation name in its output is not reused either. Compile metrics count shared and unshared models.
… analyses
A reference written next to other name characters, as in
__source("string")_split(...), becomes part of a longer name whose meaning
depends on the relation name, so the share key now requires every
relation stub to stand alone, with fixed-width positions so trailing
digits cannot run into it. Names must also start with a letter: on
Snowflake a quoted name starting with an underscore and an unquoted one
sort in an order neither name order in the key captures.
…fier helper The share key's name rule is the lowercase identifier rule sqlbuild-core already provides, so call is_lowercase_identifier instead of keeping a second copy.
Native analysis sharing moves into the analysis route's analysis helpers, next to the per-model SQL, analysis and outcome steps it composes, and the session reaches it through one entry, native_outcomes. The renamed native normalisation gets its own semantic-validation entry, the share tests become one table-driven test with their helpers in tests/helpers, and fallible steps are matched explicitly. Behaviour is unchanged.
Contributor
Author
|
Merging past one failed job in Verify run 42815134266 on 6f25453; every other job passed, including the 10,000-model fresh-process cache guard and the DuckDB CLI E2E suite. Failed job: "Fresh-process compile cache (3000 models)", case Same-machine A/B of 70ae4d9 vs 6f25453 on that case (fresh builds, 4 pinned CPUs,
The CI miss reflects runner speed, not this change. Making the absolute wall budgets relative to the same-runner control is tracked as a follow-up. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Model analysis moves from Polyglot to SQLQuack's native parsers and semantics for every shipped adapter. Custom adapters and dialects without a native module get no SQL analysis.
Changes
sqb formaton a Databricks project rewrites layout and keyword case.Verification
make test: 12,125 passed. 6 signal-timing tests fail locally on main too; CI is the reference.