Repository navigation
Conversation
The native dialect trait now has parse and normalize_type, with read_whole and type_match provided on top. read_whole refuses parser gaps, empty input, more than one statement and statements the parser kept unparsed. type_match reports NotCompared when either side has no normalized type, so the contract check makes no finding instead of guessing. Parser gaps carry the parser's message and whether the failure is a syntax error, and the P026 skip appends that message. AnalysisRoute gains an Unavailable arm, which skips a model with P026 instead of analysing it. The contracts type comparer goes through the route's type_match, and normalize_text_type is public so native dialects can reuse the shared text type rules.
A native dialect now names its sqlquack-semantics profile, and the route provides analyse and macro_body on top of read_whole. analyse turns the query's lineage and findings into the facts the analysis session keeps: output columns typed through the dialect's normalize_type, lineage rows over the model's referenced resources, the columns deciding rows, the star flags, and findings in SQLBuild's codes through the same sqlbuild_code the Polyglot path uses, so implicit-conversion findings stay blocking. Unknown functions a dialect declares as extensions are not reported. StandInSchema gives probe queries, such as a macro body's, one relation of known columns. read_whole now reports SQL with no statement, such as a lone comment, as no statement rather than more than one.
The text fallback gave family Timestamp to any type whose text held TIMESTAMP, so an ARRAY, MAP or STRUCT holding a timestamp read as a timestamp where Polyglot reads it as Other. The family now comes from the leading type name, so TIMESTAMP(3), TIMESTAMP WITH TIME ZONE and TIMESTAMP_NTZ stay timestamps and complex types do not.
…p stubs in code type_match now takes the inferred type as optional, and NotCompared keeps whether the declared type alone forbids null. A ClickHouse column declared without Nullable that receives a nullable value of unknown type is flagged again, while no type finding is made. ClickHouse types are compared without whitespace outside quoted strings, so Decimal(10, 2) is Decimal(10,2). Respelling cursor stubs for a native parser now leaves text inside strings and comments, and longer names that end in a stub's text, as written.
…alects The dynamic pivot proof reads Polyglot's parse of the model. A dialect on a native or unavailable analysis route now gets a proof failure naming why, instead of a Polyglot parse. Every dialect is still on the Polyglot route, so no proof changes yet.
…lysed dialects" The pivot proof moves onto SQLQuack's parser for every dialect in its own change, so no route needs to skip it.
…ules require Each analysis_route and semantic_validation entry module now exposes one function, with ClickHouse's spelling helpers and the read_whole gap in _helpers. Doc comments are one line, the route's tests run every case from a named test case table, and report_skip takes the P026 code and message together. The analysis_route module no longer re-exports its entries. normalize_text_type and clickhouse_type_match keep an exception for having no importer yet: their native dialect modules land in their own changes.
…eference scan On a native analysis route the gap gate reads the model SQL and hooks whole, so assembly no longer reruns Polyglot's syntax validation there: a model the native parser reads gets no Polyglot P001. The gate now runs after reference scanning, so a located scanner error wins over a raw normalisation failure, and a model whose reference calls already failed P012 gets no P026. The Polyglot route keeps its order and validation.
…wn parser A wave whose dialect routes natively now analyses each model in parallel on the analysis pool through the dialect's analyse, with the session's schema and nullability, instead of Polyglot. Placeholder defaults are filled before the parse, SQL macro checks still run on the outcome, and the Polyglot completion and cache attachment steps are skipped for native routes.
…ting rules Finding which cursor stubs and hook parameters sit in code used one quoting policy for every dialect, with backslash escapes in single-quoted strings. A DuckDB or Postgres string ending in a literal backslash, such as 'C:\', then read as unclosed and the model failed with a P001. The scan now uses the dialect's lexical syntax, so E'' escape strings, Snowflake backslash escapes and // comments still hide stubs. The dialect syntax mapping is the same one the rules crate moves to sqlbuild-sqltext.
A native route skipped the analysis cache, so every compile re-analysed every model and a warm run could not answer from stored outcomes. The cache now attaches on every route. A native wave analyses only the models the cache misses, a hit's cleaned SQL is the dialect's normalised SQL, and the Polyglot catalog is only prepared for the Polyglot route.
On a native route a model without lineage facts, such as one with SQL analysis turned off, got no fast lineage, where Polyglot used to parse it. Its lineage is now read by the dialect's own parser and analysis over the fast lineage schema, with its placeholder defaults filled. SQL the parser cannot read is still left out, and the debug event gives P026's reason with the construct, line and column.
The text fallback now takes a type's family from its top-level type, so a STRUCT, ARRAY or MAP holding a timestamp reads as other, not timestamp. Only those 666 dialect entries move; every top-level timestamp is unchanged.
The same project traced after a cold compile, from the analysis cache alone and from compile reuse, gives the compile report and traces of a run without any cache, so a restore keeps every analysed model's lineage facts.
A native route's analysis gate ran before hook expansion, when a named SQL hook is still only its name, so it read inline hooks but never named ones. A named hook the parser cannot read then reached the resource SQL check, whose Polyglot parse reported an unlocated P001, while an inline hook got P026. The gate now runs after hook expansion and reads both kinds, and on a native route the resource SQL check no longer turns a Polyglot rejection of hook SQL into a P001: the gate has given P026 for SQL the dialect's parser cannot read.
A native route analysed every model with no extension functions, so a function the adapter declares a return type for was reported unknown. The session now passes those function names to the dialect's analysis.
…s as fensu requires A native model outcome and a native fast lineage are now entry modules of the analysis route, analysis_route::main::model_outcome and ::fast_lineage, each with one visible function. The session builds its own schema for the outcome, fast lineage passes in the references it scanned, and where a parser gap is and why it is unreadable live in one route helper. The warm-restore lineage test takes its stores from local test cases. Behaviour is unchanged.
…ative routes Type recovery revalidated each model downstream of a poisoned output through the Polyglot binding catalog. On a native route that catalog was never prepared, so preparing it while the semantic stage held a borrow on it panicked with "Already borrowed" whenever a model with its own binding error sat downstream of a blocking type error. On a native route, revalidation now reads each planned model's analysed SQL again with its dialect's analysis, over the schema the session bound and with the poisoned outputs untyped. The findings come back at the session's own positions, so type recovery matches them against the raw rows directly, and the Polyglot catalog is no longer on that path. The finished session keeps the model order and the declared functions it analysed with. The Polyglot route is unchanged.
…ve routes Fast lineage on a native route passed relation markers to the parser as written. Markers now read as indexed names through the shared relation marker reader, each tied to the resource it calls, so a model and a source of the same name stay apart. A source column's relation is compared as the dialect's profile resolves names, so upper-case folding dialects such as Snowflake keep their sources too, in fast lineage and in session lineage alike, and repeated sources of one column are listed once. Native analysis now adds SQLBuild's own text checks that the Polyglot route runs: truth-value tests the warehouse rejects (B103) and SQLite's temporal casts (B104). Output column facts keep the analysis's own type spelling, such as INTEGER or VARCHAR, and use the dialect's normalised name only to decide whether the type is known. The relation marker reader is the same as on the parser train branch.
…ect scans A T-SQL `[...]` name was code to the dialect scan, so an apostrophe, `--` or `/*` inside one opened a string or comment that hid everything after it. On a native route a cursor stub after such a name was not respelt, and a valid model was skipped with P026. Lexical rules gain bracket identifiers, closing at `]` with `]]` for one bracket, and the T-SQL rules set them. Adapter records never set the flag, so their keys and scans are unchanged. ClickHouse's rules now follow its lexer: a backslash escapes inside single quotes, double quotes and backticks, and `#` starts a comment only before a space or `!`.
…ble revalidations Native fast lineage gave every relation marker call its own name, so a union of two branches over one model read two relations and its columns became expressions. Calls of the same marker now share their first call's name, padded to keep positions. Native revalidation in type recovery failed the compile when the dialect's parser could not read a planned model again. Such a model now keeps its own findings, as an unrevised model would.
A native route's P026 for a hook now names the hook and its file, as
the Polyglot syntax error did, such as post_hooks[0] sql("record_refresh")
(hooks/sql/record_refresh.sql), and places the gap in the hook's SQL.
A gap in the model's query is placed at its line and column in the
model file when the text maps back through stub spelling, placeholders
and expansion; otherwise the message says the position is in the
analysed SQL.
NativeDialect gains session_type, the identity by default: the session schema and revalidation give each known column type to the parser in the spelling the dialect reads, for dialects whose parser cannot read every type SQLBuild keeps.
…d case setting A native route analysed every model against a schema with no function facts, quoted output names lost their quoting, and Snowflake's QUOTED_IDENTIFIERS_IGNORE_CASE was ignored. - The session schema now answers the adapter's declared function return types, respelt for the dialect, and its first_arg, conditional_result and safe_cast nullability rules. - A stored name in double quotes becomes a quoted schema column, with "" read as one quote, wherever the session, revalidation or fast lineage builds one. - The session request carries the profile's quoted-identifier case setting into the cache key and each dialect's analysis, so a dialect can pick a profile for it. - Semantic completion no longer panics when the binding catalog is already borrowed: unknown relations stay per-request. - An adapter's own Python nullability rules are not called on a native route; compile now warns, naming the adapter and the functions. The callback tests run on the Polyglot route, and the payload-only completion check expects a native route to require its session.
SQLQuack quotes an unknown column's name in lower case, so a quoted "MiSsPeLlEd" was reported as 'misspelled'. A native route now puts the authored name back into the message.
…on as a compiler error
…ions A native route gave SQLQuack only relations whose full shape was known, so a source or seed with declared column types but no authoritative shape was read untyped: `status + 1` over a declared VARCHAR went unreported, where Polyglot's rich analysis used the declared types. The session schema now passes each such relation's declared columns, with the nullability the session knows, as open columns. They type the columns a query reads without making the relation's column list complete, so star expansion and unknown-column checks are unchanged. Revalidation passes the same open columns. The analysis cache format moves to v4, as outcomes for the same inputs change.
The native route's dependency update moved tempfile onto windows-sys 0.52, which sqb and the Python extension then built with different features.
Contributor
Author
|
Superseded by wip/native-cutover, which contains this branch's head 098499d; the branch is kept. |
kvlonge
added a commit
that referenced
this pull request
Nov 10, 2026
…antics (#1397) * chore(deps): bump sqlquack to a9dde6a0 * feat(analysis): give native dialects a parse and type contract The native dialect trait now has parse and normalize_type, with read_whole and type_match provided on top. read_whole refuses parser gaps, empty input, more than one statement and statements the parser kept unparsed. type_match reports NotCompared when either side has no normalized type, so the contract check makes no finding instead of guessing. Parser gaps carry the parser's message and whether the failure is a syntax error, and the P026 skip appends that message. AnalysisRoute gains an Unavailable arm, which skips a model with P026 instead of analysing it. The contracts type comparer goes through the route's type_match, and normalize_text_type is public so native dialects can reuse the shared text type rules. * feat(analysis): analyse native dialect queries with sqlquack-semantics A native dialect now names its sqlquack-semantics profile, and the route provides analyse and macro_body on top of read_whole. analyse turns the query's lineage and findings into the facts the analysis session keeps: output columns typed through the dialect's normalize_type, lineage rows over the model's referenced resources, the columns deciding rows, the star flags, and findings in SQLBuild's codes through the same sqlbuild_code the Polyglot path uses, so implicit-conversion findings stay blocking. Unknown functions a dialect declares as extensions are not reported. StandInSchema gives probe queries, such as a macro body's, one relation of known columns. read_whole now reports SQL with no statement, such as a lone comment, as no statement rather than more than one. * fix(types): take a fallback type's family from its top-level type The text fallback gave family Timestamp to any type whose text held TIMESTAMP, so an ARRAY, MAP or STRUCT holding a timestamp read as a timestamp where Polyglot reads it as Other. The family now comes from the leading type name, so TIMESTAMP(3), TIMESTAMP WITH TIME ZONE and TIMESTAMP_NTZ stay timestamps and complex types do not. * fix(analysis): compare ClickHouse nullability without a type, and keep stubs in code type_match now takes the inferred type as optional, and NotCompared keeps whether the declared type alone forbids null. A ClickHouse column declared without Nullable that receives a nullable value of unknown type is flagged again, while no type finding is made. ClickHouse types are compared without whitespace outside quoted strings, so Decimal(10, 2) is Decimal(10,2). Respelling cursor stubs for a native parser now leaves text inside strings and comments, and longer names that end in a stub's text, as written. * fix(analysis): skip the Polyglot pivot proof for natively analysed dialects The dynamic pivot proof reads Polyglot's parse of the model. A dialect on a native or unavailable analysis route now gets a proof failure naming why, instead of a Polyglot parse. Every dialect is still on the Polyglot route, so no proof changes yet. * Revert "fix(analysis): skip the Polyglot pivot proof for natively analysed dialects" The pivot proof moves onto SQLQuack's parser for every dialect in its own change, so no route needs to skip it. * refactor(analysis): shape the native analysis route as fensu's Rust rules require Each analysis_route and semantic_validation entry module now exposes one function, with ClickHouse's spelling helpers and the read_whole gap in _helpers. Doc comments are one line, the route's tests run every case from a named test case table, and report_skip takes the P026 code and message together. The analysis_route module no longer re-exports its entries. normalize_text_type and clickhouse_type_match keep an exception for having no importer yet: their native dialect modules land in their own changes. * chore(deps): bump sqlquack to 19c51fa2 sqlquack main 19c51fa2 carries the Postgres parser work and the train that followed a9dde6a0. sqlquack-syntax-postgres now depends on stacker, which re-resolves a few Windows-only windows-sys edges in the lock. * chore(deps): pin sqlquack-semantics to 19c51fa2 with the other sqlquack crates * feat(analysis): analyse Snowflake and DuckDB natively with SQLQuack Snowflake, DuckDB and MotherDuck models now take the native analysis route. Each arm parses with its SQLQuack parser, analyses under its sqlquack-semantics profile, and compares types through respellings that keep Polyglot's type names. The SQLite arm is in place but stays on the legacy route until its parser is in SQLQuack's depth probe. SQLQuack crates are pinned to main edfa30b2, which adds the DuckDB and SQLite parsers. * test(differential): keep the seed opt-out model unreadable to the native DuckDB parser The native DuckDB parser reads WHERE id = ANY [1, 2], which DuckDB runs, so the seed's sql_analysis false model no longer needed its opt-out and require_sql_analysis rejected it. A NEAREST join is valid DuckDB that the parser keeps as a gap, and the Postgres and Snowflake parsers refuse it too. Only the fixture's SQL text changes in the six seed goldens. * fix(analysis): let a native route's parser decide syntax, after the reference scan On a native analysis route the gap gate reads the model SQL and hooks whole, so assembly no longer reruns Polyglot's syntax validation there: a model the native parser reads gets no Polyglot P001. The gate now runs after reference scanning, so a located scanner error wins over a raw normalisation failure, and a model whose reference calls already failed P012 gets no P026. The Polyglot route keeps its order and validation. * fix(analysis): native type spellings take Polyglot's other dialects' names A full comparison with Polyglot's normalize_type showed the DuckDB, Snowflake and SQLite tables missing names from other dialects, such as TIMESTAMP_NTZ on DuckDB, and multi-word names such as TIMESTAMP WITHOUT TIME ZONE, which came back unknown. Two spellings Polyglot treats as one type could then compare as different and raise K002. Every bare name now matches Polyglot except Snowflake's BIT, whose Polyglot family no alias can give. Type names may span five words. The deep-nesting fallback-columns case drops to 100 calls: the native DuckDB parser's nesting guard refuses 200, so that model is skipped with P026. * feat(analysis): analyse models on a native route with the dialect's own parser A wave whose dialect routes natively now analyses each model in parallel on the analysis pool through the dialect's analyse, with the session's schema and nullability, instead of Polyglot. Placeholder defaults are filled before the parse, SQL macro checks still run on the outcome, and the Polyglot completion and cache attachment steps are skipped for native routes. * fix(analysis): find a native route's stubs with the dialect's own quoting rules Finding which cursor stubs and hook parameters sit in code used one quoting policy for every dialect, with backslash escapes in single-quoted strings. A DuckDB or Postgres string ending in a literal backslash, such as 'C:\', then read as unclosed and the model failed with a P001. The scan now uses the dialect's lexical syntax, so E'' escape strings, Snowflake backslash escapes and // comments still hide stubs. The dialect syntax mapping is the same one the rules crate moves to sqlbuild-sqltext. * fix(analysis): keep the analysis cache on a native route A native route skipped the analysis cache, so every compile re-analysed every model and a warm run could not answer from stored outcomes. The cache now attaches on every route. A native wave analyses only the models the cache misses, a hit's cleaned SQL is the dialect's normalised SQL, and the Polyglot catalog is only prepared for the Polyglot route. * fix(lineage): build fast lineage natively for models without facts On a native route a model without lineage facts, such as one with SQL analysis turned off, got no fast lineage, where Polyglot used to parse it. Its lineage is now read by the dialect's own parser and analysis over the fast lineage schema, with its placeholder defaults filled. SQL the parser cannot read is still left out, and the debug event gives P026's reason with the construct, line and column. * test(types): re-record the generated type golden for nested timestamps The text fallback now takes a type's family from its top-level type, so a STRUCT, ARRAY or MAP holding a timestamp reads as other, not timestamp. Only those 666 dialect entries move; every top-level timestamp is unchanged. * test(lineage): trace fast lineage after a warm restore The same project traced after a cold compile, from the analysis cache alone and from compile reuse, gives the compile report and traces of a run without any cache, so a restore keeps every analysed model's lineage facts. * fix(analysis): gate a native route on a model's named hooks too A native route's analysis gate ran before hook expansion, when a named SQL hook is still only its name, so it read inline hooks but never named ones. A named hook the parser cannot read then reached the resource SQL check, whose Polyglot parse reported an unlocated P001, while an inline hook got P026. The gate now runs after hook expansion and reads both kinds, and on a native route the resource SQL check no longer turns a Polyglot rejection of hook SQL into a P001: the gate has given P026 for SQL the dialect's parser cannot read. * fix(analysis): know the adapter's declared functions on a native route A native route analysed every model with no extension functions, so a function the adapter declares a return type for was reported unknown. The session now passes those function names to the dialect's analysis. * refactor(analysis): shape the native route's model and lineage entries as fensu requires A native model outcome and a native fast lineage are now entry modules of the analysis route, analysis_route::main::model_outcome and ::fast_lineage, each with one visible function. The session builds its own schema for the outcome, fast lineage passes in the references it scanned, and where a parser gap is and why it is unreadable live in one route helper. The warm-restore lineage test takes its stores from local test cases. Behaviour is unchanged. * feat(analysis): analyse SQLite natively with SQLQuack Pins SQLQuack at main eba39976 and lists SQLite among the native dialects. The generated-contracts golden records 47 K002 rows that the native type comparison now leaves NotCompared: declared or inferred types it cannot normalise, such as CHAR(3) under DuckDB or malformed spellings. * build(analysis): pin SQLQuack at the DuckDB window, PIVOT and arity parity link Not a SQLQuack main commit yet; the pin moves to main once the link lands. * test(analysis): re-record the unsupported-cast golden at the statement start The native DuckDB route places the invalid DATE literal's B218 at the statement start, because SQLQuack literals carry no span. Code, line and message are unchanged; the column, end position and the literal's note go. * WIP (not shipping): Databricks' arm on the NativeDialect trait, on a SQLQuack validation pin Databricks' module parses with SQLQuack's Databricks SQL parser (parse_strict), analyses under the Databricks semantics profile, and types spellings through normalize_text_type after folding the Databricks synonyms Polyglot spells another way (INTEGER, REAL, DOUBLE PRECISION, NUMERIC, TIMESTAMP_LTZ, TIMESTAMP_NTZ). It joins NATIVE_DIALECTS under the adapter's analysis dialect name, databricks. A byte the scanner reads as no token is a syntax error, as on BigQuery's arm. Every sqlquack crate is pinned to a37cc362, a validation pin: one commit on SQLQuack main eba39976 adding the Databricks semantics profile, which is not on main yet. Repin every sqlquack crate to the SQLQuack main commit that carries the profile. The syntax error cases use SELECT 1 FROM t WHERE ) rather than FROM FROM: Databricks does not enforce ANSI reserved words, and its reference reads SELECT where from FROM where select as a query, so the parser correctly reads FROM FROM as a table named FROM. * WIP (not shipping): drop the fensu exception Databricks' arm resolves With Databricks' module typing through normalize_text_type, its FPRSL105 exception no longer matches a fault and is removed. * fix(analysis): revalidate type recovery with the native analysis on native routes Type recovery revalidated each model downstream of a poisoned output through the Polyglot binding catalog. On a native route that catalog was never prepared, so preparing it while the semantic stage held a borrow on it panicked with "Already borrowed" whenever a model with its own binding error sat downstream of a blocking type error. On a native route, revalidation now reads each planned model's analysed SQL again with its dialect's analysis, over the schema the session bound and with the poisoned outputs untyped. The findings come back at the session's own positions, so type recovery matches them against the raw rows directly, and the Polyglot catalog is no longer on that path. The finished session keeps the model order and the declared functions it analysed with. The Polyglot route is unchanged. * fix(analysis): keep marker lineage and SQLBuild's text checks on native routes Fast lineage on a native route passed relation markers to the parser as written. Markers now read as indexed names through the shared relation marker reader, each tied to the resource it calls, so a model and a source of the same name stay apart. A source column's relation is compared as the dialect's profile resolves names, so upper-case folding dialects such as Snowflake keep their sources too, in fast lineage and in session lineage alike, and repeated sources of one column are listed once. Native analysis now adds SQLBuild's own text checks that the Polyglot route runs: truth-value tests the warehouse rejects (B103) and SQLite's temporal casts (B104). Output column facts keep the analysis's own type spelling, such as INTEGER or VARCHAR, and use the dialect's normalised name only to decide whether the type is known. The relation marker reader is the same as on the parser train branch. * test(analysis): re-record native-route goldens and pin SQLQuack at the parity merge Pins SQLQuack at ed14b717 (main fa55d9a1 with the DuckDB parity link); the pin moves to main once that link lands. Golden changes: - model-sql-syntax, syntax-hook, syntax-inline-hook: SQL the native parser cannot read is P026 rather than P001. The cases now set require_sql_analysis so P026 is the failing error. - seed 0, 6 and 10: amount passes through a CAST in a CTE, so its trace is cast rather than direct; seed 10 is now high confidence. - analysis-window-without-over, analysis-unknown-function, analysis-aggregate-in-where: the finding is placed on the call itself. * test(lineage): an unterminated DuckDB query reports where the native parser stops * fix(sqltext): read T-SQL bracket names and ClickHouse quoting in dialect scans A T-SQL `[...]` name was code to the dialect scan, so an apostrophe, `--` or `/*` inside one opened a string or comment that hid everything after it. On a native route a cursor stub after such a name was not respelt, and a valid model was skipped with P026. Lexical rules gain bracket identifiers, closing at `]` with `]]` for one bracket, and the T-SQL rules set them. Adapter records never set the flag, so their keys and scans are unchanged. ClickHouse's rules now follow its lexer: a backslash escapes inside single quotes, double quotes and backticks, and `#` starts a comment only before a space or `!`. * fix(analysis): read repeated markers as one relation and keep unreadable revalidations Native fast lineage gave every relation marker call its own name, so a union of two branches over one model read two relations and its columns became expressions. Calls of the same marker now share their first call's name, padded to keep positions. Native revalidation in type recovery failed the compile when the dialect's parser could not read a planned model again. Such a model now keeps its own findings, as an unrevised model would. * feat(analysis): read SQL Server models with SQLQuack's T-SQL parser The tsql analysis dialect now routes natively. TsqlDialect implements NativeDialect: parse reads with sqlquack-syntax-sqlserver, an unknown one-part function staying a syntax error as SQL Server's 195 is; profile is sqlquack-semantics' SQL Server profile; and normalize_type gives Polyglot's T-SQL canonical names, or None for a type neither SQL Server's built-in types nor SQLBuild's logical types know. SQLQuack is pinned to a commit carrying the SQL Server semantics profile. * test(analysis): state the T-SQL parse and type cases as named test cases TsqlDialect now calls normalize_text_type, so the rule exception that waited for a native dialect module to call it is removed. * chore(deps): pin SQLQuack to the main commit carrying the SQL Server semantics profile * test(analysis): keep T-SQL cursor stubs after bracket names holding quotes and comment openers Gap-test rows put a quote, an escaped bracket and both comment openers in a bracket name before a cursor stub; a SQL Server project requiring SQL analysis compiles a microbatch model whose stub follows [x--y]. * Route ClickHouse analysis to SQLQuack's ClickHouse parser and profile - ClickHouse implements the native dialect: SQLQuack's ClickHouse parser with its named gaps, the ClickHouse semantics profile, types normalized as Polyglot normalizes them, and contract types compared after both sides are respelt as ClickHouse prints them, so BIGINT is a non-null Int64. - A declared type ClickHouse refuses compares nothing; an inferred type it refuses or does not know still meets a non-null declaration. - Every SQLQuack crate is pinned to SQLQuack main edfa30b2, which carries the ClickHouse parser, and ClickHouse is listed as a native dialect. - The ClickHouse gap-gate tests cover an apostrophe in a `#` and a `#!` comment, which ClickHouse reads as comments. * test(analysis): re-record native spellings and the DuckDB union guard Typed markers now report DuckDB's own VARCHAR and INTEGER. A deep union of one marker resolves its columns at high confidence, and the DuckDB parser reads 8,000 UNION ALL branches and refuses 8,001. * test(analysis): re-record type notes in the engine's own spellings Native analysis keeps the dialect's type names, so DuckDB notes and messages say VARCHAR and INTEGER where Polyglot printed its comparison spellings TEXT and INT. * fix(analysis): name the hook and the model file line in P026 A native route's P026 for a hook now names the hook and its file, as the Polyglot syntax error did, such as post_hooks[0] sql("record_refresh") (hooks/sql/record_refresh.sql), and places the gap in the hook's SQL. A gap in the model's query is placed at its line and column in the model file when the text maps back through stub spelling, placeholders and expansion; otherwise the message says the position is in the analysed SQL. * feat(analysis): let a native dialect respell session column types NativeDialect gains session_type, the identity by default: the session schema and revalidation give each known column type to the parser in the spelling the dialect reads, for dialects whose parser cannot read every type SQLBuild keeps. * WIP (not shipping): BigQuery's arm on the NativeDialect trait, pinned to SQLQuack main edfa30b2 * build(analysis): pin SQLQuack at main 38be86ba * WIP (not shipping): align the BigQuery arm's tests with the native dialect trait The trait's shared read_whole places "more than one statement" at the start of the SQL for every native dialect, and the text fallback now takes a type's family from its top-level type, so RANGE<TIMESTAMP> and the complex types holding one read as Other on both routes. With BigQuery's module typing through normalize_text_type, its fensu exception no longer matches a fault and is removed. * WIP (not shipping): read session types BigQuery cannot spell by their family The session keeps SQLBuild's spellings, such as TEXT for an expression source's string columns, VARCHAR(n) and DOUBLE, which BigQuery's parser cannot read, so every column downstream of them lost its type on the native route. BigQuery overrides #1369's NativeDialect::session_type to respell an unreadable spelling as its family's BigQuery type. Two BigQuery e2e cases held SQL that GoogleSQL refuses: WINDOW is reserved as an alias, and EXCEPT and INTERSECT need DISTINCT or ALL. * WIP (not shipping): let BigQuery casts name RANGE types SQLQuack's cast check knows a named type only through an engine family or the schema's known_type, and no family names RANGE, so a valid CAST(x AS RANGE<DATE>) was blocked as B218 on the native route. BigQuery's analyse now reads the query against a schema that also knows every type BigQuery's type grammar reads; RANGE<INT64> stays B218. The BigQuery set-operation models move into a constant, as fensu requires of test modules. * WIP (not shipping): re-record the BigQuery entries of the generated contract and analysis goldens BigQuery now analyses natively, so only its entries move; every None, DuckDB, Postgres, Snowflake, Databricks and T-SQL entry is unchanged. contracts_generated (32 of 60 entries, only the bigquery key): 56 K002 findings (49 errors, 7 warnings) no longer appear and none is added. Each pairs a type GoogleSQL refuses, such as REAL, CHAR(3), VARIANT, MAP, TIMESTAMPTZ, int4 or VARCHAR, or an empty spelling, with the other side; the native comparison leaves such pairs uncompared. analysis_generated (the two cases with a BigQuery profile; 36 BigQuery groups): column types take SQLQuack's BigQuery spellings (INT64, STRING, BOOL); casts to VARCHAR or TEXT, which GoogleSQL refuses, and the columns passing them on are untyped, as is COALESCE over INT64 and STRING; more outputs are proven non-null; B002 messages drop Polyglot's context suffix; SELECT ... AS "Mixed_Case", which GoogleSQL refuses, fails analysis as a parser gap rather than as B000; and B002 on s.side over a star subquery below that gap is not reported, though a compile of the same shape reports it. * test(analysis): re-record P026 messages and fast-lineage golden for native arms The P026 rows now name the model file or the hook with its file, with the gap placed in that text. The generated fast-lineage golden moves with the native route: root stars resolved, medium confidence now high, the schema's own column spelling, set-operation columns that read different branch columns now expression, and edges for models that had none. * WIP (not shipping): route Postgres analysis to SQLQuack's PostgreSQL parser and profile PostgresDialect reads model SQL with sqlquack-syntax-postgres's strict parser, and normalises a type only when PostgreSQL's type grammar reads it. Every SQLQuack crate is pinned to main edfa30b2. * WIP (not shipping): shape the Databricks arm's type test and constant as fensu requires The type test case names Polyglot's spelling of each type as its expected name, so its assertion reads an expected field as well as comparing the whole type with Polyglot's, and the alias constant keeps a one-line doc comment. * test(analysis): cover a root star over an opted-out model in fast lineage Native analysis resolves every root star over a relation with known columns, so the generated projects no longer leave stars for fast lineage and their star minimum goes. A dedicated case keeps a root star over a model without SQL analysis, whose facts leave the star unresolved, and checks that restored lineage equals the catalogued lineage on every dialect. The arm gap tests set the new gap offset. * WIP (not shipping): compare Postgres types as PostgreSQL names them PostgresDialect::normalize_type reads a type with PostgreSQL's type grammar and names it as format_type does, so INT, int4 and INTEGER are one type, before SQLBuild's comparison shape. * WIP (not shipping): type Databricks spellings through Databricks' own type grammar normalize_type now decides whether a type is known, so it must refuse what Databricks refuses. The arm reads a spelling as the target of SELECT CAST(NULL AS ...) with Databricks' parser, as BigQuery's arm does, and abstains on anything else: TEXT, BOOL, INT64, FLOAT64, DATETIME, JSON, TIMESTAMPTZ, INTEGER[], MAP(K, V), NUMBER(p, s), DOUBLE PRECISION (the DOUBLE type's documented syntax is DOUBLE alone) and the empty spelling. Such pairs, and the columns typed with them, are left uncompared. A read type is respelt from its parsed form, so Databricks' synonyms meet: INTEGER and INT, LONG and BIGINT, SHORT and SMALLINT, BYTE and TINYINT, DEC and NUMERIC and DECIMAL with Databricks' (10,0) default and scale 0 after a lone precision, TIMESTAMP_LTZ and TIMESTAMP, TIMESTAMP_NTZ and TIMESTAMP WITHOUT TIME ZONE, VARCHAR and CHAR without a length and STRING, and a STRUCT field written with or without its colon. Field constraints, comments and collations are not part of the type. Where Polyglot keeps LONG, SHORT, DEC or TIMESTAMPLTZ apart, the native route follows Databricks' synonym table and the semantics profile's own type synonyms. Databricks types a cast to CHAR(n) or VARCHAR(n) as STRING and reads a CHAR or VARCHAR column as STRING; those types are a length check on STRING when a table is written. type_match therefore takes CHAR(n) or VARCHAR(n) on one side and STRING on the other as one type. Without this every contracted VARCHAR(n) column of the generated seed projects raised K002 on the native route. * WIP (not shipping): re-record the Databricks entries of the generated contract golden Databricks now analyses natively, so only its entries move: 32 of 60 entries of contracts_generated change, only under the databricks key, and every None, BigQuery, DuckDB, Postgres, Snowflake and T-SQL entry is unchanged. analysis_generated is unchanged for both of its cases. 55 K002 findings (50 errors, 5 warnings) no longer appear and none is added. 54 pair a spelling Databricks' type grammar refuses with the other side: TEXT, BOOL, INT64, FLOAT64, int4, JSON, DATETIME, TIMESTAMPTZ, TIMESTAMP WITH TIME ZONE (Databricks reads only WITH LOCAL TIME ZONE), INTEGER[], MAP(VARCHAR, INTEGER), NUMBER(38, 0), "not a type (" or an empty spelling. The native comparison leaves such pairs uncompared, as on BigQuery's arm. The other is a warning on CHAR(3) inferred against VARCHAR declared: Databricks reads a CHAR(n) column as STRING, and VARCHAR without a length is STRING, so the two are one type there. * WIP (not shipping): match the Databricks type probe's parse failures explicitly, as fensu requires * test(analysis): re-record native-arm spellings, a B231 position and a refused DuckDB audit Under the native DuckDB route, inferred types keep DuckDB's own names: INTEGER for a cast to INTEGER, VARCHAR for a string or a cast to VARCHAR, where the legacy analysis spelled INT and TEXT. B231 for an aggregate in WHERE now points at the call, as the COUNT(*) row already did. A relationships audit from a string key to an integer key now reports B301 and B217: DuckDB 2.0 refuses the compiled audit's NOT IN with "Cannot compare values of type VARCHAR and INTEGER in IN/ANY/ALL clause", so the case expects both findings and a failed compile. * WIP (not shipping): Databricks reads session types it cannot spell by their family, where the family has one type The session keeps SQLBuild's spellings, such as TEXT for an expression source's string columns, which Databricks' parser refuses, so the columns downstream lost their type on the native route. session_type keeps a spelling Databricks reads and respells STRING, BOOLEAN and DATE families, which have one Databricks type each. Integer, decimal, float and timestamp spellings stay unread: Databricks has several types in each family, and a family alone cannot say which. The Databricks gap test cases carry the source offset the native route now records. * WIP (not shipping): validation pin of every sqlquack crate to 02bd392d 02bd392d is the approved Databricks semantics link (the profile and its LATERAL VIEW and set-operation ORDER BY fix) on SQLQuack main 38be86ba; it is not on main yet. Repin every sqlquack crate to the SQLQuack main commit that carries it. * fix(analysis): give native routes the session's functions, quoting and case setting A native route analysed every model against a schema with no function facts, quoted output names lost their quoting, and Snowflake's QUOTED_IDENTIFIERS_IGNORE_CASE was ignored. - The session schema now answers the adapter's declared function return types, respelt for the dialect, and its first_arg, conditional_result and safe_cast nullability rules. - A stored name in double quotes becomes a quoted schema column, with "" read as one quote, wherever the session, revalidation or fast lineage builds one. - The session request carries the profile's quoted-identifier case setting into the cache key and each dialect's analysis, so a dialect can pick a profile for it. - Semantic completion no longer panics when the binding catalog is already borrowed: unknown relations stay per-request. - An adapter's own Python nullability rules are not called on a native route; compile now warns, naming the adapter and the functions. The callback tests run on the Polyglot route, and the payload-only completion check expects a native route to require its session. * fix(analysis): quote an unknown quoted column as written SQLQuack quotes an unknown column's name in lower case, so a quoted "MiSsPeLlEd" was reported as 'misspelled'. A native route now puts the authored name back into the message. * feat(analysis): fold quoted Snowflake names under QUOTED_IDENTIFIERS_IGNORE_CASE With the session parameter set, the Snowflake arm analyses through SQLQuack's quoted-ignoring-case profile, so a quoted name resolves to upper case like an unquoted one, as Snowflake does. * test(analysis): re-record CREATE TABLE AS and INSERT ... SELECT columns as unknown Native analysis reads only queries, so a model whose SQL is a CREATE TABLE AS or an INSERT ... SELECT gets no columns rather than its inner SELECT's. * test(analysis): expect the native refusal to complete without a session as a compiler error * test(analysis): capture each generated semantics project under its own dialect The generated failing-projects test compiled every project under DuckDB and re-read the captured inputs as each other dialect, so a dialect's golden held DuckDB's analysis of its models. Each dialect now compiles the project under its own analysis dialect, with DuckDB's function rules kept for all. * WIP (not shipping): pin every sqlquack crate to SQLQuack main 3b497dfa SQLQuack main 3b497dfa carries the Databricks semantics link 02bd392d, so the validation pin is replaced by a main commit. * test(analysis): re-record the native digest goldens and pin the open native gaps Re-record the Python-oracle digest goldens the native DuckDB, Snowflake and SQLite routes change: engine type spellings and result types, nullability proven by filters and literals, unknown columns for stars over relations without an authoritative shape, per-output types for duplicate output names, and messages that keep the authored spelling. Keep the open gaps as strict expected failures beside passing controls: Snowflake's invalid coercible literal and date arithmetic, and declared source types that native analysis does not read. The adapter-rule case is renamed, because adapter callbacks are not called on a native route. Native routes record no legacy lineage enrichment, so those minimums are 0 where every profile is native. * test(analysis): expect a native wave to analyse each equal model query A native wave analyses every model on its own, so the equal-query case expects no shared or re-analysed queries. Outcomes are unchanged; only the repeated work differs. * fix(analysis): type a native route's reads of declared but open relations A native route gave SQLQuack only relations whose full shape was known, so a source or seed with declared column types but no authoritative shape was read untyped: `status + 1` over a declared VARCHAR went unreported, where Polyglot's rich analysis used the declared types. The session schema now passes each such relation's declared columns, with the nullability the session knows, as open columns. They type the columns a query reads without making the relation's column list complete, so star expansion and unknown-column checks are unchanged. Revalidation passes the same open columns. The analysis cache format moves to v4, as outcomes for the same inputs change. * WIP (not shipping): bind the Databricks timestamp's zone flag in one arm The setting-help scan reads a source line holding both a word followed by ': true' and a string as a loosely named setting. The two timestamp arms become one that matches on the zone flag; the spellings are unchanged. * fix(rules): read relation markers as tables in custom-rule syntax trees Compiled query_sql keeps relation markers such as __ref("orders"). Snowflake does not accept a bare function call in FROM, so newer SQLQuack parsers refuse every Snowflake model that references another. A shared helper in sqlbuild-analysis (relation_markers::markers_as_names) now writes each marker call in code as a short indexed name, padded so all later text keeps its line and column, and returns the calls by index. sql_tree reads SQL through it for every dialect. The rules crate's dialect lexical syntax moves to sqlbuild-sqltext so both use one mapping. * feat(sql-macros): parse SQL macros and SQL-test planning with the native parser SQL macro bodies and SQL-test planning parsed with Polyglot. Both now parse with the native parser that analysis_route gives the project's SQL analysis dialect, and a dialect with no native parser reports the feature as unavailable rather than falling back. - Macro bodies parse with the project's parser instead of trying eight Polyglot dialects. Declarations and USING slot reads are masked as names of the same length, so spans line up. A body that only names a type is read as a cast's type. - Macro probes type through NativeDialect::analyse over stand-in relations; returns, ANY-family, argument and slot checks read its output types, and type findings are SQLBuild's B210-B219. - SQL-test planning parses with sql_test_parser(dialect). The planning template reads relation marker calls as plain names, so Snowflake accepts them in FROM, and a list after IN is read as a list. - Polyglot's output-typing entry points, used only by SQL macros, and the token-based simple-template fast path are removed. * refactor(sql-tests): read planning relation markers through the shared markers_as_names The planning template now names marker calls with relation_markers::markers_as_names, which sql_tree also uses, and reads them back by its MARKER_NAME_PREFIX. The SQL-test patterns no longer carry their own marker regex. * test(sql-macros): name the tracked list-typing work in the list-body xfail reasons * test(sql-macros): expect the unavailable error for adapters without a native parser Expansion cases for an adapter whose dialect has no native parser were filtered out, so they checked nothing. They now assert the P015 that says SQL macros are unavailable, and run as written once the adapter's arm lands. * docs(sql-macros): say macros and SQL tests need a native parser, and drop partly-known returns checks Macros and SQL tests now read SQL only with the adapter's native parser, so the pages say what happens without one. Native analysis does not type list literals, so a list body trusts returns rather than failing P024 by shape, and the BigQuery NUMERIC example goes until BigQuery has an arm. * test(sql-macros): describe the list-typing work in the list-body xfail reasons without a tracker reference * build(analysis): pin sqlquack to c2e78ffb The pin carries DuckDB value keywords, Databricks semantics and the DuckDB and SQLite star order over USING and NATURAL joins. * test(analysis): re-record the native goldens for declared open columns and the new pin Declared but open relations now type the columns read from them, so the pending declared-source case passes in both lineage modes and joins its control. Stars over them list the declared names, untyped and still unresolved, and a cyclic project types its declared source columns. DuckDB and SQLite stars over a USING join now keep the key in place. * fix(deps): resolve tempfile against the workspace's windows-sys again The native route's dependency update moved tempfile onto windows-sys 0.52, which sqb and the Python extension then built with different features. * chore(analysis): pin every native arm to one SQLQuack train and share the unquoted stub prefix * chore(analysis): lock the Databricks syntax crate at the shared SQLQuack train * test(sql-macros): run every macro case through its adapter's native arm Every shipped adapter now has a native arm, so no case expects the unavailable error any more. * refactor(analysis): group the native route's helpers into analysis and dialect modules With every arm in, the helpers outgrew fensu's ten-module container. The PostgreSQL type probe now matches its parse failure explicitly. * feat(analysis): leave a dialect no native module reads unanalysed Every shipped adapter's dialect now has a native arm, so analysis_route sends any other dialect, or none, to Unavailable rather than to Polyglot. Tests that relied on the no-dialect path now use DuckDB. - Fast lineage names an output the SQL leaves unnamed from the compiled column at its position, as the Polyglot path did. - An adapter's own nullability rules are never called on a native route; the two callback tests now check the compile warning instead. - The analysis session, cache and fast lineage expectations are re-recorded for the native route; each move was checked against the SQL. - The arms' gap tests and the macro type probes fill the shared route's newer fields. * test: expect every shipped dialect's native arm - SQL Server plans SQL tests natively, so its two planner cases are no longer xfailed. - Databricks, BigQuery and PostgreSQL macro return checks run natively. - Completing from the session alone is always refused on a native route. * test(analysis): xfail PostgreSQL's invalid-literal comparison until SQLQuack checks it PostgreSQL refuses `amount > 'placed'` while parsing, but SQLQuack's PostgreSQL profile does not yet check literal coercion. The DuckDB row stays the passing control. * test(analysis): compile each generated project under its profile's own dialect The compile context named no analysis dialect, so the model loop ran on the no-dialect path whatever profile the session used. Each run now compiles under the dialect it analyses, as an adapter does. * fix(analysis): route PostgreSQL's assembly dialect name to its native module Compile assembly hands the analysis session the Polyglot alias 'postgresql', not the adapter's 'postgres'. The route table only knew 'postgres', so the session took the Polyglot path for PostgreSQL models, and after unknown dialects became unavailable a warm cache skipped the catalog preparation that path needs. Both names now route to the PostgreSQL module. * test(analysis): re-record PostgreSQL goldens under its native session With PostgreSQL analysed natively, its keys move to the engine's type names (INTEGER, TEXT for MAX over VARCHAR, NUMERIC for SUM over DECIMAL), fold unquoted names in findings, and drop follow-on findings of a CTE whose body already failed, as PostgreSQL reports one error per statement. Pending rows, each strict with a passing DuckDB control: BigQuery's invalid timestamp literal, a PostgreSQL CTE over a failed upstream model that is left with no finding, and an expression source whose alias case PostgreSQL folds. The session completion test asserts that payload-only completion answers some projects and matches the session wherever it does; the legacy enrichment floor is zero on all-native profiles. * fix(analysis): route every analysis alias to its native module The alias tables also name MotherDuck and SQL Server as 'motherduck' and 'sqlserver'. Route both to the DuckDB and T-SQL modules, and check each route name both as given and after compile assembly's alias mapping, so a respelt name cannot fall back again. * test(analysis): prove each shipped adapter's session analyses natively Equal model queries share one analysis on the Polyglot batch path and never on a native wave, so a session that falls back to Polyglot under any shipped adapter's own profile fails this test. * fix(analysis): read no resource SQL where no native parser reads the dialect A custom adapter's dialect gets no SQL analysis, but the semantic stage still sent its hooks, audits and SQL tests to the Polyglot binding catalog, so a hook Polyglot could not parse failed the compile with P001. Only the audits of declared columns remain on that route. The differential case for an unbuilt dialect now requires SQL analysis, so its compile fails on P026 instead. * test(differential): record SQL test mocks in the model's reference order The native SQL test planner lists a model's mocks in the order the model reads them; Polyglot listed refs before seeds. The mocks don't read each other, so the comparison SQL means the same. * test(semantic): re-record unknown-dialect metadata checks without resource SQL Projects whose dialect no native parser reads no longer have their audits' SQL bound, so the relationships and accepted_values findings Polyglot raised there are gone; SQL test column checks remain. * build: pin SQLQuack to main 395ced6d * docs(analysis): keep the resource SQL summary to one line * test(analysis): build test compile contexts with the adapter's analysis dialect Compile inputs built without an analysis dialect take the unavailable route, so these helpers compiled every project without SQL analysis. Each now passes its adapter's dialect, as the compile pipeline does. * test(analysis): expect native-route behaviour where tests expected Polyglot's With every shipped dialect on a native route, a model or hook SQL the dialect's parser cannot read is skipped with P026 instead of raising a syntax error, and a dialect without a native parser gets no fast lineage and compares contract columns by name only. The tests now expect that. Fast lineage tests pass a native dialect. The truncated T-SQL hook is attached under T-SQL, so its parser reports it. PostgreSQL star shapes gain the types PostgreSQL itself gives (INT and TEXT). Re-recorded: the generated fast-lineage golden (PostgreSQL, which now folds the unquoted alias Up to up as PostgreSQL does; no-dialect entries now empty), the generated contract golden (PostgreSQL, Databricks, T-SQL and no-dialect entries, which now match the native DuckDB findings or drop comparisons with types foreign to the dialect), one assembly edge whose model is now skipped so its seed error raises, and two attachment summaries whose hooks DuckDB itself rejects (GRANT, PIVOT ANY). * test(sql-macros): expect native-route results in the macro e2e cases Native DuckDB analysis does not type list literals and reports EXTRACT as an unknown function, so those cases move to strict xfails with a date_part control. The BigQuery regexp case now passes a GoogleSQL argument type. * fix(sql-macros): name the BigQuery NUMERIC limit for types its parser refuses BigQuery's parser refuses a NUMERIC wider than it holds, so the limit check now reads SQLBuild's own spelling of the type before falling back to the generic unknown-type message. * docs(skill): regenerate the docs skill references for SQL macros and testing * wip: keep recorded fingerprints of unchanged definitions across fingerprint scheme changes Untested beyond compile and the analysis unit tests. * test: pin SQLite and MotherDuck query fingerprints before the SQLClean re-pin * fix(executor): treat SQLite columns without type evidence as compatible SQLite reports a column with no declared type as ANY. An incremental or snapshot schema comparison now treats that marker as compatible with any type instead of reporting a type change. * fix(analysis): know project functions by their bare names on native routes Native model analysis and type recovery now read the profile's declared project function names, so a bare call to a project function is no longer an unknown-function error. Shapes share one case-insensitive lookup. * fix(lineage): trace fast lineage and review impact in the adapter's dialect * fix(analysis): place native type findings that carry no span Type findings without a native span are placed at the expression that raises them, and SQLBuild's own text checks replace a native row with the same code and start. * fix(compile): report SQL syntax errors as P001 on native routes A gap the native parser marks as a syntax error, in a model query or a hook, is now a P001 compile error rather than a skipped-analysis warning. * refactor(sql-macros): share one statement parse for macro bodies and CTE renames * fix(contracts): compare declared types as each engine stores them DuckDB and Snowflake contract checks now meet each engine's documented synonyms and defaults, and DuckDB's runtime comparison treats every string length as the VARCHAR it stores. On Snowflake, a narrower scale-0 NUMBER satisfies a wider declared scale-0 NUMBER only where the build casts its output to the declared types: staged tables and non-microbatch incrementals. The complexity-guard fixture declares INTEGER, since NUMBER is no DuckDB type. * test(e2e): expect the native route's explained results F15 is the syntax error DuckDB reports, U01 the B212 the engine raises, the typed null and the TIMESTAMP_NTZ help use the native spellings, CHECKPOINT parses on DuckDB, Snowflake star outputs keep the upstream's spelling, and the SQL Server delta-compile fixture uses T-SQL spellings. * wip: re-pin SQLClean 0.6.0 and fingerprint every dialect through SQLQuack Query fingerprints now come from sqlquack-fingerprint at the pinned SQLQuack rev, the same function SQLClean 0.6.0 uses. Dialects SQLQuack does not tokenize (SQL Server, ClickHouse, Databricks, BigQuery) read generic tokens, so their fingerprints change. Recorded fingerprints whose stored definition re-fingerprints to the current hash are kept, so unchanged models do not rebuild; rows recorded without a definition report one query change. The local copies of the canonical token stream and keyword lists are removed. The type-finding locator keeps the Polyglot-token projection spans SQLClean 0.6.0 no longer exports. Release notes: On SQL Server, ClickHouse, Databricks and BigQuery, the web UI's node status may show unchanged models as changed until they next rebuild with a definition change or full refresh; plans and runs are unaffected. On SQL Server and BigQuery, changing only the letter case of common keywords (41 on SQL Server and 55 on BigQuery, such as JOIN, CASE, ORDER BY and LIMIT) now counts as a query change, so the first `sqb format` that upper-cases them rebuilds each reformatted model once. Dialect keyword lists in the fingerprint will remove this. Validated: cargo check, sqlbuild-analysis lib tests, clippy on the analysis, rules and Python crates; fingerprint, planner, executor and microbatch Python suites (1,866 passed). * test: keep recorded hashes of unchanged models after a fingerprint scheme change * fix(rules): wrap Snowflake table functions in TABLE( ) before parsing The rules engine stubs __table_fn calls before parsing a model. SQLQuack's Snowflake grammar, like Snowflake, accepts a table function in FROM only inside TABLE( ), which is how SQLBuild renders the call, so a bare stub made every built-in Rule report the model unevaluated. Wrap the stub the same way unless the model already wrote TABLE( ), and assert in the Snowflake evaluation tests that the model was actually evaluated. * fix(rules): read Snowflake table function markers inside TABLE( ) in rule trees Custom Rules parse a model through the rule tree, which renames each relation marker to a placeholder name first. SQLQuack's Snowflake grammar refuses the bare table function call that leaves, so a custom Rule on a Snowflake model with a table function failed to parse it. Wrap those calls in TABLE( ) over the marker's own padding and a following space, so node positions keep their line and column, and share the TABLE( ) detection with the built-in Rules' stubbing. * fix(rules): keep columns after a wrapped table function in rule trees When no space or tab follows a Snowflake table function call, the rule tree inserts its closing parenthesis. Write the TABLE( ) wrapper over one more padding space in that case, so text after the call keeps its column. * fix(rules): keep a table function's argument columns when its call ends the line The TABLE( ) wrapper borrows one more padding space only so text after the call keeps its column. At the end of a line or of the input the inserted closing parenthesis shifts nothing, so the extra space only moved the arguments one column left. * fix(rules): keep a Snowflake table function call whole when linting and formatting Snowflake reads a table function call only inside TABLE( ). Lint and format replaced only the __table_fn(..) reference with a sentinel, so the parser saw a bare call with arguments and refused the body. Outside TABLE( ) the sentinel now covers the call's arguments too. * test: expect a Snowflake bare VALUES CTE to be a syntax error as a strict xfail Native Snowflake parse errors carry no syntax flag, so the case reports P026 and rules-unevaluated instead of P001. The compile still fails. * test: keep the recorded-fingerprint and SQLite schema-change tests in the house test shape Move module helpers into helpers.py, assert against expected fields and replace conditionals with lookups, so fensu check passes. * fix(test): accept a typed null fixture's own column order under a SELECT * closure The schema-order rule protects the columns completion appends after a star's leading relation columns. A fixture whose only completion is casting its bare NULLs appends nothing, so its order cannot shift the star's columns. Native analysis types more upstream columns than before, which made DuckShop's complete fixtures with bare NULLs fail inspection. * fix: lint and format scalar function bodies and quoted hook arguments Scalar SQL function bodies are expressions, so lint and format now read them as the one item of a SELECT, as expression macro bodies already are. Table function bodies stay full queries. A quoted @'name' argument now stands in as a string sentinel, as audit parameters already do, so grammar positions that need a string, such as COMMENT ... IS, parse. * refactor: bring the native analysis code and tests into the project layout rules Name the engine type spellings the type comparisons decide on, extract nested iterator closures into named functions, read an out-of-range Snowflake precision explicitly, keep the relation-columns lookup with the catalog helpers, and write the engine-type, bare-function and baseline-fingerprint tests as named test cases in their test_types modules. * refactor: read adapter dialects and project settings without Python calls * fix(rules): keep a Snowflake table function's arguments after LATERAL The parser reads LATERAL followed by a call but not by a bare name, so a table function call after LATERAL keeps its arguments, as one already in TABLE( ) does. The TABLE( ) check now uses the shared word-bounded follows_table_wrapper. * test: check outside the xfail that a Snowflake syntax error still fails compile Only the P001 code stays a strict xfail; the exit code and the reported error are asserted by a test that runs. * fix: frame only scalar function bodies as expressions in lint and format A function body was read as a SELECT item whenever it did not start with WITH or SELECT, so a table function body opening with a parenthesis or VALUES was reformatted as an expression. The header now decides: only a function whose returns is not a table is framed. An expression body after a leading comment is dedented from its first line of code, so the comment no longer leaves the body indented. * test(duckshop): read the edge-case table function after LATERAL * test: list the Snowflake syntax-error case beside each test that reads it * fix(analysis): drop LATERAL before a table function's stub relation Analysis replaces __table_fn("f")(args) with a bare stub relation, so a call after LATERAL became LATERAL <name>, which no parser reads. The stub now replaces the keyword too; LATERAL has no meaning once the arguments are gone. A lateral inside a comment or string, or a word ending in lateral, is kept. * test(duckshop): unit-test the LATERAL table-function edge model * deps: pin SQLQuack 804864d2 and SQLClean 0.6.1 Every SQLQuack crate moves to 804864d2073ef869cbfe4a9334f7e48b20072326 and SQLClean to 0.6.1, which is built on the same SQLQuack rev. The SQL Server bracket-identifier fingerprint golden returns to the value 0.204.0 stores. BigQuery, Databricks, ClickHouse and SQL Server fingerprints use SQLQuack's tokenizers. Models whose stored fingerprint row has its query text, which is every row written by v0.47.0 or later, keep their fingerprints and plan as before; an older row plans as query_changed once. * fix(sql-tests): drop LATERAL before a mocked table function A unit test that mocks a table function swaps the call for the mock's CTE name. A LATERAL before the call was left in front of that bare relation, which DuckDB and Snowflake both refuse. The keyword now goes with the call, as analysis normalisation already does; a rendered call keeps its LATERAL. Both read the keyword through one shared lateral_start. * test(semantics): expect PostgreSQL's B218 on a text literal compared with a number SQLQuack's PostgreSQL profile now refuses the comparison, so the case leaves the strict xfail of pending dialect semantics and joins the passing cases. * test(assembly): expect P001 where the parser rejects model and hook SQL Native routes report a parser's syntax error in a model or hook as a P001 compile error, so assembly no longer reaches a skipped-analysis warning for these projects. Each engine rejects the SQL too: DuckDB 2.0.0 refuses `SELECT FROM WHERE (` and `SELECT FROM (`, and SQL Server 2022 refuses `SELECT IF(region > 1, re` with Msg 156. The truncated T-SQL hook still makes no Python resource call. * test(analysis): write each generated project in its profile's dialect A generated project is compiled under DuckDB, PostgreSQL, Snowflake and BigQuery, and a syntax error is now a P001 rather than a skipped model, so its SQL has to be valid in each. The draws are the same for every profile; only the spelling changes. BigQuery quotes with backticks, writes DISTINCT after INTERSECT, EXCEPT and UNION, and excludes with `* EXCEPT`; PostgreSQL names the one kept column instead of `* EXCLUDE`. Pivots are written only where the dialect has PIVOT: DuckDB's shapes for DuckDB and MotherDuck, and Snowflake's for Snowflake. The CTE pivot that read `IN (ANY ORDER BY status)` under DuckDB now lists its values, since DuckDB 2.0.0 rejects ANY there. Pivot proofs come from those two dialects only, so the broad cases expect at least 72 rather than 80, and the analysis-session corpus reads the listed pivot through the session. * test: re-record the python-oracle goldens on the merged native route Recorded on this head, then checked item by item against the same tests run on the cutover before the merge. Only three goldens move with the merge, each in dialects main's own type and dialect changes reach: types (MySQL, MotherDuck, PostgreSQL, BigQuery and Databricks spellings), formerly deferred metadata and semantics (ClickHouse, and the MySQL, Oracle and Trino rows main's tests now read). The rest were already stale on the cutover or follow its own changes: PostgreSQL's new B218, syntax errors reported as P001, and the generated projects now written in each profile's dialect. * test(compile): write the header-contract pivot in DuckDB's PIVOT ON form WIP: the header-contract expected summary is not yet re-recorded. DuckDB 2.0.0 rejects PIVOT ... IN (ANY ORDER BY ...), which is Snowflake syntax; the simplified PIVOT ... ON ... USING form is DuckDB's dynamic pivot. * test(compile): write DuckDB-valid hook SQL and expect P001 for rejected queries DuckDB 2.0.0 has no GRANT statement and rejects it with a parser error, so the grant_read hook fixture now records the grant in an audit table with unnest over the role list, which DuckDB runs. The header-contract and SQL-hook summaries are re-recorded: the only change is that the P026 skips for the old pivot and GRANT text are gone and both models are analysed. The two immediate-validation cases are SQL DuckDB rejects ("SELECT FROM WHERE (" and a WHERE ending in "AND ("), so they expect the native route's P001 compile error again, in its parser's wording. * fix(analysis): spell ClickHouse's BINARY(n) as FixedString(n) on the native route ClickHouse registers BINARY as an alias of FixedString, and main's type normaliser now spells BINARY(n) as FixedString(n). The native ClickHouse alias table still carried Polyglot's Nullable(BINARY(n)), so the two disagreed after the merge. The parity test now names its oracle, the shared type normaliser, and records the two spellings ClickHouse refuses (bare BINARY and a Nullable inside a Nullable), where the shared normaliser keeps the written text with a parse error. * test(analysis): key dialect spellings by template and accept an unnamed dialect The type checker cannot see __name__ through the template alias, so the per-dialect spellings are keyed by the template functions themselves. The generator takes the same optional dialect as compile_inputs; a project without one gets the plain spelling and no pivots. * chore: settle fensu on the merged head The pivot generator draws its shapes once and renders them through a per-dialect table, so the helper has no conditional; every generated project is unchanged. The ClickHouse alias table's doc comment is one line again. Two pending budgets take main's lower counts, which the merge's union of fensu.toml had kept at the cutover's values. * test(compile): write hook SQL the project's engine reads DuckDB 2.0.0 rejects GRANT and CREATE FUNCTION ... RETURNS ... LANGUAGE SQL with a parser error, so the two compile-input cases now record the grant with an INSERT and create a schema-qualified macro, keeping the destination interpolation they test, and expect no P026. The adapter-dialect hook validation case writes T-SQL bracket names, so it now compiles its inputs as a SQL Server project rather than a DuckDB one; the native route rejected the brackets with the DuckDB parser before assembly could validate them as T-SQL. * fix(format): leave Databricks models as written with a format-unavailable warning SQLClean returns `unavailable` for a dialect SQLQuack does not format yet, which is Databricks today. sqb format read only `reason`, so every Databricks model became a format-unsafe error. It now leaves the file as written and reports a format-unavailable warning naming the reason, as the sqlclean CLI does, and the command does not fail on it. Databricks formatting returns with a later SQLClean release. * test: expect the re-pinned engines' behaviour and the native route's P001 - A quoted parameter neutralises to a string sentinel, and a trailing comment after the last clause starts at column 0 (SQLClean 0.5.0 on). - Databricks identifiers named Any, Some or Order fingerprint without case at SQLQuack 804864d2; those nine cases are strict xfail until the next SQLQuack pin. - ClickHouse and SQL Server models get one non-failing L007 each, as main's built-in rule facts give every dialect SQL lint cannot parse. - DuckDB stores VARCHAR(5) and VARCHAR(100) as VARCHAR, so a longer text macro body meets a VARCHAR(5) contract; the decimal case keeps the inferred-type check. - The failure corpus's syntax cases expect P001: DuckDB 2.0.0 rejects "SELECT order_id,, ..." and "SELECT FROM WHERE" with parser errors. * chore: settle fensu on the format-unavailable warning and the known fingerprint folds The warning's message prefix and remediation are constants, chosen without a four-element tuple. The nine Databricks keyword-named cases move to their own strict-xfail test, selected by the case helper rather than by per-case marks. * build: pin SQLClean 0.6.2 SQLClean 0.6.2 restores CTE macro placeholders that sit on both sides of a comment, so `sqb format` formats a CTE list whose macros carry attached comments instead of refusing it. It is still built on SQLQuack 804864d2, so the lockfile keeps one SQLQuack source and only the sqlclean entry moves. * perf(compile): parse native gap checks on the compile pool On a native analysis route each model's query and hook statements are parsed once in the attachment loop, to decide between P001, P026 and analysis. That parse ran in turn on the main thread, so a 5,000-model DuckDB compile spent about 7.5 s attaching inputs where it had spent 1.8 s when DuckDB's syntax check waited for the parallel analysis. The loop now hands each parse to the model loop's compile pool and carries on. Before it returns, it settles the results in model order: the first syntax gap still raises P001 ahead of any error a later model raised, each P026 is reported as before, and a skipped model's input record and facts are marked unanalysed. Diagnostics are collected by key, so their output order does not change. Reusing the gap parse for analysis would keep every model's tree alive until the analysis phase, and moving the decision into that phase would stop checking hooks and unselected models, so the parse stays separate. On the 5,000-model fresh-process project, attachment drops from 7.5 s to 2.9 s and the compile from 17.4 s to 13.0 s wall; its output and semantic fingerprint are unchanged. * perf(compile): keep native read-whole verdicts in the compile cache A cached compile still parsed every model's query and hooks to decide between P001, P026 and analysis, although the analysis itself came from the cache. On a warm 5,000-model compile that parse kept attachment at 2.8 s, where it had been 1.7 s when DuckDB's syntax check rode in the cached analysis. Statements the native parser reads whole are now recorded in a native store in the compile cache folder, keyed by dialect, SQL and placeholders under the build's code identity. A model whose statements are all recorded is not parsed again. Gaps are never recorded, so a model with one is parsed every time and keeps its P001 or P026, and an edited model's new SQL misses the store. --no-cache neither reads nor writes it. On the 5,000-model project a warm compile takes 6.1 s instead of 7.5 s, with attachment at 1.7 s; its output is unchanged. * test(perf): record the native route's fresh-process compile fingerprints The fresh-process compile and compile cache guards hash each generated project's compile JSON and compiled artifacts. On the native DuckDB route two things in them change, and both are correct. Column lineage gains edges. A metric such as `CASE WHEN status = 'priority' THEN ...` beside a `status` alias reads input.status, which is how DuckDB binds an unqualified name in the same SELECT; Polyglot's lineage left that edge out. Every changed model goes from 3 + 2k to 3 + 3k edges, and no other compile JSON field changes. Column lineage does not feed planning: model identity is the query fingerprint, upstream model keys, config and contract, so these edges cause no replans. In 70 compiled SQL test files at 5,000 models the CTEs come in a different order, with no CTE read before it is defined. The order is the same on every run: two compiles give byte-identical artifacts, and warm and edited compiles reproduce each recorded fingerprint. * perf(contracts): compare each native type pair once per project On a native route, contract type checks asked the dialect to match the declared and inferred types of every column, and the dialect parses both spellings each time. The Polyglot route already normalized each spelling once per project. A dense 3,000-model DuckDB project spent 220 ms in contracts against 57 ms on the Polyglot route. The project's type comparer now keeps each native match by declared type, inferred type and nullability, so a repeated pair is parsed once. Contracts on that project take 61 ms; findings are unchanged. * perf(sql-tests): read bounded planning templates from tokens again SQL-test planning read every model template with the native parser's strict parse after the Polyglot token fast path was removed, so dense projects spent their write phase waiting on template parses. simple_analysis_template restores the bounded recognizer for DuckDB and Snowflake on SQLQuack's dialect tokens. It reads the template after markers_as_names, so a re…
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Before any dialect switches, a native module needs the parse tree, its types and the analysis Polyglot gives today from one shared place, so routes can't disagree on codes and types.
Changes
NativeDialectgainsparse,profileandnormalize_type. The route providesread_whole,analyse,macro_bodyandtype_match.analyseruns sqlquack-semantics lineage and findings and returns the session's facts. Findings take SQLBuild's codes through the samesqlbuild_codeas the Polyglot path, so implicit-conversion findings stay blocking. Declared extension functions get no unknown-function finding.type_matchreportsNotComparedwhen a type is unknown or unspellable, so contracts make no type finding; a declared type may still forbid null.StandInSchemagives probe queries one relation of known columns.AnalysisRoutegainsUnavailable.normalize_text_typeis public. The text fallback takes a type's family from its top-level type, so an ARRAY, MAP or STRUCT holding a timestamp is not a timestamp.No dialect is on a native route, so compile output is unchanged except the fallback family fix, which applies only to types Polyglot cannot parse.
Verification
cargo test -p sqlbuild-analysis: 312 passed. The fallback-family and stub tests fail without their fixes.cargo test -p sqlbuild-pythonpasses; clippy-D warningsis clean on both crates.fensu check --no-cache: 0 faults.Stacked on #1368 (SQLQuack main 19c51fa2);
sqlquack-semanticsis pinned to the same commit. Draft until the route is wired into the model loop.