Repository navigation
Conversation
The native dialect trait now has parse and normalize_type, with read_whole and type_match provided on top. read_whole refuses parser gaps, empty input, more than one statement and statements the parser kept unparsed. type_match reports NotCompared when either side has no normalized type, so the contract check makes no finding instead of guessing. Parser gaps carry the parser's message and whether the failure is a syntax error, and the P026 skip appends that message. AnalysisRoute gains an Unavailable arm, which skips a model with P026 instead of analysing it. The contracts type comparer goes through the route's type_match, and normalize_text_type is public so native dialects can reuse the shared text type rules.
A native dialect now names its sqlquack-semantics profile, and the route provides analyse and macro_body on top of read_whole. analyse turns the query's lineage and findings into the facts the analysis session keeps: output columns typed through the dialect's normalize_type, lineage rows over the model's referenced resources, the columns deciding rows, the star flags, and findings in SQLBuild's codes through the same sqlbuild_code the Polyglot path uses, so implicit-conversion findings stay blocking. Unknown functions a dialect declares as extensions are not reported. StandInSchema gives probe queries, such as a macro body's, one relation of known columns. read_whole now reports SQL with no statement, such as a lone comment, as no statement rather than more than one.
The text fallback gave family Timestamp to any type whose text held TIMESTAMP, so an ARRAY, MAP or STRUCT holding a timestamp read as a timestamp where Polyglot reads it as Other. The family now comes from the leading type name, so TIMESTAMP(3), TIMESTAMP WITH TIME ZONE and TIMESTAMP_NTZ stay timestamps and complex types do not.
…p stubs in code type_match now takes the inferred type as optional, and NotCompared keeps whether the declared type alone forbids null. A ClickHouse column declared without Nullable that receives a nullable value of unknown type is flagged again, while no type finding is made. ClickHouse types are compared without whitespace outside quoted strings, so Decimal(10, 2) is Decimal(10,2). Respelling cursor stubs for a native parser now leaves text inside strings and comments, and longer names that end in a stub's text, as written.
…alects The dynamic pivot proof reads Polyglot's parse of the model. A dialect on a native or unavailable analysis route now gets a proof failure naming why, instead of a Polyglot parse. Every dialect is still on the Polyglot route, so no proof changes yet.
…lysed dialects" The pivot proof moves onto SQLQuack's parser for every dialect in its own change, so no route needs to skip it.
…ules require Each analysis_route and semantic_validation entry module now exposes one function, with ClickHouse's spelling helpers and the read_whole gap in _helpers. Doc comments are one line, the route's tests run every case from a named test case table, and report_skip takes the P026 code and message together. The analysis_route module no longer re-exports its entries. normalize_text_type and clickhouse_type_match keep an exception for having no importer yet: their native dialect modules land in their own changes.
sqlquack main 19c51fa2 carries the Postgres parser work and the train that followed a9dde6a0. sqlquack-syntax-postgres now depends on stacker, which re-resolves a few Windows-only windows-sys edges in the lock.
Snowflake, DuckDB and MotherDuck models now take the native analysis route. Each arm parses with its SQLQuack parser, analyses under its sqlquack-semantics profile, and compares types through respellings that keep Polyglot's type names. The SQLite arm is in place but stays on the legacy route until its parser is in SQLQuack's depth probe. SQLQuack crates are pinned to main edfa30b2, which adds the DuckDB and SQLite parsers.
…ive DuckDB parser The native DuckDB parser reads WHERE id = ANY [1, 2], which DuckDB runs, so the seed's sql_analysis false model no longer needed its opt-out and require_sql_analysis rejected it. A NEAREST join is valid DuckDB that the parser keeps as a gap, and the Postgres and Snowflake parsers refuse it too. Only the fixture's SQL text changes in the six seed goldens.
…eference scan On a native analysis route the gap gate reads the model SQL and hooks whole, so assembly no longer reruns Polyglot's syntax validation there: a model the native parser reads gets no Polyglot P001. The gate now runs after reference scanning, so a located scanner error wins over a raw normalisation failure, and a model whose reference calls already failed P012 gets no P026. The Polyglot route keeps its order and validation.
…names A full comparison with Polyglot's normalize_type showed the DuckDB, Snowflake and SQLite tables missing names from other dialects, such as TIMESTAMP_NTZ on DuckDB, and multi-word names such as TIMESTAMP WITHOUT TIME ZONE, which came back unknown. Two spellings Polyglot treats as one type could then compare as different and raise K002. Every bare name now matches Polyglot except Snowflake's BIT, whose Polyglot family no alias can give. Type names may span five words. The deep-nesting fallback-columns case drops to 100 calls: the native DuckDB parser's nesting guard refuses 200, so that model is skipped with P026.
…wn parser A wave whose dialect routes natively now analyses each model in parallel on the analysis pool through the dialect's analyse, with the session's schema and nullability, instead of Polyglot. Placeholder defaults are filled before the parse, SQL macro checks still run on the outcome, and the Polyglot completion and cache attachment steps are skipped for native routes.
…ting rules Finding which cursor stubs and hook parameters sit in code used one quoting policy for every dialect, with backslash escapes in single-quoted strings. A DuckDB or Postgres string ending in a literal backslash, such as 'C:\', then read as unclosed and the model failed with a P001. The scan now uses the dialect's lexical syntax, so E'' escape strings, Snowflake backslash escapes and // comments still hide stubs. The dialect syntax mapping is the same one the rules crate moves to sqlbuild-sqltext.
A native route skipped the analysis cache, so every compile re-analysed every model and a warm run could not answer from stored outcomes. The cache now attaches on every route. A native wave analyses only the models the cache misses, a hit's cleaned SQL is the dialect's normalised SQL, and the Polyglot catalog is only prepared for the Polyglot route.
On a native route a model without lineage facts, such as one with SQL analysis turned off, got no fast lineage, where Polyglot used to parse it. Its lineage is now read by the dialect's own parser and analysis over the fast lineage schema, with its placeholder defaults filled. SQL the parser cannot read is still left out, and the debug event gives P026's reason with the construct, line and column.
The text fallback now takes a type's family from its top-level type, so a STRUCT, ARRAY or MAP holding a timestamp reads as other, not timestamp. Only those 666 dialect entries move; every top-level timestamp is unchanged.
The same project traced after a cold compile, from the analysis cache alone and from compile reuse, gives the compile report and traces of a run without any cache, so a restore keeps every analysed model's lineage facts.
A native route's analysis gate ran before hook expansion, when a named SQL hook is still only its name, so it read inline hooks but never named ones. A named hook the parser cannot read then reached the resource SQL check, whose Polyglot parse reported an unlocated P001, while an inline hook got P026. The gate now runs after hook expansion and reads both kinds, and on a native route the resource SQL check no longer turns a Polyglot rejection of hook SQL into a P001: the gate has given P026 for SQL the dialect's parser cannot read.
A native route analysed every model with no extension functions, so a function the adapter declares a return type for was reported unknown. The session now passes those function names to the dialect's analysis.
…s as fensu requires A native model outcome and a native fast lineage are now entry modules of the analysis route, analysis_route::main::model_outcome and ::fast_lineage, each with one visible function. The session builds its own schema for the outcome, fast lineage passes in the references it scanned, and where a parser gap is and why it is unreadable live in one route helper. The warm-restore lineage test takes its stores from local test cases. Behaviour is unchanged.
Pins SQLQuack at main eba39976 and lists SQLite among the native dialects. The generated-contracts golden records 47 K002 rows that the native type comparison now leaves NotCompared: declared or inferred types it cannot normalise, such as CHAR(3) under DuckDB or malformed spellings.
…ack-io/sqlbuild into wip/native-arms # Conflicts: # crates/sqlbuild-analysis/src/analysis_route/_helpers/mod.rs # crates/sqlbuild-analysis/src/analysis_route/tests.rs # crates/sqlbuild-analysis/src/analysis_route/tests/test_types.rs
…arity link Not a SQLQuack main commit yet; the pin moves to main once the link lands.
…t start The native DuckDB route places the invalid DATE literal's B218 at the statement start, because SQLQuack literals carry no span. Code, line and message are unchanged; the column, end position and the literal's note go.
…ed queries
DuckDB 2.0.0 has no GRANT statement and rejects it with a parser error,
so the grant_read hook fixture now records the grant in an audit table
with unnest over the role list, which DuckDB runs. The header-contract
and SQL-hook summaries are re-recorded: the only change is that the
P026 skips for the old pivot and GRANT text are gone and both models
are analysed.
The two immediate-validation cases are SQL DuckDB rejects
("SELECT FROM WHERE (" and a WHERE ending in "AND ("), so they expect
the native route's P001 compile error again, in its parser's wording.
…native route ClickHouse registers BINARY as an alias of FixedString, and main's type normaliser now spells BINARY(n) as FixedString(n). The native ClickHouse alias table still carried Polyglot's Nullable(BINARY(n)), so the two disagreed after the merge. The parity test now names its oracle, the shared type normaliser, and records the two spellings ClickHouse refuses (bare BINARY and a Nullable inside a Nullable), where the shared normaliser keeps the written text with a parse error.
…ed dialect The type checker cannot see __name__ through the template alias, so the per-dialect spellings are keyed by the template functions themselves. The generator takes the same optional dialect as compile_inputs; a project without one gets the plain spelling and no pivots.
The pivot generator draws its shapes once and renders them through a per-dialect table, so the helper has no conditional; every generated project is unchanged. The ClickHouse alias table's doc comment is one line again. Two pending budgets take main's lower counts, which the merge's union of fensu.toml had kept at the cutover's values.
DuckDB 2.0.0 rejects GRANT and CREATE FUNCTION ... RETURNS ... LANGUAGE SQL with a parser error, so the two compile-input cases now record the grant with an INSERT and create a schema-qualified macro, keeping the destination interpolation they test, and expect no P026. The adapter-dialect hook validation case writes T-SQL bracket names, so it now compiles its inputs as a SQL Server project rather than a DuckDB one; the native route rejected the brackets with the DuckDB parser before assembly could validate them as T-SQL.
…able warning SQLClean returns `unavailable` for a dialect SQLQuack does not format yet, which is Databricks today. sqb format read only `reason`, so every Databricks model became a format-unsafe error. It now leaves the file as written and reports a format-unavailable warning naming the reason, as the sqlclean CLI does, and the command does not fail on it. Databricks formatting returns with a later SQLClean release.
…P001 - A quoted parameter neutralises to a string sentinel, and a trailing comment after the last clause starts at column 0 (SQLClean 0.5.0 on). - Databricks identifiers named Any, Some or Order fingerprint without case at SQLQuack 804864d2; those nine cases are strict xfail until the next SQLQuack pin. - ClickHouse and SQL Server models get one non-failing L007 each, as main's built-in rule facts give every dialect SQL lint cannot parse. - DuckDB stores VARCHAR(5) and VARCHAR(100) as VARCHAR, so a longer text macro body meets a VARCHAR(5) contract; the decimal case keeps the inferred-type check. - The failure corpus's syntax cases expect P001: DuckDB 2.0.0 rejects "SELECT order_id,, ..." and "SELECT FROM WHERE" with parser errors.
…ingerprint folds The warning's message prefix and remediation are constants, chosen without a four-element tuple. The nine Databricks keyword-named cases move to their own strict-xfail test, selected by the case helper rather than by per-case marks.
SQLClean 0.6.2 restores CTE macro placeholders that sit on both sides of a comment, so `sqb format` formats a CTE list whose macros carry attached comments instead of refusing it. It is still built on SQLQuack 804864d2, so the lockfile keeps one SQLQuack source and only the sqlclean entry moves.
On a native analysis route each model's query and hook statements are parsed once in the attachment loop, to decide between P001, P026 and analysis. That parse ran in turn on the main thread, so a 5,000-model DuckDB compile spent about 7.5 s attaching inputs where it had spent 1.8 s when DuckDB's syntax check waited for the parallel analysis. The loop now hands each parse to the model loop's compile pool and carries on. Before it returns, it settles the results in model order: the first syntax gap still raises P001 ahead of any error a later model raised, each P026 is reported as before, and a skipped model's input record and facts are marked unanalysed. Diagnostics are collected by key, so their output order does not change. Reusing the gap parse for analysis would keep every model's tree alive until the analysis phase, and moving the decision into that phase would stop checking hooks and unselected models, so the parse stays separate. On the 5,000-model fresh-process project, attachment drops from 7.5 s to 2.9 s and the compile from 17.4 s to 13.0 s wall; its output and semantic fingerprint are unchanged.
A cached compile still parsed every model's query and hooks to decide between P001, P026 and analysis, although the analysis itself came from the cache. On a warm 5,000-model compile that parse kept attachment at 2.8 s, where it had been 1.7 s when DuckDB's syntax check rode in the cached analysis. Statements the native parser reads whole are now recorded in a native store in the compile cache folder, keyed by dialect, SQL and placeholders under the build's code identity. A model whose statements are all recorded is not parsed again. Gaps are never recorded, so a model with one is parsed every time and keeps its P001 or P026, and an edited model's new SQL misses the store. --no-cache neither reads nor writes it. On the 5,000-model project a warm compile takes 6.1 s instead of 7.5 s, with attachment at 1.7 s; its output is unchanged.
The fresh-process compile and compile cache guards hash each generated project's compile JSON and compiled artifacts. On the native DuckDB route two things in them change, and both are correct. Column lineage gains edges. A metric such as `CASE WHEN status = 'priority' THEN ...` beside a `status` alias reads input.status, which is how DuckDB binds an unqualified name in the same SELECT; Polyglot's lineage left that edge out. Every changed model goes from 3 + 2k to 3 + 3k edges, and no other compile JSON field changes. Column lineage does not feed planning: model identity is the query fingerprint, upstream model keys, config and contract, so these edges cause no replans. In 70 compiled SQL test files at 5,000 models the CTEs come in a different order, with no CTE read before it is defined. The order is the same on every run: two compiles give byte-identical artifacts, and warm and edited compiles reproduce each recorded fingerprint.
On a native route, contract type checks asked the dialect to match the declared and inferred types of every column, and the dialect parses both spellings each time. The Polyglot route already normalized each spelling once per project. A dense 3,000-model DuckDB project spent 220 ms in contracts against 57 ms on the Polyglot route. The project's type comparer now keeps each native match by declared type, inferred type and nullability, so a repeated pair is parsed once. Contracts on that project take 61 ms; findings are unchanged.
SQL-test planning read every model template with the native parser's strict parse after the Polyglot token fast path was removed, so dense projects spent their write phase waiting on template parses. simple_analysis_template restores the bounded recognizer for DuckDB and Snowflake on SQLQuack's dialect tokens. It reads the template after markers_as_names, so a relation named for a marker call reads back as that call, as named_marker_calls does. Text outside the tokens must be whitespace and no token may carry a comment; anything else, and every other dialect, keeps the full parse. A differential test compares the recognized facts with parsed_analysis_template over generated templates and their token deletions and insertions. Compiled output is byte-identical on the dense 3000-model project and DuckShop. Dense 3000 write_ms: 670-800 ms before, 96-124 ms after; main is 103 ms.
fensu.toml takes main's control-store read allowlist entry whole. The earlier merge of c650fcd had kept the cutover's shorter list, which dropped eight of main's paths.
The native analysis route makes cold compiles slower than main on the same runner. Kevin accepted a temporary allowance on the three gates it fails; CHI-1177 restores 1.10 and removes these lines. The limits are the ratios measured at b827e88 against main a1b06e6 on an 8-core runner, the higher of two rounds, plus 10%: - dense cold: wall 1.326/1.347 -> 1.49, CPU 1.642/1.677 -> 1.85 - fresh cold: wall 1.285/1.153 -> 1.42, CPU 1.551/1.404 -> 1.71 - dense model_analysis_cpu_ms (--select=*): 2.660/2.554 -> 2.93 compare_compile_performance gains --cold-max-ratio and --cold-max-cpu-ratio, the cold counterparts of the edit overrides, so the warm and edit modes of the same jobs keep 1.10 and their existing edit limits. The contracts and custom-rules phase gates, the absolute budgets and every other ratio are unchanged.
…lk dupes entry The allowance comments name their removal condition instead of a ticket. fensu.toml gets back main's allowlist entry for the three JSON syntax tree walks (type_trees.rs, dict_walk.rs and walk.rs), which an earlier merge of main had replaced.
…te recognizer as fensu requires The native parse store logs through the SQL-test scan store's debug helper, both on the sqlbuild.compile logger, instead of a copy that called into Python on its own. The bounded template recognizer and its tests handle a failed scan or tokenize explicitly, pass the dialect instead of matching on its name, and keep doc comments to one line. Behaviour is unchanged.
SQLQuack now builds each dialect only with its feature, so SQLBuild enables the ones it reads: every dialect's semantics, and fingerprints for the seven dialects SQLQuack fingerprints, Databricks included. The Snowflake-family parse entry points (parse_strict, parse_statements_with, dialect_tokens, tokenize) are reached through sqlquack-parse. SQLClean moves to the commit pinning the same SQLQuack, so Cargo.lock holds one SQLQuack source. ClickHouse's engine type spelling is no longer public in SQLQuack. The ClickHouse arm keeps it as a local helper copied from SQLQuack with its family and alias tables; across 38,603 spellings (5,837 known types) it returns exactly what the previous public function returned. Databricks fingerprints now keep the case of columns named Any, Some and Order, so their nine strict xfails become ordinary cases.
v0.6.4 is the release of the commit already pinned: it changes only SQLClean's version and pins the same SQLQuack f9e4ae6e.
It went unused once the store logged through the shared debug helper.
kvlonge
added
the
full-verify
Amp for GitHub
kvlonge
enabled auto-merge (squash)
November 9, 2026 22:44
Contributor
Author
|
Superseded by #1397, the same head opened from a branch name the metadata check accepts. |
auto-merge was automatically disabled
November 9, 2026 22:46
Pull request was closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SQLBuild's model analysis moves from Polyglot to SQLQuack's native parsers and semantics for every shipped adapter: DuckDB/MotherDuck, Snowflake, SQLite, PostgreSQL, BigQuery, ClickHouse, SQL Server and Databricks (beta). Custom adapters, and any dialect without a native module, get no SQL analysis. This PR also includes the SQL macros work (#1371) and the follow-up fixes needed to make the DuckShop benchmark project compile cleanly.
cargo update -p sqlcleanalso moved windows-sys versions in unrelated packages.cargo metadata --lockedpasses.format-unavailablewarning, the same as the SQLClean CLI, instead of failing withformat-unsafe. Databricks formatting comes back in a later SQLClean release.Temporary compile-ratio allowance
Native analysis makes cold compiles slower than main on the same runner. A temporary allowance applies to the three gates it fails. Each limit is the higher of two rounds measured at b827e88 against main a1b06e6 on an 8-core runner, plus 10%:
model_analysis_cpu_ms(--select=*)On the final head, two more rounds measured dense cold 1.276/1.267 wall and 1.590/1.557 CPU, fresh cold 1.156/1.141 wall and 1.381/1.385 CPU, and model analysis 2.645/2.678, all within the limits.
compare_compile_performancegains--cold-max-ratioand--cold-max-cpu-ratio, so these limits apply to cold compiles only. Warm and edit stay at 1.10, edit keeps its existing 1.20/1.75, and the contracts and custom-rules phase gates and the absolute budgets are unchanged. Warm and edit were within 1.10 in both rounds. A follow-up restores 1.10 once the native analysis path is optimised.The dense 5000 all-rules guard passes with identical fingerprints. Head cold/warm/edit is 18.29/8.53/8.81 s against main's 13.85/8.74/9.16 s on the same machine.
Behaviour changes for the release notes
require_sql_analysis. Polyglot raised B000.status = 'priority'now binds toinput.status. Lineage doesn't feed planning.format-unavailablewarning.integer + dateand COALESCE over mixed families are untyped, where Polyglot guessed a type.Merge checklist
make teston the final code: 12,125 passed. 6 interrupt and SIGTERM timing tests failed on the measuring machine, and main a1b06e6 fails the same tests there in isolation; CI is the reference for them.sqb format --checkreports FAULT=0 and WARN=0.make check-ciand full-verify CI are green.