Repository navigation
Commit c642221
authored
feat(analysis): analyse models with SQLQuack's native parsers and semantics (#1397)
* chore(deps): bump sqlquack to a9dde6a0
* feat(analysis): give native dialects a parse and type contract
The native dialect trait now has parse and normalize_type, with read_whole
and type_match provided on top. read_whole refuses parser gaps, empty input,
more than one statement and statements the parser kept unparsed. type_match
reports NotCompared when either side has no normalized type, so the contract
check makes no finding instead of guessing.
Parser gaps carry the parser's message and whether the failure is a syntax
error, and the P026 skip appends that message. AnalysisRoute gains an
Unavailable arm, which skips a model with P026 instead of analysing it.
The contracts type comparer goes through the route's type_match, and
normalize_text_type is public so native dialects can reuse the shared
text type rules.
* feat(analysis): analyse native dialect queries with sqlquack-semantics
A native dialect now names its sqlquack-semantics profile, and the route
provides analyse and macro_body on top of read_whole. analyse turns the
query's lineage and findings into the facts the analysis session keeps:
output columns typed through the dialect's normalize_type, lineage rows
over the model's referenced resources, the columns deciding rows, the star
flags, and findings in SQLBuild's codes through the same sqlbuild_code the
Polyglot path uses, so implicit-conversion findings stay blocking. Unknown
functions a dialect declares as extensions are not reported.
StandInSchema gives probe queries, such as a macro body's, one relation of
known columns. read_whole now reports SQL with no statement, such as a lone
comment, as no statement rather than more than one.
* fix(types): take a fallback type's family from its top-level type
The text fallback gave family Timestamp to any type whose text held
TIMESTAMP, so an ARRAY, MAP or STRUCT holding a timestamp read as a
timestamp where Polyglot reads it as Other. The family now comes from the
leading type name, so TIMESTAMP(3), TIMESTAMP WITH TIME ZONE and
TIMESTAMP_NTZ stay timestamps and complex types do not.
* fix(analysis): compare ClickHouse nullability without a type, and keep stubs in code
type_match now takes the inferred type as optional, and NotCompared keeps
whether the declared type alone forbids null. A ClickHouse column declared
without Nullable that receives a nullable value of unknown type is flagged
again, while no type finding is made. ClickHouse types are compared without
whitespace outside quoted strings, so Decimal(10, 2) is Decimal(10,2).
Respelling cursor stubs for a native parser now leaves text inside strings
and comments, and longer names that end in a stub's text, as written.
* fix(analysis): skip the Polyglot pivot proof for natively analysed dialects
The dynamic pivot proof reads Polyglot's parse of the model. A dialect on
a native or unavailable analysis route now gets a proof failure naming
why, instead of a Polyglot parse. Every dialect is still on the Polyglot
route, so no proof changes yet.
* Revert "fix(analysis): skip the Polyglot pivot proof for natively analysed dialects"
The pivot proof moves onto SQLQuack's parser for every dialect in its own
change, so no route needs to skip it.
* refactor(analysis): shape the native analysis route as fensu's Rust rules require
Each analysis_route and semantic_validation entry module now exposes one
function, with ClickHouse's spelling helpers and the read_whole gap in
_helpers. Doc comments are one line, the route's tests run every case from
a named test case table, and report_skip takes the P026 code and message
together. The analysis_route module no longer re-exports its entries.
normalize_text_type and clickhouse_type_match keep an exception for having
no importer yet: their native dialect modules land in their own changes.
* chore(deps): bump sqlquack to 19c51fa2
sqlquack main 19c51fa2 carries the Postgres parser work and the train that
followed a9dde6a0. sqlquack-syntax-postgres now depends on stacker, which
re-resolves a few Windows-only windows-sys edges in the lock.
* chore(deps): pin sqlquack-semantics to 19c51fa2 with the other sqlquack crates
* feat(analysis): analyse Snowflake and DuckDB natively with SQLQuack
Snowflake, DuckDB and MotherDuck models now take the native analysis
route. Each arm parses with its SQLQuack parser, analyses under its
sqlquack-semantics profile, and compares types through respellings that
keep Polyglot's type names. The SQLite arm is in place but stays on the
legacy route until its parser is in SQLQuack's depth probe.
SQLQuack crates are pinned to main edfa30b2, which adds the DuckDB and
SQLite parsers.
* test(differential): keep the seed opt-out model unreadable to the native DuckDB parser
The native DuckDB parser reads WHERE id = ANY [1, 2], which DuckDB runs,
so the seed's sql_analysis false model no longer needed its opt-out and
require_sql_analysis rejected it. A NEAREST join is valid DuckDB that the
parser keeps as a gap, and the Postgres and Snowflake parsers refuse it
too. Only the fixture's SQL text changes in the six seed goldens.
* fix(analysis): let a native route's parser decide syntax, after the reference scan
On a native analysis route the gap gate reads the model SQL and hooks whole,
so assembly no longer reruns Polyglot's syntax validation there: a model the
native parser reads gets no Polyglot P001. The gate now runs after reference
scanning, so a located scanner error wins over a raw normalisation failure,
and a model whose reference calls already failed P012 gets no P026. The
Polyglot route keeps its order and validation.
* fix(analysis): native type spellings take Polyglot's other dialects' names
A full comparison with Polyglot's normalize_type showed the DuckDB,
Snowflake and SQLite tables missing names from other dialects, such as
TIMESTAMP_NTZ on DuckDB, and multi-word names such as TIMESTAMP WITHOUT
TIME ZONE, which came back unknown. Two spellings Polyglot treats as one
type could then compare as different and raise K002. Every bare name now
matches Polyglot except Snowflake's BIT, whose Polyglot family no alias
can give. Type names may span five words.
The deep-nesting fallback-columns case drops to 100 calls: the native
DuckDB parser's nesting guard refuses 200, so that model is skipped with
P026.
* feat(analysis): analyse models on a native route with the dialect's own parser
A wave whose dialect routes natively now analyses each model in parallel on
the analysis pool through the dialect's analyse, with the session's schema and
nullability, instead of Polyglot. Placeholder defaults are filled before the
parse, SQL macro checks still run on the outcome, and the Polyglot completion
and cache attachment steps are skipped for native routes.
* fix(analysis): find a native route's stubs with the dialect's own quoting rules
Finding which cursor stubs and hook parameters sit in code used one quoting
policy for every dialect, with backslash escapes in single-quoted strings. A
DuckDB or Postgres string ending in a literal backslash, such as 'C:\', then
read as unclosed and the model failed with a P001.
The scan now uses the dialect's lexical syntax, so E'' escape strings,
Snowflake backslash escapes and // comments still hide stubs. The dialect
syntax mapping is the same one the rules crate moves to sqlbuild-sqltext.
* fix(analysis): keep the analysis cache on a native route
A native route skipped the analysis cache, so every compile re-analysed every
model and a warm run could not answer from stored outcomes. The cache now
attaches on every route. A native wave analyses only the models the cache
misses, a hit's cleaned SQL is the dialect's normalised SQL, and the Polyglot
catalog is only prepared for the Polyglot route.
* fix(lineage): build fast lineage natively for models without facts
On a native route a model without lineage facts, such as one with SQL
analysis turned off, got no fast lineage, where Polyglot used to parse it.
Its lineage is now read by the dialect's own parser and analysis over the
fast lineage schema, with its placeholder defaults filled.
SQL the parser cannot read is still left out, and the debug event gives
P026's reason with the construct, line and column.
* test(types): re-record the generated type golden for nested timestamps
The text fallback now takes a type's family from its top-level type, so a
STRUCT, ARRAY or MAP holding a timestamp reads as other, not timestamp. Only
those 666 dialect entries move; every top-level timestamp is unchanged.
* test(lineage): trace fast lineage after a warm restore
The same project traced after a cold compile, from the analysis cache alone
and from compile reuse, gives the compile report and traces of a run without
any cache, so a restore keeps every analysed model's lineage facts.
* fix(analysis): gate a native route on a model's named hooks too
A native route's analysis gate ran before hook expansion, when a named SQL
hook is still only its name, so it read inline hooks but never named ones. A
named hook the parser cannot read then reached the resource SQL check, whose
Polyglot parse reported an unlocated P001, while an inline hook got P026.
The gate now runs after hook expansion and reads both kinds, and on a native
route the resource SQL check no longer turns a Polyglot rejection of hook SQL
into a P001: the gate has given P026 for SQL the dialect's parser cannot read.
* fix(analysis): know the adapter's declared functions on a native route
A native route analysed every model with no extension functions, so a
function the adapter declares a return type for was reported unknown. The
session now passes those function names to the dialect's analysis.
* refactor(analysis): shape the native route's model and lineage entries as fensu requires
A native model outcome and a native fast lineage are now entry modules of the
analysis route, analysis_route::main::model_outcome and ::fast_lineage, each
with one visible function. The session builds its own schema for the
outcome, fast lineage passes in the references it scanned, and where a
parser gap is and why it is unreadable live in one route helper. The
warm-restore lineage test takes its stores from local test cases. Behaviour
is unchanged.
* feat(analysis): analyse SQLite natively with SQLQuack
Pins SQLQuack at main eba39976 and lists SQLite among the native
dialects. The generated-contracts golden records 47 K002 rows that the
native type comparison now leaves NotCompared: declared or inferred types
it cannot normalise, such as CHAR(3) under DuckDB or malformed spellings.
* build(analysis): pin SQLQuack at the DuckDB window, PIVOT and arity parity link
Not a SQLQuack main commit yet; the pin moves to main once the link lands.
* test(analysis): re-record the unsupported-cast golden at the statement start
The native DuckDB route places the invalid DATE literal's B218 at the
statement start, because SQLQuack literals carry no span. Code, line and
message are unchanged; the column, end position and the literal's note go.
* WIP (not shipping): Databricks' arm on the NativeDialect trait, on a SQLQuack validation pin
Databricks' module parses with SQLQuack's Databricks SQL parser
(parse_strict), analyses under the Databricks semantics profile, and
types spellings through normalize_text_type after folding the Databricks
synonyms Polyglot spells another way (INTEGER, REAL, DOUBLE PRECISION,
NUMERIC, TIMESTAMP_LTZ, TIMESTAMP_NTZ). It joins NATIVE_DIALECTS under
the adapter's analysis dialect name, databricks. A byte the scanner reads
as no token is a syntax error, as on BigQuery's arm.
Every sqlquack crate is pinned to a37cc362, a validation pin: one commit
on SQLQuack main eba39976 adding the Databricks semantics profile, which
is not on main yet. Repin every sqlquack crate to the SQLQuack main
commit that carries the profile.
The syntax error cases use SELECT 1 FROM t WHERE ) rather than FROM
FROM: Databricks does not enforce ANSI reserved words, and its reference
reads SELECT where from FROM where select as a query, so the parser
correctly reads FROM FROM as a table named FROM.
* WIP (not shipping): drop the fensu exception Databricks' arm resolves
With Databricks' module typing through normalize_text_type, its FPRSL105
exception no longer matches a fault and is removed.
* fix(analysis): revalidate type recovery with the native analysis on native routes
Type recovery revalidated each model downstream of a poisoned output through
the Polyglot binding catalog. On a native route that catalog was never
prepared, so preparing it while the semantic stage held a borrow on it
panicked with "Already borrowed" whenever a model with its own binding error
sat downstream of a blocking type error.
On a native route, revalidation now reads each planned model's analysed SQL
again with its dialect's analysis, over the schema the session bound and with
the poisoned outputs untyped. The findings come back at the session's own
positions, so type recovery matches them against the raw rows directly, and
the Polyglot catalog is no longer on that path. The finished session keeps
the model order and the declared functions it analysed with. The Polyglot
route is unchanged.
* fix(analysis): keep marker lineage and SQLBuild's text checks on native routes
Fast lineage on a native route passed relation markers to the parser as
written. Markers now read as indexed names through the shared relation
marker reader, each tied to the resource it calls, so a model and a source of
the same name stay apart. A source column's relation is compared as the
dialect's profile resolves names, so upper-case folding dialects such as
Snowflake keep their sources too, in fast lineage and in session lineage
alike, and repeated sources of one column are listed once.
Native analysis now adds SQLBuild's own text checks that the Polyglot route
runs: truth-value tests the warehouse rejects (B103) and SQLite's temporal
casts (B104). Output column facts keep the analysis's own type spelling, such
as INTEGER or VARCHAR, and use the dialect's normalised name only to decide
whether the type is known.
The relation marker reader is the same as on the parser train branch.
* test(analysis): re-record native-route goldens and pin SQLQuack at the parity merge
Pins SQLQuack at ed14b717 (main fa55d9a1 with the DuckDB parity link);
the pin moves to main once that link lands.
Golden changes:
- model-sql-syntax, syntax-hook, syntax-inline-hook: SQL the native
parser cannot read is P026 rather than P001. The cases now set
require_sql_analysis so P026 is the failing error.
- seed 0, 6 and 10: amount passes through a CAST in a CTE, so its trace
is cast rather than direct; seed 10 is now high confidence.
- analysis-window-without-over, analysis-unknown-function,
analysis-aggregate-in-where: the finding is placed on the call itself.
* test(lineage): an unterminated DuckDB query reports where the native parser stops
* fix(sqltext): read T-SQL bracket names and ClickHouse quoting in dialect scans
A T-SQL `[...]` name was code to the dialect scan, so an apostrophe, `--`
or `/*` inside one opened a string or comment that hid everything after it.
On a native route a cursor stub after such a name was not respelt, and
a valid model was skipped with P026. Lexical rules gain bracket identifiers,
closing at `]` with `]]` for one bracket, and the T-SQL rules set them.
Adapter records never set the flag, so their keys and scans are unchanged.
ClickHouse's rules now follow its lexer: a backslash escapes inside single
quotes, double quotes and backticks, and `#` starts a comment only before a
space or `!`.
* fix(analysis): read repeated markers as one relation and keep unreadable revalidations
Native fast lineage gave every relation marker call its own name, so a
union of two branches over one model read two relations and its columns
became expressions. Calls of the same marker now share their first call's
name, padded to keep positions.
Native revalidation in type recovery failed the compile when the dialect's
parser could not read a planned model again. Such a model now keeps its own
findings, as an unrevised model would.
* feat(analysis): read SQL Server models with SQLQuack's T-SQL parser
The tsql analysis dialect now routes natively. TsqlDialect implements
NativeDialect: parse reads with sqlquack-syntax-sqlserver, an unknown
one-part function staying a syntax error as SQL Server's 195 is;
profile is sqlquack-semantics' SQL Server profile; and normalize_type
gives Polyglot's T-SQL canonical names, or None for a type neither SQL
Server's built-in types nor SQLBuild's logical types know.
SQLQuack is pinned to a commit carrying the SQL Server semantics profile.
* test(analysis): state the T-SQL parse and type cases as named test cases
TsqlDialect now calls normalize_text_type, so the rule exception that
waited for a native dialect module to call it is removed.
* chore(deps): pin SQLQuack to the main commit carrying the SQL Server semantics profile
* test(analysis): keep T-SQL cursor stubs after bracket names holding quotes and comment openers
Gap-test rows put a quote, an escaped bracket and both comment openers in
a bracket name before a cursor stub; a SQL Server project requiring SQL
analysis compiles a microbatch model whose stub follows [x--y].
* Route ClickHouse analysis to SQLQuack's ClickHouse parser and profile
- ClickHouse implements the native dialect: SQLQuack's ClickHouse parser
with its named gaps, the ClickHouse semantics profile, types normalized
as Polyglot normalizes them, and contract types compared after both
sides are respelt as ClickHouse prints them, so BIGINT is a non-null
Int64.
- A declared type ClickHouse refuses compares nothing; an inferred type
it refuses or does not know still meets a non-null declaration.
- Every SQLQuack crate is pinned to SQLQuack main edfa30b2, which carries
the ClickHouse parser, and ClickHouse is listed as a native dialect.
- The ClickHouse gap-gate tests cover an apostrophe in a `#` and a `#!`
comment, which ClickHouse reads as comments.
* test(analysis): re-record native spellings and the DuckDB union guard
Typed markers now report DuckDB's own VARCHAR and INTEGER. A deep union
of one marker resolves its columns at high confidence, and the DuckDB
parser reads 8,000 UNION ALL branches and refuses 8,001.
* test(analysis): re-record type notes in the engine's own spellings
Native analysis keeps the dialect's type names, so DuckDB notes and
messages say VARCHAR and INTEGER where Polyglot printed its comparison
spellings TEXT and INT.
* fix(analysis): name the hook and the model file line in P026
A native route's P026 for a hook now names the hook and its file, as
the Polyglot syntax error did, such as post_hooks[0] sql("record_refresh")
(hooks/sql/record_refresh.sql), and places the gap in the hook's SQL.
A gap in the model's query is placed at its line and column in the
model file when the text maps back through stub spelling, placeholders
and expansion; otherwise the message says the position is in the
analysed SQL.
* feat(analysis): let a native dialect respell session column types
NativeDialect gains session_type, the identity by default: the session
schema and revalidation give each known column type to the parser in the
spelling the dialect reads, for dialects whose parser cannot read every
type SQLBuild keeps.
* WIP (not shipping): BigQuery's arm on the NativeDialect trait, pinned to SQLQuack main edfa30b2
* build(analysis): pin SQLQuack at main 38be86ba
* WIP (not shipping): align the BigQuery arm's tests with the native dialect trait
The trait's shared read_whole places "more than one statement" at the
start of the SQL for every native dialect, and the text fallback now takes
a type's family from its top-level type, so RANGE<TIMESTAMP> and the
complex types holding one read as Other on both routes. With BigQuery's
module typing through normalize_text_type, its fensu exception no longer
matches a fault and is removed.
* WIP (not shipping): read session types BigQuery cannot spell by their family
The session keeps SQLBuild's spellings, such as TEXT for an expression
source's string columns, VARCHAR(n) and DOUBLE, which BigQuery's parser
cannot read, so every column downstream of them lost its type on the
native route. BigQuery overrides #1369's NativeDialect::session_type to
respell an unreadable spelling as its family's BigQuery type.
Two BigQuery e2e cases held SQL that GoogleSQL refuses: WINDOW is reserved
as an alias, and EXCEPT and INTERSECT need DISTINCT or ALL.
* WIP (not shipping): let BigQuery casts name RANGE types
SQLQuack's cast check knows a named type only through an engine family or
the schema's known_type, and no family names RANGE, so a valid
CAST(x AS RANGE<DATE>) was blocked as B218 on the native route. BigQuery's
analyse now reads the query against a schema that also knows every type
BigQuery's type grammar reads; RANGE<INT64> stays B218.
The BigQuery set-operation models move into a constant, as fensu requires
of test modules.
* WIP (not shipping): re-record the BigQuery entries of the generated contract and analysis goldens
BigQuery now analyses natively, so only its entries move; every None,
DuckDB, Postgres, Snowflake, Databricks and T-SQL entry is unchanged.
contracts_generated (32 of 60 entries, only the bigquery key): 56 K002
findings (49 errors, 7 warnings) no longer appear and none is added. Each
pairs a type GoogleSQL refuses, such as REAL, CHAR(3), VARIANT, MAP,
TIMESTAMPTZ, int4 or VARCHAR, or an empty spelling, with the other side;
the native comparison leaves such pairs uncompared.
analysis_generated (the two cases with a BigQuery profile; 36 BigQuery
groups): column types take SQLQuack's BigQuery spellings (INT64, STRING,
BOOL); casts to VARCHAR or TEXT, which GoogleSQL refuses, and the columns
passing them on are untyped, as is COALESCE over INT64 and STRING; more
outputs are proven non-null; B002 messages drop Polyglot's context
suffix; SELECT ... AS "Mixed_Case", which GoogleSQL refuses, fails
analysis as a parser gap rather than as B000; and B002 on s.side over a
star subquery below that gap is not reported, though a compile of the same
shape reports it.
* test(analysis): re-record P026 messages and fast-lineage golden for native arms
The P026 rows now name the model file or the hook with its file, with the
gap placed in that text. The generated fast-lineage golden moves with the
native route: root stars resolved, medium confidence now high, the
schema's own column spelling, set-operation columns that read different
branch columns now expression, and edges for models that had none.
* WIP (not shipping): route Postgres analysis to SQLQuack's PostgreSQL parser and profile
PostgresDialect reads model SQL with sqlquack-syntax-postgres's strict
parser, and normalises a type only when PostgreSQL's type grammar reads
it. Every SQLQuack crate is pinned to main edfa30b2.
* WIP (not shipping): shape the Databricks arm's type test and constant as fensu requires
The type test case names Polyglot's spelling of each type as its expected
name, so its assertion reads an expected field as well as comparing the
whole type with Polyglot's, and the alias constant keeps a one-line doc
comment.
* test(analysis): cover a root star over an opted-out model in fast lineage
Native analysis resolves every root star over a relation with known
columns, so the generated projects no longer leave stars for fast lineage
and their star minimum goes. A dedicated case keeps a root star over a
model without SQL analysis, whose facts leave the star unresolved, and
checks that restored lineage equals the catalogued lineage on every
dialect. The arm gap tests set the new gap offset.
* WIP (not shipping): compare Postgres types as PostgreSQL names them
PostgresDialect::normalize_type reads a type with PostgreSQL's type
grammar and names it as format_type does, so INT, int4 and INTEGER are
one type, before SQLBuild's comparison shape.
* WIP (not shipping): type Databricks spellings through Databricks' own type grammar
normalize_type now decides whether a type is known, so it must refuse what
Databricks refuses. The arm reads a spelling as the target of
SELECT CAST(NULL AS ...) with Databricks' parser, as BigQuery's arm does,
and abstains on anything else: TEXT, BOOL, INT64, FLOAT64, DATETIME, JSON,
TIMESTAMPTZ, INTEGER[], MAP(K, V), NUMBER(p, s), DOUBLE PRECISION (the
DOUBLE type's documented syntax is DOUBLE alone) and the empty spelling.
Such pairs, and the columns typed with them, are left uncompared.
A read type is respelt from its parsed form, so Databricks' synonyms meet:
INTEGER and INT, LONG and BIGINT, SHORT and SMALLINT, BYTE and TINYINT,
DEC and NUMERIC and DECIMAL with Databricks' (10,0) default and scale 0
after a lone precision, TIMESTAMP_LTZ and TIMESTAMP, TIMESTAMP_NTZ and
TIMESTAMP WITHOUT TIME ZONE, VARCHAR and CHAR without a length and STRING,
and a STRUCT field written with or without its colon. Field constraints,
comments and collations are not part of the type. Where Polyglot keeps
LONG, SHORT, DEC or TIMESTAMPLTZ apart, the native route follows
Databricks' synonym table and the semantics profile's own type synonyms.
Databricks types a cast to CHAR(n) or VARCHAR(n) as STRING and reads a
CHAR or VARCHAR column as STRING; those types are a length check on
STRING when a table is written. type_match therefore takes CHAR(n) or
VARCHAR(n) on one side and STRING on the other as one type. Without this
every contracted VARCHAR(n) column of the generated seed projects raised
K002 on the native route.
* WIP (not shipping): re-record the Databricks entries of the generated contract golden
Databricks now analyses natively, so only its entries move: 32 of 60
entries of contracts_generated change, only under the databricks key, and
every None, BigQuery, DuckDB, Postgres, Snowflake and T-SQL entry is
unchanged. analysis_generated is unchanged for both of its cases.
55 K002 findings (50 errors, 5 warnings) no longer appear and none is
added. 54 pair a spelling Databricks' type grammar refuses with the other
side: TEXT, BOOL, INT64, FLOAT64, int4, JSON, DATETIME, TIMESTAMPTZ,
TIMESTAMP WITH TIME ZONE (Databricks reads only WITH LOCAL TIME ZONE),
INTEGER[], MAP(VARCHAR, INTEGER), NUMBER(38, 0), "not a type (" or an
empty spelling. The native comparison leaves such pairs uncompared, as on
BigQuery's arm. The other is a warning on CHAR(3) inferred against
VARCHAR declared: Databricks reads a CHAR(n) column as STRING, and VARCHAR
without a length is STRING, so the two are one type there.
* WIP (not shipping): match the Databricks type probe's parse failures explicitly, as fensu requires
* test(analysis): re-record native-arm spellings, a B231 position and a refused DuckDB audit
Under the native DuckDB route, inferred types keep DuckDB's own names:
INTEGER for a cast to INTEGER, VARCHAR for a string or a cast to VARCHAR,
where the legacy analysis spelled INT and TEXT. B231 for an aggregate in
WHERE now points at the call, as the COUNT(*) row already did.
A relationships audit from a string key to an integer key now reports B301
and B217: DuckDB 2.0 refuses the compiled audit's NOT IN with "Cannot compare
values of type VARCHAR and INTEGER in IN/ANY/ALL clause", so the case
expects both findings and a failed compile.
* WIP (not shipping): Databricks reads session types it cannot spell by their family, where the family has one type
The session keeps SQLBuild's spellings, such as TEXT for an expression
source's string columns, which Databricks' parser refuses, so the columns
downstream lost their type on the native route. session_type keeps a
spelling Databricks reads and respells STRING, BOOLEAN and DATE families,
which have one Databricks type each. Integer, decimal, float and timestamp
spellings stay unread: Databricks has several types in each family, and a
family alone cannot say which.
The Databricks gap test cases carry the source offset the native route
now records.
* WIP (not shipping): validation pin of every sqlquack crate to 02bd392d
02bd392d is the approved Databricks semantics link (the profile and its
LATERAL VIEW and set-operation ORDER BY fix) on SQLQuack main 38be86ba;
it is not on main yet. Repin every sqlquack crate to the SQLQuack main
commit that carries it.
* fix(analysis): give native routes the session's functions, quoting and case setting
A native route analysed every model against a schema with no function
facts, quoted output names lost their quoting, and Snowflake's
QUOTED_IDENTIFIERS_IGNORE_CASE was ignored.
- The session schema now answers the adapter's declared function return
types, respelt for the dialect, and its first_arg, conditional_result and
safe_cast nullability rules.
- A stored name in double quotes becomes a quoted schema column, with ""
read as one quote, wherever the session, revalidation or fast lineage
builds one.
- The session request carries the profile's quoted-identifier case setting
into the cache key and each dialect's analysis, so a dialect can pick a
profile for it.
- Semantic completion no longer panics when the binding catalog is already
borrowed: unknown relations stay per-request.
- An adapter's own Python nullability rules are not called on a native
route; compile now warns, naming the adapter and the functions. The
callback tests run on the Polyglot route, and the payload-only completion
check expects a native route to require its session.
* fix(analysis): quote an unknown quoted column as written
SQLQuack quotes an unknown column's name in lower case, so a quoted
"MiSsPeLlEd" was reported as 'misspelled'. A native route now puts the
authored name back into the message.
* feat(analysis): fold quoted Snowflake names under QUOTED_IDENTIFIERS_IGNORE_CASE
With the session parameter set, the Snowflake arm analyses through
SQLQuack's quoted-ignoring-case profile, so a quoted name resolves to upper
case like an unquoted one, as Snowflake does.
* test(analysis): re-record CREATE TABLE AS and INSERT ... SELECT columns as unknown
Native analysis reads only queries, so a model whose SQL is a CREATE TABLE AS
or an INSERT ... SELECT gets no columns rather than its inner SELECT's.
* test(analysis): expect the native refusal to complete without a session as a compiler error
* test(analysis): capture each generated semantics project under its own dialect
The generated failing-projects test compiled every project under DuckDB and
re-read the captured inputs as each other dialect, so a dialect's golden
held DuckDB's analysis of its models. Each dialect now compiles the project
under its own analysis dialect, with DuckDB's function rules kept for all.
* WIP (not shipping): pin every sqlquack crate to SQLQuack main 3b497dfa
SQLQuack main 3b497dfa carries the Databricks semantics link 02bd392d, so
the validation pin is replaced by a main commit.
* test(analysis): re-record the native digest goldens and pin the open native gaps
Re-record the Python-oracle digest goldens the native DuckDB, Snowflake
and SQLite routes change: engine type spellings and result types,
nullability proven by filters and literals, unknown columns for stars
over relations without an authoritative shape, per-output types for
duplicate output names, and messages that keep the authored spelling.
Keep the open gaps as strict expected failures beside passing controls:
Snowflake's invalid coercible literal and date arithmetic, and declared
source types that native analysis does not read.
The adapter-rule case is renamed, because adapter callbacks are not
called on a native route. Native routes record no legacy lineage
enrichment, so those minimums are 0 where every profile is native.
* test(analysis): expect a native wave to analyse each equal model query
A native wave analyses every model on its own, so the equal-query case
expects no shared or re-analysed queries. Outcomes are unchanged; only
the repeated work differs.
* fix(analysis): type a native route's reads of declared but open relations
A native route gave SQLQuack only relations whose full shape was known,
so a source or seed with declared column types but no authoritative shape
was read untyped: `status + 1` over a declared VARCHAR went unreported,
where Polyglot's rich analysis used the declared types.
The session schema now passes each such relation's declared columns, with
the nullability the session knows, as open columns. They type the columns a
query reads without making the relation's column list complete, so star
expansion and unknown-column checks are unchanged. Revalidation passes the
same open columns. The analysis cache format moves to v4, as outcomes for
the same inputs change.
* WIP (not shipping): bind the Databricks timestamp's zone flag in one arm
The setting-help scan reads a source line holding both a word followed by
': true' and a string as a loosely named setting. The two timestamp arms
become one that matches on the zone flag; the spellings are unchanged.
* fix(rules): read relation markers as tables in custom-rule syntax trees
Compiled query_sql keeps relation markers such as __ref("orders"). Snowflake
does not accept a bare function call in FROM, so newer SQLQuack parsers refuse
every Snowflake model that references another.
A shared helper in sqlbuild-analysis (relation_markers::markers_as_names) now
writes each marker call in code as a short indexed name, padded so all later
text keeps its line and column, and returns the calls by index. sql_tree reads
SQL through it for every dialect. The rules crate's dialect lexical syntax moves
to sqlbuild-sqltext so both use one mapping.
* feat(sql-macros): parse SQL macros and SQL-test planning with the native parser
SQL macro bodies and SQL-test planning parsed with Polyglot. Both now
parse with the native parser that analysis_route gives the project's SQL
analysis dialect, and a dialect with no native parser reports the feature
as unavailable rather than falling back.
- Macro bodies parse with the project's parser instead of trying eight
Polyglot dialects. Declarations and USING slot reads are masked as
names of the same length, so spans line up. A body that only names a
type is read as a cast's type.
- Macro probes type through NativeDialect::analyse over stand-in
relations; returns, ANY-family, argument and slot checks read its
output types, and type findings are SQLBuild's B210-B219.
- SQL-test planning parses with sql_test_parser(dialect). The planning
template reads relation marker calls as plain names, so Snowflake
accepts them in FROM, and a list after IN is read as a list.
- Polyglot's output-typing entry points, used only by SQL macros, and the
token-based simple-template fast path are removed.
* refactor(sql-tests): read planning relation markers through the shared markers_as_names
The planning template now names marker calls with relation_markers::markers_as_names, which sql_tree also uses, and reads them back by its MARKER_NAME_PREFIX. The SQL-test patterns no longer carry their own marker regex.
* test(sql-macros): name the tracked list-typing work in the list-body xfail reasons
* test(sql-macros): expect the unavailable error for adapters without a native parser
Expansion cases for an adapter whose dialect has no native parser were filtered out, so they checked nothing. They now assert the P015 that says SQL macros are unavailable, and run as written once the adapter's arm lands.
* docs(sql-macros): say macros and SQL tests need a native parser, and drop partly-known returns checks
Macros and SQL tests now read SQL only with the adapter's native parser, so the pages say what happens without one. Native analysis does not type list literals, so a list body trusts returns rather than failing P024 by shape, and the BigQuery NUMERIC example goes until BigQuery has an arm.
* test(sql-macros): describe the list-typing work in the list-body xfail reasons without a tracker reference
* build(analysis): pin sqlquack to c2e78ffb
The pin carries DuckDB value keywords, Databricks semantics and the
DuckDB and SQLite star order over USING and NATURAL joins.
* test(analysis): re-record the native goldens for declared open columns and the new pin
Declared but open relations now type the columns read from them, so the
pending declared-source case passes in both lineage modes and joins its
control. Stars over them list the declared names, untyped and still
unresolved, and a cyclic project types its declared source columns.
DuckDB and SQLite stars over a USING join now keep the key in place.
* fix(deps): resolve tempfile against the workspace's windows-sys again
The native route's dependency update moved tempfile onto windows-sys 0.52, which
sqb and the Python extension then built with different features.
* chore(analysis): pin every native arm to one SQLQuack train and share the unquoted stub prefix
* chore(analysis): lock the Databricks syntax crate at the shared SQLQuack train
* test(sql-macros): run every macro case through its adapter's native arm
Every shipped adapter now has a native arm, so no case expects the unavailable
error any more.
* refactor(analysis): group the native route's helpers into analysis and dialect modules
With every arm in, the helpers outgrew fensu's ten-module container. The
PostgreSQL type probe now matches its parse failure explicitly.
* feat(analysis): leave a dialect no native module reads unanalysed
Every shipped adapter's dialect now has a native arm, so analysis_route sends any
other dialect, or none, to Unavailable rather than to Polyglot. Tests that relied on
the no-dialect path now use DuckDB.
- Fast lineage names an output the SQL leaves unnamed from the compiled column at its
position, as the Polyglot path did.
- An adapter's own nullability rules are never called on a native route; the two
callback tests now check the compile warning instead.
- The analysis session, cache and fast lineage expectations are re-recorded for the
native route; each move was checked against the SQL.
- The arms' gap tests and the macro type probes fill the shared route's newer fields.
* test: expect every shipped dialect's native arm
- SQL Server plans SQL tests natively, so its two planner cases are no longer xfailed.
- Databricks, BigQuery and PostgreSQL macro return checks run natively.
- Completing from the session alone is always refused on a native route.
* test(analysis): xfail PostgreSQL's invalid-literal comparison until SQLQuack checks it
PostgreSQL refuses `amount > 'placed'` while parsing, but SQLQuack's PostgreSQL
profile does not yet check literal coercion. The DuckDB row stays the passing control.
* test(analysis): compile each generated project under its profile's own dialect
The compile context named no analysis dialect, so the model loop ran on the no-dialect
path whatever profile the session used. Each run now compiles under the dialect it
analyses, as an adapter does.
* fix(analysis): route PostgreSQL's assembly dialect name to its native module
Compile assembly hands the analysis session the Polyglot alias
'postgresql', not the adapter's 'postgres'. The route table only knew
'postgres', so the session took the Polyglot path for PostgreSQL models,
and after unknown dialects became unavailable a warm cache skipped the
catalog preparation that path needs. Both names now route to the
PostgreSQL module.
* test(analysis): re-record PostgreSQL goldens under its native session
With PostgreSQL analysed natively, its keys move to the engine's type
names (INTEGER, TEXT for MAX over VARCHAR, NUMERIC for SUM over DECIMAL),
fold unquoted names in findings, and drop follow-on findings of a CTE
whose body already failed, as PostgreSQL reports one error per
statement.
Pending rows, each strict with a passing DuckDB control: BigQuery's
invalid timestamp literal, a PostgreSQL CTE over a failed upstream model
that is left with no finding, and an expression source whose alias case
PostgreSQL folds. The session completion test asserts that payload-only
completion answers some projects and matches the session wherever it
does; the legacy enrichment floor is zero on all-native profiles.
* fix(analysis): route every analysis alias to its native module
The alias tables also name MotherDuck and SQL Server as 'motherduck' and
'sqlserver'. Route both to the DuckDB and T-SQL modules, and check each
route name both as given and after compile assembly's alias mapping, so
a respelt name cannot fall back again.
* test(analysis): prove each shipped adapter's session analyses natively
Equal model queries share one analysis on the Polyglot batch path and
never on a native wave, so a session that falls back to Polyglot under
any shipped adapter's own profile fails this test.
* fix(analysis): read no resource SQL where no native parser reads the dialect
A custom adapter's dialect gets no SQL analysis, but the semantic stage
still sent its hooks, audits and SQL tests to the Polyglot binding
catalog, so a hook Polyglot could not parse failed the compile with
P001. Only the audits of declared columns remain on that route. The
differential case for an unbuilt dialect now requires SQL analysis, so
its compile fails on P026 instead.
* test(differential): record SQL test mocks in the model's reference order
The native SQL test planner lists a model's mocks in the order the model
reads them; Polyglot listed refs before seeds. The mocks don't read each
other, so the comparison SQL means the same.
* test(semantic): re-record unknown-dialect metadata checks without resource SQL
Projects whose dialect no native parser reads no longer have their
audits' SQL bound, so the relationships and accepted_values findings
Polyglot raised there are gone; SQL test column checks remain.
* build: pin SQLQuack to main 395ced6d
* docs(analysis): keep the resource SQL summary to one line
* test(analysis): build test compile contexts with the adapter's analysis dialect
Compile inputs built without an analysis dialect take the unavailable
route, so these helpers compiled every project without SQL analysis.
Each now passes its adapter's dialect, as the compile pipeline does.
* test(analysis): expect native-route behaviour where tests expected Polyglot's
With every shipped dialect on a native route, a model or hook SQL the
dialect's parser cannot read is skipped with P026 instead of raising a
syntax error, and a dialect without a native parser gets no fast lineage
and compares contract columns by name only. The tests now expect that.
Fast lineage tests pass a native dialect. The truncated T-SQL hook is
attached under T-SQL, so its parser reports it. PostgreSQL star shapes
gain the types PostgreSQL itself gives (INT and TEXT).
Re-recorded: the generated fast-lineage golden (PostgreSQL, which now
folds the unquoted alias Up to up as PostgreSQL does; no-dialect entries
now empty), the generated contract golden (PostgreSQL, Databricks, T-SQL
and no-dialect entries, which now match the native DuckDB findings or
drop comparisons with types foreign to the dialect), one assembly edge
whose model is now skipped so its seed error raises, and two attachment
summaries whose hooks DuckDB itself rejects (GRANT, PIVOT ANY).
* test(sql-macros): expect native-route results in the macro e2e cases
Native DuckDB analysis does not type list literals and reports EXTRACT
as an unknown function, so those cases move to strict xfails with a
date_part control. The BigQuery regexp case now passes a GoogleSQL
argument type.
* fix(sql-macros): name the BigQuery NUMERIC limit for types its parser refuses
BigQuery's parser refuses a NUMERIC wider than it holds, so the limit
check now reads SQLBuild's own spelling of the type before falling back
to the generic unknown-type message.
* docs(skill): regenerate the docs skill references for SQL macros and testing
* wip: keep recorded fingerprints of unchanged definitions across fingerprint scheme changes
Untested beyond compile and the analysis unit tests.
* test: pin SQLite and MotherDuck query fingerprints before the SQLClean re-pin
* fix(executor): treat SQLite columns without type evidence as compatible
SQLite reports a column with no declared type as ANY. An incremental or
snapshot schema comparison now treats that marker as compatible with any
type instead of reporting a type change.
* fix(analysis): know project functions by their bare names on native routes
Native model analysis and type recovery now read the profile's declared
project function names, so a bare call to a project function is no longer
an unknown-function error. Shapes share one case-insensitive lookup.
* fix(lineage): trace fast lineage and review impact in the adapter's dialect
* fix(analysis): place native type findings that carry no span
Type findings without a native span are placed at the expression that
raises them, and SQLBuild's own text checks replace a native row with the
same code and start.
* fix(compile): report SQL syntax errors as P001 on native routes
A gap the native parser marks as a syntax error, in a model query or a
hook, is now a P001 compile error rather than a skipped-analysis warning.
* refactor(sql-macros): share one statement parse for macro bodies and CTE renames
* fix(contracts): compare declared types as each engine stores them
DuckDB and Snowflake contract checks now meet each engine's documented
synonyms and defaults, and DuckDB's runtime comparison treats every string
length as the VARCHAR it stores. On Snowflake, a narrower scale-0 NUMBER
satisfies a wider declared scale-0 NUMBER only where the build casts its
output to the declared types: staged tables and non-microbatch incrementals.
The complexity-guard fixture declares INTEGER, since NUMBER is no DuckDB type.
* test(e2e): expect the native route's explained results
F15 is the syntax error DuckDB reports, U01 the B212 the engine raises, the
typed null and the TIMESTAMP_NTZ help use the native spellings, CHECKPOINT
parses on DuckDB, Snowflake star outputs keep the upstream's spelling, and
the SQL Server delta-compile fixture uses T-SQL spellings.
* wip: re-pin SQLClean 0.6.0 and fingerprint every dialect through SQLQuack
Query fingerprints now come from sqlquack-fingerprint at the pinned SQLQuack
rev, the same function SQLClean 0.6.0 uses. Dialects SQLQuack does not
tokenize (SQL Server, ClickHouse, Databricks, BigQuery) read generic tokens,
so their fingerprints change. Recorded fingerprints whose stored definition
re-fingerprints to the current hash are kept, so unchanged models do not
rebuild; rows recorded without a definition report one query change.
The local copies of the canonical token stream and keyword lists are removed.
The type-finding locator keeps the Polyglot-token projection spans SQLClean
0.6.0 no longer exports.
Release notes: On SQL Server, ClickHouse, Databricks and BigQuery, the web
UI's node status may show unchanged models as changed until they next
rebuild with a definition change or full refresh; plans and runs are
unaffected. On SQL Server and BigQuery, changing only the letter case of
common keywords (41 on SQL Server and 55 on BigQuery, such as JOIN, CASE,
ORDER BY and LIMIT) now counts as a query change, so the first `sqb format`
that upper-cases them rebuilds each reformatted model once. Dialect keyword
lists in the fingerprint will remove this.
Validated: cargo check, sqlbuild-analysis lib tests, clippy on the analysis,
rules and Python crates; fingerprint, planner, executor and microbatch
Python suites (1,866 passed).
* test: keep recorded hashes of unchanged models after a fingerprint scheme change
* fix(rules): wrap Snowflake table functions in TABLE( ) before parsing
The rules engine stubs __table_fn calls before parsing a model. SQLQuack's
Snowflake grammar, like Snowflake, accepts a table function in FROM only
inside TABLE( ), which is how SQLBuild renders the call, so a bare stub made
every built-in Rule report the model unevaluated. Wrap the stub the same way
unless the model already wrote TABLE( ), and assert in the Snowflake
evaluation tests that the model was actually evaluated.
* fix(rules): read Snowflake table function markers inside TABLE( ) in rule trees
Custom Rules parse a model through the rule tree, which renames each
relation marker to a placeholder name first. SQLQuack's Snowflake grammar
refuses the bare table function call that leaves, so a custom Rule on a
Snowflake model with a table function failed to parse it. Wrap those calls
in TABLE( ) over the marker's own padding and a following space, so node
positions keep their line and column, and share the TABLE( ) detection with
the built-in Rules' stubbing.
* fix(rules): keep columns after a wrapped table function in rule trees
When no space or tab follows a Snowflake table function call, the rule tree
inserts its closing parenthesis. Write the TABLE( ) wrapper over one more
padding space in that case, so text after the call keeps its column.
* fix(rules): keep a table function's argument columns when its call ends the line
The TABLE( ) wrapper borrows one more padding space only so text after the
call keeps its column. At the end of a line or of the input the inserted
closing parenthesis shifts nothing, so the extra space only moved the
arguments one column left.
* fix(rules): keep a Snowflake table function call whole when linting and formatting
Snowflake reads a table function call only inside TABLE( ). Lint and
format replaced only the __table_fn(..) reference with a sentinel, so the
parser saw a bare call with arguments and refused the body. Outside
TABLE( ) the sentinel now covers the call's arguments too.
* test: expect a Snowflake bare VALUES CTE to be a syntax error as a strict xfail
Native Snowflake parse errors carry no syntax flag, so the case reports
P026 and rules-unevaluated instead of P001. The compile still fails.
* test: keep the recorded-fingerprint and SQLite schema-change tests in the house test shape
Move module helpers into helpers.py, assert against expected fields and
replace conditionals with lookups, so fensu check passes.
* fix(test): accept a typed null fixture's own column order under a SELECT * closure
The schema-order rule protects the columns completion appends after a
star's leading relation columns. A fixture whose only completion is
casting its bare NULLs appends nothing, so its order cannot shift the
star's columns. Native analysis types more upstream columns than before,
which made DuckShop's complete fixtures with bare NULLs fail inspection.
* fix: lint and format scalar function bodies and quoted hook arguments
Scalar SQL function bodies are expressions, so lint and format now read
them as the one item of a SELECT, as expression macro bodies already
are. Table function bodies stay full queries.
A quoted @'name' argument now stands in as a string sentinel, as audit
parameters already do, so grammar positions that need a string, such
as COMMENT ... IS, parse.
* refactor: bring the native analysis code and tests into the project layout rules
Name the engine type spellings the type comparisons decide on, extract
nested iterator closures into named functions, read an out-of-range
Snowflake precision explicitly, keep the relation-columns lookup with
the catalog helpers, and write the engine-type, bare-function and
baseline-fingerprint tests as named test cases in their test_types
modules.
* refactor: read adapter dialects and project settings without Python calls
* fix(rules): keep a Snowflake table function's arguments after LATERAL
The parser reads LATERAL followed by a call but not by a bare name, so a
table function call after LATERAL keeps its arguments, as one already in
TABLE( ) does. The TABLE( ) check now uses the shared word-bounded
follows_table_wrapper.
* test: check outside the xfail that a Snowflake syntax error still fails compile
Only the P001 code stays a strict xfail; the exit code and the reported
error are asserted by a test that runs.
* fix: frame only scalar function bodies as expressions in lint and format
A function body was read as a SELECT item whenever it did not start with
WITH or SELECT, so a table function body opening with a parenthesis or
VALUES was reformatted as an expression. The header now decides: only a
function whose returns is not a table is framed.
An expression body after a leading comment is dedented from its first
line of code, so the comment no longer leaves the body indented.
* test(duckshop): read the edge-case table function after LATERAL
* test: list the Snowflake syntax-error case beside each test that reads it
* fix(analysis): drop LATERAL before a table function's stub relation
Analysis replaces __table_fn("f")(args) with a bare stub relation, so a
call after LATERAL became LATERAL <name>, which no parser reads. The stub
now replaces the keyword too; LATERAL has no meaning once the arguments
are gone. A lateral inside a comment or string, or a word ending in
lateral, is kept.
* test(duckshop): unit-test the LATERAL table-function edge model
* deps: pin SQLQuack 804864d2 and SQLClean 0.6.1
Every SQLQuack crate moves to 804864d2073ef869cbfe4a9334f7e48b20072326 and
SQLClean to 0.6.1, which is built on the same SQLQuack rev. The SQL Server
bracket-identifier fingerprint golden returns to the value 0.204.0 stores.
BigQuery, Databricks, ClickHouse and SQL Server fingerprints use SQLQuack's
tokenizers. Models whose stored fingerprint row has its query text, which is
every row written by v0.47.0 or later, keep their fingerprints and plan as
before; an older row plans as query_changed once.
* fix(sql-tests): drop LATERAL before a mocked table function
A unit test that mocks a table function swaps the call for the mock's CTE
name. A LATERAL before the call was left in front of that bare relation,
which DuckDB and Snowflake both refuse. The keyword now goes with the call,
as analysis normalisation already does; a rendered call keeps its LATERAL.
Both read the keyword through one shared lateral_start.
* test(semantics): expect PostgreSQL's B218 on a text literal compared with a number
SQLQuack's PostgreSQL profile now refuses the comparison, so the case leaves
the strict xfail of pending dialect semantics and joins the passing cases.
* test(assembly): expect P001 where the parser rejects model and hook SQL
Native routes report a parser's syntax error in a model or hook as a P001
compile error, so assembly no longer reaches a skipped-analysis warning for
these projects. Each engine rejects the SQL too: DuckDB 2.0.0 refuses
`SELECT FROM WHERE (` and `SELECT FROM (`, and SQL Server 2022 refuses
`SELECT IF(region > 1, re` with Msg 156. The truncated T-SQL hook still
makes no Python resource call.
* test(analysis): write each generated project in its profile's dialect
A generated project is compiled under DuckDB, PostgreSQL, Snowflake and
BigQuery, and a syntax error is now a P001 rather than a skipped model, so
its SQL has to be valid in each. The draws are the same for every profile;
only the spelling changes. BigQuery quotes with backticks, writes DISTINCT
after INTERSECT, EXCEPT and UNION, and excludes with `* EXCEPT`; PostgreSQL
names the one kept column instead of `* EXCLUDE`.
Pivots are written only where the dialect has PIVOT: DuckDB's shapes for
DuckDB and MotherDuck, and Snowflake's for Snowflake. The CTE pivot that
read `IN (ANY ORDER BY status)` under DuckDB now lists its values, since
DuckDB 2.0.0 rejects ANY there. Pivot proofs come from those two dialects
only, so the broad cases expect at least 72 rather than 80, and the
analysis-session corpus reads the listed pivot through the session.
* test: re-record the python-oracle goldens on the merged native route
Recorded on this head, then checked item by item against the same tests run
on the cutover before the merge. Only three goldens move with the merge,
each in dialects main's own type and dialect changes reach: types
(MySQL, MotherDuck, PostgreSQL, BigQuery and Databricks spellings),
formerly deferred metadata and semantics (ClickHouse, and the MySQL,
Oracle and Trino rows main's tests now read).
The rest were already stale on the cutover or follow its own changes:
PostgreSQL's new B218, syntax errors reported as P001, and the generated
projects now written in each profile's dialect.
* test(compile): write the header-contract pivot in DuckDB's PIVOT ON form
WIP: the header-contract expected summary is not yet re-recorded.
DuckDB 2.0.0 rejects PIVOT ... IN (ANY ORDER BY ...), which is Snowflake
syntax; the simplified PIVOT ... ON ... USING form is DuckDB's dynamic pivot.
* test(compile): write DuckDB-valid hook SQL and expect P001 for rejected queries
DuckDB 2.0.0 has no GRANT statement and rejects it with a parser error,
so the grant_read hook fixture now records the grant in an audit table
with unnest over the role list, which DuckDB runs. The header-contract
and SQL-hook summaries are re-recorded: the only change is that the
P026 skips for the old pivot and GRANT text are gone and both models
are analysed.
The two immediate-validation cases are SQL DuckDB rejects
("SELECT FROM WHERE (" and a WHERE ending in "AND ("), so they expect
the native route's P001 compile error again, in its parser's wording.
* fix(analysis): spell ClickHouse's BINARY(n) as FixedString(n) on the native route
ClickHouse registers BINARY as an alias of FixedString, and main's type
normaliser now spells BINARY(n) as FixedString(n). The native ClickHouse
alias table still carried Polyglot's Nullable(BINARY(n)), so the two
disagreed after the merge. The parity test now names its oracle, the
shared type normaliser, and records the two spellings ClickHouse refuses
(bare BINARY and a Nullable inside a Nullable), where the shared
normaliser keeps the written text with a parse error.
* test(analysis): key dialect spellings by template and accept an unnamed dialect
The type checker cannot see __name__ through the template alias, so the
per-dialect spellings are keyed by the template functions themselves.
The generator takes the same optional dialect as compile_inputs; a
project without one gets the plain spelling and no pivots.
* chore: settle fensu on the merged head
The pivot generator draws its shapes once and renders them through a
per-dialect table, so the helper has no conditional; every generated
project is unchanged. The ClickHouse alias table's doc comment is one
line again. Two pending budgets take main's lower counts, which the
merge's union of fensu.toml had kept at the cutover's values.
* test(compile): write hook SQL the project's engine reads
DuckDB 2.0.0 rejects GRANT and CREATE FUNCTION ... RETURNS ... LANGUAGE
SQL with a parser error, so the two compile-input cases now record the
grant with an INSERT and create a schema-qualified macro, keeping the
destination interpolation they test, and expect no P026.
The adapter-dialect hook validation case writes T-SQL bracket names, so
it now compiles its inputs as a SQL Server project rather than a DuckDB
one; the native route rejected the brackets with the DuckDB parser
before assembly could validate them as T-SQL.
* fix(format): leave Databricks models as written with a format-unavailable warning
SQLClean returns `unavailable` for a dialect SQLQuack does not format
yet, which is Databricks today. sqb format read only `reason`, so every
Databricks model became a format-unsafe error. It now leaves the file as
written and reports a format-unavailable warning naming the reason, as
the sqlclean CLI does, and the command does not fail on it. Databricks
formatting returns with a later SQLClean release.
* test: expect the re-pinned engines' behaviour and the native route's P001
- A quoted parameter neutralises to a string sentinel, and a trailing
comment after the last clause starts at column 0 (SQLClean 0.5.0 on).
- Databricks identifiers named Any, Some or Order fingerprint without
case at SQLQuack 804864d2; those nine cases are strict xfail until the
next SQLQuack pin.
- ClickHouse and SQL Server models get one non-failing L007 each, as
main's built-in rule facts give every dialect SQL lint cannot parse.
- DuckDB stores VARCHAR(5) and VARCHAR(100) as VARCHAR, so a longer text
macro body meets a VARCHAR(5) contract; the decimal case keeps the
inferred-type check.
- The failure corpus's syntax cases expect P001: DuckDB 2.0.0 rejects
"SELECT order_id,, ..." and "SELECT FROM WHERE" with parser errors.
* chore: settle fensu on the format-unavailable warning and the known fingerprint folds
The warning's message prefix and remediation are constants, chosen
without a four-element tuple. The nine Databricks keyword-named cases
move to their own strict-xfail test, selected by the case helper rather
than by per-case marks.
* build: pin SQLClean 0.6.2
SQLClean 0.6.2 restores CTE macro placeholders that sit on both sides of
a comment, so `sqb format` formats a CTE list whose macros carry
attached comments instead of refusing it. It is still built on SQLQuack
804864d2, so the lockfile keeps one SQLQuack source and only the
sqlclean entry moves.
* perf(compile): parse native gap checks on the compile pool
On a native analysis route each model's query and hook statements are
parsed once in the attachment loop, to decide between P001, P026 and
analysis. That parse ran in turn on the main thread, so a 5,000-model
DuckDB compile spent about 7.5 s attaching inputs where it had spent
1.8 s when DuckDB's syntax check waited for the parallel analysis.
The loop now hands each parse to the model loop's compile pool and
carries on. Before it returns, it settles the results in model order:
the first syntax gap still raises P001 ahead of any error a later model
raised, each P026 is reported as before, and a skipped model's input
record and facts are marked unanalysed. Diagnostics are collected by
key, so their output order does not change.
Reusing the gap parse for analysis would keep every model's tree alive
until the analysis phase, and moving the decision into that phase would
stop checking hooks and unselected models, so the parse stays separate.
On the 5,000-model fresh-process project, attachment drops from 7.5 s
to 2.9 s and the compile from 17.4 s to 13.0 s wall; its output and
semantic fingerprint are unchanged.
* perf(compile): keep native read-whole verdicts in the compile cache
A cached compile still parsed every model's query and hooks to decide
between P001, P026 and analysis, although the analysis itself came from
the cache. On a warm 5,000-model compile that parse kept attachment at
2.8 s, where it had been 1.7 s when DuckDB's syntax check rode in the
cached analysis.
Statements the native parser reads whole are now recorded in a native
store in the compile cache folder, keyed by dialect, SQL and
placeholders under the build's code identity. A model whose statements
are all recorded is not parsed again. Gaps are never recorded, so a
model with one is parsed every time and keeps its P001 or P026, and an
edited model's new SQL misses the store. --no-cache neither reads nor
writes it.
On the 5,000-model project a warm compile takes 6.1 s instead of 7.5 s,
with attachment at 1.7 s; its output is unchanged.
* test(perf): record the native route's fresh-process compile fingerprints
The fresh-process compile and compile cache guards hash each generated
project's compile JSON and compiled artifacts. On the native DuckDB
route two things in them change, and both are correct.
Column lineage gains edges. A metric such as
`CASE WHEN status = 'priority' THEN ...` beside a `status` alias reads
input.status, which is how DuckDB binds an unqualified name in the same
SELECT; Polyglot's lineage left that edge out. Every changed model goes
from 3 + 2k to 3 + 3k edges, and no other compile JSON field changes.
Column lineage does not feed planning: model identity is the query
fingerprint, upstream model keys, config and contract, so these edges
cause no replans.
In 70 compiled SQL test files at 5,000 models the CTEs come in a
different order, with no CTE read before it is defined. The order is the
same on every run: two compiles give byte-identical artifacts, and warm
and edited compiles reproduce each recorded fingerprint.
* perf(contracts): compare each native type pair once per project
On a native route, contract type checks asked the dialect to match the
declared and inferred types of every column, and the dialect parses
both spellings each time. The Polyglot route already normalized each
spelling once per project. A dense 3,000-model DuckDB project spent
220 ms in contracts against 57 ms on the Polyglot route.
The project's type comparer now keeps each native match by declared
type, inferred type and nullability, so a repeated pair is parsed once.
Contracts on that project take 61 ms; findings are unchanged.
* perf(sql-tests): read bounded planning templates from tokens again
SQL-test planning read every model template with the native parser's
strict parse after the Polyglot token fast path was removed, so dense
projects spent their write phase waiting on template parses.
simple_analysis_template restores the bounded recognizer for DuckDB and
Snowflake on SQLQuack's dialect tokens. It reads the template after
markers_as_names, so a re…1 parent 714fa4e commit c642221
397 files changed
Lines changed: 30950 additions & 17931 deletions
File tree
- .github/workflows
- benchmarks/duckshop
- project
- functions/sql/edge_cases
- models/intermediate/edge_cases
- tests/unit/intermediate/edge_cases
- crates
- sqlbuild-analysis
- src
- analysis_route
- _helpers
- analysis
- dialects
- main
- tests
- assembly/analysis_session
- _helpers
- tests
- compiled_project
- _helpers
- main
- compiler
- _helpers/sql_tests
- main
- tests
- relation_markers
- simple_template
- contracts
- _helpers
- main
- tests
- lineage
- main
- tests
- planner_fixtures
- main
- tests
- query_analysis
- _helpers
- main
- relation_markers
- main
- semantic_checks/_helpers/resource_sql
- semantic_validation
- _helpers
- main
- tests
- sql_macros
- _helpers
- parsing
- rendering
- typing
- tests
- sql_test_attachment/tests
- sql_tokens
- _helpers
- main
- reserved_keywords
- tests
- sql_tree
- _helpers
- tests
- type_system
- _helpers
- main
- tests
- warehouse_types/_helpers
- sqlbuild-core/src/compile_timings
- tests
- sqlbuild-python/src/bindings/_helpers
- analysis_session
- analysis
- cache
- change_selection
- compile_attachment
- compile_driver
- compiled_project
- consumption
- contracts
- input_loops
- lineage
- location_command
- native_host
- planner_changes
- planner_entries
- planner_identity
- planner_phases
- planner_records
- planner_strategy
- project_assembly
- render
- review_command
- rules
- semantic_checks
- sql_test_glue
- sqltext
- sqlbuild-rules
- src
- dsl_headers
- main
- lint_text
- _helpers
- main
- tests
- query_facts/_helpers
- sqlbuild-sqltext/src/sql_scan
- _helpers
- main
- tests
- scripts
- compile_performance_ratio/main
- compiler_differential
- _helpers
- analysis_corpus
- corpus
- classes
- delta_compile_series
- main
- src/sqlbuild
- .agents/skills/sqlbuild/references/docs/concepts
- macros
- adapters/sqlite/classes
- adapter
- contract/classes
- type_system/_helpers
- compiler/compile
- executor/run/_helpers
- materializations
- reuse
- tests
- e2e/src/sqlbuild/cli/commands/main
- build
- compile
- fast_lineage
- fixtures/semantic
- sql_macros
- rules
- test
- goldens
- compiler
- example
- failure
- fixture
- seed
- python_oracle
- integration/src/sqlbuild
- cli/commands/main
- compiler
- analysis_session
- compiled_project
- expected
- compile
- contracts
- explicit_references
- lineage
- model_config
- pipeline
- planner/_helpers/resolve
- project_assembly
- semantic_checks
- sql_test_glue
- integrations/dbt
- unit/src/sqlbuild
- adapter/type_system
- cli/commands/main/compile
- compiler
- compile
- _helpers
- fingerprints/main
- lineage
- planner
- _helpers
- changes
- main/identity
- executor/run/_helpers
- lint/_helpers
- website/src/content/docs/docs/concepts
- macros
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
845 | 845 | | |
846 | 846 | | |
847 | 847 | | |
| 848 | + | |
| 849 | + | |
| 850 | + | |
| 851 | + | |
| 852 | + | |
848 | 853 | | |
849 | 854 | | |
850 | 855 | | |
851 | 856 | | |
| 857 | + | |
| 858 | + | |
| 859 | + | |
| 860 | + | |
| 861 | + | |
852 | 862 | | |
853 | 863 | | |
854 | 864 | | |
| |||
857 | 867 | | |
858 | 868 | | |
859 | 869 | | |
860 | | - | |
| 870 | + | |
| 871 | + | |
| 872 | + | |
861 | 873 | | |
862 | 874 | | |
863 | 875 | | |
| |||
922 | 934 | | |
923 | 935 | | |
924 | 936 | | |
| 937 | + | |
| 938 | + | |
925 | 939 | | |
926 | 940 | | |
927 | 941 | | |
928 | 942 | | |
929 | 943 | | |
930 | 944 | | |
| 945 | + | |
| 946 | + | |
931 | 947 | | |
932 | 948 | | |
933 | 949 | | |
| |||
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
174 | 174 | | |
175 | 175 | | |
176 | 176 | | |
177 | | - | |
178 | | - | |
179 | | - | |
180 | | - | |
181 | | - | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
182 | 193 | | |
183 | 194 | | |
184 | 195 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
124 | 124 | | |
125 | 125 | | |
126 | 126 | | |
127 | | - | |
128 | | - | |
| 127 | + | |
| 128 | + | |
129 | 129 | | |
130 | 130 | | |
131 | 131 | | |
| |||
137 | 137 | | |
138 | 138 | | |
139 | 139 | | |
140 | | - | |
| 140 | + | |
141 | 141 | | |
142 | 142 | | |
143 | 143 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
| 8 | + | |
8 | 9 | | |
0 commit comments