Skip to content

Commit 97667a1

Browse files
committed
Add the native DuckDB 2.0.0 parser and link it into the syntax coverage checker
sqlquack-syntax-duckdb scans and parses DuckDB 2.0.0 SQL into the shared tree, keeping the transformer's refusals with DuckDB's messages and every statement it does not build as a named gap. It bounds nesting depth, type depth, flat AND/OR/set-operation chains (1,000 links, until the shared tree's derived impls are safe on them) and the finished tree's depth (8,000 levels), and it is clean at every depth of the deep-nesting probe. The shared tree gains DuckDB-only fields, each last and skipped when empty: catalog parts on a table name, qualified star modifiers, CTE key items, an aggregate's qualified name and quoting, EXPORT_STATE, a method call's DISTINCT and ORDER BY, and a macro's further overloads. The syntax coverage checker can now use DuckDB as its local parser, and the depth probe includes it.
1 parent fe5b932 commit 97667a1

109 files changed

Lines changed: 20076 additions & 26 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎Cargo.lock‎

Lines changed: 10 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ members = [
2020
"crates/sqlquack-fingerprint",
2121
"crates/sqlquack-format",
2222
"crates/sqlquack-format-check",
23+
"crates/sqlquack-syntax-duckdb",
2324
"crates/sqlquack-python",
2425
]
2526
exclude = ["fuzz"]
@@ -76,6 +77,7 @@ sqlquack-syntax = { path = "crates/sqlquack-syntax" }
7677
sqlquack-syntax-derive = { path = "crates/sqlquack-syntax-derive" }
7778
sqlquack-syntax-postgres = { path = "crates/sqlquack-syntax-postgres" }
7879
sqlquack-syntax-bigquery = { path = "crates/sqlquack-syntax-bigquery" }
80+
sqlquack-syntax-duckdb = { path = "crates/sqlquack-syntax-duckdb" }
7981
stacker = "0.1"
8082
syn = { version = "2", features = ["full"] }
8183
tempfile = "3"

‎THIRD_PARTY_NOTICES‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,9 @@ fixtures at v30.14.0 (tests/fixtures/identity.sql, tests/test_parser.py and test
1616
as extracted by polyglot's fixture tooling. polyglot, below, is itself inspired by sqlglot and
1717
ships sqlglot's licence. corpora/polyglot-check/sqlglot_postgres.json holds the SQL of
1818
tests/dialects/test_postgres.py at the same version, and corpora/polyglot-check/sqlglot_sqlite.json
19-
that of tests/dialects/test_sqlite.py.
19+
that of tests/dialects/test_sqlite.py. corpora/duckdb-core/extras.jsonl repeats the DuckDB and
20+
generic statements of corpora/polyglot-check/sqlglot.json and polyglot_tests.json with DuckDB's
21+
verdict on each.
2022

2123
The referee corpus referee/corpora/sqlglot.json holds the query statements passed to
2224
`validate_identity` and `validate_all` in tests/dialects/test_snowflake.py at the sqlglot commit

‎corpora/duckdb-core/README.md‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
# DuckDB core corpus (WIP)
2+
3+
**Work in progress, not yet usable as a regression test.** This will be the acceptance bar for
4+
SQLQuack's strict DuckDB parser: SQLBuild's DuckShop benchmark project ported to native DuckDB
5+
2.0.0 SQL, plus common extras, with each statement tagged with the DuckDB syntax-catalogue forms it
6+
uses (`coverage/syntax-duckdb/catalogue/*.toml`). It follows `corpora/databricks-core`.
7+
8+
## Current state
9+
10+
`duckshop.jsonl` holds 877 statements: 413 models, 452 unit tests, 10 functions and 2 singular
11+
audits of SQLBuild <https://github.com/sqlstack-io/sqlbuild> at commit
12+
`cf07c4016ea4b8595dcc6d2d9e5dcfbb21021403`, project `benchmarks/duckshop/project` (written in
13+
Snowflake SQL). They were rendered offline with `sqb compile` from `sqlbuild` 0.192.0, with `ref`,
14+
`source` and `seed` resolved to `rel__<name>` relations as in `corpora/databricks-core`, and
15+
incremental cursor bounds fixed to `TIMESTAMP '2026-01-01 00:00:00'` and
16+
`TIMESTAMP '2026-01-02 00:00:00'`. Each was then ported with sqlglot 30.30.0 (Snowflake to DuckDB) plus a rewrite pass:
17+
`LATERAL FLATTEN` becomes `json_each`/`json_tree` over `to_json(...)` (`.index` read from
18+
`key`, `OUTER => TRUE` as `LEFT JOIN LATERAL ... ON TRUE`), `ASOF JOIN ... MATCH_CONDITION (c)
19+
ON k` becomes `ASOF JOIN ... ON (c) AND k`, `OBJECT_INSERT` becomes `json_merge_patch` with
20+
`json_object`, untyped `ARRAY`, `OBJECT` and `VARIANT` casts become `JSON`, the `TO_TIMESTAMP_*`
21+
and `IS_<type>` functions, `OBJECT_AGG`, `ARRAY_SIZE` and `RATIO_TO_REPORT` get DuckDB
22+
equivalents, and the functions are hand-ported to DuckDB macros (`CREATE MACRO`, table macros
23+
for the table functions).
24+
25+
Validation runs every statement on DuckDB 2.0.0: sources and seeds as empty tables of their
26+
declared types, functions as macros, models as views (DuckDB binds a view when it is created),
27+
then every unit test and audit executed. At this commit 824 of 877 bind and run. "Runs" means the
28+
statement executed; a unit test's rows are not yet compared with its expected rows.
29+
30+
47 statements carry `excluded` with the reason: Snowflake behaviour DuckDB has no equivalent for,
31+
each excluded with the unit tests of the model that uses it. They are HASH_AGG, the XML functions,
32+
CONNECT BY (the only two statements DuckDB cannot parse), `GET_IGNORE_CASE`, per-row `TO_DATE`
33+
formats, the object-editing functions, `FLATTEN`'s `PATH` column, `SYSTEM$TYPEOF` and `TYPEOF`'s
34+
type names, Snowflake's week parameters, blank-filled number formats, per-value time-zone offsets,
35+
positional column references with derived column names, and regular expressions RE2 refuses.
36+
`rpt_edge_case_model_keys` reads seven excluded models, so the model is excluded but its unit test,
37+
which supplies them as fixtures, is kept.
38+
39+
6 unit tests carry `port_unfinished` with DuckDB's error: three whose fixture lacks a column the
40+
model reads (the rendered Snowflake fixture lacks it too), two whose expected timestamps fall
41+
outside DuckDB's nanosecond range, and one that casts Snowflake's boolean words in a JSON string.
42+
43+
## Extras
44+
45+
`extras.jsonl` holds the 4,133 distinct statements of `corpora/polyglot-check/sqlglot.json` and
46+
`corpora/polyglot-check/polyglot_tests.json` written for DuckDB or for both dialects (sqlglot
47+
v30.14.0's fixtures, polyglot's tests at `7d5fe2c`), each with its `origin` (corpus file, entry
48+
index and the test it came from) and DuckDB 2.0.0's verdict from `duckdb.extract_statements`:
49+
`accepted`, or `refused` with the first line of DuckDB's message. 2,473 are accepted and 1,660
50+
refused. Duplicate statements keep their first entry.
51+
52+
Still to do: the scenarios and hooks, and form ids.

‎corpora/duckdb-core/duckshop.jsonl‎

Lines changed: 877 additions & 0 deletions
Large diffs are not rendered by default.

0 commit comments

Comments
 (0)