Repository navigation
Conversation
Add sqlquack-format, a whitespace-only formatter over SQLQuack's own tokenizer in SQLClean's documented layout: clauses on their own lines, one select item per line, CTEs separated by a blank line, joins with ON below, CASE and brackets broken only when they do not fit in 100 columns, four-space indent. Token text is never changed; output that would not keep the input's tokens is returned unchanged. - sqlquack convert --pretty lays out the DuckDB SQL; the default output is byte-identical. - The playground engine returns the laid-out SQL, which the playground shows with a Copy button. - A test lays out every distinct converted case statement and checks its tokens and DuckDB parse tree are unchanged. - scripts/measure_format_conformance.py measures agreement with an installed sqlclean format; a measurement, not a gate.
Add corpora/everyday: 514 neutral, synthetic queries over a shop schema (literals, SELECT *, joins, CTEs, aggregates, windows and QUALIFY, dates, IFF/CASE/COALESCE, strings, semi-structured paths and FLATTEN, dbt-style models, DDL and DML, typos and invalid SQL). Each records what SQLQuack does with it under the built-in default policy: its parse or check diagnostic, its first refusal, or translated. Queries whose Snowflake validity is not yet recorded are marked. A test runs each through check_source and convert_report, the paths the command line and Python use, and fails naming every query whose outcome changed.
The browser engine has no DuckDB bind, so SQL with a column missing from GROUP BY, or an aggregate in WHERE, showed stage 2 failing and stage 3 translating, where the command line refuses it. Translation is now skipped when the checks report an error; warnings do not stop it.
The playground now starts on the repository's sqlquack.toml with a UTC session, and stage 3's header names the policy in force. A one-click switch turns to strict, the built-in policy that refuses every known divergence, so refusing rather than guessing stays easy to show; ?policy=strict opens the page that way. The everyday corpus records each query's outcome under both policies (expect and expect_repository), and its test checks both and the translation rate of each: 285 of 434 valid queries strict, 354 of 434 under the repository policy.
…in validity Each query ran once over the corpus schema as empty CTEs: EXPLAIN first, then the compiling ones executed for their result columns.
The 25 queries Snowflake compiled or rejected carry its verdict in a snowflake field, which decides whether they count as valid SQL in the translation rates: 287 of 436 strict and 356 of 436 repository. The test also lists the 8 queries Snowflake rejects that pass SQLQuack's parse and checks, so closing each gap updates the list.
…aders (D150)
- check takes --schema. With one, a relation the schema does not declare is
SQ0004 at its name, and a call to a function that is neither a documented
Snowflake function (from the syntax catalogue) nor a schema function is
the new SQ0205 unknown_function. Without a schema, names stay unchecked.
The playground's checks report the same as E200/E202 findings.
- An unresolved double-quoted name gets help: use single quotes for a string
literal: '<value>'.
- SQ0001 says what was expected and what was found in SQL's words, points
an unclosed ( at where it opens, points a failure at the end of the input
past the last token, and adds help to render dbt/Jinja templates when the
SQL holds {{ }} or {% %}.
The everyday corpus records the three outcomes this changes: FROM order and
FROM orderz are now check SQ0004, NOSUCHFUNCTION is check SQ0205.
Snowflake rejects `status == 'shipped'` (2139) but the tokenizer read `==` as equality in every dialect, so SQLQuack translated it. Snowflake now tokenizes it as two `=`, which fails to parse with SQ0001 "unexpected `=`" and the help "Snowflake compares with a single `=`". Other dialects keep `==`. The everyday corpus query moves from translated to parse SQ0001.
…171)
Snowflake rejects `status IN 'shipped'`, `total >> 10` and `1 << 2` with
syntax error 1003. In Snowflake's faithful modes IN and NOT IN now need `(`
("expected `(`, found `'shipped'`"), and >> and << are no shift operators
("unexpected `>>`"). `&`, `|` and `^` still parse: Snowflake parses them and
fails at compile (2139, 2). Scripting's FOR ... IN is parsed elsewhere and
is unchanged: none of the 56,748 receipted statements changes verdict. The
== cases move to the faithful-mode tests with the new ones, and the two
everyday corpus queries move to parse SQ0001.
Snowflake rejects `COUNT(*) FILTER (WHERE ...)` with syntax error 1003
at the `(`. In Snowflake's faithful modes no call takes a FILTER clause
any more, so the error points there ("unexpected `(`"); the FILTER
function over an array is unchanged, and no receipted statement changes
verdict. The everyday corpus query moves to parse SQ0001.
…med (D171) Three queries Snowflake rejects without a schema now fail `check` instead of reaching translation, as structural Snowflake rejections that also stop the playground: - SQ0206 duplicate_alias: two FROM items or joins under one alias (Snowflake 2027), compared as Snowflake stores names, at the second alias. - SQ0207 too_many_arguments: a call passing more arguments than every catalogue signature allows (939). snowflake_function_arities.txt lists the 772 functions whose call grammars all have a fixed number of plain arguments; a test derives it from the catalogue. Calls with named arguments, and too few arguments, are not checked: receipts show two documented minimums (ARRAYS_ZIP, FORMAT_STRING_TEMPLATE) are not enforced. - SQ0208 pivot_consumed_column: reading the column a lone PIVOT aggregates or pivots on (904), outside SELECT-list aliases and nested queries. None of the three fires on any of the 24,472 parsed receipted statements or the 4,243 statements of the translation cases. Four everyday corpus queries move to check SQ0206, SQ0207 and SQ0208; ordered_at > 5 still needs a schema-typed check and stays a translation refusal.
The examples test read public/playground/sqlquack.toml, which only the site build copies there, and CI tests the engine before it builds the site. It now reads sqlquack.toml at the repository root, as the other tests in the file do.
Contributor
Author
|
Closing: SQLQuack lands through local-gate trains, not PR merges (D94), so open PRs only add clutter (D628/D630). The branch is kept. Any work here that isn't on main yet is tracked by the train owner and lands through a train. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
D150 diagnostics, plus D171's parse and check fixes (D177): SQL that Snowflake rejects now fails
sqlquack checkinstead of reaching translation, and parse failures are worded for readers.Stacking. This branch sits on the corpus series (
wip/formatter-corpus-series4070812), which lands first. Until it does, this PR's diff includes the series' commits. Each of the five commits below stands on its own and is meant to be fast-forwarded as is.Commits
check --schema: a relation the schema doesn't declare is SQ0004. A call to a function that is neither a documented Snowflake function nor a schema function is the new SQ0205 unknown_function (Snowflake 2140). Without a schema, names stay unchecked.(at where it opens, points end-of-input failures past the last token, and adds help for dbt/Jinja templates.==is a parse error in Snowflake SQL (Snowflake 2139), with the help line "Snowflake compares with a single=". Other dialects keep==.IN 'x',>>and<<are parse errors (Snowflake 1003). Scripting'sFOR … INis unchanged.&,|and^still parse, because Snowflake parses them and fails at compile (2139 and 2, import 163).FILTER (WHERE …)clause is a parse error (Snowflake 1003, at the(). TheFILTERfunction over an array is unchanged.duplicate_aliastoo_many_argumentssnowflake_function_arities.txtlists the 772 functions whose call grammars all have a fixed number of plain arguments; a test derives it from the catalogue. Calls with named arguments, and calls with too few arguments, are not checked: receipts show two documented minimums (ARRAYS_ZIP, FORMAT_STRING_TEMPLATE) aren't enforced.pivot_consumed_columnEvidence that nothing valid is refused
==,IN,>>,<<, FILTER): the parse verdict changes for none of the 56,748 receipted statements (through import 161). Receipts exercise these paths: 3,981 statements have a bareIN, mostly scriptingFOR … IN, and 11 have->>.ZIPF ( 1 , 2 , RANDOM ( ) )compiles; ZIPF is excluded because its page grammar has an empty slot.Everyday corpus
7 of the 8 queries that Snowflake rejects but SQLQuack used to pass now fail at parse or check. Their
expectlines are re-derived, and their entries are removed fromexpected_rejections_not_caught.WHERE status == 'shipped'WHERE status IN 'shipped'WHERE total >> 10COUNT(*) FILTER (WHERE …)orders o, orders oLENGTH(order_id, 2)totalFROM order,FROM orderzNOSUCHFUNCTION(order_id)WHERE ordered_at > 5(1038) stays a translation refusal: catching it at check needs typed checking over the schema. LIMIT read as a table alias waits on Snowflake receipts and will follow in its own change.Checks run
-D warnings.website/scripts/generate_diagnostics.py, which regeneratedreference.mdx.