Skip to content

feat(planner): prove migrations and diffs from SQLQuack parse trees - #1363

Closed
kvlonge wants to merge 5 commits into
wip/analysis-route-clickhousefrom
feat/planner-proofs-sqlquack
Closed

kvlonge wants to merge 5 commits into
wip/analysis-route-clickhousefrom
feat/planner-proofs-sqlquack

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Draft, stacked on the ClickHouse analysis-route arm; it moves onto the analysis route's follow-up branch once that exists.

Why

The planner's proofs parse SQL with Polyglot directly. These are migration shapes and rename fingerprints, raw diff queries, relation names, fixture star keys, Snowflake's metadata-only CREATE, and cursor types. Each proof has to choose its parser through analysis_route, as model analysis does. Unreadable SQL gives no proof and the conservative plan.

Changes

  • NativeDialect gains parse. read_whole becomes a provided method over it, which refuses gaps, more than one statement, and a lone Command.
  • Each proof matches on analysis_route(dialect). A native route reads SQLQuack's tree, and the legacy route keeps the Polyglot code unchanged. A comment has the per-proof fallback table.
  • A star key whose qualifier names several relations in the query takes all of them.
  • SQLQuack is repinned to a ClickHouse parser that reads TRUNCATE, OPTIMIZE, CREATE TEMPORARY TABLE, ALTER … DELETE/UPDATE and INTO OUTFILE, for relation names in Python-node SQL.

Release note:

  • A ClickHouse model renamed in the same change as this upgrade isn't matched against fingerprints stored earlier. It's planned as a new model.
  • In ClickHouse Python-node SQL, some forms still give no relation names, so their name check is skipped as for unreadable SQL: SETTINGS, FORMAT or INTO OUTFILE after ALTER, TRUNCATE or OPTIMIZE; IN PARTITION; DEDUPLICATE BY; and TRUNCATE of a database, of TABLES FROM, or of a temporary table. The full list is in a comment.
  • OPTIMIZE and ALTER … UPDATE read as commands naming no relation, as with Polyglot, so their names are not checked.

Verification

  • Before/after on 716 ClickHouse bodies, 137 types and 168 Python-node statements, with every difference in a comment.
  • cargo test -p sqlbuild-analysis, clippy -D warnings, fensu and the Python planner and relation-guard suites.

Migration query shapes and fingerprints, raw diff query shapes, relation
names, fixture star keys and Snowflake's metadata-only CREATE now parse a
native dialect's SQL with SQLQuack, and keep Polyglot only on the legacy
route. SQL the native parser cannot read whole gets no proof, so the
planner takes its conservative path. Cursor types parse unknown warehouse
types with SQLQuack's parser.

NativeDialect gains parse, and read_whole becomes a provided method over
it, as the analysis route's follow-up defines them.

kvlonge commented Nov 8, 2026

Copy link
Copy Markdown
Contributor Author

Fallback per proof

Proof Native route SQL the native parser can't read
Migration query shape (query_shape) SQLQuack's tree, with a set operation's leading WITH moved from its first SELECT onto the set operation, where ClickHouse's parser does not put it no shape, so no column migration proof
Migration fingerprint the canonical tree, encoded as sorted JSON with a sqlquack-tree: prefix (SQLQuack has no SQL generator) no fingerprint, so no automatic rename
Raw diff query shape root kind, node kinds and write or lock clauses from SQLQuack's tree the query is refused as not one valid statement
Relation names tables, CTE names and temporary tables from every statement None, the existing "cannot parse" answer; empty SQL still gives no relations
Fixture star keys projected stars in each SELECT: a qualified star takes its relation, and a bare star takes every relation in the query every relation given
Snowflake metadata-only CREATE the same CREATE rules over SQLQuack's tree not metadata-only, so billed as warehouse work
Cursor type of an unknown warehouse type read as a cast's type by SQLQuack's parser (DuckDB grammar) unclassified, with the existing warning

Differences from the Polyglot path

These come from a before/after run over the 716 compiled ClickHouse model bodies in SQLBuild's e2e projects and seed corpus, plus 137 warehouse type spellings. "Before" is this branch's base with every proof on Polyglot; "after" is this branch.

  • Migration query shapes: identical on all 716 (projection names, references, passthrough, positional use, input relations, local columns and clause references).
  • Relation names: identical on all 716.
  • Fixture star keys: identical on all 215 bodies with a star.
  • Cursor types: identical on all 137 spellings.
  • Raw diff query shape:
    • Root kind and write or lock clauses are identical on all 716.
    • The read-only verdict is identical: the node kinds differ only in function nodes (for example a generic function instead of upper), and no mutating kind appears or disappears.
    • Refusal messages for empty SQL differ in wording only (30 bodies): the clickhouse parser cannot read Empty query instead of Expected 1 statement, found 0.
  • Migration fingerprints: the same 30 bodies get none.
    • The native grouping refines Polyglot's: no two bodies Polyglot fingerprinted differently now match.
    • 3 Polyglot groups (30 models) split, because Polyglot renders VARCHAR(19) and VARCHAR(23) as the same ClickHouse type and the native tree keeps the length. So a native match is always a pair Polyglot also called equal.

…native

# Conflicts:
#	crates/sqlbuild-analysis/src/analysis_route/constants.rs
…r aliases

Repin SQLQuack to the ClickHouse parser that reads TRUNCATE, OPTIMIZE,
CREATE TEMPORARY TABLE, ALTER ... DELETE/UPDATE and INTO OUTFILE, so the
P008 check and the runtime relation guard read the relation names
Polyglot read for them. Forms the parser still gaps keep the existing
cannot-parse answer.

A star qualifier that names several relations in one query, such as an
alias reused across UNION branches or CTEs, now selects every relation
it names, keeping star keys a superset of what each star reads.

The native star-key walk moves into a helper, the Polyglot parse in the
migration proofs returns to its original lines, the unused Polyglot type
argument reader is removed, and the tests take fensu's case form.

kvlonge commented Nov 8, 2026

Copy link
Copy Markdown
Contributor Author

Python-node SQL: relation names before and after

The scheduler review found that ClickHouse Python-node SQL the native parser gapped gave no relation names, so the P008 check and the runtime relation guard skipped it. The ClickHouse parser now reads those statements, and this branch pins it.

The run calls extract_relation_names(sql, "clickhouse") on this branch's base, where Polyglot reads ClickHouse, and on this branch. The P008 check and the runtime guard both read its result. Each statement's verdict comes from ClickHouse 26.1 (clickhouse format), run twice with identical results.

Statements ClickHouse accepts Same names Polyglot read, native gives none Other
Python-node SQL in tests and fixtures (75) 70 70 0 0
ditto, refused by ClickHouse 5 1 4 0
The five forms' spellings (93) 67 45 20 2
ditto, refused by ClickHouse 26 2 24 0

The 4 Python-node statements ClickHouse refuses are DuckDB's CREATE TEMP TABLE (twice), a DELETE with no WHERE, and a deliberately broken statement.

Accepted forms that still give no names (named in the release note; their check is skipped, as for unreadable SQL):

  • SETTINGS, FORMAT or INTO OUTFILE after ALTER, TRUNCATE or OPTIMIZE (7 spellings), such as ALTER TABLE t DELETE WHERE … SETTINGS mutations_sync = 2;
  • IN PARTITION in ALTER … DELETE or UPDATE (2);
  • OPTIMIZE … DEDUPLICATE BY (2);
  • TRUNCATE of a database, of TABLES FROM or ALL TABLES FROM, or of a temporary table, and TRUNCATE … NO DELAY or IF EMPTY (6);
  • CREATE TEMPORARY VIEW (1);
  • INTO OUTFILE after a set operation or a bracketed query (2).

Reading the first group needs a settings field on the shared AlterTable and Truncate nodes, which would be a follow-up.

The 2 other differences:

  • ALTER TABLE orders DELETE WHERE id IN (SELECT id FROM analytics.customers) now also names analytics.customers. Polyglot skipped the subquery.
  • CREATE TEMPORARY TABLE scratch (no columns), which ClickHouse accepts, now names scratch as a temporary table. Polyglot couldn't read it.

OPTIMIZE and an ALTER with an UPDATE name no relation on either route, because both parsers read them as commands.

Other fixes since round 1

  • A star key whose qualifier names several relations in the query, such as t in SELECT t.* FROM orders t UNION ALL SELECT t.* FROM customers t, now takes all of them.
  • The Polyglot parse in proofs/parse.rs is back on its original lines, which clears the fensu dupes failure.
  • The native star-key walk moved into planner_fixtures/_helpers/native_star_keys.rs, the new tests take fensu's test_cases form, and type_system::main::polyglot_type_args is removed: its only caller was the cursor-type code this PR moved to SQLQuack. fensu now passes with its Rust rules evaluated.
  • A raw diff query with INTO OUTFILE now reads as writing, so it is refused as before, now for writing rather than for a gap.

kvlonge commented Nov 8, 2026

Copy link
Copy Markdown
Contributor Author

Repin to the reviewed parser head

SQLQuack is now pinned to 7e5e1b1d. Its ClickHouse parser also refuses a plain three-part table in FROM or JOIN (SELECT * FROM a.b.c), as ClickHouse 26.1 does with a syntax error. Three-part table functions and three-part names in ARRAY JOIN or IN are unchanged.

Python-node relation names at this head, against the base:

  • Python-node statements ClickHouse accepts (70): all identical.
  • Python-node statements ClickHouse refuses (5): all now give no names. One more than at dd735a0, because SELECT count(*) FROM memory.main.customers, a DuckDB fixture statement, is now refused as ClickHouse refuses it.
  • The five forms' spellings: unchanged from the earlier comment.

The release note now also names OPTIMIZE and ALTER … UPDATE, which read as commands naming no relation on both routes.

cargo test -p sqlbuild-analysis: 309 passed.

kvlonge commented Nov 8, 2026

Copy link
Copy Markdown
Contributor Author

Closed because its base, wip/analysis-route-clickhouse, is a ClickHouse arm superseded by wip/native-cutover's arm. The branch is kept; a fresh PR will follow once it is rebased onto the cutover's arm.

@kvlonge kvlonge closed this Nov 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant