Repository navigation
Conversation
Every recursive path now passes through the parser's one depth counter: FROM items, function and JSON_OBJECT keys read outside the operator climb, statements inside BEGIN ATOMIC and EXPLAIN, and type names, whose [] bounds count against the same limit. Each level runs on a stack that grows on demand, so the 300-level limit, not the thread's stack size, decides. Once one reading passes the limit, every other reading fails at once and the statement is refused with 54001. Flat chains of operators other than AND and OR, casts and indirections are refused past 1,000 links. A parenthesized query and JSON_OBJECT's arguments are parsed once per start token and replayed, so nested IN subqueries and json_object calls no longer backtrack exponentially.
…ack overflow A data type nested past 512 levels is refused with the bracket-nesting guard (SQ0103). Every level counts toward one depth: [] and [n] suffixes, ARRAY<...>, and ARRAY(, OBJECT(, MAP( and STRUCT( levels, so alternating short runs of each can no longer nest without bound. Suffix runs are checked as they are built, so 100,000 [] suffixes stop at level 513. Serializing a binary operator's operands and a data type's nested types moves to a fresh stack segment when the stack runs low, so a flat chain of 100,000 + terms and a 512-level type serialize on a 2 MiB thread. The JSON is unchanged. Dropping such chains was already iterative. The guard tests now serialize every tree they parse, and cover these spellings at 512, 513, 20,000 and 100,000 levels and a 100,000-term + chain on a 2 MiB thread.
The faithful-call cache clones each parsed call, and the derived Clone of a binary operator recursed once per level, so a few thousand + terms inside a function call overflowed a 2 MiB stack. Cloning an operator now moves to a fresh stack segment when the stack runs low, the same way its serialization does. The 2 MiB chain test gains 50,000-term + chains as the argument of f, SUM, COALESCE and ARRAY_CONSTRUCT.
The derived Clone, PartialEq and Debug of Expression each keep a frame of several kilobytes per level in an unoptimised build, so on a 2 MiB thread equality overflowed at 150 nested CASE, UNION ALL, parenthesis, NOT or array levels, Debug at 300 and Clone at 500, all inside the 512-level AST depth guard, and on long operator chains. A new StackSafe derive generates the same three impls for Expression with each level run through on_grown_stack. Every recursive path passes through Expression, so this covers chains and nesting alike, and BinaryOp goes back to its derived impls. The test runs clone, equality, Debug, serialization and drop on a 2 MiB thread over 100,000-term chains and nesting just inside each depth guard.
One shared function for the SQL change fingerprints that SQLBuild and SQLClean both compute, so their hashes agree by construction rather than by test. canonical_tokens reads the significant tokens of SQL with the lossless tokenizer (generic, DuckDB, Snowflake, SQLite) or the native PostgreSQL scanner, and upper-cases unquoted reserved keywords and called names outside name positions. A word after '.', ':', '@', '$' or AS, a '__' placeholder call, the relation of an INTO or TABLE column list and a CTE name before its column list keep their case. query_fingerprint is the SHA-256 of those tokens, each framed as '<byte length>:<text>'. is_reserved_keyword exposes the per-dialect keyword lists. The digests match SQLBuild's recorded fingerprints, and every query of an 8,617-model corpus fingerprints identically on this revision and on 4d0c6bc.
Chains nested in another chain's first operand escaped the per-chain and per-level counts, so a statement could build a tree tens of thousands of levels deep, which aborts recursive serialisation. Each statement's tree depth is now checked after parsing and refused past 8,000 levels with the stack depth error.
Expression's Serialize was still serde's derive, and only BinaryOp operands and DataType moved to a fresh stack segment. Any other nesting, such as casts, CASE or parentheses, overflowed a 2 MiB debug thread at a few hundred levels once a parser accepts it. The StackSafe derive now also generates Serialize, in serde's externally tagged snake_case form, with each level run through on_grown_stack. It refuses to compile for other serde attributes. Expression drops serde's Serialize derive, and BinaryOp drops its per-field serialize_with. The derive moves into the ast_node domain next to AstNode. The serialized JSON and Debug output of 63,375 corpus parses are byte-identical to before. The test adds 8,000-level casts, CASE and parentheses, parsed with raised guards, to the trees it clones, compares, formats, serializes and drops on a 2 MiB thread.
…nes them on a grown stack A remembered parenthesised query or JSON_OBJECT argument list was stored and replayed with a recursive clone, which overflowed the stack on a deep tree, such as a long UNION ALL chain in parentheses, before the statement tree depth was checked. A remembered tree past the depth limit is now refused before it is kept, and every stored or replayed tree is cloned on a stack grown to its depth.
syntax-coverage depth-probe nests 27 constructs (FROM items, subqueries, parenthesised queries and expressions, CASE, function arguments, casts, subscripts, type suffixes and constructors, and flat chains) at depths from 50 to 100,000, and parses each through the isolated worker on a 2 MiB thread with a 15 s bound. It fails on any crash, panic or timeout coverage/deep-nesting.toml does not list at its depth, and on any listed entry that no longer fails. Each entry names its parser, construct, build profile, depth and owning crate. check-ci runs it on the gate build; make depth-probe-debug runs it on a debug build, whose larger frames fail at shallower depths.
…array types A probe thread that ends without a result no longer drops its construct silently: the run reports a problem unless every construct gave one. A new construct nests CAST( levels, each to INT with a run of [], so the combined type depth is probed in every parser, including Postgres.
The depth probe now links the BigQuery parser and nests chains of casts, additions, OR, UNION ALL and pipe operators inside the first operand of another chain, 60 links per parenthesised level, plus one flat UNION ALL chain inside a FROM subquery. A per-chain link count misses these trees. With the shared tree cloned, compared and serialised on a grown stack, the Snowflake parser has no known failure left, in either profile. The Postgres and BigQuery entries are their parsers' own recursion, which their depth limits remove when they land.
…rees in its memo A parenthesized query's successful parse is no longer copied into the memo, so nested queries no longer hold a copy of their subtree at every level. Keeping a JSON_OBJECT argument tree drops the kept trees nested inside it, so kept trees never overlap, and their total nodes are bounded per statement.
A BigQuery pipe chain that ends in a syntax error is dropped on the error path before any depth guard runs, and PipeOperator's derived drop recursed once per step, so 20,000 steps aborted a 2 MiB debug thread. drop_iteratively! now covers PipeOperator and the other nodes a parser wraps around its own result in a loop: AT TIME ZONE, ** (BinaryFunc), XOR, the Snowflake ->> flow operator, model attributes, .: casts, array slices and method calls. Paren and Annotated stay derived: the parser moves out of them, the token guard bounds parentheses, and an annotation wraps one operand once. The BigQuery parser's with_ctes moved the input out of a PipeOperator, which a Drop impl forbids; it now walks to the chain's first input and replaces it in place, without recursion. A test builds a 100,000-step chain of each newly covered kind and drops it on a 2 MiB thread; it aborts without this change.
The BigQuery parser wraps the relation in a new TableSample or MatchRecognize once per repeated clause after FROM, through a helper constructor, so FROM t followed by 20,000 TABLESAMPLE clauses built a chain whose derived drop aborted a 2 MiB debug thread, including on the syntax error path. Both now drop iteratively, and the 100,000-step drop test covers them; it aborts without this change.
…Snowflake layout The formatter lays out SQL over SQLQuack's lossless tokens and parse tree, copying only whitespace and keyword case, and checks token identity and idempotency before it returns. It is gated on SQLGlot's formatting fixtures (MIT, see FORMAT_CORPUS_NOTICE.md) for Snowflake.
Adds DuckDB to the formatter, gated on SQLGlot's DuckDB formatting fixtures, and sqlquack-format-check, which compares the result columns DuckDB reports for the original and formatted SQL.
…an does Called names are upper-cased like keywords, except SQLBuild placeholders, quoted names, relations and CTEs with column lists, DuckDB macro names, and Snowflake select items without an alias, whose text names the column. INSERT targets stay on the first line, angle-bracket type parameters stay touching, and struct field names, identifiers and binary operators are found in linear time.
Add the sqlquack-lint crate: SQLClean's lint rules, ported from its Polyglot engine onto SQLQuack's tokens and parse trees, so SQLClean and SQLBuild lint without Polyglot. Snowflake and DuckDB lint natively; dialects SQLQuack does not parse report lint as unavailable instead of failing. sqlquack-syntax gains two public entry points the engine needs: dialect_tokens, the token stream of a dialect, and parse_expressions, which reads already guarded tokens as expressions. The rule walkers ported from SQLClean keep its MIT notice in THIRD_PARTY_NOTICES.
Contributor
Author
|
Closing: SQLQuack lands through local-gate trains, not PR merges (D94), so open PRs only add clutter (D628/D630). The branch is kept. Any work here that isn't on main yet is tracked by the train owner and lands through a train. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Postgres parser could abort the process with a stack overflow on deeply nested input, for example
SELECT * FROMfollowed by 1,500 parentheses, 200 nested subqueries on a 2 MiB thread, or 100,000[]suffixes on a type. Some recursive paths skipped the depth counter, and the counter alone could not protect a small thread stack. This change makes every such input end in a parse error instead.What changes
c_exprwhen read outside the operator climb (such asJSON_OBJECTkeys), statements insideBEGIN ATOMICandEXPLAIN, and type names now pass throughdeeper. A type's[]bounds count against the same 300-level limit.stacker::maybe_growbehind a defaultstackerfeature, the same pattern assqlquack-syntax. The depth limit decides, not the caller's thread size.ANDandOR, casts and indirections are refused past 1,000 links.AND,ORand set operations aren't counted.JSON_OBJECT's arguments are parsed once per start token and the outcome is replayed. Before, 25 unclosedIN ((SELECTlevels took 17 s, and nestedjson_object(calls hung.Evidence
test_depth.rs: 39 cases run on a 256 KiB debug thread. They cover FROM parentheses, joins, derived tables and LATERAL; scalar, EXISTS and IN subqueries; parenthesized and WITH queries; parentheses, calls, CASE, ARRAY, unary minus, NOT and subscripts;JSON_OBJECTkeys; type modifiers;[]and[3]suffixes in casts and column types; casts nesting array types 60 bounds deep (the combined type depth);BEGIN ATOMIC; flat chains; and the backtracking cases. Run against the unfixed parser, the same file aborts with a stack overflow.cargo test, clippy-D warnings, fmt andfensu checkpass for the crate. Onlysqlquack-syntax-postgresand itsCargo.lockentry change, so Snowflake output can't move.