Skip to content

Tokenizer and depth fixes, iterative tree drop, account and property quoting, interval qualifier precision, anonymous procedure gaps - #63

Merged
kvlonge merged 8 commits into
mainfrom
fix/tokenizer-depth
Nov 5, 2026
Merged

kvlonge merged 8 commits into
mainfrom
fix/tokenizer-depth

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Parser batch after gaps-f (#61): the tokenizer and depth fixes, three auditor withholds, and the anonymous-procedure part of the D133 remainder.

Tokenizer and depth (product lane)

  • An unterminated /* comment is refused with SQ0001 "Unterminated block comment" and its span.
  • The AST depth guard counts a flat left-associative operator chain as one level. A 50,000-term + chain parses, and dropping it doesn't recurse. 600 nested unary minus signs still give E_GUARD_AST_DEPTH_EXCEEDED.

Stack overflow when dropping chained trees (fuzz, 45e21bd)

  • A 70k-term chain of ::, [1], .a, COLLATE, IS NULL or IN, or one mixing them with =/+, overflowed the stack while its tree was dropped, even after the depth guard had refused it. The AstNode derive now generates take_expressions, and BinaryOp, Cast, CollationExpr, DotAccess, In, IsNull, IsTrueFalse, JsonExtractFunc, LikeOp, Subscript, UnaryOp, Alias and Between drop through one shared worklist. This replaces the BinaryOp-only worklist.
  • Regression test on a 2 MB thread at 70k terms: + and || parse; ::, =, [1], .a and =/IS NULL are refused with E_GUARD_AST_DEPTH_EXCEEDED; a :: chain cut off by a syntax error is refused. Without the fix the test process aborts with a stack overflow.

Auditor withholds

  • ACCOUNTS lists (release channels, release directives, replication and failover) keep [ org . ] account as a table reference, with each part quoted separately.
  • ALTER SET/UNSET property kinds upper-case only unquoted names, so ALTER ORGANIZATION ACCOUNT UNSET "v1 Q" keeps v1 Q.
  • INTERVAL literal qualifiers with a field precision become one qualifier node holding the fields and the leading and fractional seconds precisions (faithful Snowflake modes only). YEAR(9) TO MONTH no longer becomes a call plus a variable, and SECOND(9), (3) no longer splits into two select items.

D133 (utility)
Snowflake's WITH … AS PROCEDURE grammar is looser than the parser allowed. Per the receipts it accepts:

  • parameters and RETURNS TABLE columns given by name alone;
  • names as RUNTIME_VERSION, HANDLER, IMPORTS and PACKAGES values;
  • an expression CTE body after the procedure.
    Quoted strings and reserved words stay refused. This clears 24 of 27 settled-accept cases. The other 3 have an invalid LANGUAGE SQL AS 'm' body, which Snowflake reaches only after its parameter error.

Each fix has a regression test that fails without it. The gate report follows in a comment.

…as one depth level

- The tokenizer refuses an unterminated /* at its opening, as it does an
  unterminated string; the refusal is the parse diagnostic SQ0001 with its span.
- The AST depth guard counts a left-associative chain such as 1 + 1 + ... as one
  level, so a 50,000-term sum parses; real nesting (600 unary minus signs) is
  still refused with E_GUARD_AST_DEPTH_EXCEEDED.
- Dropping a long operator chain frees it with a worklist, so it cannot overflow
  the stack.
ACCOUNTS lists in release channels, release directives and replication
or failover now keep [ organization . ] account as a table reference with
each part quoted separately, instead of one joined identifier.

ALTER SET and UNSET property kinds upper-case only unquoted names, so a
quoted name such as "v1 Q" keeps its case.
Receipts show Snowflake parses a parameter or RETURNS TABLE column given
by its name alone, a name as a RUNTIME_VERSION, HANDLER, IMPORTS or
PACKAGES value, and an expression CTE body after an anonymous procedure,
as WITH p AS PROCEDURE ... , c AS ( 1 ) CALL p ( ). A lone part now has
an unknown type; quoted strings and reserved words stay rejected.
INTERVAL '1' YEAR(9) TO MONTH parsed as a YEAR call spanning a MONTH
variable, and SECOND(9), (3) left the fractional precision as a second
select item. In Snowflake's faithful modes a qualifier with a field
precision is now an interval qualifier node with its fields and its
leading and fractional seconds precisions. Qualifiers without a
precision keep their span or simple shape.
A 70,000-term chain of ::, [1], .a, COLLATE, IS NULL or IN, or one
mixing them with = or +, overflowed the stack while its tree was dropped,
even after the depth guard had refused it. The AstNode derive now
generates take_expressions, which moves a node's direct expression
children out. BinaryOp, Cast, CollationExpr, DotAccess, In, IsNull,
IsTrueFalse, JsonExtractFunc, LikeOp, Subscript, UnaryOp, Alias and
Between drop through one worklist built on it, so any node between two
chain links is freed iteratively too. This replaces the BinaryOp-only
worklist.
@kvlonge kvlonge changed the title Tokenizer and depth fixes, account and property quoting, interval qualifier precision, anonymous procedure gaps Tokenizer and depth fixes, iterative tree drop, account and property quoting, interval qualifier precision, anonymous procedure gaps Nov 5, 2026
@kvlonge
kvlonge merged commit f28341e into main Nov 5, 2026

kvlonge commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

Gate (D94) and landing record.

  • Command: make check-merge RECORDER=0 on the exact head f28341e6c37687759cad688d5556b31d98dc00dc, one build job. Exit 0.
  • The head contains main d10844175 (feat: DuckShop batch 6 (projection EXCLUDE, SYSTEM$TYPEOF stripped in a later query) #64), merged in. Two earlier gate runs on this branch were stopped before they finished: on 26de8c4 for the fuzz drop crash, and on 52d15aa because feat: DuckShop batch 6 (projection EXCLUDE, SYSTEM$TYPEOF stripped in a later query) #64 landed.
  • Landed by fast-forward: main = f28341e6. main tree = head tree = f098b1d7005128810072dc544ac1564789f76464.
  • Coverage diff against the committed baseline in the gate: no regression, DOCUMENTED or CATALOGUE lines; 0 forms lost, 0 obligations lost.
  • Measured against main (parser at f3c75ab, unchanged by feat: DuckShop batch 6 (projection EXCLUDE, SYSTEM$TYPEOF stripped in a later query) #64) on catalogue e914d92 with evidence 21e794d: +43 obligations, 0 lost, 0 forms gained or lost. Receipted Snowflake-parsed cases now rejected: 0. Snowflake syntax errors now accepted: 0. Documented (D113) refused cases now accepted: 0.
  • The 43 gained obligations: 40 in sf.utility.call-with.* (anonymous procedures), 2 for the long + chain (operators-arithmetic.add.o007, .o008) and 1 in functions.concat.concat-subscript-dot. The quoting and interval fixes change trees only and gain no obligations until review.
  • Fuzz drop crash: 12 chain kinds at 70k terms on a 2 MB thread (+, ||, ::, =, [1], .a, COLLATE, IS NULL, IN, : path, =/IS NULL, and a :: chain cut off by a syntax error) now parse or are refused cleanly. Without the fix the test aborts with a stack overflow. Also checked by hand at 300k terms (::, [1], =/IS NULL, IN, +).
  • Stale reviews: the gate printed none. Reviews that bind the parser tree go stale on any parser commit until the auditor renews them in D118. The INTERVAL qualifier, ACCOUNTS and UNSET withholds need re-review there.

kvlonge added a commit that referenced this pull request Nov 5, 2026
Approves 8c9639e (functions and query regenerated on import 155) and
records d118-21: 69 forms read, 58 approved, 10 findings, 1 withhold.
2,248 of 3,348 forms verified on evidence 7223ab4.
kvlonge added a commit that referenced this pull request Nov 5, 2026
…hold

2 receipts from run ivl-q: the CAST form is syntax error 1003; the control
executes as two select items.
@kvlonge
kvlonge deleted the fix/tokenizer-depth branch November 9, 2026 21:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant