Repository navigation
D215: LAG/LEAD negative offsets, POSITION start, SPLIT_PART column delimiter, GENERATOR constants - #98
Closed
kvlonge wants to merge 108 commits into
Closed
D215: LAG/LEAD negative offsets, POSITION start, SPLIT_PART column delimiter, GENERATOR constants#98kvlonge wants to merge 108 commits into
kvlonge wants to merge 108 commits into
Conversation
…nctions D151: per-form translation obligations for the 45 translatable numeric forms (7 arithmetic operators, 38 numeric functions), 291 obligations, each met by one case with inline data under cases/translation: - type_combination 148: the argument types each page documents (integer NUMBER, NUMBER with a scale, FLOAT, DECFLOAT, text where the page converts it), never what a conversion rule claims; - null_argument 65: NULL in each argument; - boundary 57: magnitude, negative, rounding_scale and empty_string; - error_parity 21: documented errors, such as LOG with base 1, a rounding_mode on FLOAT and division by zero. Each obligation records the wrong behaviour its case catches (D153) and the page text it reads. Inputs are asymmetric so a wrong translation differs: truncating vs floor modulo, half-away vs half-even rounding, swapped arguments, integer vs decimal division. No case has a Snowflake result yet.
Answers the auditor's review of a62b1f3 (45 non-pass lines). No obligation id moves; 12 obligations are added at the end of their forms. - Discriminating rows: ROUND ties at 1450 and at scale 2, more than 17 significant digits and 0.29 for CEIL/FLOOR/TRUNC/TRUNCATE on DECFLOAT, a non-perfect cube for CBRT on DECFLOAT, rows where a FLOAT operand or a capped scale changes the value for + and *, and a quadrant IV row for ATAN2. - Negative zero is written as a FLOAT literal (-0e0); DIV0's zero-result case uses FLOAT and DIV0NULL's adds a quotient below the result scale. - Overflow rows really overflow (COSH/SINH at 711), and DEGREES/RADIANS rows overflow only under the wrong evaluation order. - Domain edges, each recording whatever Snowflake returns: ACOS and ASIN outside [-1, 1], ACOSH below 1, ATANH at and past 1, SQRT(-1), COT(0), POW of a negative base with a fractional exponent, and MOD and % by zero. - Notes and catches corrected where they did not match their rows.
ATANH t004 adds 1e-17, where a naive log formula gives 0; CEIL t004 adds 0.28, which FLOAT rounds up at scale 2; the zero-divisor notes on MOD and % say they move to boundary if Snowflake returns a value.
The case loader accepts only lower-case identifiers, so the hyphenated ids were rejected. Obligation ids are unchanged; only case ids and the obligation case refs move from t001-slug to t001_slug.
97 string forms, 541 obligations, one case each: type combinations from each page's argument text, NULL in each argument, boundaries (empty strings, negative and out-of-range positions, unsigned 64-bit digests) and error parity. Inputs are chosen so the named wrong behaviour gives a different answer: non-ASCII text where bytes and characters differ, asymmetric argument pairs, NULL beside a set row.
Snowflake rejects ATAN2 of a BOOLEAN (1044) and ROUND with EXPR => but no SCALE => (944) at compile time, so the recorder needs compile_error set to record them.
49 date and time forms, 353 obligations, one case each, in the utc-v1 profile. Boundaries cover leap days (including 1900 and 2000), ISO-week year ends, month-end clamping, pre-1970 epochs and DST gaps and overlaps through explicit zones and TIMESTAMP_TZ offsets. Inputs are chosen so UTC-instant versus wall-clock and elapsed versus boundary-count readings give different answers.
Resolves the review of 28714ad: discriminating rows for the weak obligations (alternation under implicit anchoring, unanchored LIKE ANY and ALL, regexp positions and groups, padding lengths, lower-case hex, binary lengths), corrected notes and catches, literal SHA2 digest sizes, and seven new obligations for the form gaps (rtrim collation compares, non-Snappy decompression, start positions 0 and -1, IPv6 SEARCH_IP).
29 conditional forms, 124 obligations, one case each. Inputs separate three-valued NULL logic from NULL-skipping and NULL-as-false readings, the first argument's type from numeric comparison in GREATEST and LEAST, DECODE's NULL match from CASE's, and first-true from last-true.
Resolves the review of 1f5392c: week parts tested on a Sunday wall clock whose UTC instant is Monday of the next ISO week, a Sunday row for dayofweek, DST rows that cross the change, the mm/mi/ms part aliases, slice lengths that do not divide a day, leap rows that can move the part, and corrected catches. TIMESTAMP_TZ results whose point is the offset also return their wall clock, since the comparator compares instants. Adds obligations for two-argument CONVERT_TIMEZONE, TIMESTAMP_NTZ_FROM_PARTS of a date and a time, and TO_TIMESTAMP of strings.
DECFLOAT results are cast to VARCHAR because the recorder cannot decode a DECFLOAT column. The out-of-domain ACOS, ASIN, ACOSH and ATANH rows and unary plus of an empty string become error parity, since Snowflake raises an error on every row. ABS and ATAN2 drop their negative-zero claims: Snowflake returns +0 and +pi there.
Batch 3: TIME_SLICE uses 11-hour and 11-minute slices, which do not align with the day on its date; week and quarter notes describe the new rows; catches on the from-parts carry, YEAROFWEEK leap rows and two-argument CONVERT_TIMEZONE are corrected. Batch 4: deciding zero rows for BOOLOR and BOOLXOR, COALESCE/IFNULL/NVL result types compared directly, typed NUMBER(10,2) branches, a NOT IN row, corrected catches, and two new obligations: VARCHAR against numeric BETWEEN bounds, and a searched CASE with a guarded division and no ELSE.
Snowflake requires the BASE64 alphabet and line length and the INITCAP delimiters to be constants. Each row of those cases is now its own call with the literal inlined, in a single input row. Also fixes the BOOLXOR NULL-argument notes.
681 obligations over 143 aggregate and window-aggregate forms (D151): per form the documented type combinations, NULL in each argument, empty input, DISTINCT on duplicates, and for window forms PARTITION BY, an empty OVER, tied ORDER BY keys under the implied RANGE frame, ASC NULLS FIRST with a ROWS frame and DESC. Sample against population statistics use groups of 8, 2 and 1 rows; regression functions use asymmetric (y, x) pairs with a NULL on each side. Sketch functions (HLL, DataSketches, t-digest, top-k, MinHash, bitmaps) are read through their estimates on inputs small enough to be exact. AGG over a semantic view has no obligations here: it needs a semantic view object.
collation t001 compared COLLATION of an uncollated value, which comes back NULL through the connector where Snowflake declares the column NOT NULL, so the recorder refused it; it now compares two explicit collations. The div0 t006 note records that Snowflake returns +0 for DIV0(-5, 0), which the obligation already names as the right answer.
Snowflake refused 14 of the 482 cases at compile time. Six error-parity cases now declare compile_error. The quoted part in the DATE_PART and EXTRACT FROM forms is a parse error (1003), so those two become compile-error parity cases. TO_TIMESTAMP's scale is a literal per call, as Snowflake requires a constant. Under the utc-v1 timestamp mapping, TIMESTAMP_FROM_PARTS with a time zone resolves to the NTZ variant and fails (1044), so its three zone cases record that refusal. IN and NOT IN against a subquery holding NULL are unsupported in the select list (2031), so that case filters with them instead.
Every non-pass line of the e615217 audit, applied by obligation number: rows that now separate the named mistakes (tied MODE partitions given unique modes, rounding rows that split truncation from half-even, NULL partition keys outside the star columns, two-partition HLL and MinHash windows), ARRAY_AGG windows without WITHIN GROUP under ORDER BY, neutral wording where the pages contradict themselves, and corrected notes. New: MEDIAN and MODE DISTINCT parity, VARIANCE window DISTINCT with ORDER BY, empty input for HLL_ACCUMULATE and DATASKETCHES_HLL_ACCUMULATE, OVER () and DISTINCT for the ARRAY_AGG window, and a two-partition MINHASH_COMBINE window. sf.functions.agg joins the snowflake-object-kinds not-translatable group with the semantic-view DDL: AGG reads a metric of a semantic view, which a case cannot create.
Wording from the 65e3f47 audit: the PERCENTILE_DISC target names 0.75, the BITXOR_AGG NULL catches name the NULL-only group (0 is the XOR identity), the BOOLOR_AGG window note describes its partitions, and the DATASKETCHES_HLL_ACCUMULATE empty-input catch no longer assumes the result. The snowflake-object-kinds sign-off moves to the group's new SHA-256 with AGG.
- HLL_ESTIMATE, HLL_EXPORT and HLL_IMPORT window forms: Snowflake refuses the documented OVER form at compile, so their cases expect the compilation error. - MINHASH with DISTINCT is rejected at parse although the page says it has no effect; VARIANCE(DISTINCT) is refused as unsupported; a literal NULL TO_TIMESTAMP scale is refused. Each now expects the error. - BOOLAND_AGG, BOOLOR_AGG and BOOLXOR_AGG refuse FLOAT input: the small non-zero value moves to NUMBER(10,1), and a new FLOAT case expects the error. - TIME results with sub-microsecond digits are cast to VARCHAR, since the connector returns TIME in microseconds.
CAST(TIME AS VARCHAR) follows the session TIME_OUTPUT_FORMAT, which has no fraction, so the TIME cases use TO_VARCHAR with HH24:MI:SS.FF9. TO_TIMESTAMP with a NULL scale runs alone, so the refusal belongs to it. Wording: BITAND_AGG text input states the recorded error, and the HLL window refusals name a result rather than estimates.
31 array forms with 118 obligations: construction, membership, bag-semantics set operations, positions and slices, sorting, and the FILTER, TRANSFORM and REDUCE lambdas. Arrays enter as JSON text; a SQL NULL element is built with ARRAY_CONSTRUCT where it must differ from a JSON null, and results in unspecified order are sorted or measured by size.
ARRAY_MIN and ARRAY_MAX number and text rows now put the answer where neither the first or last element nor a text comparison gives it. The JSON null, ARRAY_SIZE object and ARRAYS_OVERLAP NULL rows state what the pages document.
AS_*, IS_*, TYPEOF, STRIP_NULL_VALUE, TO_ARRAY, TO_OBJECT and CHECK_JSON: values of each type beside text that looks like them, typed values cast to VARIANT, and SQL NULL against JSON null.
AVG, COUNT, HASH, NULLIF, TRY_TO_TIMESTAMP, DATE_PART ... FROM, EXTRACT ... FROM and MINHASH each have a recorded case that the converter does not yet reproduce, or that the parser does not read. Their cases and obligations land once the converter is fixed.
Their GROUPING SETS cases produce a construct key with a space, which the link file cannot name yet.
Casts to DECFLOAT, IPADDRESS, PERIOD and VECTOR, ROLLUP and CUBE, the COLLATE operator, NTH_VALUE with its FROM and NULLS clauses, and unary plus each link to their catalogue form.
The per-form obligation cases measure 94 of 1,182 translatable forms as
matching (8.0%); main's baseline measures 142 of 1,103 (12.9%). Form
transitions against main's baseline:
- 410 unmeasured to refused
- 81 not translatable to unmeasured (the forms returned to the denominator)
- 50 matching to refused (listed below)
- 5 syntax only to refused
- 2 syntax only to matching (hex-decode-string, length.len)
- 2 unmeasured to not translatable (agg, md5-number)
The 50 regressions are refusals the stronger obligations expose: each
form now has a case outside the converter's proven type conditions, a
missing rule, or a known divergence the policy refuses. None is a
converter change. They are, exactly:
- sf.expressions.data-types-datetime.interval.interval-constants
- sf.functions.avg.avg-over
- sf.functions.between
- sf.functions.bitand
- sf.functions.bitor
- sf.functions.bitxor
- sf.functions.booland-agg.booland-agg
- sf.functions.boolor-agg.boolor-agg
- sf.functions.case.end.syntax
- sf.functions.current-timestamp.current-timestamp
- sf.functions.dayname
- sf.functions.exp
- sf.functions.generator.generator
- sf.functions.generator.generator-rowcount
- sf.functions.greatest
- sf.functions.hex-encode
- sf.functions.hour-minute-second.hour
- sf.functions.hour-minute-second.minute
- sf.functions.hour-minute-second.second
- sf.functions.lag
- sf.functions.lead
- sf.functions.least
- sf.functions.ntile
- sf.functions.object-keys
- sf.functions.parse-url
- sf.functions.percentile-disc.by-over
- sf.functions.position.position
- sf.functions.ratio-to-report
- sf.functions.regexp
- sf.functions.regexp-substr
- sf.functions.regexp-substr-all
- sf.functions.rlike.rlike.syntax
- sf.functions.seq1.seq4
- sf.functions.seq1.seq8
- sf.functions.split-part
- sf.functions.timeadd
- sf.functions.timediff
- sf.functions.timestampadd
- sf.functions.timestampdiff
- sf.functions.to-boolean
- sf.functions.to-date.date
- sf.functions.to-time.time
- sf.functions.to-time.to-time
- sf.functions.trunc.truncate
- sf.functions.try-base64-decode-binary
- sf.functions.try-to-double
- sf.functions.year.day
- sf.functions.year.month
- sf.functions.year.quarter
- sf.functions.year.year
To check that the gate's REGRESSION lines against main are exactly this
list:
git show origin/main:coverage/translation/results-baseline.json > base.json
syntax-coverage results target/translation-statements.json --against base.json 2>&1 \
| grep -o 'against the target: REGRESSION form [^:]*' | awk '{print $NF}' | sort
The output equals the list above, line for line, and no statement
regresses.
Snowflake reads LAG(x, -n) as LEAD(x, n) and LEAD(x, -n) as LAG(x, n). The converter refused any offset but a bare integer literal; a negated integer literal now swaps the function. With IGNORE NULLS a negative offset stays refused, since no recording shows it. The recorded slice-1 cases lag.toml#t003_negative_offset and lead.toml#t003_negative_offset now replay as matching.
The converter refused a third argument, citing a recording of Snowflake rejecting it; slice 1 records Snowflake accepting POSITION(s, t, start). The start is a whole number, read as 1 below 1, and NULL gives NULL; the result counts from the text's start. The recorded cases position.position.toml#t002_start_pos, t007_null_start_pos, t008_start_past_end and t009_start_zero_negative now replay as matching.
SPLIT_PART converted only with a literal delimiter. A delimiter from a column now converts too: an empty delimiter leaves the text as its one part (parts 0, 1 and -1 give the text, others empty), as Snowflake documents, and a NULL argument gives NULL. The recorded cases split-part.toml#t001_varchar, t004_null_delimiter and t006_empty_parts now replay as matching.
ROWCOUNT may be a constant integer expression (+, -, *, parentheses, negation). A NULL or absent ROWCOUNT/TIMELIMIT yields 0 rows; ROWCOUNT is kept beside a TIMELIMIT of a whole second or more. TIMELIMIT alone and a negative ROWCOUNT (Snowflake compile error 1046) stay refused with specific reasons. generator.generator t001/t003 and generator.generator-rowcount t001/t003 now replay as matching; no other GENERATOR case changes.
kvlonge
commented
Nov 6, 2026
kvlonge
left a comment
kvlonge
left a comment
Contributor
Author
There was a problem hiding this comment.
Converter review from the backend lane (read-only: diff and call flow, plus one DuckDB probe).
Needs changes
-
The rule catalogue still describes the old limits.
crates/sqlquack-rules/src/catalogue/constants.rsis the published rule contract, and four entries are now wrong:- GENERATOR (around line 1014) says "TIMELIMIT and a ROWCOUNT other than a literal are not converted".
- REPLACE/SPLIT_PART (around line 1048) says "SPLIT_PART takes a literal delimiter".
- CHARINDEX/POSITION (around line 1160) says "without a start position".
- WINDOW_OFFSET (around line 1297) says nothing about a negative offset turning LAG into LEAD.
Please update each entry's conditions in this PR.
-
position_fromevaluates its operands more than once.{found}appears twice, so the haystack and needle are each evaluated twice, and{start}appears three times. A volatile or costly argument, such as RANDOM(), UUID_STRING() or a scalar subquery, could give different values in different branches. Please bind the operands once, with thelist_transform([struct_pack(...)], lambda s: ...)[1]pattern this PR already uses insplit_part_by_column. -
Empty needle with a start past the end. DuckDB returns
strpos(substr('abc', 10), '') = 1, so the emitted SQL givesPOSITION('', 'abc', 10)= 10, and also returns the start for an empty haystack. Is there a receipt for an empty needle with a start beyond the text? If not, refuse that case or record it. The catalogue already pins "1 for an empty s" for the two-argument form.
Please confirm
- GENERATOR keeps ROWCOUNT beside a TIMELIMIT of 1s or more. That is timing-dependent in principle. Which receipt shows n rows at the 100,000 maximum with a TIMELIMIT?
- Please add verification evidence to the PR body:
make verify-recordedon the stacked head with 0 failures, and the translation-measure diff, listing which forms move to matching and confirming nothing regressed.
The LAG/LEAD flip, SPLIT_PART NULL and empty-delimiter handling, and constant_integer overflow checks look right to me.
The start-position form evaluated its operands up to three times; they are now bound once in a struct_pack/list_transform lambda, as SPLIT_PART by column does. Same 8 recorded cases pass.
Re-recorded with syntax-coverage results --write-baseline against land/d151-s1: 0 regressions, 24 changes (charindex, lag, lead, position.position and split-part forms refused -> matching).
…de the recorded range
…P_SUBSTR column arguments Snowflake results for the 12 cases in cases/translation/functions/position.empty-needle-start.toml and regexp-substr.column-arguments.toml at 5476cd2.
A's receipts (position.empty-needle-start t001/t002) show an empty needle matching at the start position only up to the text's length; a start of 0 or below reads as 1.
position.empty-needle-start (2, matching) and regexp-substr.column-arguments (10, refused until the computed REGEXP_SUBSTR arguments land). 0 regressions, 36 changes against land/d151-s1.
kvlonge
commented
Nov 6, 2026
kvlonge
left a comment
kvlonge
left a comment
Contributor
Author
There was a problem hiding this comment.
Follow-up review of e140c0f (only the earlier findings and the code they touch).
Resolved: the catalogue conditions (05cd56c), operands bound once (0e33af3), the empty needle past the end now backed by receipts, the ROWCOUNT-beside-TIMELIMIT receipt, and the verification in the body.
One new correctness issue in position_from: a NULL text with a start above 1 returns 0 instead of NULL. DuckDB's greatest(NULL, 1) ignores the NULL, so greatest(length(s['h']), 1) is 1 and f > 1 is true, which takes the THEN 0 branch. DuckDB probe on the emitted expression:
POSITION('a', NULL, 3)→ 0 (expected NULL)POSITION('', NULL, 3)→ 0 (expected NULL)POSITION(NULL, 'abc', 3)→ NULL (correct)
Smallest fix: add WHEN s['h'] IS NULL OR s['n'] IS NULL THEN NULL before the existing branches, or guard the p check the same way. A case with NULL text and a start of 2 or more would pin it.
DuckDB's greatest(NULL, 1) is 1, so a NULL text with a start above 1 reached the past-the-end branch and gave 0. Guard both operands, as the start already was; the recorded POSITION/CHARINDEX cases are unchanged.
Contributor
Author
|
Re-checked 4ce4490: position_from now returns NULL when the text, needle or start is NULL, and the two new cases cover NULL text with a start past 1. This resolves my remaining finding. Approved for the gate once slice 1 lands. |
This was referenced Nov 6, 2026
Contributor
Author
|
Closing: SQLQuack lands through local-gate trains, not PR merges (D94), so open PRs only add clutter (D628/D630). The branch is kept. Any work here that isn't on main yet is tracked by the train owner and lands through a train. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
D215 converter fixes, part 1 (lane B groups). Stacked on land/d151-s1 (73359ae): this diff includes slice 1 until slice 1 lands on main. After that, it is merged up to main (not rebased).
Each change counts only through recorded Snowflake cases that replay as matching on DuckDB.
24 * 7), and NULL or absent arguments give 0 rows. ROWCOUNT is kept beside a TIMELIMIT of 1 s or more. TIMELIMIT alone and a negative ROWCOUNT stay refused, with specific reasons.Evidence
ROWCOUNT => 5, TIMELIMIT => 60, recorded 5 rows). A TIMELIMIT under 1 s beside ROWCOUNT has no receipt and stays refused.--require-verified --strict-errors, at 05cd56c with the same converter as ca57db4):11594 cases: 7654 passed, 1031 diverged as expected, 2792 refused, 36 typed, 81 pending, 0 unproven, 0 failed; 220 of 220 equivalences and rules verified; error checks:385 agree, 113 divergence raises, 0 findings.0 regressions, 24 other changes. Forms charindex, lag, lead, position.position and split-part go refused → matching: 94 → 99 of 1182 translatable forms. 19 statements go refused → matching. The GENERATOR forms stay refused through other cases, such as TIMELIMIT alone and the negative-count compile error.Receipts added (A, evidence/d215-receipts 2df6cb0, cherry-picked as 7134598, 0fb6475, 74f379f)
POSITION start: an empty needle past the end of the text gives 0) matches this, and both cases pass.11606 cases: 7656 passed, 1031 diverged as expected, 2802 refused, 36 typed, 81 pending, 0 unproven, 0 failed; 220 of 220 equivalences and rules verified. Measure against land/d151-s1:0 regressions, 36 other changes, the 24 above plus the 12 new statements, with the baseline re-recorded in e140c0f.Still refused, with no change here: position t003_binary; GREATEST/LEAST first-type (a declared
text_number_scaledivergence); BITAND/BITOR/BITXOR with casts or binary arguments; REGEXP_SUBSTR with non-literal arguments.