Repository navigation
test(corpus): realistic queries, dbt-style models and generated basics keep their outcomes - #135
Merged
Merged
Conversation
…s keep their outcomes Three corpora under corpora/, each checked under the built-in policy and the repository policy in a UTC session, so a statement that once translated cannot silently start refusing, and a change in what refuses shows in review: - everyday: 615 hand-written analyst queries, 537 of them valid in Snowflake (receipts in snowflake-receipts.json). - everyday-models: 499 dbt-style models, compiled dbt package output and incremental, snapshot and MERGE shapes. - basics: 1832 generated statements a first-time user types (literals in every spelling, columns, arithmetic, casts, common functions, aggregates, GROUP BY, ORDER BY, LIMIT, CTEs, joins, UNION, windows), each also spelled with other case, whitespace, a final semicolon, comments and a quoted table name. scripts/generate_basics_corpus.py writes it and records each outcome with a given sqlquack binary. Outcomes and rates are recorded on main 78910c3.
kvlonge
added a commit
that referenced
this pull request
Nov 6, 2026
…command The everyday, models and basics corpora now fail only when a statement that translated refuses, a refusal changes code or span, or a translated statement's result on the basics fixture rows changes. A refused statement that now translates is reported without failing. `make corpus-rerecord` rewrites the recorded outcomes and results, keeping each file's comments, and the failure message names it. The basics generator now writes statements with unrecorded outcomes; `make corpus-rerecord` records them with the converter. The receipts' schema descriptions name the corpus schema files.
kvlonge
added a commit
that referenced
this pull request
Nov 6, 2026
Only a refusal that now translates, or an unrecorded outcome, is reported without failing; a query recorded with a parse or check diagnostic that starts translating fails, since Snowflake may reject it. The corpus headers describe the asymmetric test and make corpus-rerecord.
kvlonge
added a commit
that referenced
this pull request
Nov 6, 2026
The basics corpus gains `WHERE ordered_at >= DATEADD(day, -30, CURRENT_DATE)` and the same over CURRENT_TIMESTAMP(), in their spellings. Both refuse with SQ0003 today and should translate under the built-in default once the date comparison and TIMESTAMP_LTZ rules land. The everyday corpora record main's clock and semi-structured guard changes: 29 more queries translate under the built-in default, and 20 refusals change code.
kvlonge
added a commit
that referenced
this pull request
Nov 7, 2026
A comment straight after SELECT no longer refuses the statement: 387 outcomes now translate, and 189 refusals move from SQ0002 node:select to the construct the statement actually stops at. Eight unaliased literals followed by a trailing comment, such as `select 1 -- trailing comment`, now refuse with SQ0009, as the fix refuses an unaliased item whose text holds a comment.
Contributor
Author
|
D94 gate passed on the exact head 99e4ab3: #135 at f1944aa (auditor record db54f0b), merged with main c2cbe9d, then
GitHub Actions can't run because of the account billing block, so this local gate is the check for this landing. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
D150/D320 regression test. Three corpora, each checked by
analysis::tests::test_everyday_corpusunder the built-in policy and the repository policy (UTC session).Changes since 521b883 (the approved head, auditor record 822f907), in b2da27d and 0076e8e:
snowflake-receipts.jsonfiles now describe the schema neutrally, naming the corpus schema files. The verdicts are unchanged.corpora/basics/fixture.sql(sorted-row hash, or the Snowflake errno it raises).refused …that now translates, or one recorded asunrecorded. A recorded parse or check diagnostic that disappears fails, because Snowflake may reject that SQL (auditor record 976fb43).make corpus-rerecordrewrites every recorded outcome and result and keeps the files' comments. The failure message names this command, so a PR that moves an outcome on purpose runs it and commits the corpora in the same commit.unrecordedoutcome, andmake corpus-rerecordfills it in. Its Python outcome recorder is removed, so the Rust test is the only recorder.No converter code changes.
Since 0076e8e (record b64ede2), in a9e5755 and f1944aa:
SELECT order_id FROM orders WHERE ordered_at >= DATEADD(day, -30, CURRENT_DATE)and the same overCURRENT_TIMESTAMP(), as 10 entries with their spellings. Both are pinned as refused under both policies:SQ0003 node:gteandSQ0003 fn:DATEADD. They should flip to translated once fix: compare naive timestamps with CURRENT_TIMESTAMP() and count sub-day DATEDIFF of a DATE #115, C's group 1 and D326 land, and each of those PRs re-derives withmake corpus-rerecord. No other basics entry changed.At the slot (247ca79, 99e4ab3): main c2cbe9d merged in, then
make corpus-rerecord. Only basics moves (D327): 387 refused → translated, 189 refusals change code, and 8 unaliased literals with a trailing comment (select 1 -- trailing commentand similar) go translated → refused with SQ0009, as D327 refuses them. D315 is expected to convert those again. No fixture result changed.Corpora:
Recorded translation rates on fc15d51, built-in / repository: everyday 376 / 459 of 537, models 293 / 404 of 458, basics 1,095 / 1,160 of 1,842. These pin SQLQuack's own outcomes. They are not Snowflake-verified support figures: only 48 of the 1,114 hand-written queries have receipts, and none of the basics do.
Checks:
make corpus-rerecordreproduces the committed basics file byte for byte, and a second re-record changes no corpus.parse SQ0001and tocheck SQ0004on translating statements both fail.-D warnings, cargo fmt, ruff, ruff format and fensu are clean, and the hygiene scan passes.The full D94 gate runs when the slot comes: merge main,
make corpus-rerecord, then gate that exact head.