Skip to content

test(corpus): realistic queries, dbt-style models and generated basics keep their outcomes - #135

Merged
kvlonge merged 7 commits into
mainfrom
land/basics-regression
Nov 7, 2026
Merged

kvlonge merged 7 commits into
mainfrom
land/basics-regression

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

D150/D320 regression test. Three corpora, each checked by analysis::tests::test_everyday_corpus under the built-in policy and the repository policy (UTC session).

Changes since 521b883 (the approved head, auditor record 822f907), in b2da27d and 0076e8e:

  1. The provenance strings in both snowflake-receipts.json files now describe the schema neutrally, naming the corpus schema files. The verdicts are unchanged.
  2. D334, the coordinator's ruling, makes the test asymmetric:
    • It fails on regressions: a statement that translated now refuses, a refusal changes its code or span, a check or parse diagnostic changes, a crash, or a translated basics statement returns a different result on corpora/basics/fixture.sql (sorted-row hash, or the Snowflake errno it raises).
    • It reports improvements without failing: only a statement recorded as refused … that now translates, or one recorded as unrecorded. A recorded parse or check diagnostic that disappears fails, because Snowflake may reject that SQL (auditor record 976fb43).
    • make corpus-rerecord rewrites every recorded outcome and result and keeps the files' comments. The failure message names this command, so a PR that moves an outcome on purpose runs it and commits the corpora in the same commit.
    • The basics generator now writes statements with an unrecorded outcome, and make corpus-rerecord fills it in. Its Python outcome recorder is removed, so the Rust test is the only recorder.

No converter code changes.

Since 0076e8e (record b64ede2), in a9e5755 and f1944aa:

  • Main fc15d51 (D316 + D322) is merged in, and the corpora are re-recorded on it: 29 more queries translate under the built-in default (clock reads, unread VARIANT columns), and 20 refusals change code. No query regressed.
  • D343: the basics generator's FIXED list gains SELECT order_id FROM orders WHERE ordered_at >= DATEADD(day, -30, CURRENT_DATE) and the same over CURRENT_TIMESTAMP(), as 10 entries with their spellings. Both are pinned as refused under both policies: SQ0003 node:gte and SQ0003 fn:DATEADD. They should flip to translated once fix: compare naive timestamps with CURRENT_TIMESTAMP() and count sub-day DATEDIFF of a DATE #115, C's group 1 and D326 land, and each of those PRs re-derives with make corpus-rerecord. No other basics entry changed.

At the slot (247ca79, 99e4ab3): main c2cbe9d merged in, then make corpus-rerecord. Only basics moves (D327): 387 refused → translated, 189 refusals change code, and 8 unaliased literals with a trailing comment (select 1 -- trailing comment and similar) go translated → refused with SQ0009, as D327 refuses them. D315 is expected to convert those again. No fixture result changed.

Corpora:

  • corpora/everyday: 615 analyst queries, 537 of them valid per the receipts.
  • corpora/everyday-models: 499 dbt-style models.
  • corpora/basics: 1,842 generated basics with surface variation, over the playground's sample schema, with fixture rows.

Recorded translation rates on fc15d51, built-in / repository: everyday 376 / 459 of 537, models 293 / 404 of 458, basics 1,095 / 1,160 of 1,842. These pin SQLQuack's own outcomes. They are not Snowflake-verified support figures: only 48 of the 1,114 hand-written queries have receipts, and none of the basics do.

Checks:

  • The three tests pass, in about 60 s, mostly the basics fixture runs.
  • Re-recording is idempotent: the generator followed by make corpus-rerecord reproduces the committed basics file byte for byte, and a second re-record changes no corpus.
  • Asymmetry, checked by hand-edited expectations and then reverted:
    • An expectation hand-flipped from translated to refused is only reported: the statement now looks like a refusal that translates.
    • Expectations hand-flipped to parse SQ0001 and to check SQ0004 on translating statements both fail.
    • A changed result hash fails, with the re-record hint.
    • A refused → translated expectation flip fails.
  • clippy -D warnings, cargo fmt, ruff, ruff format and fensu are clean, and the hygiene scan passes.

The full D94 gate runs when the slot comes: merge main, make corpus-rerecord, then gate that exact head.

…s keep their outcomes

Three corpora under corpora/, each checked under the built-in policy and
the repository policy in a UTC session, so a statement that once
translated cannot silently start refusing, and a change in what refuses
shows in review:

- everyday: 615 hand-written analyst queries, 537 of them valid in
  Snowflake (receipts in snowflake-receipts.json).
- everyday-models: 499 dbt-style models, compiled dbt package output and
  incremental, snapshot and MERGE shapes.
- basics: 1832 generated statements a first-time user types (literals in
  every spelling, columns, arithmetic, casts, common functions,
  aggregates, GROUP BY, ORDER BY, LIMIT, CTEs, joins, UNION, windows),
  each also spelled with other case, whitespace, a final semicolon,
  comments and a quoted table name. scripts/generate_basics_corpus.py
  writes it and records each outcome with a given sqlquack binary.

Outcomes and rates are recorded on main 78910c3.
kvlonge added a commit that referenced this pull request Nov 6, 2026
…command

The everyday, models and basics corpora now fail only when a statement
that translated refuses, a refusal changes code or span, or a translated
statement's result on the basics fixture rows changes. A refused statement
that now translates is reported without failing. `make corpus-rerecord`
rewrites the recorded outcomes and results, keeping each file's comments,
and the failure message names it.

The basics generator now writes statements with unrecorded outcomes;
`make corpus-rerecord` records them with the converter. The receipts'
schema descriptions name the corpus schema files.
Only a refusal that now translates, or an unrecorded outcome, is reported
without failing; a query recorded with a parse or check diagnostic that
starts translating fails, since Snowflake may reject it. The corpus
headers describe the asymmetric test and make corpus-rerecord.
The basics corpus gains `WHERE ordered_at >= DATEADD(day, -30, CURRENT_DATE)`
and the same over CURRENT_TIMESTAMP(), in their spellings. Both refuse with
SQ0003 today and should translate under the built-in default once the date
comparison and TIMESTAMP_LTZ rules land. The everyday corpora record main's
clock and semi-structured guard changes: 29 more queries translate under the
built-in default, and 20 refusals change code.
kvlonge added a commit that referenced this pull request Nov 7, 2026
A comment straight after SELECT no longer refuses the statement: 387
outcomes now translate, and 189 refusals move from SQ0002 node:select to
the construct the statement actually stops at. Eight unaliased literals
followed by a trailing comment, such as `select 1 -- trailing comment`,
now refuse with SQ0009, as the fix refuses an unaliased item whose text
holds a comment.

kvlonge commented Nov 7, 2026

Copy link
Copy Markdown
Contributor Author

D94 gate passed on the exact head 99e4ab3: #135 at f1944aa (auditor record db54f0b), merged with main c2cbe9d, then make corpus-rerecord.

  • make check-merge RECORDER=0: exit 0 in 41m36s. fensu reported 0 faults, and the three corpus tests passed.
  • make verify-recorded: exit 0 in 22m53s. 12,033 cases: 7,997 passed, 1,035 diverged as expected, 2,964 refused, 37 typed, 0 pending, 0 unproven, 0 failed; 226 of 226 equivalences and rules verified. The translation results have 0 regressions against the baseline.
  • The public-repository hygiene scan is clean.

GitHub Actions can't run because of the account billing block, so this local gate is the check for this landing.

@kvlonge
kvlonge merged commit 99e4ab3 into main Nov 7, 2026
@kvlonge
kvlonge deleted the land/basics-regression branch November 9, 2026 21:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant