Skip to content

fix: keep converted SQL on one line where its layout cannot be checked - #128

Merged
kvlonge merged 3 commits into
mainfrom
fix/layout-fallback
Nov 6, 2026
Merged

kvlonge merged 3 commits into
mainfrom
fix/layout-fallback

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Why

The layout's tokenizer models single-quoted strings, quoted names and --//* */ comments. It does not model DuckDB's $$/$tag$ strings, prefixed strings such as E'…\'…', or nested comments. Converted SQL contains none of them today, but convert, the Python API and the playground called layout_sql unchecked. If a later rule emits one, a break or collapsed space could land inside a literal and change the SQL silently. The existing token check can't see that, because it re-tokenizes both sides with the same tokenizer.

Changes

  • checked_layout first refuses SQL holding text the layout does not model: a $, a string directly after a word (E'', X''), an unclosed quote, or an unclosed or nested comment. Then it checks, as before, that the tokens are equal ignoring whitespace and that a second layout changes nothing.
  • layout_sql, which the CLI, Python and wasm call, now returns checked_layout's result, or the SQL unchanged on one line when the check fails.
  • The raw walk moves to a crate-private helper, so the check and the layout no longer call each other.

Verification

  • A unit test forces the fallback with a $$ string, an E'' string with an escaped quote, and a nested comment. Each holds a clause word and doubled spaces in a statement long enough to break. The test asserts the refusal reason and that layout_sql returns the input unchanged.
  • verify lays out every converted case through checked_layout, so no converted case falls back.
  • make check-ci, make test-rust, make verify-recorded.

layout_sql, which convert, the Python API and the playground call, now
returns checked_layout's result or the SQL unchanged. checked_layout
first refuses text its tokenizer does not model: a $, a prefixed
string such as E'', an unclosed quote, or an unclosed or nested
comment. Its token comparison re-tokenizes both sides with the same
tokenizer, so it cannot see a literal that tokenizer misreads.
kvlonge added a commit that referenced this pull request Nov 6, 2026

kvlonge commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

Pre-slot gate at bb134a2 (main d19c7a5 merged in):

  • make check-ci: exit 0
  • make test-rust: every suite passed
  • make verify-recorded: 11800 cases, 7780 passed, 1031 diverged as expected, 2952 refused, 37 typed, 0 pending, 0 unproven, 0 failed; 220 of 220 rules. Every converted case passes checked_layout, so the new guard refuses nothing conversion emits today (audit condition). Translation baseline: 0 regressions, 0 changes.

Lands after the D272c slot; if main moves, it re-merges and re-gates first.

kvlonge commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

Landing gate at 9ecf9fe (main df143dd merged in):

  • make check-ci: exit 0
  • make test-rust: every suite passed
  • make verify-recorded: 11870 cases, 7829 passed, 1032 diverged as expected, 2972 refused, 37 typed, 0 pending, 0 unproven, 0 failed; 220 of 220 rules. Every converted case passes checked_layout (audit condition). Translation baseline: 0 regressions, 0 changes. Hosted CI failures are the Actions billing block (job not started).

@kvlonge
kvlonge merged commit 9ecf9fe into main Nov 6, 2026
1 check failed
@kvlonge
kvlonge deleted the fix/layout-fallback branch November 9, 2026 21:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant