feat(format)!: layout-only token formatter with canonical SQL fingerprints (#902)

* fix(format): treat dollar-quoted literals as opaque in SQL site scanners

The formatter's interpolation scanner and the native lint preparation did
not recognise $$...$$ or $tag$...$tag$ literals. An apostrophe inside one
flipped quote state, so a later __ref(...) call was not protected and was
uppercased by the formatter, breaking compilation.

Lint preparation now scans with the shared sql_scan quote policy, whose
dollar quotes match the compiler's, and the Python fallback scanner uses
the same dollar-quote rule.

* feat(format): print authored SQL tokens with a parse-tree layout

The formatter regenerated each statement from its syntax tree and then ran
repair passes to restore what regeneration lost, so any spelling the SQL
library canonicalised could reach the user's file, and shapes it
restructured (PostgreSQL comma SUBSTRING, SELECT ALL) were refused.

The pretty-printed statement is now only a layout oracle. Authored tokens
are aligned to it, and the formatter prints the authored tokens in their
original order with the oracle's whitespace and word case. Tokens the oracle
dropped or respelled keep their authored text with deterministic spacing,
and tokens that were adjacent in the source stay adjacent. Comments stay
attached to the token they precede or follow.

A permanent invariant compares the canonical token stream (whitespace and
comments removed, unquoted words upper-cased, everything else exact) and
the comment attachments of input and output; any difference leaves the
file unchanged with a named fault.

Formatting is now layout-only. It no longer adds AS to
implicit aliases or drops statement terminators and trailing select-list
commas. Already formatted files are unchanged.

* perf(format): check idempotence against the existing layout oracle

The idempotence check formatted the output from scratch, parsing and
pretty-printing every statement a second time. Formatted output has the
same tokens, so it is re-printed against the oracle already computed.
Comment attachment and the comment invariant now use binary search
instead of a scan over all tokens per comment.

* test(format): pin formatter output over a dialect corpus and example projects

A corpus of 1,066 SELECT and WITH queries from the SQLGlot dialect
identity fixtures (MIT, notice included) records the formatted output or
refusal for each query and checks that every formatted layout, including
the previous release's, is a fixed point. Formatting each repository
example project must leave compiled dependencies, column lineage and
column counts unchanged and pass format --check.

* test(format): bound unformatted model formatting time

Adds performance guards for 1,300 mixed unformatted models, one
6,000-line model and linear scaling across doublings, and runs the
formatter performance job when the native crate changes.

* feat(planner): fingerprint compiled SQL by its canonical token stream

Change detection hashed compiled SQL with whitespace collapsed, so a
comment, a keyword-case change or a formatter run rebuilt the model. The
stored definition_hash is now the SHA-256 of the dialect token stream with
whitespace and comments dropped and unquoted words upper-cased; string
literals, quoted identifiers, numbers and optimizer hints stay exact.
Models, SQL functions, query-diff artifacts and the dbt manifest checksum
share it; non-SQL function bodies are still hashed exactly.

The model version identity now hashes the same fingerprint, and SQL
function bodies enter dependent identities as fingerprints. Previously a
formatting-only edit kept definition_hash but changed version_hash, which
superseded a pending concurrent microbatch replay requirement and opened a
second one.

SQL the dialect tokenizer rejects is a planning error (S025) naming the
model or function. Both hashes change in place once on upgrade, so every
tracked model and function is planned as changed one time.

* feat(format): wrap SQL and header audit lines longer than the line width

[format] line_width (default 100) now also bounds SQL bodies. A longer line
is broken at its outermost structure: before top-level AND/OR (not the AND
of BETWEEN), then before arithmetic operators, then by putting the
arguments of the outermost bracket group that crosses the limit one per
line, with OVER (...) split before ORDER BY/ROWS/RANGE. Top-level commas
are the last resort. String literals, comments and lines carrying a
comment are never broken, and interpolation sentinels count with the
width of the SQLBuild syntax they stand in for.

Wrapping runs inside the printer, so the token invariant and the
idempotence check cover it. MODEL header audits [...] lists over the width
put each audit and then each argument on its own line, kept only when the
parsed header values are unchanged.

* feat(format): apply compiler-verified Rule fixes with format --fix

Ports the --fix implementation from the earlier enforceable-SQL-quality
work. sqb format --fix runs the SQBRSQL rules to a fixed point in memory,
applies non-overlapping native edits from the reviewed safe set, and
recompiles after every pass. A pass is accepted only when output columns
(types and nullability), dependencies and column lineage are unchanged;
otherwise the original files are kept and a
format-fix-verification-failed fault is reported. Edits that touch macro
or intrinsic expansions are refused. Applied, refused and unavailable
fixes are listed in human output and in the JSON rule_fixes array, and
--check and --diff report without writing.

* feat(format): verify Rule fixes per file and extend the safe set

When a --fix pass cannot be proved, the edits of each file are now
recompiled on their own. Files whose edits fail keep their original
contents and report format-fix-verification-failed; the other files are
still fixed, so one unprovable model no longer blocks the project.
Verification compares only edited models and their downstream models, so
an unrelated model without column inference no longer blocks every fix.

COUNT(1) to COUNT(*) (SQBRSQL017) and JOIN to INNER JOIN (SQBRSQL025)
join the safe set after passing recompile verification and result
comparison; SQBRSQL023 alias removal stays refused. The format docs list
each Rule decision.

* fix(planner): report untokenizable SQL in column-migration planning as S025

Column-migration candidate detection and the renamed-snapshot overlay
hashed model SQL with compute_query_hash directly, so a tokenizer failure
escaped as a raw fingerprint error. Both now go through
model_definition_hash, which raises the S025 planning error naming the
model.

* fix(format): recase only keywords and built-in calls; keep dialect comments

The fingerprint upper-cased every unquoted word, so case changes of
identifiers, aliases, qualified names and JSON path keys did not change
it. The printer recased unaligned keyword-typed words even where they
named columns or tables, such as inventory.view or AS index.

One shared classification now decides which tokens are case-insensitive:
keyword-typed tokens in keyword position (not qualified, not a path key
or cast type, not an alias, not a relation name or operand), and unquoted
names called as built-in functions according to the dialect function
catalogue and parser. The printer recases only those tokens, the token
invariant compares every other token exactly, and the fingerprint folds
exactly that set. Golden hashes change once more.

Comments in every dialect form are kept: any non-whitespace text the
tokenizer skips between tokens is carried as a comment, so Snowflake //
and BigQuery # comments are no longer dropped. Adjacent string literals
separated by a line break keep the break, and the invariant refuses
output that joins them.

* fix(format): fold only reserved keywords and built-in calls

The case classification folded keyword-typed words by default and
excluded contexts with a deny-list, so struct-literal keys, keyword-named
operands before unlisted operators and comma-joined keyword-named tables
were still recased by the printer and folded by the fingerprint.

It is now an allow-list. A word is case-insensitive only when it is a
reserved keyword of the dialect (a word that can never be an unquoted
identifier there), or a built-in function name directly followed by an
opening parenthesis. Words after `.`, `:`, `@`, `$` or `AS` stay exact,
because DuckDB, PostgreSQL and BigQuery accept reserved words as field
names and column aliases. Everything else, including non-reserved
keywords, keeps its authored case in the printer, the token invariant and
the fingerprint.

Reserved keywords come from per-dialect data files with their sources:
duckdb_keywords() reserved, PostgreSQL kwlist.h RESERVED_KEYWORD,
BigQuery, Snowflake and T-SQL documented lists, none for Databricks, and
their intersection for the generic dialect.

* fix(format): upper-case parse-confirmed keywords; keep fingerprint folding strict

Formatting now prints every keyword in upper case when the layout guide
confirms it at that position: an authored word aligned with a guide token
that the tokenizer types as a keyword and that the generator prints in
upper case, outside the name positions after `.`, `:`, `@`, `$` and `AS`.
The generator quotes or keeps the authored spelling of identifiers, so
keyword-named columns, aliases, tables and struct keys keep their case.

The printer records each such recase, and the token invariant accepts a
case change only for fold-set tokens or for a recorded recase whose guide
token is the same word as an upper-case keyword. The change fingerprint
is unchanged: it still folds only reserved keywords and built-in calls,
so formatting lowercase SQL may change a fingerprint once. The corpus
property now checks that formatting formatted SQL keeps its fingerprint
and that raw SQL differs only by upper-cased words.

* test(e2e): assign formatter and fingerprint E2E files to CI shards

* feat(compile): reject built-in function spellings the dialect does not accept

The SQL parser maps alias spellings such as STARTS_WITH and STARTSWITH to
one typed node, so compile accepted spellings the warehouse rejects.
Check each authored call-site name against a SQLBuild-owned Snowflake
spelling list and the DuckDB function catalogue, and report B101 with the
dialect spelling when the parser maps the name to the same built-in as an
accepted spelling. Unknown names, UDF sentinels, qualified and method calls,
quoted names, type positions and keywords are never flagged.

* test(compile): cover dialect function spellings end to end and bound their cost

Compile real Snowflake and DuckDB projects through the CLI to prove that
unsupported spellings fail with the dialect spelling while accepted
synonyms and an alias-named project UDF compile. Guard the wide-query scan
and a many-error compile against non-linear scaling.

* docs(compile): document B101 built-in spelling checks

* fix(compile): tell built-in calls from relation and column lists

A CTE column list, a derived-table or table alias without AS, and INSERT
or CREATE column lists put an identifier before a parenthesis without
calling a function, so alias-named relations were rejected as B101. Track
the previous token and the FROM and DEFINE clause at each depth, and treat
a name as a call only where an expression can start. Also check object
literal values after a string key, MATCH_RECOGNIZE DEFINE conditions, and
names separated from their parenthesis by a comment.

* fix(compile): treat materialized CTE column lists as non-calls

DuckDB CTEs may write AS MATERIALIZED ( or AS NOT MATERIALIZED ( after a
column list, so alias-named CTEs with those hints were rejected as B101.

* feat(format): clause layout rules and trailing-aware line wrapping

Printed SQL now goes through one token/gap layout pass before and after
line wrapping. Only whitespace between tokens changes, so the token
invariant and fingerprints are unaffected.

- A select list with one item, and WHERE/QUALIFY/HAVING with one
  condition, stay on the clause line when they fit.
- CTEs close with `),` and one blank line; one blank line separates the
  last CTE from the statement (nested WITH included, comments kept above
  their CTE).
- A VALUES list that does not fit puts each row on its own line; rows
  are never split.
- Type parameter lists such as NUMBER(38, 2) are never split (CHI-461).
- DECODE breaks into the first argument, then search/result pairs.
- An operator whose operand spans lines starts the next line.
- Wrapping counts trailing aliases and commas, breaks the widest
  outermost group (including groups that close on a later line and
  one-line CASE expressions), joins AND/OR breaks inside call arguments,
  and falls back to the outermost commas on the line.

* docs(format): describe clause layout rules and wrapping of trailing tokens

* fix(format): never space a function name from its parenthesis

When the layout guide prints a call in another shape (the generic
dialect prints IF/IFF as CASE, Databricks reorders TRIM, Snowflake
GENERATOR and window frames), the printer copied the guide's spacing
onto the unaligned authored tokens and produced `IF ( x, 1, 2)`. A
word directly followed by `(` in the authored SQL now stays attached,
and no space is printed after `(` or before `)` unless the line breaks.

A property test formats a two-, one- or zero-argument call of every
built-in name the formatter's catalogues and allow-lists know, plus
keyword-like call names, in all seven dialects at a normal and a
breaking width; no name needs a skip.

Author: kvlonge

.github/workflows/verify.yml

@@ -81,6 +81,7 @@ jobs:
               crates/*|Cargo.toml|Cargo.lock|pyproject.toml|uv.lock)
                 broad=true
                 performance=true
+                format_performance=true
                 rules_performance=true
                 ;;
               src/sqlbuild/compiler/*|src/sqlbuild/adapter/type_system/*)

Cargo.lock

@@ -29,6 +29,9 @@ polyglot-sql = { git = "https://github.com/chio-labs/polyglot.git", rev = "25ad0
     "semantic",
     "stacker",
 ] }
+polyglot-sql-function-catalogs = { git = "https://github.com/chio-labs/polyglot.git", rev = "25ad011c49f2c7cd48f4427abb8fc0a6af2e4120", default-features = false, features = [
+    "dialect-duckdb",
+] }
 quote = "1"
 rayon = "1"
 regex = "1"

Cargo.toml

@@ -82,6 +82,7 @@ E2E_DUCKDB_BUILD_CORE_PATHS := \
 	tests/e2e/src/sqlbuild/cli/commands/main/build/test_python_node_identity.py \
 	tests/e2e/src/sqlbuild/cli/commands/main/build/test_sql_hook_identity.py \
 	tests/e2e/src/sqlbuild/cli/commands/main/build/test_query_change_tracking.py \
+	tests/e2e/src/sqlbuild/cli/commands/main/build/test_query_fingerprint_changes.py \
 	tests/e2e/src/sqlbuild/cli/commands/main/build/test_query_propagation.py \
 	tests/e2e/src/sqlbuild/cli/commands/main/build/test_remove_column_semantics.py \
 	tests/e2e/src/sqlbuild/cli/commands/main/build/test_reusable_model_schemas.py \
@@ -131,6 +132,9 @@ E2E_DUCKDB_CLI_PATHS := \
 	tests/e2e/src/sqlbuild/cli/commands/main/databricks \
 	tests/e2e/src/sqlbuild/cli/commands/main/dbt \
 	tests/e2e/src/sqlbuild/cli/commands/main/debug \
+	tests/e2e/src/sqlbuild/cli/commands/main/format/test_format_example_projects.py \
+	tests/e2e/src/sqlbuild/cli/commands/main/format/test_rule_fixes.py \
+	tests/e2e/src/sqlbuild/cli/commands/main/format/test_token_preservation.py \
 	tests/e2e/src/sqlbuild/cli/commands/main/freshness \
 	tests/e2e/src/sqlbuild/cli/commands/main/init \
 	tests/e2e/src/sqlbuild/cli/commands/main/rules \

Makefile

@@ -19,6 +19,7 @@ fensu-policy.workspace = true
 globset.workspace = true
 mimalloc.workspace = true
 polyglot-sql.workspace = true
+polyglot-sql-function-catalogs.workspace = true
 pyo3.workspace = true
 rayon.workspace = true
 regex.workspace = true