feat(rules): run SQL Rules through the SQLClean engine (#1283)

* feat(rules): run SQL Rules through the SQLClean engine (work in progress)

Checkpoint of the SQLClean integration: SQLCLEAN Rule codes, default
rule selection with the human-only notice, shared expansion inputs for
early lint, and SQL Rules skipped under --no-sql-analysis. e2e fixture
classification and Fensu faults are still open.

* feat(rules): finish the SQLClean engine integration

Pin sqlclean to v0.4.1, lint with the compile target and vars, skip
audit and SQL test files that opt out of SQL analysis, skip the row-rule
fact build for SQL-only selections, align fixtures and DuckShop with the
new findings, and update the docs.

* fix(rules): lint with the compiled target and vars, keep format fix eligibility

SQL Rules now expand with the compiled project effective target and vars
on every compile path instead of falling back to the default target when
the early-lint reuse is missing. Commands without target flags use the
project default explicitly. Restore the eight-code format --fix allowlist
and its documented decisions, and cover --target and --vars end to end.

* test(compile): split perf guards into default Rules and compiler-only

Default SQL Rules lint every compiled model, a measured 1.14x to 1.19x
cold wall cost over select = []. Compiler-only cases keep the previous
budgets and the 1.10 same-runner ratio; default-Rules cases scale the
cold budgets by the measured cost. Also pin sqlclean to the PIVOT USING
fix and keep two fixtures' derived tables out of SQLCLEAN039.

* test(compile): keep compiler-only ratios valid after default Rules land

Pass compiler_only to the base generator when the base supports it, so
the compiler-only job keeps both sides without Rules once main runs them.
Mark the default-Rules cold allowance as temporary, reject
--compiler-only with the dense project at parse time, assert that
default-Rules guard cases run Rules, and pin sqlclean to the v0.4.2
release commit.

* test(compile): split the 5,000- and 10,000-model guards for default Rules

Default SQL Rules cost 1.09x on the 10,000-model compile and 1.20x and
1.25x on the 5,000- and 10,000-model cache guards' cold runs, measured
against select = [] on the same machine. Scale those cold budgets and
keep the previous budgets as compiler-only cases, as for 3,000 models.

* test(compile): keep one compiler-only guard and calibrate Rules budgets on CI

Replace the compiler-only twin at every guard size with one paired guard: the
3,000-model cache guard runs default Rules and select = [] in the same job and
covers cold, warm, edit and project-config edit compiles at the previous
budgets. Drop the compiler-only same-runner ratio job and the --compiler-only
option of the ratio script.

Scale each default Rules budget by the larger of the CI same-job and local
alternating medians, excluding pairs whose Rules-independent phases exceed
twice their median. Give the project-config edit its own budget, since it
reruns every rule.

* docs(compile): correct the 5,000-model project-config budget and cold budget scope

The stated method gives 1.24x for the 5,000-model project-config edit, so its
budget is 18.0 s. The cold budget bounds the cold run as well as the
cache-disabled timeout.

Author: kvlonge

.github/workflows/verify.yml

@@ -671,8 +671,15 @@ jobs:
             max_ratio: "1.10"
             edit_max_ratio: "1.20"
             edit_max_cpu_ratio: "1.75"
+          # Temporary migration allowance; restore 1.10 once the base also runs default Rules.
+          # Default SQL Rules add a cold cost over select = [] (1.19x to 1.20x wall, 1.32x to
+          # 1.33x CPU in this job), so only the cold limits scale; warm and edit keep 1.10.
+          # The compiler-only cache guard at 3,000 models keeps the select = [] budgets.
           - kind: fresh
+            label: fresh default Rules
             max_ratio: "1.10"
+            cold_max_ratio: "1.25"
+            cold_max_cpu_ratio: "1.41"
           - kind: dense
             label: contracts CPU vs base
             gate_phase: contracts_cpu_ms
@@ -746,12 +753,16 @@ jobs:
         env:
           KIND: ${{ matrix.kind }}
           MAX_RATIO: ${{ matrix.max_ratio }}
+          COLD_MAX_RATIO: ${{ matrix.cold_max_ratio }}
+          COLD_MAX_CPU_RATIO: ${{ matrix.cold_max_cpu_ratio }}
           EDIT_MAX_RATIO: ${{ matrix.edit_max_ratio }}
           EDIT_MAX_CPU_RATIO: ${{ matrix.edit_max_cpu_ratio }}
         run: |
           uv run python -m scripts.compare_compile_performance \
             --kind "${KIND}" \
             --modes cold warm edit \
+            ${COLD_MAX_RATIO:+--cold-max-ratio "$COLD_MAX_RATIO"} \
+            ${COLD_MAX_CPU_RATIO:+--cold-max-cpu-ratio "$COLD_MAX_CPU_RATIO"} \
             ${EDIT_MAX_RATIO:+--edit-max-ratio "$EDIT_MAX_RATIO"} \
             ${EDIT_MAX_CPU_RATIO:+--edit-max-cpu-ratio "$EDIT_MAX_CPU_RATIO"} \
             --max-ratio "${MAX_RATIO}" \

Cargo.lock

@@ -139,6 +139,8 @@ sqlbuild-recall = { path = "crates/sqlbuild-recall" }
 sqlbuild-scopes = { path = "crates/sqlbuild-scopes" }
 sqlbuild-sqltext = { path = "crates/sqlbuild-sqltext" }
 sqlbuild-state = { path = "crates/sqlbuild-state" }
+# SQL lint Rules, quality Rules, the formatter and the canonical token stream.
+sqlclean = { git = "https://github.com/sqlstack-io/sqlclean.git", rev = "571e40a746b3d8d20d6118620f05428eca45243c" }
 stacker = "0.1"
 syn = { version = "2", features = ["full", "visit", "extra-traits"] }
 tempfile = "3"

Cargo.toml

@@ -34,7 +34,7 @@ average.
 | Python macros | Jinja macros producing the same SQL |
 | `audits [not_null]` | `data_tests: [not_null]` |
 | SQL tests with fixture CTEs | `unit_tests` with the same input and expected rows |
-| `[rules] select = ["SQBR", "XSQBR"]` (built-in Rules plus one custom Rule) | none |
+| `[rules] select = ["SQBR", "SQLCLEAN", "XSQBR"]` (built-in Rules, SQLClean Rules and one custom Rule) | none |
 
 `sqlbuild-norules/` is the same project with `[rules] select = []`. That turns off configurable Rules;
 column binding, type inference and contract checks still run.

benchmarks/dbt-compare/README.md

@@ -66,7 +66,7 @@ def write_norules(project: Path) -> Path:
     shutil.copytree(project, target)
     toml = target / "sqlbuild_project.toml"
     text = toml.read_text()
-    new = text.replace('select = ["SQBR", "XSQBR"]', "select = []")
+    new = text.replace('select = ["SQBR", "SQLCLEAN", "XSQBR"]', "select = []")
     assert new != text
     toml.write_text(new)
     return target

benchmarks/dbt-compare/write_dbt_dense.py

@@ -1 +1 @@
-generator 2 files 10874 sha256 7dd651a573bce90f9816889cdeece75b9f837137813a033125daafa28b7b70eb
+generator 3 files 10874 sha256 5f3d30b82ebf52f2eb869494f52f77d1699f9a08743febd6ee9ca0cb22ed0611

benchmarks/duckshop-10k/checksum.txt

@@ -4,6 +4,6 @@
 regenerate checksum.txt, whenever a change alters any generated file.
 """
 
-GENERATOR_VERSION = "2"
+GENERATOR_VERSION = "3"
 SEED = 20261024
 PROJECT_NAME = "duckshop_10k"

benchmarks/duckshop-10k/duckshop10k/__init__.py

@@ -321,23 +321,23 @@ def union_relations(relations: list[SqlResourceRef], columns: list[str]) -> str:
 
 MACRO_EXCEPTIONS: tuple[tuple[str, str, str], ...] = (
     (
-        "SQBRSQL018",
+        "SQLCLEAN018",
         "cdc_latest",
         """
 The macro cannot see its callers' keys on its own. Every call site is still checked, against the
 unique, non-null _seq that each change-log source declares.
 """,
     ),
     (
-        "SQBRSQL018",
+        "SQLCLEAN018",
         "cdc_history",
         """
 The macro cannot see its callers' keys on its own. _seq is unique and non-null in every change-log
 source, so ordering by it within a key and load is deterministic.
 """,
     ),
     (
-        "SQBRSQL034",
+        "SQLCLEAN034",
         "load_metadata",
         """
 A column-list macro body is a select list by construction. Its columns are inserted into the

benchmarks/duckshop-10k/duckshop10k/declarations.py

@@ -192,12 +192,12 @@ def project_toml() -> str:
 lookup_customer_id = "C000001"
 
 [rules]
-select = ["SQBRSQL", "SQBRGRAPH", "SQBRCONTRACT", "SQBRTEST", "XSQBRDUCK"]
-# SQBRSQL021: `SELECT * FROM final` and the change-log macro's pass-through CTEs are the house
+select = ["SQLCLEAN", "SQBRGRAPH", "SQBRCONTRACT", "SQBRTEST", "XSQBRDUCK"]
+# SQLCLEAN021: `SELECT * FROM final` and the change-log macro's pass-through CTEs are the house
 #   style; each model's columns block, not its terminal SELECT, owns the output shape.
 # SQBRCONTRACT101: exact contracts are reserved for published dim_ and fct_ models.
 # SQBRTEST202: a sample of models carries unit tests, as in a project of this size.
-ignore = ["SQBRSQL021", "SQBRCONTRACT101", "SQBRTEST202"]
+ignore = ["SQLCLEAN021", "SQBRCONTRACT101", "SQBRTEST202"]
 '''
 
 

benchmarks/duckshop-10k/duckshop10k/project.py

@@ -48,7 +48,7 @@ The core project, without the [edge-case area](#edge-cases):
 - 5 seeds, 2 SQL functions (a scalar UDF and a table function) and a Python UDF.
 - SQL and Python macros, folder-scoped constants and enums, generic and singular audits, model
   schemas, hooks, incremental models with cursor windows, and snapshots with both strategies.
-- Rules select `SQBRSQL`, `SQBRGRAPH`, `SQBRCONTRACT`, `SQBRTEST` and the project's own
+- Rules select `SQLCLEAN`, `SQBRGRAPH`, `SQBRCONTRACT`, `SQBRTEST` and the project's own
   `XSQBRDUCK` rules, with reviewed exceptions in `sqlbuild_project.toml`. Every model must have at least one unit test (`SQBRTEST202`).
 
 ## Tests and scenarios
@@ -225,7 +225,7 @@ noted here:
   static types of the columns it reads from a CTE through a table alias (`FROM listings AS l`), so
   `rpt_edge_partner_listing_export` declares only the name and audits of `"listingId"`, without
   types.
-- `SQBRSQL018` can only take proof from single-column keys. Append-only feeds redeliver rows, so
+- `SQLCLEAN018` can only take proof from single-column keys. Append-only feeds redeliver rows, so
   their id is unique only together with `_loaded_at` (the `unique_per_delivery` audit), and the
   staging models that keep the latest delivery carry reviewed exceptions.