From 55e5914ef91ef1080ed7be6c64af25bd14efc894 Mon Sep 17 00:00:00 2001 From: Amp Date: Sat, 31 Oct 2026 05:26:35 +0000 Subject: [PATCH 1/3] docs: restructure the README around the SQLBuild layout Co-authored-by: Kevin Longe --- README.md | 257 +++++++++++++++--------------------------------------- 1 file changed, 69 insertions(+), 188 deletions(-) diff --git a/README.md b/README.md index 80d14da..29d0c9c 100644 --- a/README.md +++ b/README.md @@ -1,228 +1,108 @@

- SQLClean + SQLClean

Clean SQL, every time. Fast linting, safe fixes and one canonical format.

-SQLClean checks `.sql` files against built-in Rules, applies their fixes and formats every file in -one canonical layout. The engine and the `sqlclean` command are written in Rust. Custom Rules -are plain Python functions that read structural facts about each statement. +

+ Website · + Docs · + Quickstart · + Benchmarks +

+ +SQLClean is a SQL linter and formatter. It flags SQL that runs but rarely does what you meant, +applies the fixes that are safe, and formats every file one way. The engine and the `sqlclean` +command are written in Rust. Custom Rules are plain Python functions, backed by a Rust API that +parses the SQL for them. It is free and open source under Apache-2.0.

- sqlclean check reports three findings, then sqlclean fix applies two fixes and formats the file + sqlclean check reports three findings, then sqlclean fix applies two fixes and formats the file

`sqlclean check` reports a bare `JOIN`, a `COUNT(1)` and a temporary table from a custom Rule. `sqlclean fix` applies the two fixes and formats the file; the custom Rule's finding has no fix, so it stays. -## Install +## Quick start ```bash uv tool install sqlclean # or: pip install sqlclean -sqlclean --version +sqlclean check # report findings in every .sql file under the directory +sqlclean fix # apply safe fixes, then format +sqlclean format # rewrite files in the canonical layout ``` -The `sqlclean` package installs the native `sqlclean` executable. Custom Rules run in the same -Python environment as that executable. +Add a `sqlclean.toml` when you want to choose a dialect, files or Rules. See the +[quickstart](https://sqlclean.io/docs/quickstart/) and +[configuration](https://sqlclean.io/docs/configuration/). -## Commands +## What it does -```bash -sqlclean check [PATHS]... # report findings; exit 1 when there are any -sqlclean fix [PATHS]... # apply safe fixes until stable, then format -sqlclean format [PATHS]... # rewrite files in the canonical layout -sqlclean rules [CODE] # list the Rules, or explain one -sqlclean skills # install the skill for AI coding agents -``` +### Catch SQL that is easy to get wrong -- With no paths, a command reads every matching file under the configuration root. -- `check` and `fix` accept `--select` and `--ignore`, which take codes or prefixes such as - `SQC001` or `SQC`, and `--no-cache`. -- `check --output-format` takes `text`, `json` or `sarif`. Use SARIF for code-scanning uploads. -- `fix --unsafe-fixes` also applies fixes that can change query results. `fix --diff` and - `format --diff` print a unified diff instead of writing. -- `format --check` fails when a file needs formatting, without writing it. -- Exit status is 0 when clean, 1 when there are findings or files to change, and 2 on errors. - -Each statement is formatted on its own. Statements are separated by one blank line, and every -file ends with a newline. Formatting is idempotent: a formatted file formats to itself. - -## Configuration - -SQLClean reads the nearest `sqlclean.toml` at or above the working directory, or the file passed -with `--config`. Paths in it are relative to the file's directory. - -```toml -dialect = "duckdb" # default "generic" -include = ["**/*.sql"] -exclude = ["vendor/**"] # files ignored by .gitignore are skipped too - -[[dialects]] # a different dialect for some paths -paths = ["warehouse/**"] -dialect = "snowflake" - -[format] -line_width = 100 -indent = 4 - -[rules] -select = ["SQC"] # omit to run the Rules enabled by default -ignore = ["SQC027"] -custom = ["sql_rules"] # Python files or directories that define custom Rules - -[rules.thresholds] -max_literal_length = 100 # SQC044 -max_ranking_order_by = 5 # SQC043 - -[rules.rule_options.XSQCNAME001] -prefixes = ["tmp_", "scratch_"] - -[[rules.rule_exceptions]] # one reviewed finding that is allowed -rule = "SQC038" -path = "models/calendar.sql" -relation = "dates" # optional: the joined relation the CROSS JOIN reads -reason = "Every pair of dates is needed." - -[[rules.rule_ignores]] # Rules that do not apply to some paths -rules = ["SQC034"] -paths = ["scratch/**"] -reason = "Exploratory queries." -``` +- **Built-in Rules.** Default Rules flag comparisons with `NULL`, `NOT IN` lists that contain + `NULL`, `ROW_NUMBER` with no deterministic order and more. Each finding points at the exact + characters and says what to write instead. See [Rules](https://sqlclean.io/docs/rules/overview/) + and the [Rule reference](https://sqlclean.io/docs/rules/reference/). +- **Reviewed exceptions.** An exception allows one Rule in one file, with a reason. When the SQL + changes and it no longer matches, `sqlclean check` fails until it is removed. See + [exceptions](https://sqlclean.io/docs/rules/overview/#exceptions). -An exception that no longer matches a finding is reported as stale, so exceptions are removed -when the SQL they cover changes. A parse error is reported as `SQC999` and cannot be ignored. - -Results are cached in `.sqlclean/cache/`, keyed by file content, dialect, selection, options and -the custom Rule sources. The directory has its own `.gitignore`. - -## Rules - -Built-in Rules have `SQC###` codes. Run `sqlclean rules` for the full list, or -`sqlclean rules SQC001` for one Rule. A finding's fix is applied by `sqlclean fix`. `SQC001` turns -`= NULL` into `IS NULL`, which changes which rows match, so it needs `--unsafe-fixes`. - -## Custom Rules - -A custom Rule is a function decorated with `@rule`. Its first keyword parameter names its -subject: - -- `statement: Statement` runs once per statement. -- `file: SqlFile` runs once per file. -- `repository: Repository` runs once over every checked file. - -Its second parameter receives a `RuleContext`. Codes are `XSQC`, optional capital letters, and -three digits, such as `XSQCNAME001`. - -```python -from sqlclean import FixSafety, RuleContext, RuleOption, Statement, rule - -PREFIXES = RuleOption.string_list( - name="prefixes", default=("tmp_",), description="Table prefixes that must not be read." -) - - -@rule( - code="XSQCNAME001", - message="Query reads a temporary table", - remediation="Read from the published table instead.", - options=(PREFIXES,), - enabled_by_default=True, -) -def temporary_table(*, statement: Statement, ctx: RuleContext) -> list: - prefixes = ctx.option(PREFIXES) - return [ - ctx.finding(subject=statement, node=table) - for table in statement.tables - if table.name.startswith(prefixes) - ] - - -@rule( - code="XSQCSTYLE001", - message="COUNT(1) should be COUNT(*)", - remediation="Write COUNT(*).", - fixable=FixSafety.SAFE, - enabled_by_default=True, -) -def count_one(*, statement: Statement, ctx: RuleContext) -> list: - counts = {f.end + 1 for f in statement.functions if f.name.upper() == "COUNT"} - return [ - ctx.finding(subject=statement, node=literal, fix=[ctx.replace(node=literal, text="*")]) - for literal in statement.literals - if literal.text == "1" and literal.start in counts - ] -``` +### Fix and format safely -- A `Statement` exposes `kind` and `text`, plus located `ctes`, `tables`, `joins`, `stars`, - `functions`, `columns` and `literals`. Each node has `start` and `end` character offsets in - the file, `line`, `column` and its `text`. -- `statement.polyglot_ast()` returns the full Polyglot syntax tree. It needs - `pip install "sqlclean[ast]"`. -- Fixes are lists of edits made with `ctx.replace`, `ctx.insert_before`, `ctx.insert_after` and - `ctx.delete`. A fixing Rule declares `fixable` as `FixSafety.SAFE`, or as `FixSafety.UNSAFE` - when the fix can change results. -- Options are declared with `RuleOption.boolean`, `integer`, `string` or `string_list`, set under - `[rules.rule_options.]`, and validated before any Rule runs. -- Files named `test_*.py` and `conftest.py` under custom Rule paths are not loaded as Rules. - -### Testing custom Rules - -```python -from sqlclean.testing import RuleCase, evaluate_rule - -from sql_rules.naming import temporary_table - - -def test_temporary_tables_are_reported() -> None: - result = evaluate_rule( - rule=temporary_table, - test_case=RuleCase( - description="reads tmp_orders", - source="SELECT * FROM tmp_orders", - dialect="duckdb", - expected_finding_count=1, - ), - ) - assert result.fixed_source == "SELECT * FROM tmp_orders" -``` +- **Safe fixes by default.** `sqlclean fix` applies safe fixes until the file is stable, then + formats it. A fix that can change which rows match, such as `= NULL` to `IS NULL`, waits for + `--unsafe-fixes`, and a fix that would break parsing is thrown away. See + [`sqlclean fix`](https://sqlclean.io/docs/cli/fix/). +- **One canonical format.** Every statement gets the same layout, and formatting a formatted file + changes nothing. See [formatting](https://sqlclean.io/docs/formatting/). -`evaluate_rule` parses the SQL with the same native engine as `sqlclean check`. It fails when the -finding count differs, and when the Rule's findings change with the order of `files`. +### Make your conventions Rules -## Editors, CI and pre-commit +- **Custom Rules in plain Python.** A Rule is a function over the tables, joins, columns and + literals of each statement, already parsed and located by the Rust engine. No regular + expressions and no parser to learn. See [custom Rules](https://sqlclean.io/docs/custom-rules/overview/). +- **Tested like any other code.** `sqlclean.testing` runs a Rule with the same engine as + `sqlclean check`. See [testing custom Rules](https://sqlclean.io/docs/custom-rules/testing/). -Run `sqlclean check` in CI. `--output-format sarif` produces a report for code-scanning tools, -and `--output-format json` produces one for other integrations. Editors can run -`sqlclean format` on save. +### Run it everywhere + +- **CI and pre-commit.** `sqlclean check` exits non-zero on findings and writes SARIF for + code-scanning annotations, or JSON for anything else. See + [integrations](https://sqlclean.io/docs/integrations/). +- **Agent skills.** `sqlclean skills` installs a skill that teaches AI coding agents the commands, + Rules and configuration of the installed version. See + [`sqlclean skills`](https://sqlclean.io/docs/cli/skills/). +- **An embeddable engine.** Tools that template or compile SQL can call the Rust crate directly. + See [embedding](https://sqlclean.io/docs/embedding/). + +## Speed + +On 10,282 Snowflake SQL files, on a Linux machine with 4 CPUs and no cache, `sqlclean check` with +its 13 default Rules took 0.81 s, against 41 s for `sqruff lint` and 919 s for +`sqlfluff lint --processes 4`. All 45 built-in Rules took 1.33 s, and the default Rules plus three +custom Python Rules took 2.27 s. A rerun with nothing changed reads the result cache and takes +0.23 s. These are SQLClean 0.0.1 measurements, each tool with its default rules; the tools don't +check the same things. See the [benchmarks](https://sqlclean.io/docs/benchmarks/) for the method, +the corpus and how to run it yourself. + +## pre-commit ```yaml # .pre-commit-config.yaml repos: - repo: https://github.com/sqlstack-io/sqlclean - rev: v0.0.1 + rev: v0.1.0 hooks: - id: sqlclean-fix - id: sqlclean-format ``` -`sqlclean-check` is also available for hooks that must not change files. - -## Using the engine from Rust - -`crates/sqlclean` is the engine without the command. Tools that template or compile SQL can call -it directly and keep their own configuration and Rule codes: - -- `sql_lint::main::lint::lint` lints one SQL text. `sql_lint::main::engine::lint_json` and - `batch_engine::lint_batch_json` do the same through JSON requests. -- `sql_lint::main::formatter::format_json` formats SQL. -- `sql_lint::main::catalogue::catalogue` and `default_rules::default_rules` describe the Rules. -- `sql_facts::main::file_facts::file_facts` returns the statement facts custom Rules read. - -Requests can carry external identifiers and declared relation keys, so a host tool can tell the -Rules about names it resolves itself. +Use `sqlclean-check` instead of `sqlclean-fix` where a hook must not change files. See +[pre-commit](https://sqlclean.io/docs/integrations/#pre-commit). ## Development @@ -234,4 +114,5 @@ make test # Rust and Python tests ## License -Apache-2.0 +SQLClean is licensed under the +[Apache License 2.0](https://github.com/sqlstack-io/sqlclean/blob/main/LICENSE). From 2fbffc57f5f9cb844601534fa401ff6ab137e147 Mon Sep 17 00:00:00 2001 From: Amp Date: Sat, 31 Oct 2026 05:33:39 +0000 Subject: [PATCH 2/3] docs: lead the README with speed and capabilities Co-authored-by: Kevin Longe --- README.md | 135 +++++++++++++++++++++++++++++------------------------- 1 file changed, 72 insertions(+), 63 deletions(-) diff --git a/README.md b/README.md index 29d0c9c..9f35cc6 100644 --- a/README.md +++ b/README.md @@ -2,21 +2,26 @@ SQLClean

+

The fastest SQL linter.

+ +

+ {{SPEED_LINE}} Benchmarks +

+

- Clean SQL, every time. Fast linting, safe fixes and one canonical format. + Catch SQL mistakes, apply safe fixes, and format consistently. Add your own rules in Python.

Website · + Get started · Docs · - Quickstart · Benchmarks

-SQLClean is a SQL linter and formatter. It flags SQL that runs but rarely does what you meant, -applies the fixes that are safe, and formats every file one way. The engine and the `sqlclean` -command are written in Rust. Custom Rules are plain Python functions, backed by a Rust API that -parses the SQL for them. It is free and open source under Apache-2.0. +```bash +uv tool install sqlclean # or: pip install sqlclean +```

sqlclean check reports three findings, then sqlclean fix applies two fixes and formats the file @@ -26,70 +31,66 @@ parses the SQL for them. It is free and open source under Apache-2.0. `sqlclean fix` applies the two fixes and formats the file; the custom Rule's finding has no fix, so it stays. -## Quick start - -```bash -uv tool install sqlclean # or: pip install sqlclean -sqlclean check # report findings in every .sql file under the directory -sqlclean fix # apply safe fixes, then format -sqlclean format # rewrite files in the canonical layout -``` - -Add a `sqlclean.toml` when you want to choose a dialect, files or Rules. See the -[quickstart](https://sqlclean.io/docs/quickstart/) and -[configuration](https://sqlclean.io/docs/configuration/). - -## What it does +## Speed -### Catch SQL that is easy to get wrong +{{SPEED_TABLE}} -- **Built-in Rules.** Default Rules flag comparisons with `NULL`, `NOT IN` lists that contain - `NULL`, `ROW_NUMBER` with no deterministic order and more. Each finding points at the exact - characters and says what to write instead. See [Rules](https://sqlclean.io/docs/rules/overview/) - and the [Rule reference](https://sqlclean.io/docs/rules/reference/). -- **Reviewed exceptions.** An exception allows one Rule in one file, with a reason. When the SQL - changes and it no longer matches, `sqlclean check` fails until it is removed. See - [exceptions](https://sqlclean.io/docs/rules/overview/#exceptions). +The tools don't check the same things, so see the +[benchmarks](https://sqlclean.io/docs/benchmarks/) for coverage, the corpus, the method and how +to run it yourself. -### Fix and format safely +## Correct and safe -- **Safe fixes by default.** `sqlclean fix` applies safe fixes until the file is stable, then - formats it. A fix that can change which rows match, such as `= NULL` to `IS NULL`, waits for - `--unsafe-fixes`, and a fix that would break parsing is thrown away. See +- **Rules for SQL that runs but is wrong.** Built-in Rules flag comparisons with `NULL`, `NOT IN` + lists that contain `NULL`, `ROW_NUMBER` with no deterministic order and more. See + [Rules](https://sqlclean.io/docs/rules/overview/). +- **Safe fixes by default.** `sqlclean fix` applies safe fixes only. A fix that can change which + rows match waits for `--unsafe-fixes`, and a fix that would break parsing is discarded. See [`sqlclean fix`](https://sqlclean.io/docs/cli/fix/). -- **One canonical format.** Every statement gets the same layout, and formatting a formatted file - changes nothing. See [formatting](https://sqlclean.io/docs/formatting/). - -### Make your conventions Rules - -- **Custom Rules in plain Python.** A Rule is a function over the tables, joins, columns and - literals of each statement, already parsed and located by the Rust engine. No regular - expressions and no parser to learn. See [custom Rules](https://sqlclean.io/docs/custom-rules/overview/). -- **Tested like any other code.** `sqlclean.testing` runs a Rule with the same engine as - `sqlclean check`. See [testing custom Rules](https://sqlclean.io/docs/custom-rules/testing/). +- **An idempotent formatter.** Every statement gets one canonical layout, and formatting a + formatted file changes nothing. See [formatting](https://sqlclean.io/docs/formatting/). + +## Custom Rules in Python + +A custom Rule is a Python function over the tables, joins, columns and literals of each +statement, already parsed and located by the Rust engine: + +```python +# sql_rules/naming.py +from sqlclean import Finding, RuleContext, Statement, rule + + +@rule( + code="XSQCNAME001", + message="Query reads a temporary table", + remediation="Read from the published table instead.", + enabled_by_default=True, +) +def temporary_table(*, statement: Statement, ctx: RuleContext) -> list[Finding]: + return [ + ctx.finding(subject=statement, node=table) + for table in statement.tables + if table.name.startswith("tmp_") + ] +``` -### Run it everywhere +Point `[rules] custom = ["sql_rules"]` in `sqlclean.toml` at it. See +[custom Rules](https://sqlclean.io/docs/custom-rules/overview/) and +[testing custom Rules](https://sqlclean.io/docs/custom-rules/testing/). -- **CI and pre-commit.** `sqlclean check` exits non-zero on findings and writes SARIF for - code-scanning annotations, or JSON for anything else. See - [integrations](https://sqlclean.io/docs/integrations/). -- **Agent skills.** `sqlclean skills` installs a skill that teaches AI coding agents the commands, - Rules and configuration of the installed version. See - [`sqlclean skills`](https://sqlclean.io/docs/cli/skills/). -- **An embeddable engine.** Tools that template or compile SQL can call the Rust crate directly. - See [embedding](https://sqlclean.io/docs/embedding/). +## Quick start -## Speed +```bash +sqlclean check # report findings in every .sql file under the directory +sqlclean fix # apply safe fixes, then format +sqlclean format # rewrite files in the canonical layout +``` -On 10,282 Snowflake SQL files, on a Linux machine with 4 CPUs and no cache, `sqlclean check` with -its 13 default Rules took 0.81 s, against 41 s for `sqruff lint` and 919 s for -`sqlfluff lint --processes 4`. All 45 built-in Rules took 1.33 s, and the default Rules plus three -custom Python Rules took 2.27 s. A rerun with nothing changed reads the result cache and takes -0.23 s. These are SQLClean 0.0.1 measurements, each tool with its default rules; the tools don't -check the same things. See the [benchmarks](https://sqlclean.io/docs/benchmarks/) for the method, -the corpus and how to run it yourself. +Add a `sqlclean.toml` to choose a dialect, files or Rules. See the +[quickstart](https://sqlclean.io/docs/quickstart/) and +[configuration](https://sqlclean.io/docs/configuration/). -## pre-commit +## Integrations ```yaml # .pre-commit-config.yaml @@ -101,8 +102,17 @@ repos: - id: sqlclean-format ``` -Use `sqlclean-check` instead of `sqlclean-fix` where a hook must not change files. See -[pre-commit](https://sqlclean.io/docs/integrations/#pre-commit). +Use `sqlclean-check` where a hook must not change files. In CI, `sqlclean check` exits non-zero +on findings and writes SARIF with `--output-format sarif` for code-scanning annotations. See +[integrations](https://sqlclean.io/docs/integrations/). `sqlclean skills` installs a skill for AI +coding agents; see [`sqlclean skills`](https://sqlclean.io/docs/cli/skills/). + +## Docs + +- [Docs home](https://sqlclean.io/docs/) +- [Rule reference](https://sqlclean.io/docs/rules/reference/) +- [Configuration](https://sqlclean.io/docs/configuration/) +- [Embedding the Rust engine](https://sqlclean.io/docs/embedding/) ## Development @@ -114,5 +124,4 @@ make test # Rust and Python tests ## License -SQLClean is licensed under the [Apache License 2.0](https://github.com/sqlstack-io/sqlclean/blob/main/LICENSE). From 8e2f216025d266e6817924ac3b9832f553b30fd0 Mon Sep 17 00:00:00 2001 From: Amp Date: Sat, 31 Oct 2026 05:53:54 +0000 Subject: [PATCH 3/3] docs: fill the README speed claim and table from the published benchmarks Co-authored-by: Kevin Longe --- README.md | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 9f35cc6..c1d5eb0 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,7 @@

The fastest SQL linter.

- {{SPEED_LINE}} Benchmarks + 50× faster than sqruff. Lint 10,000 SQL files in under a second. Benchmarks

@@ -33,7 +33,17 @@ so it stays. ## Speed -{{SPEED_TABLE}} +Linting the same 10,282 Snowflake SQL files on 4 CPUs, with no cache: + +| Command | Wall time | vs `sqlclean check` | +|---|---:|---:| +| `sqlclean check` (default Rules) | **0.81 s** | 1× | +| `sqlclean check --select SQC` (all Rules) | **1.33 s** | 1.6× | +| `sqlclean check` with three custom Python Rules | **2.27 s** | 2.8× | +| `sqruff lint` | 41.0 s | 51× | +| `sqlfluff lint --processes 4` | 919 s | 1,135× | + +Run again with nothing changed, `sqlclean check` reads its result cache and finishes in 0.23 s. The tools don't check the same things, so see the [benchmarks](https://sqlclean.io/docs/benchmarks/) for coverage, the corpus, the method and how