Skip to content

Commit a37459b

Browse files
kvlongeampagent
kvlonge
and
ampagent
authored
docs: restructure the README around the SQLBuild layout (#7)
* docs: restructure the README around the SQLBuild layout Co-authored-by: Kevin Longe <kvlonge@gmail.com> * docs: lead the README with speed and capabilities Co-authored-by: Kevin Longe <kvlonge@gmail.com> * docs: fill the README speed claim and table from the published benchmarks Co-authored-by: Kevin Longe <kvlonge@gmail.com> --------- Co-authored-by: Amp <amp@ampcode.com>
1 parent a128aec commit a37459b

1 file changed

Lines changed: 69 additions & 169 deletions

File tree

‎README.md‎

Lines changed: 69 additions & 169 deletions
Original file line numberDiff line numberDiff line change
@@ -1,228 +1,128 @@
11
<p align="center">
2-
<img src=".github/sqlclean-logo.png" alt="SQLClean" width="100%">
2+
<img src="https://raw.githubusercontent.com/sqlstack-io/sqlclean/main/.github/sqlclean-logo.png" alt="SQLClean" width="100%">
33
</p>
44

5+
<h2 align="center">The fastest SQL linter.</h2>
6+
57
<p align="center">
6-
<strong>Clean SQL, every time.</strong> Fast linting, safe fixes and one canonical format.
8+
<strong>50× faster than sqruff. Lint 10,000 SQL files in under a second.</strong> <a href="https://sqlclean.io/docs/benchmarks/">Benchmarks</a>
79
</p>
810

9-
SQLClean checks `.sql` files against built-in Rules, applies their fixes and formats every file in
10-
one canonical layout. The engine and the `sqlclean` command are written in Rust. Custom Rules
11-
are plain Python functions that read structural facts about each statement.
12-
1311
<p align="center">
14-
<img src=".github/demos/check-fix.gif" alt="sqlclean check reports three findings, then sqlclean fix applies two fixes and formats the file" width="100%">
12+
Catch SQL mistakes, apply safe fixes, and format consistently. Add your own rules in Python.
1513
</p>
1614

17-
`sqlclean check` reports a bare `JOIN`, a `COUNT(1)` and a temporary table from a custom Rule.
18-
`sqlclean fix` applies the two fixes and formats the file; the custom Rule's finding has no fix,
19-
so it stays.
20-
21-
## Install
15+
<p align="center">
16+
<a href="https://sqlclean.io">Website</a> ·
17+
<a href="https://sqlclean.io/docs/quickstart/">Get started</a> ·
18+
<a href="https://sqlclean.io/docs/">Docs</a> ·
19+
<a href="https://sqlclean.io/docs/benchmarks/">Benchmarks</a>
20+
</p>
2221

2322
```bash
2423
uv tool install sqlclean # or: pip install sqlclean
25-
sqlclean --version
2624
```
2725

28-
The `sqlclean` package installs the native `sqlclean` executable. Custom Rules run in the same
29-
Python environment as that executable.
26+
<p align="center">
27+
<img src="https://raw.githubusercontent.com/sqlstack-io/sqlclean/main/.github/demos/check-fix.gif" alt="sqlclean check reports three findings, then sqlclean fix applies two fixes and formats the file" width="100%">
28+
</p>
3029

31-
## Commands
30+
`sqlclean check` reports a bare `JOIN`, a `COUNT(1)` and a temporary table from a custom Rule.
31+
`sqlclean fix` applies the two fixes and formats the file; the custom Rule's finding has no fix,
32+
so it stays.
3233

33-
```bash
34-
sqlclean check [PATHS]... # report findings; exit 1 when there are any
35-
sqlclean fix [PATHS]... # apply safe fixes until stable, then format
36-
sqlclean format [PATHS]... # rewrite files in the canonical layout
37-
sqlclean rules [CODE] # list the Rules, or explain one
38-
sqlclean skills # install the skill for AI coding agents
39-
```
34+
## Speed
4035

41-
- With no paths, a command reads every matching file under the configuration root.
42-
- `check` and `fix` accept `--select` and `--ignore`, which take codes or prefixes such as
43-
`SQC001` or `SQC`, and `--no-cache`.
44-
- `check --output-format` takes `text`, `json` or `sarif`. Use SARIF for code-scanning uploads.
45-
- `fix --unsafe-fixes` also applies fixes that can change query results. `fix --diff` and
46-
`format --diff` print a unified diff instead of writing.
47-
- `format --check` fails when a file needs formatting, without writing it.
48-
- Exit status is 0 when clean, 1 when there are findings or files to change, and 2 on errors.
49-
50-
Each statement is formatted on its own. Statements are separated by one blank line, and every
51-
file ends with a newline. Formatting is idempotent: a formatted file formats to itself.
52-
53-
## Configuration
54-
55-
SQLClean reads the nearest `sqlclean.toml` at or above the working directory, or the file passed
56-
with `--config`. Paths in it are relative to the file's directory.
57-
58-
```toml
59-
dialect = "duckdb" # default "generic"
60-
include = ["**/*.sql"]
61-
exclude = ["vendor/**"] # files ignored by .gitignore are skipped too
62-
63-
[[dialects]] # a different dialect for some paths
64-
paths = ["warehouse/**"]
65-
dialect = "snowflake"
66-
67-
[format]
68-
line_width = 100
69-
indent = 4
70-
71-
[rules]
72-
select = ["SQC"] # omit to run the Rules enabled by default
73-
ignore = ["SQC027"]
74-
custom = ["sql_rules"] # Python files or directories that define custom Rules
75-
76-
[rules.thresholds]
77-
max_literal_length = 100 # SQC044
78-
max_ranking_order_by = 5 # SQC043
79-
80-
[rules.rule_options.XSQCNAME001]
81-
prefixes = ["tmp_", "scratch_"]
82-
83-
[[rules.rule_exceptions]] # one reviewed finding that is allowed
84-
rule = "SQC038"
85-
path = "models/calendar.sql"
86-
relation = "dates" # optional: the joined relation the CROSS JOIN reads
87-
reason = "Every pair of dates is needed."
88-
89-
[[rules.rule_ignores]] # Rules that do not apply to some paths
90-
rules = ["SQC034"]
91-
paths = ["scratch/**"]
92-
reason = "Exploratory queries."
93-
```
36+
Linting the same 10,282 Snowflake SQL files on 4 CPUs, with no cache:
9437

95-
An exception that no longer matches a finding is reported as stale, so exceptions are removed
96-
when the SQL they cover changes. A parse error is reported as `SQC999` and cannot be ignored.
38+
| Command | Wall time | vs `sqlclean check` |
39+
|---|---:|---:|
40+
| `sqlclean check` (default Rules) | **0.81 s** | 1× |
41+
| `sqlclean check --select SQC` (all Rules) | **1.33 s** | 1.6× |
42+
| `sqlclean check` with three custom Python Rules | **2.27 s** | 2.8× |
43+
| `sqruff lint` | 41.0 s | 51× |
44+
| `sqlfluff lint --processes 4` | 919 s | 1,135× |
9745

98-
Results are cached in `.sqlclean/cache/`, keyed by file content, dialect, selection, options and
99-
the custom Rule sources. The directory has its own `.gitignore`.
46+
Run again with nothing changed, `sqlclean check` reads its result cache and finishes in 0.23 s.
10047

101-
## Rules
48+
The tools don't check the same things, so see the
49+
[benchmarks](https://sqlclean.io/docs/benchmarks/) for coverage, the corpus, the method and how
50+
to run it yourself.
10251

103-
Built-in Rules have `SQC###` codes. Run `sqlclean rules` for the full list, or
104-
`sqlclean rules SQC001` for one Rule. A finding's fix is applied by `sqlclean fix`. `SQC001` turns
105-
`= NULL` into `IS NULL`, which changes which rows match, so it needs `--unsafe-fixes`.
52+
## Correct and safe
10653

107-
## Custom Rules
54+
- **Rules for SQL that runs but is wrong.** Built-in Rules flag comparisons with `NULL`, `NOT IN`
55+
lists that contain `NULL`, `ROW_NUMBER` with no deterministic order and more. See
56+
[Rules](https://sqlclean.io/docs/rules/overview/).
57+
- **Safe fixes by default.** `sqlclean fix` applies safe fixes only. A fix that can change which
58+
rows match waits for `--unsafe-fixes`, and a fix that would break parsing is discarded. See
59+
[`sqlclean fix`](https://sqlclean.io/docs/cli/fix/).
60+
- **An idempotent formatter.** Every statement gets one canonical layout, and formatting a
61+
formatted file changes nothing. See [formatting](https://sqlclean.io/docs/formatting/).
10862

109-
A custom Rule is a function decorated with `@rule`. Its first keyword parameter names its
110-
subject:
63+
## Custom Rules in Python
11164

112-
- `statement: Statement` runs once per statement.
113-
- `file: SqlFile` runs once per file.
114-
- `repository: Repository` runs once over every checked file.
115-
116-
Its second parameter receives a `RuleContext`. Codes are `XSQC`, optional capital letters, and
117-
three digits, such as `XSQCNAME001`.
65+
A custom Rule is a Python function over the tables, joins, columns and literals of each
66+
statement, already parsed and located by the Rust engine:
11867

11968
```python
120-
from sqlclean import FixSafety, RuleContext, RuleOption, Statement, rule
121-
122-
PREFIXES = RuleOption.string_list(
123-
name="prefixes", default=("tmp_",), description="Table prefixes that must not be read."
124-
)
69+
# sql_rules/naming.py
70+
from sqlclean import Finding, RuleContext, Statement, rule
12571

12672

12773
@rule(
12874
code="XSQCNAME001",
12975
message="Query reads a temporary table",
13076
remediation="Read from the published table instead.",
131-
options=(PREFIXES,),
13277
enabled_by_default=True,
13378
)
134-
def temporary_table(*, statement: Statement, ctx: RuleContext) -> list:
135-
prefixes = ctx.option(PREFIXES)
79+
def temporary_table(*, statement: Statement, ctx: RuleContext) -> list[Finding]:
13680
return [
13781
ctx.finding(subject=statement, node=table)
13882
for table in statement.tables
139-
if table.name.startswith(prefixes)
140-
]
141-
142-
143-
@rule(
144-
code="XSQCSTYLE001",
145-
message="COUNT(1) should be COUNT(*)",
146-
remediation="Write COUNT(*).",
147-
fixable=FixSafety.SAFE,
148-
enabled_by_default=True,
149-
)
150-
def count_one(*, statement: Statement, ctx: RuleContext) -> list:
151-
counts = {f.end + 1 for f in statement.functions if f.name.upper() == "COUNT"}
152-
return [
153-
ctx.finding(subject=statement, node=literal, fix=[ctx.replace(node=literal, text="*")])
154-
for literal in statement.literals
155-
if literal.text == "1" and literal.start in counts
83+
if table.name.startswith("tmp_")
15684
]
15785
```
15886

159-
- A `Statement` exposes `kind` and `text`, plus located `ctes`, `tables`, `joins`, `stars`,
160-
`functions`, `columns` and `literals`. Each node has `start` and `end` character offsets in
161-
the file, `line`, `column` and its `text`.
162-
- `statement.polyglot_ast()` returns the full Polyglot syntax tree. It needs
163-
`pip install "sqlclean[ast]"`.
164-
- Fixes are lists of edits made with `ctx.replace`, `ctx.insert_before`, `ctx.insert_after` and
165-
`ctx.delete`. A fixing Rule declares `fixable` as `FixSafety.SAFE`, or as `FixSafety.UNSAFE`
166-
when the fix can change results.
167-
- Options are declared with `RuleOption.boolean`, `integer`, `string` or `string_list`, set under
168-
`[rules.rule_options.<CODE>]`, and validated before any Rule runs.
169-
- Files named `test_*.py` and `conftest.py` under custom Rule paths are not loaded as Rules.
87+
Point `[rules] custom = ["sql_rules"]` in `sqlclean.toml` at it. See
88+
[custom Rules](https://sqlclean.io/docs/custom-rules/overview/) and
89+
[testing custom Rules](https://sqlclean.io/docs/custom-rules/testing/).
17090

171-
### Testing custom Rules
91+
## Quick start
17292

173-
```python
174-
from sqlclean.testing import RuleCase, evaluate_rule
175-
176-
from sql_rules.naming import temporary_table
177-
178-
179-
def test_temporary_tables_are_reported() -> None:
180-
result = evaluate_rule(
181-
rule=temporary_table,
182-
test_case=RuleCase(
183-
description="reads tmp_orders",
184-
source="SELECT * FROM tmp_orders",
185-
dialect="duckdb",
186-
expected_finding_count=1,
187-
),
188-
)
189-
assert result.fixed_source == "SELECT * FROM tmp_orders"
93+
```bash
94+
sqlclean check # report findings in every .sql file under the directory
95+
sqlclean fix # apply safe fixes, then format
96+
sqlclean format # rewrite files in the canonical layout
19097
```
19198

192-
`evaluate_rule` parses the SQL with the same native engine as `sqlclean check`. It fails when the
193-
finding count differs, and when the Rule's findings change with the order of `files`.
99+
Add a `sqlclean.toml` to choose a dialect, files or Rules. See the
100+
[quickstart](https://sqlclean.io/docs/quickstart/) and
101+
[configuration](https://sqlclean.io/docs/configuration/).
194102

195-
## Editors, CI and pre-commit
196-
197-
Run `sqlclean check` in CI. `--output-format sarif` produces a report for code-scanning tools,
198-
and `--output-format json` produces one for other integrations. Editors can run
199-
`sqlclean format` on save.
103+
## Integrations
200104

201105
```yaml
202106
# .pre-commit-config.yaml
203107
repos:
204108
- repo: https://github.com/sqlstack-io/sqlclean
205-
rev: v0.0.1
109+
rev: v0.1.0
206110
hooks:
207111
- id: sqlclean-fix
208112
- id: sqlclean-format
209113
```
210114
211-
`sqlclean-check` is also available for hooks that must not change files.
212-
213-
## Using the engine from Rust
214-
215-
`crates/sqlclean` is the engine without the command. Tools that template or compile SQL can call
216-
it directly and keep their own configuration and Rule codes:
115+
Use `sqlclean-check` where a hook must not change files. In CI, `sqlclean check` exits non-zero
116+
on findings and writes SARIF with `--output-format sarif` for code-scanning annotations. See
117+
[integrations](https://sqlclean.io/docs/integrations/). `sqlclean skills` installs a skill for AI
118+
coding agents; see [`sqlclean skills`](https://sqlclean.io/docs/cli/skills/).
217119

218-
- `sql_lint::main::lint::lint` lints one SQL text. `sql_lint::main::engine::lint_json` and
219-
`batch_engine::lint_batch_json` do the same through JSON requests.
220-
- `sql_lint::main::formatter::format_json` formats SQL.
221-
- `sql_lint::main::catalogue::catalogue` and `default_rules::default_rules` describe the Rules.
222-
- `sql_facts::main::file_facts::file_facts` returns the statement facts custom Rules read.
120+
## Docs
223121

224-
Requests can carry external identifiers and declared relation keys, so a host tool can tell the
225-
Rules about names it resolves itself.
122+
- [Docs home](https://sqlclean.io/docs/)
123+
- [Rule reference](https://sqlclean.io/docs/rules/reference/)
124+
- [Configuration](https://sqlclean.io/docs/configuration/)
125+
- [Embedding the Rust engine](https://sqlclean.io/docs/embedding/)
226126

227127
## Development
228128

@@ -234,4 +134,4 @@ make test # Rust and Python tests
234134

235135
## License
236136

237-
Apache-2.0
137+
[Apache License 2.0](https://github.com/sqlstack-io/sqlclean/blob/main/LICENSE).

0 commit comments

Comments
 (0)