Repository navigation
feat: add the SQLClean engine, command and Python custom Rules - #1
Merged
Merged
Conversation
Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <kvlonge@gmail.com>
Built-in Rules use SQC codes. The engine exposes JSON lint, batch and format entries so host tools can embed it with their own codes. Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <kvlonge@gmail.com>
Configuration comes from the nearest sqlclean.toml. Results are cached per file, fixes run until stable and unsafe fixes need --unsafe-fixes. Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <kvlonge@gmail.com>
Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <kvlonge@gmail.com>
Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <kvlonge@gmail.com>
Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <kvlonge@gmail.com>
Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <kvlonge@gmail.com>
Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <kvlonge@gmail.com>
A finding whose only fix is unsafe was counted as fixable with plain sqlclean fix, which leaves it unchanged. Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <kvlonge@gmail.com>
A table fact for raw.payments covered only payments, so findings and fixes at the node left the schema out. Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <kvlonge@gmail.com>
Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <kvlonge@gmail.com>
sqlclean format formatted one file at a time while check already used rayon. On 10,282 files format --check drops from about 10 s to 2.7 s on 4 CPUs; output keeps file order. Co-authored-by: Kevin Longe <kvlonge@gmail.com>
- Formatted SQL is indented four spaces per step by default, matching SQLFluff and sqruff; [format] indent chooses 1 to 8. The engine keeps two spaces unless a request passes indent, so embedding tools keep their current layout. - The parse-tree layout broke every parenthesis level of a long expression. Groups that fit on their line are now rejoined, innermost first, so only the levels that overflow break. A group that fits once its leading operator starts the line breaks before that operator. - A leading comma after a line comment moves to the end of the commented line instead of standing on its own line. - A file that holds only comments keeps its final newline. Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <kvlonge@gmail.com>
- website/: an Astro Starlight site with a homepage and 18 docs pages covering every command, every Rule, configuration, custom Rules, integrations, embedding and benchmarks. Example output comes from the sqlclean command. A workflow builds the site and checks its links. - The homepage and README describe what SQLClean checks, without claiming it catches every query that returns wrong rows. - Examples and the README demo show the four-space indent, and the docs cover [format] indent. Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <kvlonge@gmail.com>
bench.sh runs each linter on the plain-SQL DuckShop 10K corpus from SQLBuild's generator, cold and, for SQLClean, warm, and summarize.py reports median wall time, CPU time, peak memory and finding counts. convert_duckshop.py turns SQLBuild's compiled models into plain Snowflake SQL. Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <kvlonge@gmail.com>
They move to private configuration. Co-authored-by: Kevin Longe <kvlonge@gmail.com>
`--config custom.toml` made the configuration root an empty path, so every command failed with ": no such file or directory". Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <kvlonge@gmail.com>
Custom Rules rebuilt each file's facts serially and ran in a single Python process. Facts are now built in parallel, and statement and file Rules split the files across one host per CPU, from 250 files a host. Repository Rules still read every file in one host. On 10,282 files with three Python Rules, check drops from 5.4 s to 2.1 s on 4 CPUs. Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <kvlonge@gmail.com>
sqlclean_custom runs the default Rules plus three Python Rules that a data team might write: layering between marts, staging and raw sources, and a ban on nondeterministic functions. Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <kvlonge@gmail.com>
- The hero shows a real-time race on the 10,282-file benchmark: the sqlclean bar fills in 0.81 s while sqruff and SQLFluff advance at their measured rates. Reduced motion shows the moment sqlclean ends. - Three feature boxes replace six: speed, custom Rules in Python and safe fixes. - The Benchmarks page adds the custom Rules row, and the custom Rules docs say statement and file Rules can run in several processes. Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <kvlonge@gmail.com>
…he Rust engine - The race labels each linter with its total time instead of a percentage. - The lede describes the NULL example in words. - The custom Rules box says the Rust engine parses SQL and builds the facts, so Python only runs the Rule's logic. Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <kvlonge@gmail.com>
fensu requires dataclass cases with an expected_ field and no conditionals in tests. Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <kvlonge@gmail.com>
SQLFluff and sqruff users type lint. The alias is hidden from --help; check stays the documented name, and the check page mentions lint. Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <kvlonge@gmail.com>
Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <kvlonge@gmail.com>
Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <kvlonge@gmail.com>
Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <kvlonge@gmail.com>
Deploying sqlclean with
|
| Latest commit: |
261abfb
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://9bbc0f4c.sqlclean.pages.dev |
| Branch Preview URL: | https://feat-initial-sqlclean.sqlclean.pages.dev |
The executable embeds a SQLClean skill with a copy of every docs page. make skills regenerates the copies from the website docs and make check-docs-skill fails while they differ. Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <kvlonge@gmail.com>
…eases Verify classifies a pull request by changed paths. Website, docs and benchmark edits run the docs skill check and ruff; bundled skill edits add a Linux build and the skills tests; anything else runs everything. The Verify job aggregates the selected suites. Release Please counts every Conventional Commit regardless of hidden changelog sections, so exclude-paths keeps website, benchmark and workflow-only commits from cutting a release. Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <kvlonge@gmail.com>
Amp-Thread-ID: https://ampcode.com/threads/T-01a18b8a-340f-717d-9ca7-6dcf564a065a Co-authored-by: Kevin Longe <kvlonge@gmail.com>
This was referenced Oct 30, 2026
kvlonge
added a commit
that referenced
this pull request
Oct 31, 2026
* build: add the Cargo workspace, Python packaging and fensu policy Co-authored-by: Kevin Longe <kvlonge@gmail.com> * feat: add the SQL lint, format and statement facts engine Built-in Rules use SQC codes. The engine exposes JSON lint, batch and format entries so host tools can embed it with their own codes. Co-authored-by: Kevin Longe <kvlonge@gmail.com> * feat: add the sqlclean check, fix, format and rules commands Configuration comes from the nearest sqlclean.toml. Results are cached per file, fixes run until stable and unsafe fixes need --unsafe-fixes. Co-authored-by: Kevin Longe <kvlonge@gmail.com> * feat: add Python custom Rules, their host and test harness Co-authored-by: Kevin Longe <kvlonge@gmail.com> * ci: add verify, release and publish workflows Co-authored-by: Kevin Longe <kvlonge@gmail.com> * docs: document commands, configuration and custom Rules Co-authored-by: Kevin Longe <kvlonge@gmail.com> * fix: satisfy clippy nonminimal_bool on the CI toolchain Co-authored-by: Kevin Longe <kvlonge@gmail.com> * docs: add the logo and a check and fix demo to the README Co-authored-by: Kevin Longe <kvlonge@gmail.com> * fix: count unsafe fixes apart in the check summary A finding whose only fix is unsafe was counted as fixable with plain sqlclean fix, which leaves it unchanged. Co-authored-by: Kevin Longe <kvlonge@gmail.com> * fix: locate qualified tables from their first name part A table fact for raw.payments covered only payments, so findings and fixes at the node left the schema out. Co-authored-by: Kevin Longe <kvlonge@gmail.com> * fix: separate the message and remediation in SARIF results Co-authored-by: Kevin Longe <kvlonge@gmail.com> * perf(cli): format files in parallel sqlclean format formatted one file at a time while check already used rayon. On 10,282 files format --check drops from about 10 s to 2.7 s on 4 CPUs; output keeps file order. Co-authored-by: Kevin Longe <kvlonge@gmail.com> * feat(format): indent four spaces and break only overflowing groups - Formatted SQL is indented four spaces per step by default, matching SQLFluff and sqruff; [format] indent chooses 1 to 8. The engine keeps two spaces unless a request passes indent, so embedding tools keep their current layout. - The parse-tree layout broke every parenthesis level of a long expression. Groups that fit on their line are now rejoined, innermost first, so only the levels that overflow break. A group that fits once its leading operator starts the line breaks before that operator. - A leading comma after a line comment moves to the end of the commented line instead of standing on its own line. - A file that holds only comments keeps its final newline. Co-authored-by: Kevin Longe <kvlonge@gmail.com> * docs: add the SQLClean website, docs and benchmark page - website/: an Astro Starlight site with a homepage and 18 docs pages covering every command, every Rule, configuration, custom Rules, integrations, embedding and benchmarks. Example output comes from the sqlclean command. A workflow builds the site and checks its links. - The homepage and README describe what SQLClean checks, without claiming it catches every query that returns wrong rows. - Examples and the README demo show the four-space indent, and the docs cover [format] indent. Co-authored-by: Kevin Longe <kvlonge@gmail.com> * chore(bench): compare SQLClean with sqruff and SQLFluff bench.sh runs each linter on the plain-SQL DuckShop 10K corpus from SQLBuild's generator, cold and, for SQLClean, warm, and summarize.py reports median wall time, CPU time, peak memory and finding counts. convert_duckshop.py turns SQLBuild's compiled models into plain Snowflake SQL. Co-authored-by: Kevin Longe <kvlonge@gmail.com> * chore: keep agent instructions and orb setup out of the repository They move to private configuration. Co-authored-by: Kevin Longe <kvlonge@gmail.com> * fix(cli): read a bare --config file name from the working directory `--config custom.toml` made the configuration root an empty path, so every command failed with ": no such file or directory". Co-authored-by: Kevin Longe <kvlonge@gmail.com> * perf(custom): evaluate custom Rules on one host per CPU Custom Rules rebuilt each file's facts serially and ran in a single Python process. Facts are now built in parallel, and statement and file Rules split the files across one host per CPU, from 250 files a host. Repository Rules still read every file in one host. On 10,282 files with three Python Rules, check drops from 5.4 s to 2.1 s on 4 CPUs. Co-authored-by: Kevin Longe <kvlonge@gmail.com> * chore(bench): add a custom Rules variant sqlclean_custom runs the default Rules plus three Python Rules that a data team might write: layering between marts, staging and raw sources, and a ban on nondeterministic functions. Co-authored-by: Kevin Longe <kvlonge@gmail.com> * docs(website): lead with the benchmark and custom Rules - The hero shows a real-time race on the 10,282-file benchmark: the sqlclean bar fills in 0.81 s while sqruff and SQLFluff advance at their measured rates. Reduced motion shows the moment sqlclean ends. - Three feature boxes replace six: speed, custom Rules in Python and safe fixes. - The Benchmarks page adds the custom Rules row, and the custom Rules docs say statement and file Rules can run in several processes. Co-authored-by: Kevin Longe <kvlonge@gmail.com> * docs(website): show race totals in time and say custom Rules run on the Rust engine - The race labels each linter with its total time instead of a percentage. - The lede describes the NULL example in words. - The custom Rules box says the Rust engine parses SQL and builds the facts, so Python only runs the Rule's logic. Co-authored-by: Kevin Longe <kvlonge@gmail.com> * test(e2e): parametrize the many-files custom Rules test fensu requires dataclass cases with an expected_ field and no conditionals in tests. Co-authored-by: Kevin Longe <kvlonge@gmail.com> * feat(cli): accept lint as an alias of check SQLFluff and sqruff users type lint. The alias is hidden from --help; check stays the documented name, and the check page mentions lint. Co-authored-by: Kevin Longe <kvlonge@gmail.com> * docs(website): shorten the hero lede Co-authored-by: Kevin Longe <kvlonge@gmail.com> * docs(website): serve the site from sqlclean.io Co-authored-by: Kevin Longe <kvlonge@gmail.com> * chore: trigger the first site preview build Co-authored-by: Kevin Longe <kvlonge@gmail.com> * feat(cli): add sqlclean skills to install a bundled agent skill The executable embeds a SQLClean skill with a copy of every docs page. make skills regenerates the copies from the website docs and make check-docs-skill fails while they differ. Co-authored-by: Kevin Longe <kvlonge@gmail.com> * ci: run only the suites a change needs and keep site edits out of releases Verify classifies a pull request by changed paths. Website, docs and benchmark edits run the docs skill check and ruff; bundled skill edits add a Linux build and the skills tests; anything else runs everything. The Verify job aggregates the selected suites. Release Please counts every Conventional Commit regardless of hidden changelog sections, so exclude-paths keeps website, benchmark and workflow-only commits from cutting a release. Co-authored-by: Kevin Longe <kvlonge@gmail.com> * test: write e2e project files with LF line endings on every platform Co-authored-by: Kevin Longe <kvlonge@gmail.com> ---------
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
SQLClean is SQLStack's standalone SQL linter and formatter, extracted from SQLBuild's engine so SQLBuild can depend on the crate and keep its own codes.
Changes
crates/sqlclean: built-in Rules (SQC###), the formatter and statement facts, with JSON entries for host tools.crates/sqlclean-cli:check,fix,format,rules,factsandskills, with config discovery, exceptions, a result cache and text, JSON or SARIF output.src/sqlclean: Python custom Rules (@rule,XSQCcodes, options, fixes), their host andsqlclean.testing.sqlclean skillsinstalls a bundled agent skill holding every docs page;make skillsregenerates it and CI fails while it is stale.website/(sqlclean.io, deployed by Cloudflare Pages) andbenchmarks/linters/(vs sqruff and SQLFluff).Verifyjob;exclude-pathskeeps website, benchmark and workflow-only commits out of releases. See.github/RELEASING.md.Verification
make verify: ruff, ty, fensu (0 faults), docs skill check, clippy, 52 Rust and 42 pytest tests.cargo +1.88 check, actionlint, website build with 0 broken links, hygiene scan clean.