diff --git a/.github/demos/check-convert.gif b/.github/demos/check-convert.gif new file mode 100644 index 000000000..537681aaf Binary files /dev/null and b/.github/demos/check-convert.gif differ diff --git a/.github/sqlquack-logo-dark.png b/.github/sqlquack-logo-dark.png new file mode 100644 index 000000000..172621611 Binary files /dev/null and b/.github/sqlquack-logo-dark.png differ diff --git a/.github/sqlquack-logo-light.png b/.github/sqlquack-logo-light.png new file mode 100644 index 000000000..5079106ff Binary files /dev/null and b/.github/sqlquack-logo-light.png differ diff --git a/README.md b/README.md index 5e694a305..6bd2746cb 100644 --- a/README.md +++ b/README.md @@ -1,29 +1,92 @@ -# SQLQuack - -SQLQuack converts Snowflake SQL into DuckDB SQL, and refuses to guess. - -Most SQL transpilers print *something* for every input. Often that output parses and runs on DuckDB but -means something different: `CONCAT` handles NULLs differently, regular expressions anchor -differently, decimal division keeps a different scale. SQLQuack takes the opposite approach. Every -conversion either comes from a rule verified against recorded Snowflake results, or it is refused -with a precise reason: the construct, where it is, and why. - -**Status: early.** A first set of rules converts the relational core, exact NUMBER arithmetic, -comparisons, conditionals, casts, basic string functions, aggregates and window functions; -everything else is refused, or stopped by a declared guard where DuckDB's result would differ. -What works today: - -- parsing Snowflake SQL with SQLQuack's own parser, derived from - [polyglot](https://github.com/sqlstack-io/polyglot), which keeps every call as written, and - classifying every function, operator, type, cast, join and clause in it; -- a structured refusal for each construct: *missing rule*, *outside a rule's type conditions*, - *unresolved name*, *not expressible in DuckDB*, *parse fidelity*, or *conversion budget - exceeded*; -- DuckDB's own parse-and-bind check of every converted statement; -- `sqlquack census`, which reports which constructs a body of SQL uses and how much of it would - convert as rules are added; -- `sqlquack index`, which maps every building block, typing function and rule to the cases and - SQL files whose conversions rely on it. +
+
+
+ by SQLStack
+
+ Run warehouse SQL locally. Verified results, or a clear no. +
+ ++ Website · + Docs · + Playground · + Quickstart +
+ +SQLQuack checks warehouse SQL and translates it into DuckDB SQL, so you can run it locally and in +CI. Today it reads Snowflake SQL. + +Most SQL transpilers print *something* for every input. Often that output runs on DuckDB but means +something different: `CONCAT` handles NULLs differently, regular expressions anchor differently, +decimal division keeps a different scale. SQLQuack converts only where a rule verified against +recorded Snowflake results covers the types involved. Everything else is refused with a code, the +exact span and the reason. + +
+
+
` prints; a code is never reused or renumbered. SQL that Snowflake parses
-but then rejects, such as `INTERSECT ALL`, `FOR UPDATE WAIT`, `QUALIFY` without a window function,
-or a `* ILIKE` `REPLACE` or `RENAME` of a column the pattern removes, is refused with its own
-`SQ02xx` code instead of being converted.
-A `VALUES` list of more than 1,000 rows is refused for now.
-`--session NAME=VALUE` passes the caller's Snowflake session parameters
-(`function` takes it too):
+listed with an explanation each in `crates/sqlquack-core/src/diagnostics/constants.rs` and in the
+[diagnostics reference](https://sqlquack.io/docs/diagnostics/reference/). A code is never reused or
+renumbered.
+
+### Sessions
+
+`--session NAME=VALUE` passes the caller's Snowflake session parameters (`function` takes it too):
- Conversion runs under the recorded session profile they match (`utc-v1`, or `utc-qiic-v1` for
`QUOTED_IDENTIFIERS_IGNORE_CASE = true`) and refuses, naming the parameters, when they match none.
@@ -103,23 +152,24 @@ A `VALUES` list of more than 1,000 rows is refused for now.
- `TIMEZONE = UTC`, or no `TIMEZONE`, leaves time-zone-dependent statements to the
`session_timezone_utc` assumption. Any other `TIMEZONE` refuses them, even with `--utc-session`.
-`--strict`, `--utc-session` and `--json` work as for the census. `census` and `index` take no
-session parameters yet and convert under `utc-v1`.
+`census` and `index` take no session parameters yet and convert under `utc-v1`.
+
+### Policy
`convert`, `function`, `census` and `index` read `sqlquack.toml` (or `--policy`); without one they
use the built-in policy, which refuses every known divergence, as `--builtin-policy` asks for.
-They share the policy flags (`--policy`, `--builtin-policy`, `--strict`, `--utc-session`,
+They share the policy flags: `--policy`, `--builtin-policy`, `--strict`, `--utc-session`,
`--confirm-declared-offsets`, `--python-runtime` and `--loud-divergences-only`, which relies only
-on divergences whose converted SQL raises rather than returns another value). Every command with a report takes
-`--format text|json`; `--json` is short for `--format json`.
+on divergences whose converted SQL raises rather than returns another value. Every command with a
+report takes `--format text|json`; `--json` is short for `--format json`.
HASH is refused by default: the converted SQL computes DuckDB's hash, which is consistent within one
run but never Snowflake's value. A project that relies on HASH only for joins, grouping,
deduplication or change detection can opt in. A policy file is complete on its own, so start from
this repository's [sqlquack.toml](sqlquack.toml): a file holding only `[divergences.hash]` is
rejected, because the comparator, session profiles and edge-case tables are required. Copy it into
-the directory you run `sqlquack` from, and change `default` in its existing `[divergences.hash]`
-table from `"refuse"` to `"warn"`:
+the directory you run `sqlquack` from, and change `default` in its `[divergences.hash]` table from
+`"refuse"` to `"warn"`:
```console
$ cp path/to/sqlquack/sqlquack.toml .
@@ -131,7 +181,9 @@ default = "warn"
strict = "refuse"
```
-A divergence the file leaves out is refused.
+A divergence the file leaves out is refused. See [Policy and sessions](https://sqlquack.io/docs/policy/).
+
+### Functions and exit status
`function` converts a SQL project function's body, read from a file or from standard input with
`-`, and prints the macro as JSON. A refused body prints `{"converted": false, "name": …,
@@ -149,29 +201,33 @@ $ cargo run --release -p sqlquack -- census path/to/sql
$ cargo run --release -p sqlquack -- census path/to/sql --json
```
-Each `.sql` file is one query, or one model compiled by
-[SQLBuild](https://github.com/sqlstack-io/sqlbuild); SQLBuild's reference calls are recognised. The
-report contains counts and construct names only, never SQL text or file names; `--names` adds
-file, model and relation names to the slowest files and the blockers. Conversion needs the
-column types of the relations a file reads: `--schema` takes a JSON file mapping relation names to
-`[column, Snowflake type]` pairs, and `--chain-models` types each converted file as the relation
-its file name references, then converts again. Where the schema file types a model that is itself
-refused, files reading it still convert from those columns: the report counts them as
-unit-testable (`convertible`) but not strict (`convertible_strict`, every model upstream converts
-too), and lists each with an `upstream_refused:` label for every refused model upstream,
-or `upstream_missing:` for a referenced model the scan holds no file for. Both counts are
-repeated without named assumptions (`convertible_no_assumptions`,
-`convertible_strict_no_assumptions`): a file relying on one, such as `session_timezone_utc` or
-`source_variant_json`, drops out, and a strict file also does when a model upstream relies on one.
-`--cursor-type timestamp|integer` replaces SQLBuild's cursor intrinsics with a bound as SQLBuild's
-Snowflake adapter renders it (timestamp bounds as text, integer bounds as integers); without it
-they stay unknown.
-`--assume-supported` (with `--chain-models`) takes construct keys or `rule.`s and reports how
-many files would then be unit-testable and strict, what blocks the rest next, and a greedy plan of
-the few constructs each step should support together. Assumptions match at construct level: a
-`rule.` expands to its construct keys, so it also clears other rules' refusals of the same
-construct, such as every refused `node:cast`. Plan steps support only missing rules and rules' type
-conditions. Every such figure is an upper bound: it counts only the refusals collected, matches
+`census` reports which constructs a body of SQL uses and how much of it would convert as rules are
+added. Each `.sql` file is one query, or one model compiled by
+[SQLBuild](https://github.com/sqlstack-io/sqlbuild), whose reference calls are recognised. The
+report holds counts and construct names only, never SQL text or file names; `--names` adds file,
+model and relation names to the slowest files and the blockers.
+
+- `--schema` takes a JSON file mapping relation names to `[column, Snowflake type]` pairs.
+ `--chain-models` types each converted file as the relation its file name references, then
+ converts again.
+- Where the schema types a model that is itself refused, files reading it still convert from those
+ columns. The report counts them as unit-testable (`convertible`) but not strict
+ (`convertible_strict`, every model upstream converts too), and labels each
+ `upstream_refused:`, or `upstream_missing:` for a referenced model the scan holds
+ no file for.
+- Both counts are repeated without named assumptions (`convertible_no_assumptions`,
+ `convertible_strict_no_assumptions`): a file relying on one, such as `session_timezone_utc` or
+ `source_variant_json`, drops out, and a strict file also does when a model upstream relies on one.
+- `--cursor-type timestamp|integer` replaces SQLBuild's cursor intrinsics with a bound as SQLBuild's
+ Snowflake adapter renders it (timestamp bounds as text, integer bounds as integers); without it
+ they stay unknown.
+- `--assume-supported` (with `--chain-models`) takes construct keys or `rule.`s and reports how
+ many files would then be unit-testable and strict, what blocks the rest next, and a greedy plan of
+ the few constructs each step should support together. Assumptions match at construct level: a
+ `rule.` expands to its construct keys, so it also clears other rules' refusals of the same
+ construct. Plan steps support only missing rules and rules' type conditions.
+
+Every `--assume-supported` figure is an upper bound: it counts only the refusals collected, matches
constructs rather than the exact condition refused, and supporting a construct may uncover others.
Each figure says how many of its unit-testable files may hold refusals collecting did not reach, and
a key matching no collected refusal is warned about.
@@ -200,11 +256,11 @@ result = sqlquack.convert(
)
```
-The `sqlquack` package (in `python/`, built with `make python-wheel`) needs Python 3.12 or newer and
-runs the same code as the command line. `check` and `convert` return frozen results with typed
-`Diagnostic`s, the text the command line renders, and, for `convert`, its exact `--json` report.
-`ConvertOptions` has one field per `convert` flag. Where the command line exits 2, they raise
-`SqlQuackError`.
+The `sqlquack` Python package lives in `python/`. Build it from source with `make python-wheel`; it
+needs Python 3.12 or newer, runs the same code as the command line and ships the DuckDB library
+beside itself. `check` and `convert` return frozen results with typed `Diagnostic`s, the text the
+command line renders, and, for `convert`, its exact `--json` report. `ConvertOptions` has one field
+per `convert` flag. Where the command line exits 2, they raise `SqlQuackError`.
## Evidence
@@ -254,6 +310,9 @@ only ever holds a tree that passed.
conversion over pinned benchmark projects, which it downloads on first use, and a fuzz smoke run,
which needs a nightly toolchain with `cargo-fuzz`.
+The README demo is recorded from a real run with the tapes in
+[website/recordings](website/recordings/README.md).
+
## License
Apache-2.0.
diff --git a/website/astro.config.mjs b/website/astro.config.mjs
index 5fb1660fd..3c107b7d6 100644
--- a/website/astro.config.mjs
+++ b/website/astro.config.mjs
@@ -21,11 +21,11 @@ export default defineConfig({
},
favicon: '/favicon.png',
head: [
- { tag: 'meta', attrs: { property: 'og:image', content: 'https://sqlquack.io/assets/og-sqlquack.png' } },
+ { tag: 'meta', attrs: { property: 'og:image', content: 'https://sqlquack.io/assets/og-sqlquack-v2.png' } },
{ tag: 'meta', attrs: { property: 'og:image:width', content: '1200' } },
{ tag: 'meta', attrs: { property: 'og:image:height', content: '630' } },
{ tag: 'meta', attrs: { property: 'og:image:alt', content: 'SQLQuack: a warehouse SQL compiler that translates to DuckDB, or refuses to guess.' } },
- { tag: 'meta', attrs: { name: 'twitter:image', content: 'https://sqlquack.io/assets/og-sqlquack.png' } },
+ { tag: 'meta', attrs: { name: 'twitter:image', content: 'https://sqlquack.io/assets/og-sqlquack-v2.png' } },
],
social: [
{ icon: 'github', label: 'GitHub', href: 'https://github.com/sqlstack-io/sqlquack' },
diff --git a/website/public/assets/og-sqlquack.png b/website/public/assets/og-sqlquack-v2.png
similarity index 100%
rename from website/public/assets/og-sqlquack.png
rename to website/public/assets/og-sqlquack-v2.png
diff --git a/website/public/index.html b/website/public/index.html
index 58e3facfe..5e771a02b 100644
--- a/website/public/index.html
+++ b/website/public/index.html
@@ -11,12 +11,12 @@
-
+
-
+
diff --git a/website/public/playground/index.html b/website/public/playground/index.html
index 68be097ac..387a8e4f1 100644
--- a/website/public/playground/index.html
+++ b/website/public/playground/index.html
@@ -11,12 +11,12 @@
-
+
-
+
diff --git a/website/recordings/.gitignore b/website/recordings/.gitignore
new file mode 100644
index 000000000..208267062
--- /dev/null
+++ b/website/recordings/.gitignore
@@ -0,0 +1,2 @@
+scratch/
+out/
diff --git a/website/recordings/README.md b/website/recordings/README.md
new file mode 100644
index 000000000..6dbf3347b
--- /dev/null
+++ b/website/recordings/README.md
@@ -0,0 +1,33 @@
+# CLI recordings
+
+Terminal recordings of `sqlquack` for the README, rendered with
+[VHS](https://github.com/charmbracelet/vhs) from the `.tape` scripts in `tapes/`. The tapes and
+scripts are the source; the rendered GIF and PNG files go to `out/`, which is not committed.
+
+| Demo | Shows |
+|------|-------|
+| `check-convert` | `sqlquack check` reports a QUALIFY clause Snowflake rejects, `sqlquack convert` converts a query to DuckDB SQL, and refuses one that needs `HASH`. |
+
+The README uses `check-convert`, copied into `.github/demos/`.
+
+## Requirements
+
+- `vhs`, `ttyd` and `ffmpeg` on `PATH`, plus a Chromium or Chrome that VHS can find.
+- DejaVu Sans Mono, the font the tapes use.
+- `sqlquack` built with `cargo build --release -p sqlquack`, or `SQLQUACK_BIN` pointing at a
+ directory that contains the `sqlquack` to record. `record.sh` puts the DuckDB library the build
+ downloaded on the library path.
+
+## Record
+
+```bash
+cd website/recordings
+./record.sh check-convert # out/check-convert.gif and out/check-convert.png
+cp out/check-convert.gif ../../.github/demos/
+```
+
+`record.sh` runs `prepare.sh ` to write the demo's SQL and schema under `scratch//`,
+then renders `tapes/.tape`. Shared terminal settings and the colour theme are in
+`tapes/common.tape`.
+
+Look at every recording before publishing it.
diff --git a/website/recordings/prepare.sh b/website/recordings/prepare.sh
new file mode 100755
index 000000000..bc50a8105
--- /dev/null
+++ b/website/recordings/prepare.sh
@@ -0,0 +1,46 @@
+#!/usr/bin/env bash
+# Build the scratch directory for one demo. Usage: prepare.sh
+set -euo pipefail
+demo="$1"
+here="$(cd "$(dirname "$0")" && pwd)"
+dir="$here/scratch/$demo"
+rm -rf "$dir"
+mkdir -p "$dir"
+cd "$dir"
+case "$demo" in
+ check-convert)
+ # A query Snowflake rejects, one that converts, and one the built-in policy refuses.
+ cat > orders.sql <<'SQL'
+SELECT customer_id, SUM(total) AS revenue
+FROM orders
+WHERE status = 'shipped'
+GROUP BY customer_id
+QUALIFY revenue > 100
+SQL
+ cat > revenue.sql <<'SQL'
+SELECT customer_id, SUM(total) AS revenue
+FROM orders
+WHERE status = 'shipped'
+GROUP BY customer_id
+HAVING SUM(total) > 100
+SQL
+ cat > bucket.sql <<'SQL'
+SELECT order_id,
+ HASH(customer_id) AS bucket
+FROM orders
+SQL
+ cat > schema.json <<'JSON'
+{
+ "ORDERS": [
+ ["ORDER_ID", "NUMBER(38,0)"],
+ ["CUSTOMER_ID", "NUMBER(38,0)"],
+ ["STATUS", "VARCHAR"],
+ ["TOTAL", "NUMBER(12,2)"]
+ ]
+}
+JSON
+ ;;
+ *)
+ echo "unknown demo: $demo" >&2
+ exit 2 ;;
+esac
diff --git a/website/recordings/record.sh b/website/recordings/record.sh
new file mode 100755
index 000000000..873d2dccd
--- /dev/null
+++ b/website/recordings/record.sh
@@ -0,0 +1,18 @@
+#!/usr/bin/env bash
+# Record one demo to out/.gif and out/.png. Usage: record.sh
+# Uses target/release/sqlquack; set SQLQUACK_BIN to a directory containing another sqlquack to use that.
+set -euo pipefail
+demo="$1"
+here="$(cd "$(dirname "$0")" && pwd)"
+repo="$(cd "$here/../.." && pwd)"
+bin="${SQLQUACK_BIN:-$repo/target/release}"
+export PATH="$bin:$PATH"
+# The binary links the DuckDB shared library the build downloads beside its profile directory.
+lib="$(dirname "$(find "$bin/../duckdb-download" -name 'libduckdb.*' | sort | tail -1)")"
+export LD_LIBRARY_PATH="$lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
+export DYLD_LIBRARY_PATH="$lib${DYLD_LIBRARY_PATH:+:$DYLD_LIBRARY_PATH}"
+"$here/prepare.sh" "$demo"
+mkdir -p "$here/out"
+cd "$here"
+# Headless Chromium's sandbox is unavailable in many containers and CI runners.
+VHS_NO_SANDBOX="${VHS_NO_SANDBOX:-true}" vhs "tapes/$demo.tape"
diff --git a/website/recordings/tapes/check-convert.tape b/website/recordings/tapes/check-convert.tape
new file mode 100644
index 000000000..5cb8b9c69
--- /dev/null
+++ b/website/recordings/tapes/check-convert.tape
@@ -0,0 +1,33 @@
+Source tapes/common.tape
+Set Height 600
+Output out/check-convert.gif
+Hide
+Type "cd scratch/check-convert && export PS1='$ ' && clear"
+Enter
+Sleep 500ms
+Show
+Type "cat orders.sql"
+Enter
+Sleep 1500ms
+Type "sqlquack check orders.sql"
+Sleep 300ms
+Enter
+Wait@60s /\$\s*$/
+Sleep 5s
+Hide
+Type "clear"
+Enter
+Sleep 300ms
+Show
+Type "sqlquack convert revenue.sql --schema schema.json"
+Sleep 300ms
+Enter
+Wait@60s /\$\s*$/
+Sleep 4s
+Type "sqlquack convert bucket.sql --schema schema.json"
+Sleep 300ms
+Enter
+Wait@60s /\$\s*$/
+Sleep 1s
+Screenshot out/check-convert.png
+Sleep 5s
diff --git a/website/recordings/tapes/common.tape b/website/recordings/tapes/common.tape
new file mode 100644
index 000000000..f551881ae
--- /dev/null
+++ b/website/recordings/tapes/common.tape
@@ -0,0 +1,13 @@
+Set Shell bash
+Set FontFamily "DejaVu Sans Mono"
+Set FontSize 17
+Set LineHeight 1.1
+Set Width 1360
+Set Height 680
+Set Padding 28
+Set Margin 24
+Set MarginFill "#0d0f12"
+Set BorderRadius 10
+Set WindowBar Colorful
+Set TypingSpeed 45ms
+Set Theme { "name": "SQLQuack", "black": "#626975", "red": "#f47089", "green": "#54cf8e", "yellow": "#ffe000", "blue": "#4f98ee", "magenta": "#a78bfa", "cyan": "#1fe0c0", "white": "#c9cdd4", "brightBlack": "#9ba1ac", "brightRed": "#f47089", "brightGreen": "#54cf8e", "brightYellow": "#fff06a", "brightBlue": "#4f98ee", "brightMagenta": "#c79bf0", "brightCyan": "#1fe0c0", "brightWhite": "#e9ebef", "background": "#111418", "foreground": "#c9cdd4", "selection": "#2a2f37", "cursor": "#e9ebef" }