diff --git a/.github/demos/contract.gif b/.github/demos/contract.gif new file mode 100644 index 0000000000..510a6b65fd Binary files /dev/null and b/.github/demos/contract.gif differ diff --git a/.github/demos/rename.gif b/.github/demos/rename.gif index 1090d4697b..dbafd7f81b 100644 Binary files a/.github/demos/rename.gif and b/.github/demos/rename.gif differ diff --git a/.github/demos/rules.gif b/.github/demos/rules.gif new file mode 100644 index 0000000000..f3bca77c61 Binary files /dev/null and b/.github/demos/rules.gif differ diff --git a/.github/demos/scope.gif b/.github/demos/scope.gif index 358013cdf7..54e43a126a 100644 Binary files a/.github/demos/scope.gif and b/.github/demos/scope.gif differ diff --git a/README.md b/README.md index 8937a7597a..d73717e728 100644 --- a/README.md +++ b/README.md @@ -22,6 +22,9 @@ manifest files and no paid tier. Moving and renaming an incremental model: sqb plan migrates the existing table instead of rebuilding it

+Move an incremental model to a new folder and name, and `sqb plan` migrates its table instead of +rebuilding it. + ## Quick start ```bash @@ -71,12 +74,47 @@ The playground runs on local DuckDB, with no warehouse credentials. - **Tidy up safely.** The [janitor](https://sqlbuild.com/docs/cli/janitor/) archives stale tables before anything is deleted. +Ingestion with Python loaders, and Python tasks, assets and checks, run in the same graph as your SQL +models. See the [docs](https://sqlbuild.com/docs/) for everything else. + +## In the terminal + +Each demo runs on local DuckDB with the projects in [`website/examples`](website/examples). The +tapes that record them are in [`website/recordings`](website/recordings). + +### A renamed column breaks the contract + +`daily_revenue` declares `contract enforced`. Renaming `waffles_sold` to `units_sold` in the +`SELECT` fails at compile time, before anything reaches the warehouse. +

- sqb scope previews a move: the enum and macro the model uses would be lost, so the move is flagged before anything breaks + sqb compile fails because units_sold is not in the enforced contract and the declared waffles_sold column is missing

-Ingestion with Python loaders, and Python tasks, assets and checks, run in the same graph as your SQL -models. See the [docs](https://sqlbuild.com/docs/) for everything else. +Both sides are reported: the new column isn't in the contract, and the declared one is gone. + +### Your own conventions as compile errors + +The project's rule says marts must read sources through staging. A mart that reads +`__source("raw__payments")` directly fails, and so does the unit test that has no mock for it. + +

+ sqb compile reports the custom rule XSQBRARCH001 on the line that reads a raw source, and a unit test with no mock for that source +

+ +The rule is [a short Python function](website/examples/waffle-shop/rules/layers.py). See +[rules](https://sqlbuild.com/docs/concepts/rules/). + +### See what a move would break + +`sqb scope --as-path` previews moving a model before you move it. + +

+ sqb scope previews moving daily_revenue: the enum and macro it uses would be lost, so both usages are invalidated +

+ +Look at `Lost` and `Invalidated usages`: the model uses an enum and a macro that are private to +`models/marts`, so moving it would break both. ## Example diff --git a/website/recordings/README.md b/website/recordings/README.md index e2949ccd10..e1c0a64dca 100644 --- a/website/recordings/README.md +++ b/website/recordings/README.md @@ -7,18 +7,21 @@ scripts are the source; the rendered GIF, MP4 and PNG files go to `out/`, which | Demo | Shows | |------|-------| | `rename` | A model is moved and renamed, and `sqb plan` migrates the existing table. | -| `compile` | `sqb compile` catches an unknown column and a type mismatch. | +| `contract` | A renamed column fails `sqb compile` against an enforced contract. | +| `rules` | A custom rule and a missing test mock fail `sqb compile`. | | `scope` | `sqb scope --as-path` previews what a move would lose. | | `janitor` | `sqb janitor` archives a stale table before anything is deleted. | -| `quickstart` | The playground: `sqb playground`, `sqb build` and `sqb test`. | +| `compile` | `sqb compile` catches an unknown column and a type mismatch. Not used yet: the column checks it shows are unreleased. | -The README uses `rename.gif` and `scope.gif`, copied into `.github/demos/`. Each demo runs locally on DuckDB, using the projects in [`../examples`](../examples). +The README uses `rename`, `contract`, `rules` and `scope`, copied into `.github/demos/`. Each demo +runs locally on DuckDB, using the projects in [`../examples`](../examples). ## Requirements - `vhs`, `ttyd` and `ffmpeg` on `PATH`, plus a Chromium or Chrome that VHS can find (it downloads one if none is on `PATH`). - DejaVu Sans Mono, the font the tapes use. +- `tree` (the rename and scope tapes show folder trees). - `sqb` on `PATH`, or `SQB_BIN` pointing at a directory that contains the `sqb` to record. ## Record diff --git a/website/recordings/prepare.sh b/website/recordings/prepare.sh index c34af34b48..f5a9ebdcdd 100755 --- a/website/recordings/prepare.sh +++ b/website/recordings/prepare.sh @@ -8,16 +8,41 @@ dir="$here/scratch/$demo" rm -rf "$dir" mkdir -p "$here/scratch" case "$demo" in - quickstart) - mkdir -p "$dir" ;; rename) - # Build the model under its old name, so the tape can move and rename it. + # Build the model under its old name, so the tape can move it back to models/finance. cp -r "$examples/hero-shop" "$dir" cd "$dir" mkdir -p models/marts mv models/finance/daily_revenue.sql models/marts/revenue.sql - rmdir models/finance sqb build >/dev/null ;; + contract) + # Give daily_revenue an enforced contract, then rename one of its columns in the SELECT. + cp -r "$examples/waffle-shop" "$dir" + cd "$dir" + python3 - <<'PY' +path = "models/marts/daily_revenue.sql" +text = open(path).read() +text = text.replace(" materialized table,\n", " materialized table,\n contract enforced,\n", 1) +text = text.replace( + " revenue_date (nullable false),\n", + " revenue_date (type DATE, nullable false),\n" + ' order_count (description "Orders placed that day"),\n' + ' waffles_sold (description "Waffles sold that day"),\n' + ' total_revenue_cents (description "Successful payments in cents"),\n' + ' total_revenue_dollars (description "Successful payments in dollars"),\n' + ' avg_order_value_cents (description "Average order value in cents"),\n', + 1, +) +text = text.replace("AS waffles_sold", "AS units_sold", 1) +open(path, "w").write(text) +PY + ;; + rules) + # Make a mart read a raw source directly, which the project's layering rule forbids. + cp -r "$examples/waffle-shop" "$dir" + cd "$dir" + sed -i 's/LEFT JOIN __ref("stg_payments") p/LEFT JOIN __source("raw__payments") p/' \ + models/marts/fact_orders.sql ;; scope) cp -r "$examples/waffle-shop" "$dir" cd "$dir" diff --git a/website/recordings/tapes/contract.tape b/website/recordings/tapes/contract.tape new file mode 100644 index 0000000000..00bc326e95 --- /dev/null +++ b/website/recordings/tapes/contract.tape @@ -0,0 +1,18 @@ +Source tapes/common.tape +Set FontSize 16 +Set Width 1600 +Set Height 820 +Output out/contract.gif +Output out/contract.mp4 +Hide +Type "cd scratch/contract && export PS1='$ ' && clear" +Enter +Sleep 500ms +Show +Type "sqb compile --select daily_revenue" +Sleep 300ms +Enter +Wait@60s /\$\s*$/ +Sleep 1s +Screenshot out/contract.png +Sleep 15s diff --git a/website/recordings/tapes/quickstart.tape b/website/recordings/tapes/quickstart.tape deleted file mode 100644 index 70f9d9d2e0..0000000000 --- a/website/recordings/tapes/quickstart.tape +++ /dev/null @@ -1,25 +0,0 @@ -Source tapes/common.tape -Output out/quickstart.gif -Output out/quickstart.mp4 -Hide -Type "cd scratch/quickstart && export PS1='$ ' && clear" -Enter -Sleep 500ms -Show -Type "sqb playground waffle-shop" -Sleep 300ms -Enter -Sleep 2s -Type "cd waffle-shop" -Enter -Sleep 500ms -Type "sqb build" -Sleep 300ms -Enter -Sleep 8s -Type "sqb test" -Sleep 300ms -Enter -Sleep 6s -Screenshot out/quickstart.png -Sleep 2s diff --git a/website/recordings/tapes/rename.tape b/website/recordings/tapes/rename.tape index 9d2e1bbbd8..e93740995c 100644 --- a/website/recordings/tapes/rename.tape +++ b/website/recordings/tapes/rename.tape @@ -1,4 +1,5 @@ Source tapes/common.tape +Set Height 800 Output out/rename.gif Output out/rename.mp4 Hide @@ -6,16 +7,17 @@ Type "cd scratch/rename && export PS1='$ ' && clear" Enter Sleep 500ms Show -Type "find models -name '*.sql'" +Type "tree --noreport models" Enter Sleep 1500ms -Type "mkdir models/finance && mv models/marts/revenue.sql models/finance/daily_revenue.sql" +Type "mv models/marts/revenue.sql models/finance/daily_revenue.sql" Sleep 300ms Enter Sleep 800ms Type "sqb plan" Sleep 300ms Enter -Sleep 5s +Wait@60s /\$\s*$/ +Sleep 1s Screenshot out/rename.png -Sleep 3s +Sleep 10s diff --git a/website/recordings/tapes/rules.tape b/website/recordings/tapes/rules.tape new file mode 100644 index 0000000000..d1528a8a1e --- /dev/null +++ b/website/recordings/tapes/rules.tape @@ -0,0 +1,18 @@ +Source tapes/common.tape +Set FontSize 16 +Set Width 1480 +Set Height 760 +Output out/rules.gif +Output out/rules.mp4 +Hide +Type "cd scratch/rules && export PS1='$ ' && clear" +Enter +Sleep 500ms +Show +Type "sqb compile --select fact_orders" +Sleep 300ms +Enter +Wait@60s /\$\s*$/ +Sleep 1s +Screenshot out/rules.png +Sleep 15s diff --git a/website/recordings/tapes/scope.tape b/website/recordings/tapes/scope.tape index bba9aba046..2075782d13 100644 --- a/website/recordings/tapes/scope.tape +++ b/website/recordings/tapes/scope.tape @@ -1,7 +1,7 @@ Source tapes/common.tape Set FontSize 16 Set Width 1400 -Set Height 1100 +Set Height 1180 Output out/scope.gif Output out/scope.mp4 Hide @@ -9,12 +9,13 @@ Type "cd scratch/scope && export PS1='$ ' && clear" Enter Sleep 500ms Show -Type "find models/marts/_sqlbuild models/marts/daily_revenue.sql -type f" +Type "tree --noreport models/marts --prune -P 'daily_revenue.sql|payment_status.sql|*.py'" Enter Sleep 1500ms Type "sqb scope model:daily_revenue --as-path models/intermediate/daily_revenue.sql" Sleep 300ms Enter -Sleep 5s +Wait@60s /\$\s*$/ +Sleep 1s Screenshot out/scope.png -Sleep 3s +Sleep 15s