Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added .github/demos/contract.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified .github/demos/rename.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added .github/demos/rules.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified .github/demos/scope.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
44 changes: 41 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,9 @@ manifest files and no paid tier.
<img src="https://raw.githubusercontent.com/chio-labs/sqlbuild/main/.github/demos/rename.gif" alt="Moving and renaming an incremental model: sqb plan migrates the existing table instead of rebuilding it" width="100%">
</p>

Move an incremental model to a new folder and name, and `sqb plan` migrates its table instead of
rebuilding it.

## Quick start

```bash
Expand Down Expand Up @@ -71,12 +74,47 @@ The playground runs on local DuckDB, with no warehouse credentials.
- **Tidy up safely.** The [janitor](https://sqlbuild.com/docs/cli/janitor/) archives stale tables
before anything is deleted.

Ingestion with Python loaders, and Python tasks, assets and checks, run in the same graph as your SQL
models. See the [docs](https://sqlbuild.com/docs/) for everything else.

## In the terminal

Each demo runs on local DuckDB with the projects in [`website/examples`](website/examples). The
tapes that record them are in [`website/recordings`](website/recordings).

### A renamed column breaks the contract

`daily_revenue` declares `contract enforced`. Renaming `waffles_sold` to `units_sold` in the
`SELECT` fails at compile time, before anything reaches the warehouse.

<p align="center">
<img src="https://raw.githubusercontent.com/chio-labs/sqlbuild/main/.github/demos/scope.gif" alt="sqb scope previews a move: the enum and macro the model uses would be lost, so the move is flagged before anything breaks" width="100%">
<img src="https://raw.githubusercontent.com/chio-labs/sqlbuild/main/.github/demos/contract.gif" alt="sqb compile fails because units_sold is not in the enforced contract and the declared waffles_sold column is missing" width="100%">
</p>

Ingestion with Python loaders, and Python tasks, assets and checks, run in the same graph as your SQL
models. See the [docs](https://sqlbuild.com/docs/) for everything else.
Both sides are reported: the new column isn't in the contract, and the declared one is gone.

### Your own conventions as compile errors

The project's rule says marts must read sources through staging. A mart that reads
`__source("raw__payments")` directly fails, and so does the unit test that has no mock for it.

<p align="center">
<img src="https://raw.githubusercontent.com/chio-labs/sqlbuild/main/.github/demos/rules.gif" alt="sqb compile reports the custom rule XSQBRARCH001 on the line that reads a raw source, and a unit test with no mock for that source" width="100%">
</p>

The rule is [a short Python function](website/examples/waffle-shop/rules/layers.py). See
[rules](https://sqlbuild.com/docs/concepts/rules/).

### See what a move would break

`sqb scope --as-path` previews moving a model before you move it.

<p align="center">
<img src="https://raw.githubusercontent.com/chio-labs/sqlbuild/main/.github/demos/scope.gif" alt="sqb scope previews moving daily_revenue: the enum and macro it uses would be lost, so both usages are invalidated" width="100%">
</p>

Look at `Lost` and `Invalidated usages`: the model uses an enum and a macro that are private to
`models/marts`, so moving it would break both.

## Example

Expand Down
9 changes: 6 additions & 3 deletions website/recordings/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,18 +7,21 @@ scripts are the source; the rendered GIF, MP4 and PNG files go to `out/`, which
| Demo | Shows |
|------|-------|
| `rename` | A model is moved and renamed, and `sqb plan` migrates the existing table. |
| `compile` | `sqb compile` catches an unknown column and a type mismatch. |
| `contract` | A renamed column fails `sqb compile` against an enforced contract. |
| `rules` | A custom rule and a missing test mock fail `sqb compile`. |
| `scope` | `sqb scope --as-path` previews what a move would lose. |
| `janitor` | `sqb janitor` archives a stale table before anything is deleted. |
| `quickstart` | The playground: `sqb playground`, `sqb build` and `sqb test`. |
| `compile` | `sqb compile` catches an unknown column and a type mismatch. Not used yet: the column checks it shows are unreleased. |

The README uses `rename.gif` and `scope.gif`, copied into `.github/demos/`. Each demo runs locally on DuckDB, using the projects in [`../examples`](../examples).
The README uses `rename`, `contract`, `rules` and `scope`, copied into `.github/demos/`. Each demo
runs locally on DuckDB, using the projects in [`../examples`](../examples).

## Requirements

- `vhs`, `ttyd` and `ffmpeg` on `PATH`, plus a Chromium or Chrome that VHS can find (it downloads
one if none is on `PATH`).
- DejaVu Sans Mono, the font the tapes use.
- `tree` (the rename and scope tapes show folder trees).
- `sqb` on `PATH`, or `SQB_BIN` pointing at a directory that contains the `sqb` to record.

## Record
Expand Down
33 changes: 29 additions & 4 deletions website/recordings/prepare.sh
Original file line number Diff line number Diff line change
Expand Up @@ -8,16 +8,41 @@ dir="$here/scratch/$demo"
rm -rf "$dir"
mkdir -p "$here/scratch"
case "$demo" in
quickstart)
mkdir -p "$dir" ;;
rename)
# Build the model under its old name, so the tape can move and rename it.
# Build the model under its old name, so the tape can move it back to models/finance.
cp -r "$examples/hero-shop" "$dir"
cd "$dir"
mkdir -p models/marts
mv models/finance/daily_revenue.sql models/marts/revenue.sql
rmdir models/finance
sqb build >/dev/null ;;
contract)
# Give daily_revenue an enforced contract, then rename one of its columns in the SELECT.
cp -r "$examples/waffle-shop" "$dir"
cd "$dir"
python3 - <<'PY'
path = "models/marts/daily_revenue.sql"
text = open(path).read()
text = text.replace(" materialized table,\n", " materialized table,\n contract enforced,\n", 1)
text = text.replace(
" revenue_date (nullable false),\n",
" revenue_date (type DATE, nullable false),\n"
' order_count (description "Orders placed that day"),\n'
' waffles_sold (description "Waffles sold that day"),\n'
' total_revenue_cents (description "Successful payments in cents"),\n'
' total_revenue_dollars (description "Successful payments in dollars"),\n'
' avg_order_value_cents (description "Average order value in cents"),\n',
1,
)
text = text.replace("AS waffles_sold", "AS units_sold", 1)
open(path, "w").write(text)
PY
;;
rules)
# Make a mart read a raw source directly, which the project's layering rule forbids.
cp -r "$examples/waffle-shop" "$dir"
cd "$dir"
sed -i 's/LEFT JOIN __ref("stg_payments") p/LEFT JOIN __source("raw__payments") p/' \
models/marts/fact_orders.sql ;;
scope)
cp -r "$examples/waffle-shop" "$dir"
cd "$dir"
Expand Down
18 changes: 18 additions & 0 deletions website/recordings/tapes/contract.tape
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
Source tapes/common.tape
Set FontSize 16
Set Width 1600
Set Height 820
Output out/contract.gif
Output out/contract.mp4
Hide
Type "cd scratch/contract && export PS1='$ ' && clear"
Enter
Sleep 500ms
Show
Type "sqb compile --select daily_revenue"
Sleep 300ms
Enter
Wait@60s /\$\s*$/
Sleep 1s
Screenshot out/contract.png
Sleep 15s
25 changes: 0 additions & 25 deletions website/recordings/tapes/quickstart.tape

This file was deleted.

10 changes: 6 additions & 4 deletions website/recordings/tapes/rename.tape
Original file line number Diff line number Diff line change
@@ -1,21 +1,23 @@
Source tapes/common.tape
Set Height 800
Output out/rename.gif
Output out/rename.mp4
Hide
Type "cd scratch/rename && export PS1='$ ' && clear"
Enter
Sleep 500ms
Show
Type "find models -name '*.sql'"
Type "tree --noreport models"
Enter
Sleep 1500ms
Type "mkdir models/finance && mv models/marts/revenue.sql models/finance/daily_revenue.sql"
Type "mv models/marts/revenue.sql models/finance/daily_revenue.sql"
Sleep 300ms
Enter
Sleep 800ms
Type "sqb plan"
Sleep 300ms
Enter
Sleep 5s
Wait@60s /\$\s*$/
Sleep 1s
Screenshot out/rename.png
Sleep 3s
Sleep 10s
18 changes: 18 additions & 0 deletions website/recordings/tapes/rules.tape
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
Source tapes/common.tape
Set FontSize 16
Set Width 1480
Set Height 760
Output out/rules.gif
Output out/rules.mp4
Hide
Type "cd scratch/rules && export PS1='$ ' && clear"
Enter
Sleep 500ms
Show
Type "sqb compile --select fact_orders"
Sleep 300ms
Enter
Wait@60s /\$\s*$/
Sleep 1s
Screenshot out/rules.png
Sleep 15s
9 changes: 5 additions & 4 deletions website/recordings/tapes/scope.tape
Original file line number Diff line number Diff line change
@@ -1,20 +1,21 @@
Source tapes/common.tape
Set FontSize 16
Set Width 1400
Set Height 1100
Set Height 1180
Output out/scope.gif
Output out/scope.mp4
Hide
Type "cd scratch/scope && export PS1='$ ' && clear"
Enter
Sleep 500ms
Show
Type "find models/marts/_sqlbuild models/marts/daily_revenue.sql -type f"
Type "tree --noreport models/marts --prune -P 'daily_revenue.sql|payment_status.sql|*.py'"
Enter
Sleep 1500ms
Type "sqb scope model:daily_revenue --as-path models/intermediate/daily_revenue.sql"
Sleep 300ms
Enter
Sleep 5s
Wait@60s /\$\s*$/
Sleep 1s
Screenshot out/scope.png
Sleep 3s
Sleep 15s
Loading