Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 14 additions & 14 deletions website/public/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ <h1>The refactorable warehouse</h1>
<span class="dim">└──</span> revenue.sql</span></span><span class="hv-b"><span class="tree"><span class="dim">models</span>
<span class="dim">└──</span> finance
<span class="dim">└──</span> <span class="info">daily_revenue.sql</span></span></span></span>
<span class="hv-l hv-l1"><span class="cmd">sqb plan</span></span><span class="hv-l hv-l2">Migrations (1)</span><span class="hv-l hv-l3"><span class="tree"><span class="dim">└──</span></span> daily_revenue migrate<span class="hv-wide"> prod.revenue -&gt; prod.daily_revenue</span></span><span class="hv-l hv-l4"><span class="tree"> <span class="dim">├──</span></span> compatibility compatible</span><span class="hv-l hv-l5"><span class="tree"> <span class="dim">├──</span></span> transfer physical copy<span class="hv-wide">, promote by staged rename</span></span><span class="hv-l hv-l6"><span class="tree"> <span class="dim">└──</span></span> discovery automatic</span></div>
<span class="hv-l hv-l1"><span class="cmd">sqb plan</span></span><span class="hv-l hv-l2"><span class="strong">Migrations</span> <span class="dim">(1)</span></span><span class="hv-l hv-l3"><span class="tree"><span class="dim">└──</span></span> daily_revenue <span class="info strong">migrate</span><span class="hv-wide"> <span class="dim">prod.revenue -&gt;</span> prod.daily_revenue</span></span><span class="hv-l hv-l4"><span class="tree"> <span class="dim">├──</span></span> <span class="dim">compatibility</span> <span class="ok">compatible</span></span><span class="hv-l hv-l5"><span class="tree"> <span class="dim">├──</span></span> <span class="dim">transfer</span> physical copy<span class="hv-wide">, promote by staged rename</span></span><span class="hv-l hv-l6"><span class="tree"> <span class="dim">└──</span></span> <span class="dim">discovery</span> automatic</span></div>
</div>

<section>
Expand Down Expand Up @@ -202,14 +202,14 @@ <h3>Reorganise without breaking anything</h3>
│ └── datetime.py <span class="cm"># timestamp_trunc</span>
└── daily_revenue.sql <span class="cm"># uses payment_status and cents_to_dollars</span></span>
<span class="cmd">sqb scope model:daily_revenue --as-path models/intermediate/daily_revenue.sql</span>
Move preview
Resource: model:daily_revenue
Destination: models/intermediate/daily_revenue.sql
Lost (3)
├─ <span class="bad">●</span> enum:payment_status models/marts/_sqlbuild/_enums/payment_status.sql:1:1
├─ <span class="bad">●</span> macro:cents_to_dollars models/marts/_sqlbuild/_macros/currency.py:4:1
└─ ○ macro:timestamp_trunc models/marts/_sqlbuild/_macros/datetime.py:4:1
Invalidated usages (2)
<span class="strong">Move preview</span>
<span class="dim">Resource:</span> model:daily_revenue
<span class="dim">Destination:</span> models/intermediate/daily_revenue.sql
<span class="warn strong">Lost</span> <span class="warn">(3)</span>
<span class="dim">├─</span> <span class="bad">●</span> enum:payment_status <span class="dim">models/marts/_sqlbuild/_enums/payment_status.sql:1</span>
<span class="dim">├─</span> <span class="bad">●</span> macro:cents_to_dollars <span class="dim">models/marts/_sqlbuild/_macros/currency.py:4</span>
<span class="dim">└─ ○</span> macro:timestamp_trunc <span class="dim">models/marts/_sqlbuild/_macros/datetime.py:4</span>
<span class="bad strong">Invalidated usages</span> <span class="bad">(2)</span>
<span class="bad">- enum:payment_status</span>
<span class="bad">- macro:cents_to_dollars</span></pre>
</div>
Expand Down Expand Up @@ -326,11 +326,11 @@ <h3>Rename models without rebuilding history</h3>
</p>
</div>
<pre class="terminal"><span class="cmd">sqb plan</span>
Migrations (1)
<span class="dim">└──</span> daily_order_rollup migrate dev.daily_activity_rollup -&gt; dev.daily_order_rollup
<span class="dim">├──</span> compatibility compatible
<span class="dim">├──</span> transfer physical copy, promote by staged rename
<span class="dim">└──</span> discovery automatic</pre>
<span class="strong">Migrations</span> <span class="dim">(1)</span>
<span class="dim">└──</span> daily_order_rollup <span class="info strong">migrate</span> <span class="dim">dev.daily_activity_rollup -&gt;</span> dev.daily_order_rollup
<span class="dim">├── compatibility</span> <span class="ok">compatible</span>
<span class="dim">├── transfer</span> physical copy, promote by staged rename
<span class="dim">└── discovery</span> automatic</pre>
</div>
<div class="step">
<div class="step-text">
Expand Down
2 changes: 2 additions & 0 deletions website/recordings/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
out/
scratch/
36 changes: 36 additions & 0 deletions website/recordings/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# CLI recordings

Terminal recordings of `sqb` for the README and the website, rendered with
[VHS](https://github.com/charmbracelet/vhs) from the `.tape` scripts in `tapes/`. The tapes and
scripts are the source; the rendered GIF, MP4 and PNG files go to `out/`, which is not committed.

| Demo | Shows |
|------|-------|
| `rename` | A model is moved and renamed, and `sqb plan` migrates the existing table. |
| `compile` | `sqb compile` catches an unknown column and a type mismatch. |
| `scope` | `sqb scope --as-path` previews what a move would lose. |
| `janitor` | `sqb janitor` archives a stale table before anything is deleted. |
| `quickstart` | The playground: `sqb playground`, `sqb build` and `sqb test`. |

Each demo runs locally on DuckDB, using the projects in [`../examples`](../examples).

## Requirements

- `vhs`, `ttyd` and `ffmpeg` on `PATH`, plus a Chromium or Chrome that VHS can find (it downloads
one if none is on `PATH`).
- DejaVu Sans Mono, the font the tapes use.
- `sqb` on `PATH`, or `SQB_BIN` pointing at a directory that contains the `sqb` to record.

## Record

```bash
cd website/recordings
./record.sh rename # out/rename.gif, out/rename.mp4 and out/rename.png
```

`record.sh` runs `prepare.sh <demo>` to build a clean project under `scratch/<demo>/`, then renders
`tapes/<demo>.tape`. Shared terminal settings and the colour theme, which matches the website's
terminal blocks, are in `tapes/common.tape`.

Look at every recording before publishing it: output such as timings, run IDs and dates changes
between runs.
62 changes: 62 additions & 0 deletions website/recordings/prepare.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
#!/usr/bin/env bash
# Build a clean scratch project for one demo. Usage: prepare.sh <demo>
set -euo pipefail
demo="$1"
here="$(cd "$(dirname "$0")" && pwd)"
examples="$here/../examples"
dir="$here/scratch/$demo"
rm -rf "$dir"
mkdir -p "$here/scratch"
case "$demo" in
quickstart)
mkdir -p "$dir" ;;
rename)
# Build the model under its old name, so the tape can move and rename it.
cp -r "$examples/hero-shop" "$dir"
cd "$dir"
mkdir -p models/marts
mv models/finance/daily_revenue.sql models/marts/revenue.sql
rmdir models/finance
sqb build >/dev/null ;;
scope)
cp -r "$examples/waffle-shop" "$dir"
cd "$dir"
sqb compile >/dev/null || true ;;
janitor)
# Build a model, then remove it, so the janitor has a stale table to archive.
cp -r "$examples/tidy-shop" "$dir"
cd "$dir"
cat > models/weekly_revenue.sql <<'SQL'
MODEL (
materialized table,
);

SELECT DATE_TRUNC('week', revenue_date) AS revenue_week, SUM(total_revenue_cents) AS total_revenue_cents
FROM __ref("daily_revenue")
GROUP BY 1
SQL
sqb build >/dev/null
rm models/weekly_revenue.sql ;;
compile)
# Introduce an unknown column and a type mismatch, checked against enforced source contracts.
cp -r "$examples/waffle-shop" "$dir"
cd "$dir"
python3 - <<'PY'
import re

path = "sources/raw.yml"
text = open(path).read()
for name in ("raw__customers", "raw__orders"):
text = re.sub(rf"(\n(\s*)- name: {name}\n)", rf"\1\2 contract: enforced\n", text, count=1)
open(path, "w").write(text)

path = "models/marts/fact_orders.sql"
text = open(path).read()
text = text.replace("o.quantity,", "o.qty,", 1).rstrip() + "\nWHERE o.ordered_at > 5\n"
open(path, "w").write(text)
PY
;;
*)
echo "unknown demo: $demo" >&2
exit 2 ;;
esac
14 changes: 14 additions & 0 deletions website/recordings/record.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
#!/usr/bin/env bash
# Record one demo to out/<demo>.gif and out/<demo>.mp4. Usage: record.sh <demo>
# Uses the sqb on PATH; set SQB_BIN to a directory containing another sqb to use that instead.
set -euo pipefail
demo="$1"
here="$(cd "$(dirname "$0")" && pwd)"
if [[ -n "${SQB_BIN:-}" ]]; then
export PATH="$SQB_BIN:$PATH"
fi
"$here/prepare.sh" "$demo"
mkdir -p "$here/out"
cd "$here"
# Headless Chromium's sandbox is unavailable in many containers and CI runners.
VHS_NO_SANDBOX="${VHS_NO_SANDBOX:-true}" vhs "tapes/$demo.tape"
13 changes: 13 additions & 0 deletions website/recordings/tapes/common.tape
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
Set Shell bash
Set FontFamily "DejaVu Sans Mono"
Set FontSize 18
Set LineHeight 1.0
Set Width 1200
Set Height 680
Set Padding 28
Set Margin 24
Set MarginFill "#0d0f12"
Set BorderRadius 10
Set WindowBar Colorful
Set TypingSpeed 45ms
Set Theme { "name": "SQLBuild", "black": "#626975", "red": "#f47089", "green": "#54cf8e", "yellow": "#f0c267", "blue": "#3788e8", "magenta": "#a78bfa", "cyan": "#4dd6c4", "white": "#c9cdd4", "brightBlack": "#9ba1ac", "brightRed": "#f47089", "brightGreen": "#54cf8e", "brightYellow": "#f0c267", "brightBlue": "#4f98ee", "brightMagenta": "#c79bf0", "brightCyan": "#4dd6c4", "brightWhite": "#e9ebef", "background": "#111418", "foreground": "#c9cdd4", "selection": "#2a2f37", "cursor": "#e9ebef" }
14 changes: 14 additions & 0 deletions website/recordings/tapes/compile.tape
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
Source tapes/common.tape
Output out/compile.gif
Output out/compile.mp4
Hide
Type "cd scratch/compile && export PS1='$ ' && clear"
Enter
Sleep 500ms
Show
Type "sqb compile"
Sleep 300ms
Enter
Sleep 6s
Screenshot out/compile.png
Sleep 3s
18 changes: 18 additions & 0 deletions website/recordings/tapes/janitor.tape
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
Source tapes/common.tape
Output out/janitor.gif
Output out/janitor.mp4
Hide
Type "cd scratch/janitor && export PS1='$ ' && clear"
Enter
Sleep 500ms
Show
Type "sqb janitor"
Sleep 300ms
Enter
Sleep 5s
Type "archive 1 and delete 1 items from prod"
Sleep 500ms
Enter
Sleep 4s
Screenshot out/janitor.png
Sleep 3s
25 changes: 25 additions & 0 deletions website/recordings/tapes/quickstart.tape
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
Source tapes/common.tape
Output out/quickstart.gif
Output out/quickstart.mp4
Hide
Type "cd scratch/quickstart && export PS1='$ ' && clear"
Enter
Sleep 500ms
Show
Type "sqb playground waffle-shop"
Sleep 300ms
Enter
Sleep 2s
Type "cd waffle-shop"
Enter
Sleep 500ms
Type "sqb build"
Sleep 300ms
Enter
Sleep 8s
Type "sqb test"
Sleep 300ms
Enter
Sleep 6s
Screenshot out/quickstart.png
Sleep 2s
21 changes: 21 additions & 0 deletions website/recordings/tapes/rename.tape
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
Source tapes/common.tape
Output out/rename.gif
Output out/rename.mp4
Hide
Type "cd scratch/rename && export PS1='$ ' && clear"
Enter
Sleep 500ms
Show
Type "find models -name '*.sql'"
Enter
Sleep 1500ms
Type "mkdir models/finance && mv models/marts/revenue.sql models/finance/daily_revenue.sql"
Sleep 300ms
Enter
Sleep 800ms
Type "sqb plan"
Sleep 300ms
Enter
Sleep 5s
Screenshot out/rename.png
Sleep 3s
17 changes: 17 additions & 0 deletions website/recordings/tapes/scope.tape
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
Source tapes/common.tape
Output out/scope.gif
Output out/scope.mp4
Hide
Type "cd scratch/scope && export PS1='$ ' && clear"
Enter
Sleep 500ms
Show
Type "find models/marts -name '*.sql' -o -name '*.py'"
Enter
Sleep 1500ms
Type "sqb scope model:daily_revenue --as-path models/intermediate/daily_revenue.sql"
Sleep 300ms
Enter
Sleep 5s
Screenshot out/scope.png
Sleep 3s
Loading