Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 19 additions & 22 deletions src/sqlbuild/.agents/skills/sqlbuild/references/docs/benchmarks.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,25 +22,23 @@ GitHub-hosted runner with 4 CPUs and 15 GiB RAM. Each cell is the median of thre
## SQLBuild vs dbt v2

Both tools bind every column and type in every model. SQLBuild runs `sqb compile` with its defaults
(configurable rules off). dbt runs `dbt compile --static-analysis strict` with its opt-in parse cache
(`--partial-parse --partial-load`).
(configurable rules off). dbt runs `dbt compile --static-analysis strict` (strict: with column and
type checks) with its opt-in cache (`--partial-parse --partial-load`).

| Scenario | SQLBuild | dbt v2 | dbt / SQLBuild |
|---|---:|---:|---:|
| Cold (no `target/`, no caches) | **15.7 s** | 64.7 s | 4.1× |
| Unchanged | **0.46 s** (replay) | 43.8 s | 95× |
| Unchanged | **0.46 s** (reused) | 43.8 s | 95× |
| One model edited | **2.4 s** | 45.9 s | 19× |
| 100 models edited (1%) | **2.4 s** | 72.9 s | 30× |

Cold, SQLBuild used 30 s of CPU and 3.1 GiB at peak; dbt used 167 s and 4.7 GiB.

- **Reuse isn't the whole gap.** With reuse and delta compile off, every SQLBuild run is a full
compile with per-model caches: 14.2 s cold and about 8 s after an edit, still 5–9× faster.
- **Every rule on.** With every built-in rule and a custom Python rule, SQLBuild compiles cold in
- **All rules on.** With every built-in rule and a custom Python rule, SQLBuild compiles cold in
19.5 s, 3.3× faster than dbt strict.
- **dbt `--dirty` wins a one-model edit.** It compiles only changed models and their descendants: 1.6 s
for the edited model and 47 descendants, against SQLBuild's 2.4 s for a report on the whole
project. With 100 edited files it falls back to a full parse and takes 60.7 s.
- **dbt's changed-models-only mode wins a one-model edit.** `--dirty` compiles only changed models
and their descendants: 1.6 s for the edited model and 47 descendants, against SQLBuild's 2.4 s for
a report on the whole project. With 100 edited files it falls back to a full parse and takes 60.7 s.

## All variants

Expand All @@ -49,15 +47,14 @@ Wall time in seconds. CPU and memory for every cell are in the run's `summary.md
| Variant | Cold | Unchanged | 1 edit | 100 edits |
|---|---:|---:|---:|---:|
| SQLBuild | 15.7 | 0.46 | 2.4 | 2.4 |
| SQLBuild, reuse and delta off | 14.2 | 8.6 | 8.0 | 8.1 |
| SQLBuild, every rule | 19.5 | 0.46 | 2.9 | 3.1 |
| dbt v2 strict, parse cache | 64.7 | 43.8 | 45.9 | 72.9 |
| dbt v2 strict, `--dirty` | 63.3 | 0.86 | 1.6 | 60.7 |
| dbt v2 strict, default | 61.5 | 67.2 | 67.4 | 65.7 |
| dbt v2 strict, sources from warehouse | 67.4 | 74.3 | 73.9 | 73.5 |
| dbt v2, parse cache | 39.5 | 21.8 | 24.0 | 47.9 |
| dbt v2, `--dirty` | 40.1 | 0.80 | 1.5 | 48.6 |
| dbt v2, default | 36.9 | 45.9 | 44.8 | 43.4 |
| SQLBuild, all rules on | 19.5 | 0.46 | 2.9 | 3.1 |
| dbt v2 strict, with cache | 64.7 | 43.8 | 45.9 | 72.9 |
| dbt v2 strict, changed models only (`--dirty`) | 63.3 | 0.86 | 1.6 | 60.7 |
| dbt v2 strict, no cache | 61.5 | 67.2 | 67.4 | 65.7 |
| dbt v2 strict, no cache, sources read from the warehouse | 67.4 | 74.3 | 73.9 | 73.5 |
| dbt v2, with cache | 39.5 | 21.8 | 24.0 | 47.9 |
| dbt v2, changed models only (`--dirty`) | 40.1 | 0.80 | 1.5 | 48.6 |
| dbt v2, no cache | 36.9 | 45.9 | 44.8 | 43.4 |

dbt v2 without `--static-analysis strict` doesn't report unknown columns.

Expand All @@ -70,7 +67,7 @@ dbt v2 without `--static-analysis strict` doesn't report unknown columns.
type mismatches and checks enforced contracts at run time.
- Sources are declared locally in both. The warehouse holds every relation, empty, as after
`dbt build`; dbt reads it and `sqb compile` doesn't.
- dbt's parse cache and `--dirty` are opt-in. SQLBuild's reuse and delta compile are on by default.
- dbt's cache and changed-models-only mode are opt-in. SQLBuild reuses unchanged work by default.

Before timing anything, the workflow checks both tools on 50-model projects, and fails if strict
analysis doesn't report the missing column in every dbt mode:
Expand All @@ -90,12 +87,12 @@ SQLBuild records what each compile depended on and checks all of it before reusi
loaded modules, and the SQLBuild build.
- **Files are checked by type, size, modification time, change time and inode**, and by content
digest when a recently touched file's stamps move. If anything doesn't match, the stored result
isn't replayed.
isn't reused.
- **Edits recompile only what they affect.** A changed model is compiled again, with every model
whose input columns, types or nullability changed. Edits outside model and SQL test files, such as
a macro, config or a new file, run a full compile.
- **The result matches a full compile.** A delta compile produces byte-identical output to
`sqb compile --no-cache`, and a replay prints the previous report again. CI compiles a copy in full
- **The result matches a full compile.** Recompiling only the edited models produces byte-identical
output to `sqb compile --no-cache`, and a reused compile prints the previous report again. CI compiles a copy in full
after every step of a seeded edit series and compares the exit code and everything printed.

Reuse assumes macros, hooks, providers, adapters and custom rules are deterministic. It has
Expand Down
41 changes: 19 additions & 22 deletions website/src/content/docs/docs/benchmarks.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -10,25 +10,23 @@ GitHub-hosted runner with 4 CPUs and 15 GiB RAM. Each cell is the median of thre
## SQLBuild vs dbt v2

Both tools bind every column and type in every model. SQLBuild runs `sqb compile` with its defaults
(configurable rules off). dbt runs `dbt compile --static-analysis strict` with its opt-in parse cache
(`--partial-parse --partial-load`).
(configurable rules off). dbt runs `dbt compile --static-analysis strict` (strict: with column and
type checks) with its opt-in cache (`--partial-parse --partial-load`).

| Scenario | SQLBuild | dbt v2 | dbt / SQLBuild |
|---|---:|---:|---:|
| Cold (no `target/`, no caches) | **15.7 s** | 64.7 s | 4.1× |
| Unchanged | **0.46 s** (replay) | 43.8 s | 95× |
| Unchanged | **0.46 s** (reused) | 43.8 s | 95× |
| One model edited | **2.4 s** | 45.9 s | 19× |
| 100 models edited (1%) | **2.4 s** | 72.9 s | 30× |

Cold, SQLBuild used 30 s of CPU and 3.1 GiB at peak; dbt used 167 s and 4.7 GiB.

- **Reuse isn't the whole gap.** With reuse and delta compile off, every SQLBuild run is a full
compile with per-model caches: 14.2 s cold and about 8 s after an edit, still 5–9× faster.
- **Every rule on.** With every built-in rule and a custom Python rule, SQLBuild compiles cold in
- **All rules on.** With every built-in rule and a custom Python rule, SQLBuild compiles cold in
19.5 s, 3.3× faster than dbt strict.
- **dbt `--dirty` wins a one-model edit.** It compiles only changed models and their descendants: 1.6 s
for the edited model and 47 descendants, against SQLBuild's 2.4 s for a report on the whole
project. With 100 edited files it falls back to a full parse and takes 60.7 s.
- **dbt's changed-models-only mode wins a one-model edit.** `--dirty` compiles only changed models
and their descendants: 1.6 s for the edited model and 47 descendants, against SQLBuild's 2.4 s for
a report on the whole project. With 100 edited files it falls back to a full parse and takes 60.7 s.

## All variants

Expand All @@ -37,15 +35,14 @@ Wall time in seconds. CPU and memory for every cell are in the run's `summary.md
| Variant | Cold | Unchanged | 1 edit | 100 edits |
|---|---:|---:|---:|---:|
| SQLBuild | 15.7 | 0.46 | 2.4 | 2.4 |
| SQLBuild, reuse and delta off | 14.2 | 8.6 | 8.0 | 8.1 |
| SQLBuild, every rule | 19.5 | 0.46 | 2.9 | 3.1 |
| dbt v2 strict, parse cache | 64.7 | 43.8 | 45.9 | 72.9 |
| dbt v2 strict, `--dirty` | 63.3 | 0.86 | 1.6 | 60.7 |
| dbt v2 strict, default | 61.5 | 67.2 | 67.4 | 65.7 |
| dbt v2 strict, sources from warehouse | 67.4 | 74.3 | 73.9 | 73.5 |
| dbt v2, parse cache | 39.5 | 21.8 | 24.0 | 47.9 |
| dbt v2, `--dirty` | 40.1 | 0.80 | 1.5 | 48.6 |
| dbt v2, default | 36.9 | 45.9 | 44.8 | 43.4 |
| SQLBuild, all rules on | 19.5 | 0.46 | 2.9 | 3.1 |
| dbt v2 strict, with cache | 64.7 | 43.8 | 45.9 | 72.9 |
| dbt v2 strict, changed models only (`--dirty`) | 63.3 | 0.86 | 1.6 | 60.7 |
| dbt v2 strict, no cache | 61.5 | 67.2 | 67.4 | 65.7 |
| dbt v2 strict, no cache, sources read from the warehouse | 67.4 | 74.3 | 73.9 | 73.5 |
| dbt v2, with cache | 39.5 | 21.8 | 24.0 | 47.9 |
| dbt v2, changed models only (`--dirty`) | 40.1 | 0.80 | 1.5 | 48.6 |
| dbt v2, no cache | 36.9 | 45.9 | 44.8 | 43.4 |

dbt v2 without `--static-analysis strict` doesn't report unknown columns.

Expand All @@ -58,7 +55,7 @@ dbt v2 without `--static-analysis strict` doesn't report unknown columns.
type mismatches and checks enforced contracts at run time.
- Sources are declared locally in both. The warehouse holds every relation, empty, as after
`dbt build`; dbt reads it and `sqb compile` doesn't.
- dbt's parse cache and `--dirty` are opt-in. SQLBuild's reuse and delta compile are on by default.
- dbt's cache and changed-models-only mode are opt-in. SQLBuild reuses unchanged work by default.

Before timing anything, the workflow checks both tools on 50-model projects, and fails if strict
analysis doesn't report the missing column in every dbt mode:
Expand All @@ -78,12 +75,12 @@ SQLBuild records what each compile depended on and checks all of it before reusi
loaded modules, and the SQLBuild build.
- **Files are checked by type, size, modification time, change time and inode**, and by content
digest when a recently touched file's stamps move. If anything doesn't match, the stored result
isn't replayed.
isn't reused.
- **Edits recompile only what they affect.** A changed model is compiled again, with every model
whose input columns, types or nullability changed. Edits outside model and SQL test files, such as
a macro, config or a new file, run a full compile.
- **The result matches a full compile.** A delta compile produces byte-identical output to
`sqb compile --no-cache`, and a replay prints the previous report again. CI compiles a copy in full
- **The result matches a full compile.** Recompiling only the edited models produces byte-identical
output to `sqb compile --no-cache`, and a reused compile prints the previous report again. CI compiles a copy in full
after every step of a seeded edit series and compares the exit code and everything printed.

Reuse assumes macros, hooks, providers, adapters and custom rules are deterministic. It has
Expand Down
Loading