Skip to content

Commit d8ea65c

Browse files
committed
docs: use plain variant names on the benchmarks page
1 parent 46bfa4a commit d8ea65c

2 files changed

Lines changed: 38 additions & 44 deletions

File tree

‎src/sqlbuild/.agents/skills/sqlbuild/references/docs/benchmarks.md‎

Lines changed: 19 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -22,25 +22,23 @@ GitHub-hosted runner with 4 CPUs and 15 GiB RAM. Each cell is the median of thre
2222
## SQLBuild vs dbt v2
2323

2424
Both tools bind every column and type in every model. SQLBuild runs `sqb compile` with its defaults
25-
(configurable rules off). dbt runs `dbt compile --static-analysis strict` with its opt-in parse cache
26-
(`--partial-parse --partial-load`).
25+
(configurable rules off). dbt runs `dbt compile --static-analysis strict` (strict: with column and
26+
type checks) with its opt-in cache (`--partial-parse --partial-load`).
2727

2828
| Scenario | SQLBuild | dbt v2 | dbt / SQLBuild |
2929
|---|---:|---:|---:|
3030
| Cold (no `target/`, no caches) | **15.7 s** | 64.7 s | 4.1× |
31-
| Unchanged | **0.46 s** (replay) | 43.8 s | 95× |
31+
| Unchanged | **0.46 s** (reused) | 43.8 s | 95× |
3232
| One model edited | **2.4 s** | 45.9 s | 19× |
3333
| 100 models edited (1%) | **2.4 s** | 72.9 s | 30× |
3434

3535
Cold, SQLBuild used 30 s of CPU and 3.1 GiB at peak; dbt used 167 s and 4.7 GiB.
3636

37-
- **Reuse isn't the whole gap.** With reuse and delta compile off, every SQLBuild run is a full
38-
compile with per-model caches: 14.2 s cold and about 8 s after an edit, still 5–9× faster.
39-
- **Every rule on.** With every built-in rule and a custom Python rule, SQLBuild compiles cold in
37+
- **All rules on.** With every built-in rule and a custom Python rule, SQLBuild compiles cold in
4038
19.5 s, 3.3× faster than dbt strict.
41-
- **dbt `--dirty` wins a one-model edit.** It compiles only changed models and their descendants: 1.6 s
42-
for the edited model and 47 descendants, against SQLBuild's 2.4 s for a report on the whole
43-
project. With 100 edited files it falls back to a full parse and takes 60.7 s.
39+
- **dbt's changed-models-only mode wins a one-model edit.** `--dirty` compiles only changed models
40+
and their descendants: 1.6 s for the edited model and 47 descendants, against SQLBuild's 2.4 s for
41+
a report on the whole project. With 100 edited files it falls back to a full parse and takes 60.7 s.
4442

4543
## All variants
4644

@@ -49,15 +47,14 @@ Wall time in seconds. CPU and memory for every cell are in the run's `summary.md
4947
| Variant | Cold | Unchanged | 1 edit | 100 edits |
5048
|---|---:|---:|---:|---:|
5149
| SQLBuild | 15.7 | 0.46 | 2.4 | 2.4 |
52-
| SQLBuild, reuse and delta off | 14.2 | 8.6 | 8.0 | 8.1 |
53-
| SQLBuild, every rule | 19.5 | 0.46 | 2.9 | 3.1 |
54-
| dbt v2 strict, parse cache | 64.7 | 43.8 | 45.9 | 72.9 |
55-
| dbt v2 strict, `--dirty` | 63.3 | 0.86 | 1.6 | 60.7 |
56-
| dbt v2 strict, default | 61.5 | 67.2 | 67.4 | 65.7 |
57-
| dbt v2 strict, sources from warehouse | 67.4 | 74.3 | 73.9 | 73.5 |
58-
| dbt v2, parse cache | 39.5 | 21.8 | 24.0 | 47.9 |
59-
| dbt v2, `--dirty` | 40.1 | 0.80 | 1.5 | 48.6 |
60-
| dbt v2, default | 36.9 | 45.9 | 44.8 | 43.4 |
50+
| SQLBuild, all rules on | 19.5 | 0.46 | 2.9 | 3.1 |
51+
| dbt v2 strict, with cache | 64.7 | 43.8 | 45.9 | 72.9 |
52+
| dbt v2 strict, changed models only (`--dirty`) | 63.3 | 0.86 | 1.6 | 60.7 |
53+
| dbt v2 strict, no cache | 61.5 | 67.2 | 67.4 | 65.7 |
54+
| dbt v2 strict, no cache, sources read from the warehouse | 67.4 | 74.3 | 73.9 | 73.5 |
55+
| dbt v2, with cache | 39.5 | 21.8 | 24.0 | 47.9 |
56+
| dbt v2, changed models only (`--dirty`) | 40.1 | 0.80 | 1.5 | 48.6 |
57+
| dbt v2, no cache | 36.9 | 45.9 | 44.8 | 43.4 |
6158

6259
dbt v2 without `--static-analysis strict` doesn't report unknown columns.
6360

@@ -70,7 +67,7 @@ dbt v2 without `--static-analysis strict` doesn't report unknown columns.
7067
type mismatches and checks enforced contracts at run time.
7168
- Sources are declared locally in both. The warehouse holds every relation, empty, as after
7269
`dbt build`; dbt reads it and `sqb compile` doesn't.
73-
- dbt's parse cache and `--dirty` are opt-in. SQLBuild's reuse and delta compile are on by default.
70+
- dbt's cache and changed-models-only mode are opt-in. SQLBuild reuses unchanged work by default.
7471

7572
Before timing anything, the workflow checks both tools on 50-model projects, and fails if strict
7673
analysis doesn't report the missing column in every dbt mode:
@@ -90,12 +87,12 @@ SQLBuild records what each compile depended on and checks all of it before reusi
9087
loaded modules, and the SQLBuild build.
9188
- **Files are checked by type, size, modification time, change time and inode**, and by content
9289
digest when a recently touched file's stamps move. If anything doesn't match, the stored result
93-
isn't replayed.
90+
isn't reused.
9491
- **Edits recompile only what they affect.** A changed model is compiled again, with every model
9592
whose input columns, types or nullability changed. Edits outside model and SQL test files, such as
9693
a macro, config or a new file, run a full compile.
97-
- **The result matches a full compile.** A delta compile produces byte-identical output to
98-
`sqb compile --no-cache`, and a replay prints the previous report again. CI compiles a copy in full
94+
- **The result matches a full compile.** Recompiling only the edited models produces byte-identical
95+
output to `sqb compile --no-cache`, and a reused compile prints the previous report again. CI compiles a copy in full
9996
after every step of a seeded edit series and compares the exit code and everything printed.
10097

10198
Reuse assumes macros, hooks, providers, adapters and custom rules are deterministic. It has

‎website/src/content/docs/docs/benchmarks.mdx‎

Lines changed: 19 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -10,25 +10,23 @@ GitHub-hosted runner with 4 CPUs and 15 GiB RAM. Each cell is the median of thre
1010
## SQLBuild vs dbt v2
1111

1212
Both tools bind every column and type in every model. SQLBuild runs `sqb compile` with its defaults
13-
(configurable rules off). dbt runs `dbt compile --static-analysis strict` with its opt-in parse cache
14-
(`--partial-parse --partial-load`).
13+
(configurable rules off). dbt runs `dbt compile --static-analysis strict` (strict: with column and
14+
type checks) with its opt-in cache (`--partial-parse --partial-load`).
1515

1616
| Scenario | SQLBuild | dbt v2 | dbt / SQLBuild |
1717
|---|---:|---:|---:|
1818
| Cold (no `target/`, no caches) | **15.7 s** | 64.7 s | 4.1× |
19-
| Unchanged | **0.46 s** (replay) | 43.8 s | 95× |
19+
| Unchanged | **0.46 s** (reused) | 43.8 s | 95× |
2020
| One model edited | **2.4 s** | 45.9 s | 19× |
2121
| 100 models edited (1%) | **2.4 s** | 72.9 s | 30× |
2222

2323
Cold, SQLBuild used 30 s of CPU and 3.1 GiB at peak; dbt used 167 s and 4.7 GiB.
2424

25-
- **Reuse isn't the whole gap.** With reuse and delta compile off, every SQLBuild run is a full
26-
compile with per-model caches: 14.2 s cold and about 8 s after an edit, still 5–9× faster.
27-
- **Every rule on.** With every built-in rule and a custom Python rule, SQLBuild compiles cold in
25+
- **All rules on.** With every built-in rule and a custom Python rule, SQLBuild compiles cold in
2826
19.5 s, 3.3× faster than dbt strict.
29-
- **dbt `--dirty` wins a one-model edit.** It compiles only changed models and their descendants: 1.6 s
30-
for the edited model and 47 descendants, against SQLBuild's 2.4 s for a report on the whole
31-
project. With 100 edited files it falls back to a full parse and takes 60.7 s.
27+
- **dbt's changed-models-only mode wins a one-model edit.** `--dirty` compiles only changed models
28+
and their descendants: 1.6 s for the edited model and 47 descendants, against SQLBuild's 2.4 s for
29+
a report on the whole project. With 100 edited files it falls back to a full parse and takes 60.7 s.
3230

3331
## All variants
3432

@@ -37,15 +35,14 @@ Wall time in seconds. CPU and memory for every cell are in the run's `summary.md
3735
| Variant | Cold | Unchanged | 1 edit | 100 edits |
3836
|---|---:|---:|---:|---:|
3937
| SQLBuild | 15.7 | 0.46 | 2.4 | 2.4 |
40-
| SQLBuild, reuse and delta off | 14.2 | 8.6 | 8.0 | 8.1 |
41-
| SQLBuild, every rule | 19.5 | 0.46 | 2.9 | 3.1 |
42-
| dbt v2 strict, parse cache | 64.7 | 43.8 | 45.9 | 72.9 |
43-
| dbt v2 strict, `--dirty` | 63.3 | 0.86 | 1.6 | 60.7 |
44-
| dbt v2 strict, default | 61.5 | 67.2 | 67.4 | 65.7 |
45-
| dbt v2 strict, sources from warehouse | 67.4 | 74.3 | 73.9 | 73.5 |
46-
| dbt v2, parse cache | 39.5 | 21.8 | 24.0 | 47.9 |
47-
| dbt v2, `--dirty` | 40.1 | 0.80 | 1.5 | 48.6 |
48-
| dbt v2, default | 36.9 | 45.9 | 44.8 | 43.4 |
38+
| SQLBuild, all rules on | 19.5 | 0.46 | 2.9 | 3.1 |
39+
| dbt v2 strict, with cache | 64.7 | 43.8 | 45.9 | 72.9 |
40+
| dbt v2 strict, changed models only (`--dirty`) | 63.3 | 0.86 | 1.6 | 60.7 |
41+
| dbt v2 strict, no cache | 61.5 | 67.2 | 67.4 | 65.7 |
42+
| dbt v2 strict, no cache, sources read from the warehouse | 67.4 | 74.3 | 73.9 | 73.5 |
43+
| dbt v2, with cache | 39.5 | 21.8 | 24.0 | 47.9 |
44+
| dbt v2, changed models only (`--dirty`) | 40.1 | 0.80 | 1.5 | 48.6 |
45+
| dbt v2, no cache | 36.9 | 45.9 | 44.8 | 43.4 |
4946

5047
dbt v2 without `--static-analysis strict` doesn't report unknown columns.
5148

@@ -58,7 +55,7 @@ dbt v2 without `--static-analysis strict` doesn't report unknown columns.
5855
type mismatches and checks enforced contracts at run time.
5956
- Sources are declared locally in both. The warehouse holds every relation, empty, as after
6057
`dbt build`; dbt reads it and `sqb compile` doesn't.
61-
- dbt's parse cache and `--dirty` are opt-in. SQLBuild's reuse and delta compile are on by default.
58+
- dbt's cache and changed-models-only mode are opt-in. SQLBuild reuses unchanged work by default.
6259

6360
Before timing anything, the workflow checks both tools on 50-model projects, and fails if strict
6461
analysis doesn't report the missing column in every dbt mode:
@@ -78,12 +75,12 @@ SQLBuild records what each compile depended on and checks all of it before reusi
7875
loaded modules, and the SQLBuild build.
7976
- **Files are checked by type, size, modification time, change time and inode**, and by content
8077
digest when a recently touched file's stamps move. If anything doesn't match, the stored result
81-
isn't replayed.
78+
isn't reused.
8279
- **Edits recompile only what they affect.** A changed model is compiled again, with every model
8380
whose input columns, types or nullability changed. Edits outside model and SQL test files, such as
8481
a macro, config or a new file, run a full compile.
85-
- **The result matches a full compile.** A delta compile produces byte-identical output to
86-
`sqb compile --no-cache`, and a replay prints the previous report again. CI compiles a copy in full
82+
- **The result matches a full compile.** Recompiling only the edited models produces byte-identical
83+
output to `sqb compile --no-cache`, and a reused compile prints the previous report again. CI compiles a copy in full
8784
after every step of a seeded edit series and compares the exit code and everything printed.
8885

8986
Reuse assumes macros, hooks, providers, adapters and custom rules are deterministic. It has

0 commit comments

Comments
 (0)