Skip to content

Commit 46bfa4a

Browse files
kvlongekvlonge
authored
docs: make the benchmarks page concise (#1192)
1 parent 8346c24 commit 46bfa4a

3 files changed

Lines changed: 166 additions & 414 deletions

File tree

‎src/sqlbuild/.agents/skills/sqlbuild/references/docs/CONTENTS.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ Bundled copies of every page on the SQLBuild documentation site, matching the in
1010
- [Quickstart](quickstart.md) (`quickstart`) - Get a SQLBuild project running locally with DuckDB in under a minute.
1111
- [Feature Comparison](feature-comparison.md) (`feature-comparison`) - Feature comparison between SQLBuild, dbt, and SQLMesh.
1212
- [Roadmap](roadmap.md) (`roadmap`) - What's coming next in SQLBuild.
13-
- [Benchmarks](benchmarks.md) (`benchmarks`) - SQLBuild and dbt v2 compiling the same 10,000-model project: cold, unchanged, one-model and 1% edits, with CPU time and peak memory.
13+
- [Benchmarks](benchmarks.md) (`benchmarks`) - SQLBuild and dbt v2 compiling the same 10,000-model project: cold, unchanged, one-model and 1% edits.
1414

1515
## dbt Compatibility
1616

‎src/sqlbuild/.agents/skills/sqlbuild/references/docs/benchmarks.md‎

Lines changed: 85 additions & 211 deletions
Original file line numberDiff line numberDiff line change
@@ -2,251 +2,125 @@
22

33
# Benchmarks
44

5-
> SQLBuild and dbt v2 compiling the same 10,000-model project: cold, unchanged, one-model and 1% edits, with CPU time and peak memory.
5+
> SQLBuild and dbt v2 compiling the same 10,000-model project: cold, unchanged, one-model and 1% edits.
66
77
Online: https://sqlbuild.com/docs/benchmarks/
88

99
## Contents
1010

11-
- SQLBuild vs dbt v2 with column checking
12-
- dbt v2 `--dirty`
13-
- dbt v2 out of the box and with remote sources
14-
- SQLBuild with every rule
15-
- Correctness
16-
- What is compared and what isn't
17-
- The project
18-
- Methodology
19-
- Reproduce the benchmark
11+
- SQLBuild vs dbt v2
12+
- All variants
13+
- What is compared
14+
- Why SQLBuild can reuse work by default
15+
- The project and how to reproduce it
2016
- Compiler guards in CI
2117

22-
These numbers come from one run of the [benchmark workflow](https://github.com/chio-labs/sqlbuild/blob/main/.github/workflows/benchmark-dbt.yml):
23-
[run 39543189999](https://github.com/chio-labs/sqlbuild/actions/runs/39543189999), October 21, 2026,
24-
on a GitHub-hosted `ubuntu-24.04` runner with 4 CPUs (AMD EPYC 9V45) and 15 GiB RAM. It compared
25-
**SQLBuild 0.164.3** with **dbt 2.0.9** (the `dbt` package on PyPI), both installed from PyPI,
26-
on the same generated 10,000-model DuckDB project. Every cell is the median of three rounds.
18+
SQLBuild 0.164.3 and dbt 2.0.9 compiling the same generated 10,000-model DuckDB project, on a
19+
GitHub-hosted runner with 4 CPUs and 15 GiB RAM. Each cell is the median of three rounds from
20+
[run 39543189999](https://github.com/chio-labs/sqlbuild/actions/runs/39543189999) (October 21, 2026).
2721

28-
## SQLBuild vs dbt v2 with column checking
22+
## SQLBuild vs dbt v2
2923

30-
Both tools here bind every column and type in every model. SQLBuild runs `sqb compile` with its
31-
defaults: compile with column and type checks, configurable rules off. dbt runs
32-
`dbt compile --static-analysis strict` with its parse cache turned on (`--partial-parse --partial-load`;
33-
the cache is opt-in and the command compiles the whole project).
24+
Both tools bind every column and type in every model. SQLBuild runs `sqb compile` with its defaults
25+
(configurable rules off). dbt runs `dbt compile --static-analysis strict` with its opt-in parse cache
26+
(`--partial-parse --partial-load`).
3427

35-
| Scenario | SQLBuild wall | CPU | Peak memory | dbt v2 strict, parse cache wall | CPU | Peak memory | dbt / SQLBuild |
36-
|---|---:|---:|---:|---:|---:|---:|---:|
37-
| Cold (no `target/`, no caches) | **15.7 s** | 30.3 s | 3.1 GiB | 64.7 s | 167.0 s | 4.7 GiB | 4.1× |
38-
| Unchanged | **0.46 s** (replay) | 0.5 s | 0.07 GiB | 43.8 s | 132.2 s | 4.1 GiB | 95× |
39-
| One model edited | **2.4 s** | 2.5 s | 0.8 GiB | 45.9 s | 137.2 s | 4.1 GiB | 19× |
40-
| 100 models edited (1%) | **2.4 s** | 2.5 s | 0.8 GiB | 72.9 s | 176.1 s | 6.9 GiB | 30× |
41-
42-
SQLBuild's unchanged run replays the stored result of the previous compile. Its edit runs compile
43-
only the changed files against the previous compile and report the same result as a full compile.
44-
45-
The gap isn't only replay. With reuse and delta compile turned off, every SQLBuild run is a full
46-
compile that takes per-model analyses from its cache:
47-
48-
| Scenario | SQLBuild full compile wall | CPU | Peak memory | dbt v2 strict, parse cache wall | dbt / SQLBuild |
49-
|---|---:|---:|---:|---:|---:|
50-
| Cold | **14.2 s** | 28.4 s | 2.8 GiB | 64.7 s | 4.6× |
51-
| Unchanged | **8.6 s** | 12.9 s | 2.8 GiB | 43.8 s | 5.1× |
52-
| One model edited | **8.0 s** | 12.4 s | 2.8 GiB | 45.9 s | 5.7× |
53-
| 100 models edited (1%) | **8.1 s** | 12.6 s | 2.8 GiB | 72.9 s | 9.0× |
54-
55-
## dbt v2 `--dirty`
56-
57-
`dbt compile --dirty` compiles only models whose files changed since the last parse-cache run,
58-
plus all their downstream dependents. It needs the cache, so its cold run is a
59-
`--partial-parse --partial-load` compile.
60-
61-
| Scenario | SQLBuild | dbt v2 `--dirty`, strict | dbt v2 `--dirty` |
28+
| Scenario | SQLBuild | dbt v2 | dbt / SQLBuild |
6229
|---|---:|---:|---:|
63-
| Cold | **15.7 s** | 63.3 s | 40.1 s |
64-
| Unchanged | **0.46 s** (replay) | 0.86 s (nothing to do) | 0.80 s (nothing to do) |
65-
| One model edited | 2.4 s | **1.6 s** (48 models) | **1.5 s** (48 models) |
66-
| 100 models edited (1%) | **2.4 s** | 60.7 s (2,800 models) | 48.6 s (2,800 models) |
67-
68-
`--dirty` is faster on a single-model edit: strict analysis of the edited model and its 47
69-
descendants took 1.6 s. SQLBuild took 2.4 s and reported the result for the whole project, the same
70-
report a full compile produces; models whose inputs didn't change kept their stored analysis. With
71-
100 edited files, dbt 2.0.9 logged `[partial-parse] ... exceeds incremental threshold (100), falling
72-
back to full parse` and compiled the 2,800 selected models after a full parse.
73-
74-
## dbt v2 out of the box and with remote sources
30+
| Cold (no `target/`, no caches) | **15.7 s** | 64.7 s | 4.1× |
31+
| Unchanged | **0.46 s** (replay) | 43.8 s | 95× |
32+
| One model edited | **2.4 s** | 45.9 s | 19× |
33+
| 100 models edited (1%) | **2.4 s** | 72.9 s | 30× |
7534

76-
For context, dbt v2 without the parse cache, and strict analysis reading source schemas from the
77-
warehouse (dbt's default) instead of from their declarations:
78-
79-
| Scenario | dbt v2 default | dbt v2 default, strict | dbt v2 default, strict, sources from warehouse |
80-
|---|---:|---:|---:|
81-
| Cold | 36.9 s | 61.5 s | 67.4 s |
82-
| Unchanged | 45.9 s | 67.2 s | 74.3 s |
83-
| One model edited | 44.8 s | 67.4 s | 73.9 s |
84-
| 100 models edited (1%) | 43.4 s | 65.7 s | 73.5 s |
35+
Cold, SQLBuild used 30 s of CPU and 3.1 GiB at peak; dbt used 167 s and 4.7 GiB.
8536

86-
dbt v2's default `compile` doesn't report unknown columns.
37+
- **Reuse isn't the whole gap.** With reuse and delta compile off, every SQLBuild run is a full
38+
compile with per-model caches: 14.2 s cold and about 8 s after an edit, still 5–9× faster.
39+
- **Every rule on.** With every built-in rule and a custom Python rule, SQLBuild compiles cold in
40+
19.5 s, 3.3× faster than dbt strict.
41+
- **dbt `--dirty` wins a one-model edit.** It compiles only changed models and their descendants: 1.6 s
42+
for the edited model and 47 descendants, against SQLBuild's 2.4 s for a report on the whole
43+
project. With 100 edited files it falls back to a full parse and takes 60.7 s.
8744

88-
## SQLBuild with every rule
45+
## All variants
8946

90-
The same project with every built-in rule and one custom Python rule turned on. These rules check
91-
contract, declaration, graph, model, project, SQL and test conventions that dbt compile doesn't
92-
check.
47+
Wall time in seconds. CPU and memory for every cell are in the run's `summary.md`.
9348

94-
| Scenario | SQLBuild with rules | CPU | Peak memory | Full compile with per-model caches |
49+
| Variant | Cold | Unchanged | 1 edit | 100 edits |
9550
|---|---:|---:|---:|---:|
96-
| Cold | 19.5 s | 42.0 s | 3.6 GiB | 18.0 s |
97-
| Unchanged | 0.46 s (replay) | 0.5 s | 0.07 GiB | 9.9 s |
98-
| One model edited | 2.9 s | 2.8 s | 0.9 GiB | 9.8 s |
99-
| 100 models edited (1%) | 3.1 s | 3.0 s | 0.9 GiB | 9.9 s |
100-
101-
Cold, it is still 3.3× faster than dbt v2 strict with its parse cache (64.7 s).
102-
103-
## Correctness
104-
105-
The workflow also runs both tools on three 50-model projects before timing anything:
106-
107-
| Project | `sqb compile` | `dbt compile` | `dbt compile --static-analysis strict` |
108-
|---|---|---|---|
109-
| Clean | exit 0 | exit 0 | exit 0 |
110-
| Reference to a missing model | exit 1, `P001` | exit 1, `dbt1048` | exit 1, `dbt1048` |
111-
| Column the upstream model doesn't have | exit 1, `B002 Unknown column 'missing_col' in b` | exit 0, nothing reported | exit 1, `dbt0227 No column b.missing_col found` |
112-
113-
Strict analysis must also report the missing column in the parse-cache and `--dirty` modes, and no
114-
strict run may mention a fallback, login or skipped analysis. Otherwise the workflow fails. SQLBuild
115-
reports the same errors with rules on and off.
116-
117-
## What is compared and what isn't
118-
119-
- Both tools bind columns and types. dbt does this only with `--static-analysis strict`.
120-
- SQLBuild compile also plans and binds its 10,000 SQL tests. dbt `compile` excludes unit tests.
121-
- SQLBuild reports contract, type and nullability problems as errors. dbt strict warns on contract
122-
type mismatches, and enforced contracts are checked at run time.
123-
- SQLBuild computes column lineage by default.
124-
- Sources are declared locally in both: dbt's sources set `schema_origin: local` and every column
125-
has a `data_type`. The remote-sources column above shows dbt's default.
126-
- The DuckDB warehouse holds every source and model relation, empty, as after `dbt build`. dbt reads
127-
it; `sqb compile` is offline and doesn't.
128-
- The measured dbt is the `dbt` distribution on PyPI. Strict static analysis isn't in its Apache-2.0
129-
build.
51+
| SQLBuild | 15.7 | 0.46 | 2.4 | 2.4 |
52+
| SQLBuild, reuse and delta off | 14.2 | 8.6 | 8.0 | 8.1 |
53+
| SQLBuild, every rule | 19.5 | 0.46 | 2.9 | 3.1 |
54+
| dbt v2 strict, parse cache | 64.7 | 43.8 | 45.9 | 72.9 |
55+
| dbt v2 strict, `--dirty` | 63.3 | 0.86 | 1.6 | 60.7 |
56+
| dbt v2 strict, default | 61.5 | 67.2 | 67.4 | 65.7 |
57+
| dbt v2 strict, sources from warehouse | 67.4 | 74.3 | 73.9 | 73.5 |
58+
| dbt v2, parse cache | 39.5 | 21.8 | 24.0 | 47.9 |
59+
| dbt v2, `--dirty` | 40.1 | 0.80 | 1.5 | 48.6 |
60+
| dbt v2, default | 36.9 | 45.9 | 44.8 | 43.4 |
61+
62+
dbt v2 without `--static-analysis strict` doesn't report unknown columns.
63+
64+
## What is compared
65+
66+
- Both tools bind columns and types; dbt only with `--static-analysis strict`. The measured dbt is
67+
the `dbt` distribution on PyPI; strict analysis isn't in its Apache-2.0 build.
68+
- SQLBuild also plans and binds its 10,000 SQL tests, reports contract, type and nullability
69+
problems as errors, and computes column lineage. dbt `compile` skips unit tests, warns on contract
70+
type mismatches and checks enforced contracts at run time.
71+
- Sources are declared locally in both. The warehouse holds every relation, empty, as after
72+
`dbt build`; dbt reads it and `sqb compile` doesn't.
13073
- dbt's parse cache and `--dirty` are opt-in. SQLBuild's reuse and delta compile are on by default.
13174

132-
### Why SQLBuild can reuse work by default
75+
Before timing anything, the workflow checks both tools on 50-model projects, and fails if strict
76+
analysis doesn't report the missing column in every dbt mode:
13377

134-
SQLBuild records what each compile depended on, then checks all of it before reusing anything:
78+
| Project | `sqb compile` | `dbt compile` | `dbt compile` strict |
79+
|---|:-:|:-:|:-:|
80+
| Clean | ✓ passes | ✓ passes | ✓ passes |
81+
| Reference to a missing model | ✓ reported | ✓ reported | ✓ reported |
82+
| Column the upstream model doesn't have | ✓ reported | ✗ not reported | ✓ reported |
13583

136-
- **Every input is recorded during the compile itself.** That covers the project's files, the
137-
command line, the variables passed with `--vars`, every environment variable a template read,
138-
provider settings files, the Python interpreter and every loaded module, and the SQLBuild build.
139-
Nothing is inferred afterwards.
84+
## Why SQLBuild can reuse work by default
85+
86+
SQLBuild records what each compile depended on and checks all of it before reusing anything:
87+
88+
- **Every input is recorded during the compile itself:** project files, the command line, `--vars`,
89+
every environment variable a template read, provider settings files, the Python interpreter and
90+
loaded modules, and the SQLBuild build.
14091
- **Files are checked by type, size, modification time, change time and inode**, and by content
14192
digest when a recently touched file's stamps move. If anything doesn't match, the stored result
14293
isn't replayed.
143-
- **Edits recompile only what they affect.** SQLBuild knows each model's references and the
144-
columns, types and nullability it reads. A changed model is compiled again, together with every
145-
model whose inputs changed shape. An edit outside model and SQL test files, such as a macro,
146-
config or a new file, runs a full compile. That choice is made before any work starts, apart from
147-
a few edits found part-way through to need a full compile.
148-
- **The result is the same as a full compile.** A delta compile produces byte-identical output to
149-
`sqb compile --no-cache`, and a replay prints the previous report again. CI checks this by
150-
compiling a copy in full after every step of a seeded edit series and comparing the exit code and
151-
everything printed.
94+
- **Edits recompile only what they affect.** A changed model is compiled again, with every model
95+
whose input columns, types or nullability changed. Edits outside model and SQL test files, such as
96+
a macro, config or a new file, run a full compile.
97+
- **The result matches a full compile.** A delta compile produces byte-identical output to
98+
`sqb compile --no-cache`, and a replay prints the previous report again. CI compiles a copy in full
99+
after every step of a seeded edit series and compares the exit code and everything printed.
152100

153101
Reuse assumes macros, hooks, providers, adapters and custom rules are deterministic. It has
154-
[documented limits](cli/compile.md#reusing-an-unchanged-compile), for example on network
155-
filesystems or with clocks that differ between machines, and `--no-cache` always compiles in full.
156-
157-
## The project
158-
159-
The project comes from the dense generator SQLBuild's compile-performance CI jobs use, with the
160-
adapter switched to DuckDB. The dbt project is derived from it file by file: the same model names,
161-
folders, SQL bodies and DAG, with Jinja macros, `_models.yml` contracts, `data_tests` and
162-
`unit_tests`.
163-
164-
| Resource | Count |
165-
|---|---:|
166-
| Models | 10,000 (8 to 472 columns, 31 on average) |
167-
| Sources | 2,400 |
168-
| Seeds | 500 |
169-
| SQL functions | 250 |
170-
| Macros | 120 |
171-
| `not_null` checks | 17,000 |
172-
| SQL tests (dbt unit tests) | 10,000 |
102+
[documented limits](cli/compile.md#reusing-an-unchanged-compile), and `--no-cache` always
103+
compiles in full.
173104

174-
Each edit changes one constant in a model's SQL body, so its columns, types and references stay the
175-
same. The 1% edit changes 100 models spread evenly across the project.
105+
## The project and how to reproduce it
176106

177-
## Methodology
107+
The project comes from the dense generator SQLBuild's compile CI uses: 10,000 models (31 columns on
108+
average), 2,400 sources, 500 seeds, 250 SQL functions, 120 macros and 10,000 SQL tests. The dbt
109+
project has the same names, folders, SQL and DAG. Each edit changes one constant in a model's SQL;
110+
the 1% edit changes 100 models spread across the project. Rounds alternate the variant order.
178111

179-
For every variant and round, the workflow copies the pristine project, runs the cold compile, runs
180-
it again unchanged, edits one model and compiles, then edits 100 models and compiles. Rounds
181-
alternate the variant order. `/usr/bin/time` records wall time, user plus system CPU time, and the
182-
peak resident memory of the largest process. dbt runs use `DBT_SEND_ANONYMOUS_USAGE_STATS=false` and
183-
a DuckDB profile with `threads: 4`.
184-
185-
The exact commands, run in each project copy:
186-
187-
```bash
188-
# SQLBuild default
189-
sqb --no-color compile
190-
191-
# SQLBuild full compile with per-model caches
192-
SQLBUILD_DISABLE_COMPILE_REUSE=1 SQLBUILD_DISABLE_DELTA_COMPILE=1 sqb --no-color compile
193-
194-
# SQLBuild with every rule: the same command in the project with [rules] select = ["SQBR", "XSQBR"]
195-
sqb --no-color compile
196-
197-
# dbt v2 strict with parse cache
198-
dbt compile --profiles-dir ../profiles --no-version-check --static-analysis strict --partial-parse --partial-load
199-
200-
# dbt v2 --dirty: the cold run builds the cache, later runs use --dirty
201-
dbt compile --profiles-dir ../profiles --no-version-check --static-analysis strict --partial-parse --partial-load
202-
dbt compile --profiles-dir ../profiles --no-version-check --static-analysis strict --partial-load --dirty
203-
204-
# dbt v2 default, with and without strict
205-
dbt compile --profiles-dir ../profiles --no-version-check
206-
dbt compile --profiles-dir ../profiles --no-version-check --static-analysis strict
207-
```
208-
209-
The non-strict parse-cache and `--dirty` rows drop `--static-analysis strict`. The remote-sources row
210-
runs the default strict command in a copy of the dbt project without `schema_origin: local`.
211-
212-
## Reproduce the benchmark
213-
214-
The harness is in [`benchmarks/dbt-compare/`](https://github.com/chio-labs/sqlbuild/tree/main/benchmarks/dbt-compare),
215-
and its README describes every variant. Start the workflow with the versions measured here:
112+
The harness and the exact command for every variant are in
113+
[`benchmarks/dbt-compare/`](https://github.com/chio-labs/sqlbuild/tree/main/benchmarks/dbt-compare).
114+
To rerun it on GitHub Actions:
216115

217116
```bash
218117
gh workflow run benchmark-dbt.yml -f sqlbuild_version=0.164.3 -f dbt_version=2.0.9 -f rounds=3
219118
```
220119

221-
The run uploads the per-run files and a `summary.md` with the environment, the correctness table
222-
and the full results table.
223-
224-
To run it locally, from a checkout of the repository:
225-
226-
```bash
227-
uv venv -p 3.12 venv-sqb && uv pip install -p venv-sqb/bin/python sqlbuild==0.164.3 pytest duckdb
228-
uv venv -p 3.12 venv-dbt2 && uv pip install -p venv-dbt2/bin/python dbt==2.0.9
229-
230-
export SQB=$PWD/venv-sqb/bin/sqb DBT_V2=$PWD/venv-dbt2/bin/dbt PYTHON=$PWD/venv-sqb/bin/python
231-
232-
"$PYTHON" benchmarks/dbt-compare/write_dbt_dense.py --models 10000 --out /tmp/p10k
233-
benchmarks/dbt-compare/correctness.sh /tmp/correctness
234-
python3 benchmarks/dbt-compare/check_strict.py correctness /tmp/correctness
235-
benchmarks/dbt-compare/bench.sh /tmp/p10k /tmp/results 3
236-
python3 benchmarks/dbt-compare/check_strict.py runs /tmp/results
237-
python3 benchmarks/dbt-compare/summarize.py /tmp/results
238-
```
239-
240-
Use an otherwise idle machine: other load changes wall times far more than CPU times.
120+
The README also covers running it locally. Use an otherwise idle machine: load changes wall times
121+
far more than CPU times.
241122

242123
## Compiler guards in CI
243124

244-
SQLBuild's CI also runs fresh-process compile guards with time and memory budgets on dense and
245-
varied projects, and a custom-Rules benchmark with 20- and 100-rule packs. Run them from the
246-
repository:
247-
248-
```bash
249-
make test-e2e-dense-compile-performance SQLBUILD_BENCHMARK_MODELS=10000
250-
make test-e2e-cache-compile-performance SQLBUILD_BENCHMARK_MODELS=5000
251-
uv run python -m scripts.benchmark_rules --models 5000 --rule-counts 20 100 --iterations 5 --output rules-5000.json
252-
```
125+
SQLBuild's CI also runs compile time and memory guards on dense projects and a custom Rules
126+
benchmark; see `make test-e2e-dense-compile-performance` and `scripts/benchmark_rules.py`.

0 commit comments

Comments
 (0)