Skip to content

Commit 742ad51

Browse files
kvlongekvlonge
committed
docs: lead the benchmarks page with the SQLBuild vs dbt v2 comparison
1 parent ee12488 commit 742ad51

5 files changed

Lines changed: 426 additions & 178 deletions

File tree

‎src/sqlbuild/.agents/skills/sqlbuild/references/docs/CONTENTS.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ Bundled copies of every page on the SQLBuild documentation site, matching the in
1010
- [Quickstart](quickstart.md) (`quickstart`) - Get a SQLBuild project running locally with DuckDB in under a minute.
1111
- [Feature Comparison](feature-comparison.md) (`feature-comparison`) - Feature comparison between SQLBuild, dbt, and SQLMesh.
1212
- [Roadmap](roadmap.md) (`roadmap`) - What's coming next in SQLBuild.
13-
- [Benchmarks](benchmarks.md) (`benchmarks`) - Complete compiler performance on projects with thousands of models, SQL tests, and audit attachments.
13+
- [Benchmarks](benchmarks.md) (`benchmarks`) - SQLBuild and dbt v2 compiling the same 10,000-model project: cold, unchanged, one-model and 1% edits, with CPU time and peak memory.
1414

1515
## dbt Compatibility
1616

‎src/sqlbuild/.agents/skills/sqlbuild/references/docs/benchmarks.md‎

Lines changed: 216 additions & 89 deletions
Original file line numberDiff line numberDiff line change
@@ -2,124 +2,251 @@
22

33
# Benchmarks
44

5-
> Complete compiler performance on projects with thousands of models, SQL tests, and audit attachments.
5+
> SQLBuild and dbt v2 compiling the same 10,000-model project: cold, unchanged, one-model and 1% edits, with CPU time and peak memory.
66
77
Online: https://sqlbuild.com/docs/benchmarks/
88

99
## Contents
1010

11-
- Compiler performance
12-
- Rules performance
11+
- SQLBuild vs dbt v2 with column checking
12+
- dbt v2 `--dirty`
13+
- dbt v2 out of the box and with remote sources
14+
- SQLBuild with every rule
15+
- Correctness
16+
- What is compared and what isn't
17+
- The project
1318
- Methodology
14-
- Reproduce the benchmarks
19+
- Reproduce the benchmark
20+
- Compiler guards in CI
1521

16-
SQLBuild compiles a 5,000-model project with nearly 5,000 SQL tests and more than 8,000 audit
17-
attachments from scratch in **19 seconds** in our CI benchmark. An unchanged recompile takes
18-
**13 seconds**.
22+
These numbers come from one run of the [benchmark workflow](https://github.com/chio-labs/sqlbuild/blob/main/.github/workflows/benchmark-dbt.yml):
23+
[run 39543189999](https://github.com/chio-labs/sqlbuild/actions/runs/39543189999), October 21, 2026,
24+
on a GitHub-hosted `ubuntu-24.04` runner with 4 CPUs (AMD EPYC 9V45) and 15 GiB RAM. It compared
25+
**SQLBuild 0.164.3** with **dbt 2.0.9** (the `dbt` package on PyPI), both installed from PyPI,
26+
on the same generated 10,000-model DuckDB project. Every cell is the median of three rounds.
27+
28+
## SQLBuild vs dbt v2 with column checking
29+
30+
Both tools here bind every column and type in every model. SQLBuild runs `sqb compile` with its
31+
defaults: compile with column and type checks, configurable rules off. dbt runs
32+
`dbt compile --static-analysis strict` with its parse cache turned on (`--partial-parse --partial-load`;
33+
the cache is opt-in and the command compiles the whole project).
34+
35+
| Scenario | SQLBuild wall | CPU | Peak memory | dbt v2 strict, parse cache wall | CPU | Peak memory | dbt / SQLBuild |
36+
|---|---:|---:|---:|---:|---:|---:|---:|
37+
| Cold (no `target/`, no caches) | **15.7 s** | 30.3 s | 3.1 GiB | 64.7 s | 167.0 s | 4.7 GiB | 4.1× |
38+
| Unchanged | **0.46 s** (replay) | 0.5 s | 0.07 GiB | 43.8 s | 132.2 s | 4.1 GiB | 95× |
39+
| One model edited | **2.4 s** | 2.5 s | 0.8 GiB | 45.9 s | 137.2 s | 4.1 GiB | 19× |
40+
| 100 models edited (1%) | **2.4 s** | 2.5 s | 0.8 GiB | 72.9 s | 176.1 s | 6.9 GiB | 30× |
41+
42+
SQLBuild's unchanged run replays the stored result of the previous compile. Its edit runs compile
43+
only the changed files against the previous compile and report the same result as a full compile.
44+
45+
The gap isn't only replay. With reuse and delta compile turned off, every SQLBuild run is a full
46+
compile that takes per-model analyses from its cache:
47+
48+
| Scenario | SQLBuild full compile wall | CPU | Peak memory | dbt v2 strict, parse cache wall | dbt / SQLBuild |
49+
|---|---:|---:|---:|---:|---:|
50+
| Cold | **14.2 s** | 28.4 s | 2.8 GiB | 64.7 s | 4.6× |
51+
| Unchanged | **8.6 s** | 12.9 s | 2.8 GiB | 43.8 s | 5.1× |
52+
| One model edited | **8.0 s** | 12.4 s | 2.8 GiB | 45.9 s | 5.7× |
53+
| 100 models edited (1%) | **8.1 s** | 12.6 s | 2.8 GiB | 72.9 s | 9.0× |
54+
55+
## dbt v2 `--dirty`
56+
57+
`dbt compile --dirty` compiles only models whose files changed since the last parse-cache run,
58+
plus all their downstream dependents. It needs the cache, so its cold run is a
59+
`--partial-parse --partial-load` compile.
60+
61+
| Scenario | SQLBuild | dbt v2 `--dirty`, strict | dbt v2 `--dirty` |
62+
|---|---:|---:|---:|
63+
| Cold | **15.7 s** | 63.3 s | 40.1 s |
64+
| Unchanged | **0.46 s** (replay) | 0.86 s (nothing to do) | 0.80 s (nothing to do) |
65+
| One model edited | 2.4 s | **1.6 s** (48 models) | **1.5 s** (48 models) |
66+
| 100 models edited (1%) | **2.4 s** | 60.7 s (2,800 models) | 48.6 s (2,800 models) |
67+
68+
`--dirty` is faster on a single-model edit: strict analysis of the edited model and its 47
69+
descendants took 1.6 s. SQLBuild took 2.4 s and reported the result for the whole project, the same
70+
report a full compile produces; models whose inputs didn't change kept their stored analysis. With
71+
100 edited files, dbt 2.0.9 logged `[partial-parse] ... exceeds incremental threshold (100), falling
72+
back to full parse` and compiled the 2,800 selected models after a full parse.
73+
74+
## dbt v2 out of the box and with remote sources
75+
76+
For context, dbt v2 without the parse cache, and strict analysis reading source schemas from the
77+
warehouse (dbt's default) instead of from their declarations:
78+
79+
| Scenario | dbt v2 default | dbt v2 default, strict | dbt v2 default, strict, sources from warehouse |
80+
|---|---:|---:|---:|
81+
| Cold | 36.9 s | 61.5 s | 67.4 s |
82+
| Unchanged | 45.9 s | 67.2 s | 74.3 s |
83+
| One model edited | 44.8 s | 67.4 s | 73.9 s |
84+
| 100 models edited (1%) | 43.4 s | 65.7 s | 73.5 s |
85+
86+
dbt v2's default `compile` doesn't report unknown columns.
87+
88+
## SQLBuild with every rule
89+
90+
The same project with every built-in rule and one custom Python rule turned on. These rules check
91+
contract, declaration, graph, model, project, SQL and test conventions that dbt compile doesn't
92+
check.
93+
94+
| Scenario | SQLBuild with rules | CPU | Peak memory | Full compile with per-model caches |
95+
|---|---:|---:|---:|---:|
96+
| Cold | 19.5 s | 42.0 s | 3.6 GiB | 18.0 s |
97+
| Unchanged | 0.46 s (replay) | 0.5 s | 0.07 GiB | 9.9 s |
98+
| One model edited | 2.9 s | 2.8 s | 0.9 GiB | 9.8 s |
99+
| 100 models edited (1%) | 3.1 s | 3.0 s | 0.9 GiB | 9.9 s |
100+
101+
Cold, it is still 3.3× faster than dbt v2 strict with its parse cache (64.7 s).
102+
103+
## Correctness
104+
105+
The workflow also runs both tools on three 50-model projects before timing anything:
106+
107+
| Project | `sqb compile` | `dbt compile` | `dbt compile --static-analysis strict` |
108+
|---|---|---|---|
109+
| Clean | exit 0 | exit 0 | exit 0 |
110+
| Reference to a missing model | exit 1, `P001` | exit 1, `dbt1048` | exit 1, `dbt1048` |
111+
| Column the upstream model doesn't have | exit 1, `B002 Unknown column 'missing_col' in b` | exit 0, nothing reported | exit 1, `dbt0227 No column b.missing_col found` |
112+
113+
Strict analysis must also report the missing column in the parse-cache and `--dirty` modes, and no
114+
strict run may mention a fallback, login or skipped analysis. Otherwise the workflow fails. SQLBuild
115+
reports the same errors with rules on and off.
116+
117+
## What is compared and what isn't
118+
119+
- Both tools bind columns and types. dbt does this only with `--static-analysis strict`.
120+
- SQLBuild compile also plans and binds its 10,000 SQL tests. dbt `compile` excludes unit tests.
121+
- SQLBuild reports contract, type and nullability problems as errors. dbt strict warns on contract
122+
type mismatches, and enforced contracts are checked at run time.
123+
- SQLBuild computes column lineage by default.
124+
- Sources are declared locally in both: dbt's sources set `schema_origin: local` and every column
125+
has a `data_type`. The remote-sources column above shows dbt's default.
126+
- The DuckDB warehouse holds every source and model relation, empty, as after `dbt build`. dbt reads
127+
it; `sqb compile` is offline and doesn't.
128+
- The measured dbt is the `dbt` distribution on PyPI. Strict static analysis isn't in its Apache-2.0
129+
build.
130+
- dbt's parse cache and `--dirty` are opt-in. SQLBuild's reuse and delta compile are on by default.
131+
132+
### Why SQLBuild can reuse work by default
133+
134+
SQLBuild records what each compile depended on, then checks all of it before reusing anything:
135+
136+
- **Every input is recorded during the compile itself.** That covers the project's files, the
137+
command line, the variables passed with `--vars`, every environment variable a template read,
138+
provider settings files, the Python interpreter and every loaded module, and the SQLBuild build.
139+
Nothing is inferred afterwards.
140+
- **Files are checked by type, size, modification time, change time and inode**, and by content
141+
digest when a recently touched file's stamps move. If anything doesn't match, the stored result
142+
isn't replayed.
143+
- **Edits recompile only what they affect.** SQLBuild knows each model's references and the
144+
columns, types and nullability it reads. A changed model is compiled again, together with every
145+
model whose inputs changed shape. An edit outside model and SQL test files, such as a macro,
146+
config or a new file, runs a full compile. That choice is made before any work starts, apart from
147+
a few edits found part-way through to need a full compile.
148+
- **The result is the same as a full compile.** A delta compile produces byte-identical output to
149+
`sqb compile --no-cache`, and a replay prints the previous report again. CI checks this by
150+
compiling a copy in full after every step of a seeded edit series and comparing the exit code and
151+
everything printed.
152+
153+
Reuse assumes macros, hooks, providers, adapters and custom rules are deterministic. It has
154+
[documented limits](cli/compile.md#reusing-an-unchanged-compile), for example on network
155+
filesystems or with clocks that differ between machines, and `--no-cache` always compiles in full.
156+
157+
## The project
158+
159+
The project comes from the dense generator SQLBuild's compile-performance CI jobs use, with the
160+
adapter switched to DuckDB. The dbt project is derived from it file by file: the same model names,
161+
folders, SQL bodies and DAG, with Jinja macros, `_models.yml` contracts, `data_tests` and
162+
`unit_tests`.
163+
164+
| Resource | Count |
165+
|---|---:|
166+
| Models | 10,000 (8 to 472 columns, 31 on average) |
167+
| Sources | 2,400 |
168+
| Seeds | 500 |
169+
| SQL functions | 250 |
170+
| Macros | 120 |
171+
| `not_null` checks | 17,000 |
172+
| SQL tests (dbt unit tests) | 10,000 |
173+
174+
Each edit changes one constant in a model's SQL body, so its columns, types and references stay the
175+
same. The 1% edit changes 100 models spread evenly across the project.
19176

20-
These measurements run the complete `sqb compile` command, including project discovery, dependency
21-
resolution, SQL expansion and analysis, contract validation, test and audit preparation, configured
22-
Rules, cache publication, and artifact generation.
23-
24-
## Compiler performance
25-
26-
| Compile time | **3k models**<br />2.9k tests · 5.1k audit attachments | **5k models**<br />4.9k tests · 8.4k audit attachments |
27-
|---|---:|---:|
28-
| First compile / cache fill | **12.48s** | **19.01s** |
29-
| Unchanged recompile | **8.79s** | **12.76s** |
30-
31-
Timings include test and audit preparation, not warehouse execution.
32-
33-
The 5,000-model compiler process used approximately **1.3 GiB peak resident memory** on its first
34-
compile and **0.7 GiB** on its unchanged recompile.
35-
36-
### Workload details
37-
38-
The deterministic generated projects exercise compiler work at scale, with synthetic SQL,
39-
contracts, macros, functions, tests, and audits.
177+
## Methodology
40178

41-
| Resource | 3k-model project | 5k-model project |
42-
|---|---:|---:|
43-
| Models | 3,000 | 5,000 |
44-
| Sources | 713 | 1,189 |
45-
| Seeds | 141 | 236 |
46-
| SQL functions | 71 | 118 |
47-
| Python macros | 37 | 61 |
48-
| SQL test cases | 2,945 | 4,908 |
49-
| Audit attachments | 5,056 | 8,427 |
179+
For every variant and round, the workflow copies the pristine project, runs the cold compile, runs
180+
it again unchanged, edits one model and compiles, then edits 100 models and compiles. Rounds
181+
alternate the variant order. `/usr/bin/time` records wall time, user plus system CPU time, and the
182+
peak resident memory of the largest process. dbt runs use `DBT_SEND_ANONYMOUS_USAGE_STATS=false` and
183+
a DuckDB profile with `threads: 4`.
50184

51-
The projects use repeated generated structures. Individual project timings depend on SQL
52-
complexity, macros, configuration, and hardware as well as model count.
185+
The exact commands, run in each project copy:
53186

54-
### Incremental compilation
187+
```bash
188+
# SQLBuild default
189+
sqb --no-color compile
55190

56-
SQLBuild reuses cached analysis for unaffected models. In the 5,000-model benchmark, a single-model
57-
file edit reused **4,999 model analyses**. A scoped Python-macro edit required fresh analysis for
58-
**7 models**. Both scenarios completed in approximately **9.8 seconds**.
191+
# SQLBuild full compile with per-model caches
192+
SQLBUILD_DISABLE_COMPILE_REUSE=1 SQLBUILD_DISABLE_DELTA_COMPILE=1 sqb --no-color compile
59193

60-
The single-model scenario appends a SQL comment to exercise file-change invalidation. Separate
61-
regression coverage checks expression-changing edits against uncached compilation results.
62-
Scenarios run in a fixed sequence and use different cache-loading paths; an edit timing below the
63-
initial unchanged timing does not mean editing inherently requires less work.
194+
# SQLBuild with every rule: the same command in the project with [rules] select = ["SQBR", "XSQBR"]
195+
sqb --no-color compile
64196

65-
## Rules performance
197+
# dbt v2 strict with parse cache
198+
dbt compile --profiles-dir ../profiles --no-version-check --static-analysis strict --partial-parse --partial-load
66199

67-
Rules participate in ordinary compilation. Model Rules are cached per model, and SQLBuild tracks
68-
their dependencies so unaffected findings can be reused after edits. Project Rules evaluate the
69-
complete project view and have broader invalidation.
200+
# dbt v2 --dirty: the cold run builds the cache, later runs use --dirty
201+
dbt compile --profiles-dir ../profiles --no-version-check --static-analysis strict --partial-parse --partial-load
202+
dbt compile --profiles-dir ../profiles --no-version-check --static-analysis strict --partial-load --dirty
70203

71-
A separate custom-Rules benchmark exercises 20- and 100-Rule packs over 5,000 models. It measures
72-
first compilation, unchanged compilation, model edits, and Rule implementation changes. This is a
73-
different workload from the compiler table above; its results should be compared within that profile.
204+
# dbt v2 default, with and without strict
205+
dbt compile --profiles-dir ../profiles --no-version-check
206+
dbt compile --profiles-dir ../profiles --no-version-check --static-analysis strict
207+
```
74208

75-
See [Execution and caching](concepts/rules/execution-and-caching.md) for the Rules lifecycle, or run
76-
the publication profile below to collect current custom-Rules measurements.
209+
The non-strict parse-cache and `--dirty` rows drop `--static-analysis strict`. The remote-sources row
210+
runs the default strict command in a copy of the dbt project without `schema_origin: local`.
77211

78-
## Methodology
212+
## Reproduce the benchmark
79213

80-
The compiler results above are individual scenario measurements from the
81-
[passing CI run on September 22, 2026](https://github.com/chio-labs/sqlbuild/actions/runs/35764207007),
82-
using the compiler code released in **SQLBuild 0.113.3** and Python 3.12 on Linux. They are not
83-
multi-run medians.
214+
The harness is in [`benchmarks/dbt-compare/`](https://github.com/chio-labs/sqlbuild/tree/main/benchmarks/dbt-compare),
215+
and its README describes every variant. Start the workflow with the versions measured here:
84216

85-
Each scenario launches a fresh process. The first compile starts with an empty project cache;
86-
subsequent scenarios reuse that cache. The CI job enforces a **4 GiB combined memory limit**.
87-
Reported peak resident memory is for the compiler process, not the entire CI job.
217+
```bash
218+
gh workflow run benchmark-dbt.yml -f sqlbuild_version=0.164.3 -f dbt_version=2.0.9 -f rounds=3
219+
```
88220

89-
CI checks cache-hit and miss expectations and pinned semantic fingerprints alongside timing and
90-
memory budgets. The varied 1,000-model guard additionally compares SQL-expression and macro edits
91-
with uncached analysis and Rules results.
221+
The run uploads the per-run files and a `summary.md` with the environment, the correctness table
222+
and the full results table.
92223

93-
These measurements cover compilation and static analysis. SQL tests and audits are prepared, not
94-
executed against a warehouse.
224+
To run it locally, from a checkout of the repository:
95225

96-
## Reproduce the benchmarks
226+
```bash
227+
uv venv -p 3.12 venv-sqb && uv pip install -p venv-sqb/bin/python sqlbuild==0.164.3 pytest duckdb
228+
uv venv -p 3.12 venv-dbt2 && uv pip install -p venv-dbt2/bin/python dbt==2.0.9
97229

98-
Run the fresh-process compiler cache guards from the
99-
[SQLBuild repository](https://github.com/chio-labs/sqlbuild):
230+
export SQB=$PWD/venv-sqb/bin/sqb DBT_V2=$PWD/venv-dbt2/bin/dbt PYTHON=$PWD/venv-sqb/bin/python
100231

101-
```bash
102-
make test-e2e-cache-compile-performance SQLBUILD_BENCHMARK_MODELS=3000
103-
make test-e2e-cache-compile-performance SQLBUILD_BENCHMARK_MODELS=5000
232+
"$PYTHON" benchmarks/dbt-compare/write_dbt_dense.py --models 10000 --out /tmp/p10k
233+
benchmarks/dbt-compare/correctness.sh /tmp/correctness
234+
python3 benchmarks/dbt-compare/check_strict.py correctness /tmp/correctness
235+
benchmarks/dbt-compare/bench.sh /tmp/p10k /tmp/results 3
236+
python3 benchmarks/dbt-compare/check_strict.py runs /tmp/results
237+
python3 benchmarks/dbt-compare/summarize.py /tmp/results
104238
```
105239

106-
Run the five-iteration custom-Rules publication profile:
240+
Use an otherwise idle machine: other load changes wall times far more than CPU times.
107241

108-
```bash
109-
uv run python -m scripts.benchmark_rules \
110-
--models 5000 \
111-
--rule-counts 20 100 \
112-
--iterations 5 \
113-
--output rules-5000.json
114-
```
242+
## Compiler guards in CI
115243

116-
Run the bounded Rules profile used by required CI:
244+
SQLBuild's CI also runs fresh-process compile guards with time and memory budgets on dense and
245+
varied projects, and a custom-Rules benchmark with 20- and 100-rule packs. Run them from the
246+
repository:
117247

118248
```bash
119-
uv run python -m scripts.benchmark_rules_ci \
120-
--output rules-performance-ci.json \
121-
--summary-output rules-performance-ci.md
249+
make test-e2e-dense-compile-performance SQLBUILD_BENCHMARK_MODELS=10000
250+
make test-e2e-cache-compile-performance SQLBUILD_BENCHMARK_MODELS=5000
251+
uv run python -m scripts.benchmark_rules --models 5000 --rule-counts 20 100 --iterations 5 --output rules-5000.json
122252
```
123-
124-
The benchmark generators, guard thresholds, and CI workflow are maintained in the public
125-
[SQLBuild repository](https://github.com/chio-labs/sqlbuild).

‎src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -227,7 +227,7 @@ Required CI exercises custom Rule packs over generated projects, checking compil
227227
memory, and cache invalidation. Separate 20- and 100-Rule profiles measure the cost of custom
228228
validation at 5,000-model scale.
229229

230-
See [Benchmarks](../benchmarks.md#rules-performance) for the benchmark profiles and reproduction commands,
230+
See [Benchmarks](../benchmarks.md#compiler-guards-in-ci) for the benchmark profiles and reproduction commands,
231231
and [Execution and caching](rules/execution-and-caching.md) for the Rules lifecycle.
232232

233233
## Continue

0 commit comments

Comments
 (0)