Skip to content

docs: lead the benchmarks page with the SQLBuild vs dbt v2 comparison #1191

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kvlonge merged 1 commit into from
Oct 21, 2026
Merged

docs: lead the benchmarks page with the SQLBuild vs dbt v2 comparison #1191

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/sqlbuild/.agents/skills/sqlbuild/references/docs/CONTENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Bundled copies of every page on the SQLBuild documentation site, matching the in
- [Quickstart](quickstart.md) (`quickstart`) - Get a SQLBuild project running locally with DuckDB in under a minute.
- [Feature Comparison](feature-comparison.md) (`feature-comparison`) - Feature comparison between SQLBuild, dbt, and SQLMesh.
- [Roadmap](roadmap.md) (`roadmap`) - What's coming next in SQLBuild.
- [Benchmarks](benchmarks.md) (`benchmarks`) - Complete compiler performance on projects with thousands of models, SQL tests, and audit attachments.
- [Benchmarks](benchmarks.md) (`benchmarks`) - SQLBuild and dbt v2 compiling the same 10,000-model project: cold, unchanged, one-model and 1% edits, with CPU time and peak memory.

## dbt Compatibility

Expand Down
305 changes: 216 additions & 89 deletions src/sqlbuild/.agents/skills/sqlbuild/references/docs/benchmarks.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,124 +2,251 @@

# Benchmarks

> Complete compiler performance on projects with thousands of models, SQL tests, and audit attachments.
> SQLBuild and dbt v2 compiling the same 10,000-model project: cold, unchanged, one-model and 1% edits, with CPU time and peak memory.

Online: https://sqlbuild.com/docs/benchmarks/

## Contents

- Compiler performance
- Rules performance
- SQLBuild vs dbt v2 with column checking
- dbt v2 `--dirty`
- dbt v2 out of the box and with remote sources
- SQLBuild with every rule
- Correctness
- What is compared and what isn't
- The project
- Methodology
- Reproduce the benchmarks
- Reproduce the benchmark
- Compiler guards in CI

SQLBuild compiles a 5,000-model project with nearly 5,000 SQL tests and more than 8,000 audit
attachments from scratch in **19 seconds** in our CI benchmark. An unchanged recompile takes
**13 seconds**.
These numbers come from one run of the [benchmark workflow](https://github.com/chio-labs/sqlbuild/blob/main/.github/workflows/benchmark-dbt.yml):
[run 39543189999](https://github.com/chio-labs/sqlbuild/actions/runs/39543189999), October 21, 2026,
on a GitHub-hosted `ubuntu-24.04` runner with 4 CPUs (AMD EPYC 9V45) and 15 GiB RAM. It compared
**SQLBuild 0.164.3** with **dbt 2.0.9** (the `dbt` package on PyPI), both installed from PyPI,
on the same generated 10,000-model DuckDB project. Every cell is the median of three rounds.

## SQLBuild vs dbt v2 with column checking

Both tools here bind every column and type in every model. SQLBuild runs `sqb compile` with its
defaults: compile with column and type checks, configurable rules off. dbt runs
`dbt compile --static-analysis strict` with its parse cache turned on (`--partial-parse --partial-load`;
the cache is opt-in and the command compiles the whole project).

| Scenario | SQLBuild wall | CPU | Peak memory | dbt v2 strict, parse cache wall | CPU | Peak memory | dbt / SQLBuild |
|---|---:|---:|---:|---:|---:|---:|---:|
| Cold (no `target/`, no caches) | **15.7 s** | 30.3 s | 3.1 GiB | 64.7 s | 167.0 s | 4.7 GiB | 4.1× |
| Unchanged | **0.46 s** (replay) | 0.5 s | 0.07 GiB | 43.8 s | 132.2 s | 4.1 GiB | 95× |
| One model edited | **2.4 s** | 2.5 s | 0.8 GiB | 45.9 s | 137.2 s | 4.1 GiB | 19× |
| 100 models edited (1%) | **2.4 s** | 2.5 s | 0.8 GiB | 72.9 s | 176.1 s | 6.9 GiB | 30× |

SQLBuild's unchanged run replays the stored result of the previous compile. Its edit runs compile
only the changed files against the previous compile and report the same result as a full compile.

The gap isn't only replay. With reuse and delta compile turned off, every SQLBuild run is a full
compile that takes per-model analyses from its cache:

| Scenario | SQLBuild full compile wall | CPU | Peak memory | dbt v2 strict, parse cache wall | dbt / SQLBuild |
|---|---:|---:|---:|---:|---:|
| Cold | **14.2 s** | 28.4 s | 2.8 GiB | 64.7 s | 4.6× |
| Unchanged | **8.6 s** | 12.9 s | 2.8 GiB | 43.8 s | 5.1× |
| One model edited | **8.0 s** | 12.4 s | 2.8 GiB | 45.9 s | 5.7× |
| 100 models edited (1%) | **8.1 s** | 12.6 s | 2.8 GiB | 72.9 s | 9.0× |

## dbt v2 `--dirty`

`dbt compile --dirty` compiles only models whose files changed since the last parse-cache run,
plus all their downstream dependents. It needs the cache, so its cold run is a
`--partial-parse --partial-load` compile.

| Scenario | SQLBuild | dbt v2 `--dirty`, strict | dbt v2 `--dirty` |
|---|---:|---:|---:|
| Cold | **15.7 s** | 63.3 s | 40.1 s |
| Unchanged | **0.46 s** (replay) | 0.86 s (nothing to do) | 0.80 s (nothing to do) |
| One model edited | 2.4 s | **1.6 s** (48 models) | **1.5 s** (48 models) |
| 100 models edited (1%) | **2.4 s** | 60.7 s (2,800 models) | 48.6 s (2,800 models) |

`--dirty` is faster on a single-model edit: strict analysis of the edited model and its 47
descendants took 1.6 s. SQLBuild took 2.4 s and reported the result for the whole project, the same
report a full compile produces; models whose inputs didn't change kept their stored analysis. With
100 edited files, dbt 2.0.9 logged `[partial-parse] ... exceeds incremental threshold (100), falling
back to full parse` and compiled the 2,800 selected models after a full parse.

## dbt v2 out of the box and with remote sources

For context, dbt v2 without the parse cache, and strict analysis reading source schemas from the
warehouse (dbt's default) instead of from their declarations:

| Scenario | dbt v2 default | dbt v2 default, strict | dbt v2 default, strict, sources from warehouse |
|---|---:|---:|---:|
| Cold | 36.9 s | 61.5 s | 67.4 s |
| Unchanged | 45.9 s | 67.2 s | 74.3 s |
| One model edited | 44.8 s | 67.4 s | 73.9 s |
| 100 models edited (1%) | 43.4 s | 65.7 s | 73.5 s |

dbt v2's default `compile` doesn't report unknown columns.

## SQLBuild with every rule

The same project with every built-in rule and one custom Python rule turned on. These rules check
contract, declaration, graph, model, project, SQL and test conventions that dbt compile doesn't
check.

| Scenario | SQLBuild with rules | CPU | Peak memory | Full compile with per-model caches |
|---|---:|---:|---:|---:|
| Cold | 19.5 s | 42.0 s | 3.6 GiB | 18.0 s |
| Unchanged | 0.46 s (replay) | 0.5 s | 0.07 GiB | 9.9 s |
| One model edited | 2.9 s | 2.8 s | 0.9 GiB | 9.8 s |
| 100 models edited (1%) | 3.1 s | 3.0 s | 0.9 GiB | 9.9 s |

Cold, it is still 3.3× faster than dbt v2 strict with its parse cache (64.7 s).

## Correctness

The workflow also runs both tools on three 50-model projects before timing anything:

| Project | `sqb compile` | `dbt compile` | `dbt compile --static-analysis strict` |
|---|---|---|---|
| Clean | exit 0 | exit 0 | exit 0 |
| Reference to a missing model | exit 1, `P001` | exit 1, `dbt1048` | exit 1, `dbt1048` |
| Column the upstream model doesn't have | exit 1, `B002 Unknown column 'missing_col' in b` | exit 0, nothing reported | exit 1, `dbt0227 No column b.missing_col found` |

Strict analysis must also report the missing column in the parse-cache and `--dirty` modes, and no
strict run may mention a fallback, login or skipped analysis. Otherwise the workflow fails. SQLBuild
reports the same errors with rules on and off.

## What is compared and what isn't

- Both tools bind columns and types. dbt does this only with `--static-analysis strict`.
- SQLBuild compile also plans and binds its 10,000 SQL tests. dbt `compile` excludes unit tests.
- SQLBuild reports contract, type and nullability problems as errors. dbt strict warns on contract
type mismatches, and enforced contracts are checked at run time.
- SQLBuild computes column lineage by default.
- Sources are declared locally in both: dbt's sources set `schema_origin: local` and every column
has a `data_type`. The remote-sources column above shows dbt's default.
- The DuckDB warehouse holds every source and model relation, empty, as after `dbt build`. dbt reads
it; `sqb compile` is offline and doesn't.
- The measured dbt is the `dbt` distribution on PyPI. Strict static analysis isn't in its Apache-2.0
build.
- dbt's parse cache and `--dirty` are opt-in. SQLBuild's reuse and delta compile are on by default.

### Why SQLBuild can reuse work by default

SQLBuild records what each compile depended on, then checks all of it before reusing anything:

- **Every input is recorded during the compile itself.** That covers the project's files, the
command line, the variables passed with `--vars`, every environment variable a template read,
provider settings files, the Python interpreter and every loaded module, and the SQLBuild build.
Nothing is inferred afterwards.
- **Files are checked by type, size, modification time, change time and inode**, and by content
digest when a recently touched file's stamps move. If anything doesn't match, the stored result
isn't replayed.
- **Edits recompile only what they affect.** SQLBuild knows each model's references and the
columns, types and nullability it reads. A changed model is compiled again, together with every
model whose inputs changed shape. An edit outside model and SQL test files, such as a macro,
config or a new file, runs a full compile. That choice is made before any work starts, apart from
a few edits found part-way through to need a full compile.
- **The result is the same as a full compile.** A delta compile produces byte-identical output to
`sqb compile --no-cache`, and a replay prints the previous report again. CI checks this by
compiling a copy in full after every step of a seeded edit series and comparing the exit code and
everything printed.

Reuse assumes macros, hooks, providers, adapters and custom rules are deterministic. It has
[documented limits](cli/compile.md#reusing-an-unchanged-compile), for example on network
filesystems or with clocks that differ between machines, and `--no-cache` always compiles in full.

## The project

The project comes from the dense generator SQLBuild's compile-performance CI jobs use, with the
adapter switched to DuckDB. The dbt project is derived from it file by file: the same model names,
folders, SQL bodies and DAG, with Jinja macros, `_models.yml` contracts, `data_tests` and
`unit_tests`.

| Resource | Count |
|---|---:|
| Models | 10,000 (8 to 472 columns, 31 on average) |
| Sources | 2,400 |
| Seeds | 500 |
| SQL functions | 250 |
| Macros | 120 |
| `not_null` checks | 17,000 |
| SQL tests (dbt unit tests) | 10,000 |

Each edit changes one constant in a model's SQL body, so its columns, types and references stay the
same. The 1% edit changes 100 models spread evenly across the project.

These measurements run the complete `sqb compile` command, including project discovery, dependency
resolution, SQL expansion and analysis, contract validation, test and audit preparation, configured
Rules, cache publication, and artifact generation.

## Compiler performance

| Compile time | **3k models**<br />2.9k tests · 5.1k audit attachments | **5k models**<br />4.9k tests · 8.4k audit attachments |
|---|---:|---:|
| First compile / cache fill | **12.48s** | **19.01s** |
| Unchanged recompile | **8.79s** | **12.76s** |

Timings include test and audit preparation, not warehouse execution.

The 5,000-model compiler process used approximately **1.3 GiB peak resident memory** on its first
compile and **0.7 GiB** on its unchanged recompile.

### Workload details

The deterministic generated projects exercise compiler work at scale, with synthetic SQL,
contracts, macros, functions, tests, and audits.
## Methodology

| Resource | 3k-model project | 5k-model project |
|---|---:|---:|
| Models | 3,000 | 5,000 |
| Sources | 713 | 1,189 |
| Seeds | 141 | 236 |
| SQL functions | 71 | 118 |
| Python macros | 37 | 61 |
| SQL test cases | 2,945 | 4,908 |
| Audit attachments | 5,056 | 8,427 |
For every variant and round, the workflow copies the pristine project, runs the cold compile, runs
it again unchanged, edits one model and compiles, then edits 100 models and compiles. Rounds
alternate the variant order. `/usr/bin/time` records wall time, user plus system CPU time, and the
peak resident memory of the largest process. dbt runs use `DBT_SEND_ANONYMOUS_USAGE_STATS=false` and
a DuckDB profile with `threads: 4`.

The projects use repeated generated structures. Individual project timings depend on SQL
complexity, macros, configuration, and hardware as well as model count.
The exact commands, run in each project copy:

### Incremental compilation
```bash
# SQLBuild default
sqb --no-color compile

SQLBuild reuses cached analysis for unaffected models. In the 5,000-model benchmark, a single-model
file edit reused **4,999 model analyses**. A scoped Python-macro edit required fresh analysis for
**7 models**. Both scenarios completed in approximately **9.8 seconds**.
# SQLBuild full compile with per-model caches
SQLBUILD_DISABLE_COMPILE_REUSE=1 SQLBUILD_DISABLE_DELTA_COMPILE=1 sqb --no-color compile

The single-model scenario appends a SQL comment to exercise file-change invalidation. Separate
regression coverage checks expression-changing edits against uncached compilation results.
Scenarios run in a fixed sequence and use different cache-loading paths; an edit timing below the
initial unchanged timing does not mean editing inherently requires less work.
# SQLBuild with every rule: the same command in the project with [rules] select = ["SQBR", "XSQBR"]
sqb --no-color compile

## Rules performance
# dbt v2 strict with parse cache
dbt compile --profiles-dir ../profiles --no-version-check --static-analysis strict --partial-parse --partial-load

Rules participate in ordinary compilation. Model Rules are cached per model, and SQLBuild tracks
their dependencies so unaffected findings can be reused after edits. Project Rules evaluate the
complete project view and have broader invalidation.
# dbt v2 --dirty: the cold run builds the cache, later runs use --dirty
dbt compile --profiles-dir ../profiles --no-version-check --static-analysis strict --partial-parse --partial-load
dbt compile --profiles-dir ../profiles --no-version-check --static-analysis strict --partial-load --dirty

A separate custom-Rules benchmark exercises 20- and 100-Rule packs over 5,000 models. It measures
first compilation, unchanged compilation, model edits, and Rule implementation changes. This is a
different workload from the compiler table above; its results should be compared within that profile.
# dbt v2 default, with and without strict
dbt compile --profiles-dir ../profiles --no-version-check
dbt compile --profiles-dir ../profiles --no-version-check --static-analysis strict
```

See [Execution and caching](concepts/rules/execution-and-caching.md) for the Rules lifecycle, or run
the publication profile below to collect current custom-Rules measurements.
The non-strict parse-cache and `--dirty` rows drop `--static-analysis strict`. The remote-sources row
runs the default strict command in a copy of the dbt project without `schema_origin: local`.

## Methodology
## Reproduce the benchmark

The compiler results above are individual scenario measurements from the
[passing CI run on September 22, 2026](https://github.com/chio-labs/sqlbuild/actions/runs/35764207007),
using the compiler code released in **SQLBuild 0.113.3** and Python 3.12 on Linux. They are not
multi-run medians.
The harness is in [`benchmarks/dbt-compare/`](https://github.com/chio-labs/sqlbuild/tree/main/benchmarks/dbt-compare),
and its README describes every variant. Start the workflow with the versions measured here:

Each scenario launches a fresh process. The first compile starts with an empty project cache;
subsequent scenarios reuse that cache. The CI job enforces a **4 GiB combined memory limit**.
Reported peak resident memory is for the compiler process, not the entire CI job.
```bash
gh workflow run benchmark-dbt.yml -f sqlbuild_version=0.164.3 -f dbt_version=2.0.9 -f rounds=3
```

CI checks cache-hit and miss expectations and pinned semantic fingerprints alongside timing and
memory budgets. The varied 1,000-model guard additionally compares SQL-expression and macro edits
with uncached analysis and Rules results.
The run uploads the per-run files and a `summary.md` with the environment, the correctness table
and the full results table.

These measurements cover compilation and static analysis. SQL tests and audits are prepared, not
executed against a warehouse.
To run it locally, from a checkout of the repository:

## Reproduce the benchmarks
```bash
uv venv -p 3.12 venv-sqb && uv pip install -p venv-sqb/bin/python sqlbuild==0.164.3 pytest duckdb
uv venv -p 3.12 venv-dbt2 && uv pip install -p venv-dbt2/bin/python dbt==2.0.9

Run the fresh-process compiler cache guards from the
[SQLBuild repository](https://github.com/chio-labs/sqlbuild):
export SQB=$PWD/venv-sqb/bin/sqb DBT_V2=$PWD/venv-dbt2/bin/dbt PYTHON=$PWD/venv-sqb/bin/python

```bash
make test-e2e-cache-compile-performance SQLBUILD_BENCHMARK_MODELS=3000
make test-e2e-cache-compile-performance SQLBUILD_BENCHMARK_MODELS=5000
"$PYTHON" benchmarks/dbt-compare/write_dbt_dense.py --models 10000 --out /tmp/p10k
benchmarks/dbt-compare/correctness.sh /tmp/correctness
python3 benchmarks/dbt-compare/check_strict.py correctness /tmp/correctness
benchmarks/dbt-compare/bench.sh /tmp/p10k /tmp/results 3
python3 benchmarks/dbt-compare/check_strict.py runs /tmp/results
python3 benchmarks/dbt-compare/summarize.py /tmp/results
```

Run the five-iteration custom-Rules publication profile:
Use an otherwise idle machine: other load changes wall times far more than CPU times.

```bash
uv run python -m scripts.benchmark_rules \
--models 5000 \
--rule-counts 20 100 \
--iterations 5 \
--output rules-5000.json
```
## Compiler guards in CI

Run the bounded Rules profile used by required CI:
SQLBuild's CI also runs fresh-process compile guards with time and memory budgets on dense and
varied projects, and a custom-Rules benchmark with 20- and 100-rule packs. Run them from the
repository:

```bash
uv run python -m scripts.benchmark_rules_ci \
--output rules-performance-ci.json \
--summary-output rules-performance-ci.md
make test-e2e-dense-compile-performance SQLBUILD_BENCHMARK_MODELS=10000
make test-e2e-cache-compile-performance SQLBUILD_BENCHMARK_MODELS=5000
uv run python -m scripts.benchmark_rules --models 5000 --rule-counts 20 100 --iterations 5 --output rules-5000.json
```

The benchmark generators, guard thresholds, and CI workflow are maintained in the public
[SQLBuild repository](https://github.com/chio-labs/sqlbuild).
2 changes: 1 addition & 1 deletion src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/rules.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -227,7 +227,7 @@ Required CI exercises custom Rule packs over generated projects, checking compil
memory, and cache invalidation. Separate 20- and 100-Rule profiles measure the cost of custom
validation at 5,000-model scale.

See [Benchmarks](../benchmarks.md#rules-performance) for the benchmark profiles and reproduction commands,
See [Benchmarks](../benchmarks.md#compiler-guards-in-ci) for the benchmark profiles and reproduction commands,
and [Execution and caching](rules/execution-and-caching.md) for the Rules lifecycle.

## Continue
Expand Down
Loading
Loading