Skip to content

Commit 0cf575c

Browse files
authored
docs: describe the benchmark project before the variant table (#1196)
1 parent 3dc05b8 commit 0cf575c

2 files changed

Lines changed: 42 additions & 37 deletions

File tree

‎src/sqlbuild/.agents/skills/sqlbuild/references/docs/benchmarks.md‎

Lines changed: 22 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -9,13 +9,14 @@ Online: https://sqlbuild.com/docs/benchmarks/
99
## Contents
1010

1111
- SQLBuild vs dbt v2
12+
- The project
1213
- All variants, 10,000 models
1314
- What is compared
1415
- Why SQLBuild can reuse work by default
15-
- The project and how to reproduce it
16+
- Reproduce
1617
- Compiler guards in CI
1718

18-
SQLBuild 0.164.3 and dbt 2.0.9 compiling the same generated 10,000-model DuckDB project, on a
19+
SQLBuild 0.164.3 and dbt 2.0.9 compiling the same generated [10,000-model DuckDB project](#the-project), on a
1920
GitHub-hosted runner with 4 CPUs and 15 GiB RAM. Each cell is the median of three rounds from
2021
[run 39543189999](https://github.com/chio-labs/sqlbuild/actions/runs/39543189999) (October 21, 2026).
2122

@@ -40,6 +41,24 @@ Cold, SQLBuild used 30 s of CPU and 3.1 GiB at peak; dbt used 167 s and 4.7 GiB.
4041
and their descendants: 1.6 s for the edited model and 47 descendants, against SQLBuild's 2.4 s for
4142
a report on the whole project. With 100 edited files it falls back to a full parse and takes 60.7 s.
4243

44+
## The project
45+
46+
The project comes from the dense generator SQLBuild's compile CI uses. The dbt project has the same
47+
names, folders, SQL and DAG.
48+
49+
| What | Count | In dbt |
50+
|---|---:|---|
51+
| Models with typed columns (8–472 columns, about 31 on average) | 10,000 | models with enforced contracts |
52+
| Sources | 2,400 | sources |
53+
| Seeds | 500 | seeds |
54+
| SQL functions | 250 | functions |
55+
| Macros | 120 | Jinja macros producing the same SQL |
56+
| `not_null` audits | 17,000 | `not_null` data tests |
57+
| SQL tests with fixture rows | 10,000 | unit tests |
58+
59+
Each edit changes one constant in a model's SQL; the 1% edit changes 100 models spread across the
60+
project. Rounds alternate the variant order.
61+
4362
## All variants, 10,000 models
4463

4564
Wall time in seconds on the same 10,000-model project. CPU and memory for every cell are in the run's `summary.md`.
@@ -99,23 +118,7 @@ Reuse assumes macros, hooks, providers, adapters and custom rules are determinis
99118
[documented limits](cli/compile.md#reusing-an-unchanged-compile), and `--no-cache` always
100119
compiles in full.
101120

102-
## The project and how to reproduce it
103-
104-
The project comes from the dense generator SQLBuild's compile CI uses. The dbt project has the same
105-
names, folders, SQL and DAG.
106-
107-
| What | Count | In dbt |
108-
|---|---:|---|
109-
| Models with typed columns (8–472 columns, about 31 on average) | 10,000 | models with enforced contracts |
110-
| Sources | 2,400 | sources |
111-
| Seeds | 500 | seeds |
112-
| SQL functions | 250 | functions |
113-
| Macros | 120 | Jinja macros producing the same SQL |
114-
| `not_null` audits | 17,000 | `not_null` data tests |
115-
| SQL tests with fixture rows | 10,000 | unit tests |
116-
117-
Each edit changes one constant in a model's SQL; the 1% edit changes 100 models spread across the
118-
project. Rounds alternate the variant order.
121+
## Reproduce
119122

120123
The harness and the exact command for every variant are in
121124
[`benchmarks/dbt-compare/`](https://github.com/chio-labs/sqlbuild/tree/main/benchmarks/dbt-compare).

‎website/src/content/docs/docs/benchmarks.mdx‎

Lines changed: 20 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ title: "Benchmarks"
33
description: "SQLBuild and dbt v2 compiling the same 10,000-model project: cold, unchanged, one-model and 1% edits."
44
---
55

6-
SQLBuild 0.164.3 and dbt 2.0.9 compiling the same generated 10,000-model DuckDB project, on a
6+
SQLBuild 0.164.3 and dbt 2.0.9 compiling the same generated [10,000-model DuckDB project](#the-project), on a
77
GitHub-hosted runner with 4 CPUs and 15 GiB RAM. Each cell is the median of three rounds from
88
[run 39543189999](https://github.com/chio-labs/sqlbuild/actions/runs/39543189999) (October 21, 2026).
99

@@ -28,6 +28,24 @@ Cold, SQLBuild used 30 s of CPU and 3.1 GiB at peak; dbt used 167 s and 4.7 GiB.
2828
and their descendants: 1.6 s for the edited model and 47 descendants, against SQLBuild's 2.4 s for
2929
a report on the whole project. With 100 edited files it falls back to a full parse and takes 60.7 s.
3030

31+
## The project
32+
33+
The project comes from the dense generator SQLBuild's compile CI uses. The dbt project has the same
34+
names, folders, SQL and DAG.
35+
36+
| What | Count | In dbt |
37+
|---|---:|---|
38+
| Models with typed columns (8–472 columns, about 31 on average) | 10,000 | models with enforced contracts |
39+
| Sources | 2,400 | sources |
40+
| Seeds | 500 | seeds |
41+
| SQL functions | 250 | functions |
42+
| Macros | 120 | Jinja macros producing the same SQL |
43+
| `not_null` audits | 17,000 | `not_null` data tests |
44+
| SQL tests with fixture rows | 10,000 | unit tests |
45+
46+
Each edit changes one constant in a model's SQL; the 1% edit changes 100 models spread across the
47+
project. Rounds alternate the variant order.
48+
3149
## All variants, 10,000 models
3250

3351
Wall time in seconds on the same 10,000-model project. CPU and memory for every cell are in the run's `summary.md`.
@@ -87,23 +105,7 @@ Reuse assumes macros, hooks, providers, adapters and custom rules are determinis
87105
[documented limits](/docs/cli/compile#reusing-an-unchanged-compile), and `--no-cache` always
88106
compiles in full.
89107

90-
## The project and how to reproduce it
91-
92-
The project comes from the dense generator SQLBuild's compile CI uses. The dbt project has the same
93-
names, folders, SQL and DAG.
94-
95-
| What | Count | In dbt |
96-
|---|---:|---|
97-
| Models with typed columns (8–472 columns, about 31 on average) | 10,000 | models with enforced contracts |
98-
| Sources | 2,400 | sources |
99-
| Seeds | 500 | seeds |
100-
| SQL functions | 250 | functions |
101-
| Macros | 120 | Jinja macros producing the same SQL |
102-
| `not_null` audits | 17,000 | `not_null` data tests |
103-
| SQL tests with fixture rows | 10,000 | unit tests |
104-
105-
Each edit changes one constant in a model's SQL; the 1% edit changes 100 models spread across the
106-
project. Rounds alternate the variant order.
108+
## Reproduce
107109

108110
The harness and the exact command for every variant are in
109111
[`benchmarks/dbt-compare/`](https://github.com/chio-labs/sqlbuild/tree/main/benchmarks/dbt-compare).

0 commit comments

Comments
 (0)