Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 22 additions & 19 deletions src/sqlbuild/.agents/skills/sqlbuild/references/docs/benchmarks.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,13 +9,14 @@ Online: https://sqlbuild.com/docs/benchmarks/
## Contents

- SQLBuild vs dbt v2
- The project
- All variants, 10,000 models
- What is compared
- Why SQLBuild can reuse work by default
- The project and how to reproduce it
- Reproduce
- Compiler guards in CI

SQLBuild 0.164.3 and dbt 2.0.9 compiling the same generated 10,000-model DuckDB project, on a
SQLBuild 0.164.3 and dbt 2.0.9 compiling the same generated [10,000-model DuckDB project](#the-project), on a
GitHub-hosted runner with 4 CPUs and 15 GiB RAM. Each cell is the median of three rounds from
[run 39543189999](https://github.com/chio-labs/sqlbuild/actions/runs/39543189999) (October 21, 2026).

Expand All @@ -40,6 +41,24 @@ Cold, SQLBuild used 30 s of CPU and 3.1 GiB at peak; dbt used 167 s and 4.7 GiB.
and their descendants: 1.6 s for the edited model and 47 descendants, against SQLBuild's 2.4 s for
a report on the whole project. With 100 edited files it falls back to a full parse and takes 60.7 s.

## The project

The project comes from the dense generator SQLBuild's compile CI uses. The dbt project has the same
names, folders, SQL and DAG.

| What | Count | In dbt |
|---|---:|---|
| Models with typed columns (8–472 columns, about 31 on average) | 10,000 | models with enforced contracts |
| Sources | 2,400 | sources |
| Seeds | 500 | seeds |
| SQL functions | 250 | functions |
| Macros | 120 | Jinja macros producing the same SQL |
| `not_null` audits | 17,000 | `not_null` data tests |
| SQL tests with fixture rows | 10,000 | unit tests |

Each edit changes one constant in a model's SQL; the 1% edit changes 100 models spread across the
project. Rounds alternate the variant order.

## All variants, 10,000 models

Wall time in seconds on the same 10,000-model project. CPU and memory for every cell are in the run's `summary.md`.
Expand Down Expand Up @@ -99,23 +118,7 @@ Reuse assumes macros, hooks, providers, adapters and custom rules are determinis
[documented limits](cli/compile.md#reusing-an-unchanged-compile), and `--no-cache` always
compiles in full.

## The project and how to reproduce it

The project comes from the dense generator SQLBuild's compile CI uses. The dbt project has the same
names, folders, SQL and DAG.

| What | Count | In dbt |
|---|---:|---|
| Models with typed columns (8–472 columns, about 31 on average) | 10,000 | models with enforced contracts |
| Sources | 2,400 | sources |
| Seeds | 500 | seeds |
| SQL functions | 250 | functions |
| Macros | 120 | Jinja macros producing the same SQL |
| `not_null` audits | 17,000 | `not_null` data tests |
| SQL tests with fixture rows | 10,000 | unit tests |

Each edit changes one constant in a model's SQL; the 1% edit changes 100 models spread across the
project. Rounds alternate the variant order.
## Reproduce

The harness and the exact command for every variant are in
[`benchmarks/dbt-compare/`](https://github.com/chio-labs/sqlbuild/tree/main/benchmarks/dbt-compare).
Expand Down
38 changes: 20 additions & 18 deletions website/src/content/docs/docs/benchmarks.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ title: "Benchmarks"
description: "SQLBuild and dbt v2 compiling the same 10,000-model project: cold, unchanged, one-model and 1% edits."
---

SQLBuild 0.164.3 and dbt 2.0.9 compiling the same generated 10,000-model DuckDB project, on a
SQLBuild 0.164.3 and dbt 2.0.9 compiling the same generated [10,000-model DuckDB project](#the-project), on a
GitHub-hosted runner with 4 CPUs and 15 GiB RAM. Each cell is the median of three rounds from
[run 39543189999](https://github.com/chio-labs/sqlbuild/actions/runs/39543189999) (October 21, 2026).

Expand All @@ -28,6 +28,24 @@ Cold, SQLBuild used 30 s of CPU and 3.1 GiB at peak; dbt used 167 s and 4.7 GiB.
and their descendants: 1.6 s for the edited model and 47 descendants, against SQLBuild's 2.4 s for
a report on the whole project. With 100 edited files it falls back to a full parse and takes 60.7 s.

## The project

The project comes from the dense generator SQLBuild's compile CI uses. The dbt project has the same
names, folders, SQL and DAG.

| What | Count | In dbt |
|---|---:|---|
| Models with typed columns (8–472 columns, about 31 on average) | 10,000 | models with enforced contracts |
| Sources | 2,400 | sources |
| Seeds | 500 | seeds |
| SQL functions | 250 | functions |
| Macros | 120 | Jinja macros producing the same SQL |
| `not_null` audits | 17,000 | `not_null` data tests |
| SQL tests with fixture rows | 10,000 | unit tests |

Each edit changes one constant in a model's SQL; the 1% edit changes 100 models spread across the
project. Rounds alternate the variant order.

## All variants, 10,000 models

Wall time in seconds on the same 10,000-model project. CPU and memory for every cell are in the run's `summary.md`.
Expand Down Expand Up @@ -87,23 +105,7 @@ Reuse assumes macros, hooks, providers, adapters and custom rules are determinis
[documented limits](/docs/cli/compile#reusing-an-unchanged-compile), and `--no-cache` always
compiles in full.

## The project and how to reproduce it

The project comes from the dense generator SQLBuild's compile CI uses. The dbt project has the same
names, folders, SQL and DAG.

| What | Count | In dbt |
|---|---:|---|
| Models with typed columns (8–472 columns, about 31 on average) | 10,000 | models with enforced contracts |
| Sources | 2,400 | sources |
| Seeds | 500 | seeds |
| SQL functions | 250 | functions |
| Macros | 120 | Jinja macros producing the same SQL |
| `not_null` audits | 17,000 | `not_null` data tests |
| SQL tests with fixture rows | 10,000 | unit tests |

Each edit changes one constant in a model's SQL; the 1% edit changes 100 models spread across the
project. Rounds alternate the variant order.
## Reproduce

The harness and the exact command for every variant are in
[`benchmarks/dbt-compare/`](https://github.com/chio-labs/sqlbuild/tree/main/benchmarks/dbt-compare).
Expand Down
Loading