From 46cda0230e889fec15c989dc72be7e97370ab30c Mon Sep 17 00:00:00 2001 From: Kevin Longe <273747162+kvlonge@users.noreply.github.com> Date: Wed, 21 Oct 2026 08:04:52 +0100 Subject: [PATCH] docs: describe the benchmark project before the variant table --- .../sqlbuild/references/docs/benchmarks.md | 41 ++++++++++--------- website/src/content/docs/docs/benchmarks.mdx | 38 +++++++++-------- 2 files changed, 42 insertions(+), 37 deletions(-) diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/benchmarks.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/benchmarks.md index f0f1b11f31..4856edf6ec 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/benchmarks.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/benchmarks.md @@ -9,13 +9,14 @@ Online: https://sqlbuild.com/docs/benchmarks/ ## Contents - SQLBuild vs dbt v2 +- The project - All variants, 10,000 models - What is compared - Why SQLBuild can reuse work by default -- The project and how to reproduce it +- Reproduce - Compiler guards in CI -SQLBuild 0.164.3 and dbt 2.0.9 compiling the same generated 10,000-model DuckDB project, on a +SQLBuild 0.164.3 and dbt 2.0.9 compiling the same generated [10,000-model DuckDB project](#the-project), on a GitHub-hosted runner with 4 CPUs and 15 GiB RAM. Each cell is the median of three rounds from [run 39543189999](https://github.com/chio-labs/sqlbuild/actions/runs/39543189999) (October 21, 2026). @@ -40,6 +41,24 @@ Cold, SQLBuild used 30 s of CPU and 3.1 GiB at peak; dbt used 167 s and 4.7 GiB. and their descendants: 1.6 s for the edited model and 47 descendants, against SQLBuild's 2.4 s for a report on the whole project. With 100 edited files it falls back to a full parse and takes 60.7 s. +## The project + +The project comes from the dense generator SQLBuild's compile CI uses. The dbt project has the same +names, folders, SQL and DAG. + +| What | Count | In dbt | +|---|---:|---| +| Models with typed columns (8–472 columns, about 31 on average) | 10,000 | models with enforced contracts | +| Sources | 2,400 | sources | +| Seeds | 500 | seeds | +| SQL functions | 250 | functions | +| Macros | 120 | Jinja macros producing the same SQL | +| `not_null` audits | 17,000 | `not_null` data tests | +| SQL tests with fixture rows | 10,000 | unit tests | + +Each edit changes one constant in a model's SQL; the 1% edit changes 100 models spread across the +project. Rounds alternate the variant order. + ## All variants, 10,000 models Wall time in seconds on the same 10,000-model project. CPU and memory for every cell are in the run's `summary.md`. @@ -99,23 +118,7 @@ Reuse assumes macros, hooks, providers, adapters and custom rules are determinis [documented limits](cli/compile.md#reusing-an-unchanged-compile), and `--no-cache` always compiles in full. -## The project and how to reproduce it - -The project comes from the dense generator SQLBuild's compile CI uses. The dbt project has the same -names, folders, SQL and DAG. - -| What | Count | In dbt | -|---|---:|---| -| Models with typed columns (8–472 columns, about 31 on average) | 10,000 | models with enforced contracts | -| Sources | 2,400 | sources | -| Seeds | 500 | seeds | -| SQL functions | 250 | functions | -| Macros | 120 | Jinja macros producing the same SQL | -| `not_null` audits | 17,000 | `not_null` data tests | -| SQL tests with fixture rows | 10,000 | unit tests | - -Each edit changes one constant in a model's SQL; the 1% edit changes 100 models spread across the -project. Rounds alternate the variant order. +## Reproduce The harness and the exact command for every variant are in [`benchmarks/dbt-compare/`](https://github.com/chio-labs/sqlbuild/tree/main/benchmarks/dbt-compare). diff --git a/website/src/content/docs/docs/benchmarks.mdx b/website/src/content/docs/docs/benchmarks.mdx index 84a2b8a13a..d09b5c9c66 100644 --- a/website/src/content/docs/docs/benchmarks.mdx +++ b/website/src/content/docs/docs/benchmarks.mdx @@ -3,7 +3,7 @@ title: "Benchmarks" description: "SQLBuild and dbt v2 compiling the same 10,000-model project: cold, unchanged, one-model and 1% edits." --- -SQLBuild 0.164.3 and dbt 2.0.9 compiling the same generated 10,000-model DuckDB project, on a +SQLBuild 0.164.3 and dbt 2.0.9 compiling the same generated [10,000-model DuckDB project](#the-project), on a GitHub-hosted runner with 4 CPUs and 15 GiB RAM. Each cell is the median of three rounds from [run 39543189999](https://github.com/chio-labs/sqlbuild/actions/runs/39543189999) (October 21, 2026). @@ -28,6 +28,24 @@ Cold, SQLBuild used 30 s of CPU and 3.1 GiB at peak; dbt used 167 s and 4.7 GiB. and their descendants: 1.6 s for the edited model and 47 descendants, against SQLBuild's 2.4 s for a report on the whole project. With 100 edited files it falls back to a full parse and takes 60.7 s. +## The project + +The project comes from the dense generator SQLBuild's compile CI uses. The dbt project has the same +names, folders, SQL and DAG. + +| What | Count | In dbt | +|---|---:|---| +| Models with typed columns (8–472 columns, about 31 on average) | 10,000 | models with enforced contracts | +| Sources | 2,400 | sources | +| Seeds | 500 | seeds | +| SQL functions | 250 | functions | +| Macros | 120 | Jinja macros producing the same SQL | +| `not_null` audits | 17,000 | `not_null` data tests | +| SQL tests with fixture rows | 10,000 | unit tests | + +Each edit changes one constant in a model's SQL; the 1% edit changes 100 models spread across the +project. Rounds alternate the variant order. + ## All variants, 10,000 models Wall time in seconds on the same 10,000-model project. CPU and memory for every cell are in the run's `summary.md`. @@ -87,23 +105,7 @@ Reuse assumes macros, hooks, providers, adapters and custom rules are determinis [documented limits](/docs/cli/compile#reusing-an-unchanged-compile), and `--no-cache` always compiles in full. -## The project and how to reproduce it - -The project comes from the dense generator SQLBuild's compile CI uses. The dbt project has the same -names, folders, SQL and DAG. - -| What | Count | In dbt | -|---|---:|---| -| Models with typed columns (8–472 columns, about 31 on average) | 10,000 | models with enforced contracts | -| Sources | 2,400 | sources | -| Seeds | 500 | seeds | -| SQL functions | 250 | functions | -| Macros | 120 | Jinja macros producing the same SQL | -| `not_null` audits | 17,000 | `not_null` data tests | -| SQL tests with fixture rows | 10,000 | unit tests | - -Each edit changes one constant in a model's SQL; the 1% edit changes 100 models spread across the -project. Rounds alternate the variant order. +## Reproduce The harness and the exact command for every variant are in [`benchmarks/dbt-compare/`](https://github.com/chio-labs/sqlbuild/tree/main/benchmarks/dbt-compare).