Skip to content

docs: list what the benchmark project contains #1195

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kvlonge merged 1 commit into from
Oct 21, 2026
Merged

docs: list what the benchmark project contains #1195

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 15 additions & 4 deletions src/sqlbuild/.agents/skills/sqlbuild/references/docs/benchmarks.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,10 +101,21 @@ compiles in full.

## The project and how to reproduce it

The project comes from the dense generator SQLBuild's compile CI uses: 10,000 models (31 columns on
average), 2,400 sources, 500 seeds, 250 SQL functions, 120 macros and 10,000 SQL tests. The dbt
project has the same names, folders, SQL and DAG. Each edit changes one constant in a model's SQL;
the 1% edit changes 100 models spread across the project. Rounds alternate the variant order.
The project comes from the dense generator SQLBuild's compile CI uses. The dbt project has the same
names, folders, SQL and DAG.

| What | Count | In dbt |
|---|---:|---|
| Models with typed columns (8–472 columns, about 31 on average) | 10,000 | models with enforced contracts |
| Sources | 2,400 | sources |
| Seeds | 500 | seeds |
| SQL functions | 250 | functions |
| Macros | 120 | Jinja macros producing the same SQL |
| `not_null` audits | 17,000 | `not_null` data tests |
| SQL tests with fixture rows | 10,000 | unit tests |

Each edit changes one constant in a model's SQL; the 1% edit changes 100 models spread across the
project. Rounds alternate the variant order.

The harness and the exact command for every variant are in
[`benchmarks/dbt-compare/`](https://github.com/chio-labs/sqlbuild/tree/main/benchmarks/dbt-compare).
Expand Down
19 changes: 15 additions & 4 deletions website/src/content/docs/docs/benchmarks.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,10 +89,21 @@ compiles in full.

## The project and how to reproduce it

The project comes from the dense generator SQLBuild's compile CI uses: 10,000 models (31 columns on
average), 2,400 sources, 500 seeds, 250 SQL functions, 120 macros and 10,000 SQL tests. The dbt
project has the same names, folders, SQL and DAG. Each edit changes one constant in a model's SQL;
the 1% edit changes 100 models spread across the project. Rounds alternate the variant order.
The project comes from the dense generator SQLBuild's compile CI uses. The dbt project has the same
names, folders, SQL and DAG.

| What | Count | In dbt |
|---|---:|---|
| Models with typed columns (8–472 columns, about 31 on average) | 10,000 | models with enforced contracts |
| Sources | 2,400 | sources |
| Seeds | 500 | seeds |
| SQL functions | 250 | functions |
| Macros | 120 | Jinja macros producing the same SQL |
| `not_null` audits | 17,000 | `not_null` data tests |
| SQL tests with fixture rows | 10,000 | unit tests |

Each edit changes one constant in a model's SQL; the 1% edit changes 100 models spread across the
project. Rounds alternate the variant order.

The harness and the exact command for every variant are in
[`benchmarks/dbt-compare/`](https://github.com/chio-labs/sqlbuild/tree/main/benchmarks/dbt-compare).
Expand Down
Loading