diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/benchmarks.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/benchmarks.md index baec97cbe8..dd2dfa673e 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/benchmarks.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/benchmarks.md @@ -22,25 +22,23 @@ GitHub-hosted runner with 4 CPUs and 15 GiB RAM. Each cell is the median of thre ## SQLBuild vs dbt v2 Both tools bind every column and type in every model. SQLBuild runs `sqb compile` with its defaults -(configurable rules off). dbt runs `dbt compile --static-analysis strict` with its opt-in parse cache -(`--partial-parse --partial-load`). +(configurable rules off). dbt runs `dbt compile --static-analysis strict` (strict: with column and +type checks) with its opt-in cache (`--partial-parse --partial-load`). | Scenario | SQLBuild | dbt v2 | dbt / SQLBuild | |---|---:|---:|---:| | Cold (no `target/`, no caches) | **15.7 s** | 64.7 s | 4.1× | -| Unchanged | **0.46 s** (replay) | 43.8 s | 95× | +| Unchanged | **0.46 s** (reused) | 43.8 s | 95× | | One model edited | **2.4 s** | 45.9 s | 19× | | 100 models edited (1%) | **2.4 s** | 72.9 s | 30× | Cold, SQLBuild used 30 s of CPU and 3.1 GiB at peak; dbt used 167 s and 4.7 GiB. -- **Reuse isn't the whole gap.** With reuse and delta compile off, every SQLBuild run is a full - compile with per-model caches: 14.2 s cold and about 8 s after an edit, still 5–9× faster. -- **Every rule on.** With every built-in rule and a custom Python rule, SQLBuild compiles cold in +- **All rules on.** With every built-in rule and a custom Python rule, SQLBuild compiles cold in 19.5 s, 3.3× faster than dbt strict. -- **dbt `--dirty` wins a one-model edit.** It compiles only changed models and their descendants: 1.6 s - for the edited model and 47 descendants, against SQLBuild's 2.4 s for a report on the whole - project. With 100 edited files it falls back to a full parse and takes 60.7 s. +- **dbt's changed-models-only mode wins a one-model edit.** `--dirty` compiles only changed models + and their descendants: 1.6 s for the edited model and 47 descendants, against SQLBuild's 2.4 s for + a report on the whole project. With 100 edited files it falls back to a full parse and takes 60.7 s. ## All variants @@ -49,15 +47,14 @@ Wall time in seconds. CPU and memory for every cell are in the run's `summary.md | Variant | Cold | Unchanged | 1 edit | 100 edits | |---|---:|---:|---:|---:| | SQLBuild | 15.7 | 0.46 | 2.4 | 2.4 | -| SQLBuild, reuse and delta off | 14.2 | 8.6 | 8.0 | 8.1 | -| SQLBuild, every rule | 19.5 | 0.46 | 2.9 | 3.1 | -| dbt v2 strict, parse cache | 64.7 | 43.8 | 45.9 | 72.9 | -| dbt v2 strict, `--dirty` | 63.3 | 0.86 | 1.6 | 60.7 | -| dbt v2 strict, default | 61.5 | 67.2 | 67.4 | 65.7 | -| dbt v2 strict, sources from warehouse | 67.4 | 74.3 | 73.9 | 73.5 | -| dbt v2, parse cache | 39.5 | 21.8 | 24.0 | 47.9 | -| dbt v2, `--dirty` | 40.1 | 0.80 | 1.5 | 48.6 | -| dbt v2, default | 36.9 | 45.9 | 44.8 | 43.4 | +| SQLBuild, all rules on | 19.5 | 0.46 | 2.9 | 3.1 | +| dbt v2 strict, with cache | 64.7 | 43.8 | 45.9 | 72.9 | +| dbt v2 strict, changed models only (`--dirty`) | 63.3 | 0.86 | 1.6 | 60.7 | +| dbt v2 strict, no cache | 61.5 | 67.2 | 67.4 | 65.7 | +| dbt v2 strict, no cache, sources read from the warehouse | 67.4 | 74.3 | 73.9 | 73.5 | +| dbt v2, with cache | 39.5 | 21.8 | 24.0 | 47.9 | +| dbt v2, changed models only (`--dirty`) | 40.1 | 0.80 | 1.5 | 48.6 | +| dbt v2, no cache | 36.9 | 45.9 | 44.8 | 43.4 | dbt v2 without `--static-analysis strict` doesn't report unknown columns. @@ -70,7 +67,7 @@ dbt v2 without `--static-analysis strict` doesn't report unknown columns. type mismatches and checks enforced contracts at run time. - Sources are declared locally in both. The warehouse holds every relation, empty, as after `dbt build`; dbt reads it and `sqb compile` doesn't. -- dbt's parse cache and `--dirty` are opt-in. SQLBuild's reuse and delta compile are on by default. +- dbt's cache and changed-models-only mode are opt-in. SQLBuild reuses unchanged work by default. Before timing anything, the workflow checks both tools on 50-model projects, and fails if strict analysis doesn't report the missing column in every dbt mode: @@ -90,12 +87,12 @@ SQLBuild records what each compile depended on and checks all of it before reusi loaded modules, and the SQLBuild build. - **Files are checked by type, size, modification time, change time and inode**, and by content digest when a recently touched file's stamps move. If anything doesn't match, the stored result - isn't replayed. + isn't reused. - **Edits recompile only what they affect.** A changed model is compiled again, with every model whose input columns, types or nullability changed. Edits outside model and SQL test files, such as a macro, config or a new file, run a full compile. -- **The result matches a full compile.** A delta compile produces byte-identical output to - `sqb compile --no-cache`, and a replay prints the previous report again. CI compiles a copy in full +- **The result matches a full compile.** Recompiling only the edited models produces byte-identical + output to `sqb compile --no-cache`, and a reused compile prints the previous report again. CI compiles a copy in full after every step of a seeded edit series and compares the exit code and everything printed. Reuse assumes macros, hooks, providers, adapters and custom rules are deterministic. It has diff --git a/website/src/content/docs/docs/benchmarks.mdx b/website/src/content/docs/docs/benchmarks.mdx index 523b81aee8..dcd157e9e2 100644 --- a/website/src/content/docs/docs/benchmarks.mdx +++ b/website/src/content/docs/docs/benchmarks.mdx @@ -10,25 +10,23 @@ GitHub-hosted runner with 4 CPUs and 15 GiB RAM. Each cell is the median of thre ## SQLBuild vs dbt v2 Both tools bind every column and type in every model. SQLBuild runs `sqb compile` with its defaults -(configurable rules off). dbt runs `dbt compile --static-analysis strict` with its opt-in parse cache -(`--partial-parse --partial-load`). +(configurable rules off). dbt runs `dbt compile --static-analysis strict` (strict: with column and +type checks) with its opt-in cache (`--partial-parse --partial-load`). | Scenario | SQLBuild | dbt v2 | dbt / SQLBuild | |---|---:|---:|---:| | Cold (no `target/`, no caches) | **15.7 s** | 64.7 s | 4.1× | -| Unchanged | **0.46 s** (replay) | 43.8 s | 95× | +| Unchanged | **0.46 s** (reused) | 43.8 s | 95× | | One model edited | **2.4 s** | 45.9 s | 19× | | 100 models edited (1%) | **2.4 s** | 72.9 s | 30× | Cold, SQLBuild used 30 s of CPU and 3.1 GiB at peak; dbt used 167 s and 4.7 GiB. -- **Reuse isn't the whole gap.** With reuse and delta compile off, every SQLBuild run is a full - compile with per-model caches: 14.2 s cold and about 8 s after an edit, still 5–9× faster. -- **Every rule on.** With every built-in rule and a custom Python rule, SQLBuild compiles cold in +- **All rules on.** With every built-in rule and a custom Python rule, SQLBuild compiles cold in 19.5 s, 3.3× faster than dbt strict. -- **dbt `--dirty` wins a one-model edit.** It compiles only changed models and their descendants: 1.6 s - for the edited model and 47 descendants, against SQLBuild's 2.4 s for a report on the whole - project. With 100 edited files it falls back to a full parse and takes 60.7 s. +- **dbt's changed-models-only mode wins a one-model edit.** `--dirty` compiles only changed models + and their descendants: 1.6 s for the edited model and 47 descendants, against SQLBuild's 2.4 s for + a report on the whole project. With 100 edited files it falls back to a full parse and takes 60.7 s. ## All variants @@ -37,15 +35,14 @@ Wall time in seconds. CPU and memory for every cell are in the run's `summary.md | Variant | Cold | Unchanged | 1 edit | 100 edits | |---|---:|---:|---:|---:| | SQLBuild | 15.7 | 0.46 | 2.4 | 2.4 | -| SQLBuild, reuse and delta off | 14.2 | 8.6 | 8.0 | 8.1 | -| SQLBuild, every rule | 19.5 | 0.46 | 2.9 | 3.1 | -| dbt v2 strict, parse cache | 64.7 | 43.8 | 45.9 | 72.9 | -| dbt v2 strict, `--dirty` | 63.3 | 0.86 | 1.6 | 60.7 | -| dbt v2 strict, default | 61.5 | 67.2 | 67.4 | 65.7 | -| dbt v2 strict, sources from warehouse | 67.4 | 74.3 | 73.9 | 73.5 | -| dbt v2, parse cache | 39.5 | 21.8 | 24.0 | 47.9 | -| dbt v2, `--dirty` | 40.1 | 0.80 | 1.5 | 48.6 | -| dbt v2, default | 36.9 | 45.9 | 44.8 | 43.4 | +| SQLBuild, all rules on | 19.5 | 0.46 | 2.9 | 3.1 | +| dbt v2 strict, with cache | 64.7 | 43.8 | 45.9 | 72.9 | +| dbt v2 strict, changed models only (`--dirty`) | 63.3 | 0.86 | 1.6 | 60.7 | +| dbt v2 strict, no cache | 61.5 | 67.2 | 67.4 | 65.7 | +| dbt v2 strict, no cache, sources read from the warehouse | 67.4 | 74.3 | 73.9 | 73.5 | +| dbt v2, with cache | 39.5 | 21.8 | 24.0 | 47.9 | +| dbt v2, changed models only (`--dirty`) | 40.1 | 0.80 | 1.5 | 48.6 | +| dbt v2, no cache | 36.9 | 45.9 | 44.8 | 43.4 | dbt v2 without `--static-analysis strict` doesn't report unknown columns. @@ -58,7 +55,7 @@ dbt v2 without `--static-analysis strict` doesn't report unknown columns. type mismatches and checks enforced contracts at run time. - Sources are declared locally in both. The warehouse holds every relation, empty, as after `dbt build`; dbt reads it and `sqb compile` doesn't. -- dbt's parse cache and `--dirty` are opt-in. SQLBuild's reuse and delta compile are on by default. +- dbt's cache and changed-models-only mode are opt-in. SQLBuild reuses unchanged work by default. Before timing anything, the workflow checks both tools on 50-model projects, and fails if strict analysis doesn't report the missing column in every dbt mode: @@ -78,12 +75,12 @@ SQLBuild records what each compile depended on and checks all of it before reusi loaded modules, and the SQLBuild build. - **Files are checked by type, size, modification time, change time and inode**, and by content digest when a recently touched file's stamps move. If anything doesn't match, the stored result - isn't replayed. + isn't reused. - **Edits recompile only what they affect.** A changed model is compiled again, with every model whose input columns, types or nullability changed. Edits outside model and SQL test files, such as a macro, config or a new file, run a full compile. -- **The result matches a full compile.** A delta compile produces byte-identical output to - `sqb compile --no-cache`, and a replay prints the previous report again. CI compiles a copy in full +- **The result matches a full compile.** Recompiling only the edited models produces byte-identical + output to `sqb compile --no-cache`, and a reused compile prints the previous report again. CI compiles a copy in full after every step of a seeded edit series and compares the exit code and everything printed. Reuse assumes macros, hooks, providers, adapters and custom rules are deterministic. It has