You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit ee12488
Browse filesBrowse the repository at this point in the historyBrowse files
ci: benchmark dbt v2 partial parse and dirty compiles (#1189)
* ci(bench): benchmark dbt v2 partial parse and dirty compiles
* style(bench): wrap a long docstring line
* ci(bench): add strict dirty dbt v2 runs and lead with cached dbt v2 rows
* ci(bench): create every model relation in the benchmark warehouse
* ci(bench): prove dbt v2 strict analysis ran and declare source columns locally
For the dbt Core reference rows, install `dbt-core dbt-duckdb` in another environment, set
97
+
`DBT_CORE` to its `dbt`, and pass `core_parse core_compile` to `bench.sh`. `correctness.sh` adds dbt
98
+
Core rows when `DBT_CORE` is set.
99
+
80
100
The generator imports the CI helpers from the checkout, so it needs a Python with `sqlbuild`,
81
101
`duckdb` and `pytest` installed, and works best with a sqlbuild release close to the checkout.
82
102
Nothing is built from source. Run on an otherwise idle machine: other load changes wall times far
@@ -88,41 +108,106 @@ For every variant and round, `bench.sh` copies the pristine project into `work/V
88
108
five seconds, then runs:
89
109
90
110
1.**cold**: the command with no `target/` and no caches.
91
-
2.**warm**: the same command again with nothing changed.
111
+
2.**warm**: the same command again with nothing changed. SQLBuild's default replays the stored
112
+
result of the previous compile; the summary labels these rows "warm (replay)".
92
113
3.**edit1**: one model's SQL body edited, then the command.
93
114
4.**edit100**: 100 models' SQL bodies edited (1% of 10,000), then the command.
94
115
95
116
Each edit changes one constant in a projection, so the query changes but its columns, types and
96
117
references don't. Rounds alternate the variant order. `SCENARIOS` limits which of warm, edit1 and
97
118
edit100 run.
98
119
99
-
Commands, run from the project copy:
120
+
Commands, run from the project copy. Every dbt command also gets
121
+
`--profiles-dir ../profiles --no-version-check`.
100
122
101
123
| Variant | Command |
102
124
|---|---|
103
-
|`sqb_rules`|`sqb --no-color compile` in `sqlbuild/`|
104
-
|`sqb_norules`|`sqb --no-color compile` in `sqlbuild-norules/`|
105
-
|`sqb_rules_full`|`sqb_rules` with `SQLBUILD_DISABLE_COMPILE_REUSE=1 SQLBUILD_DISABLE_DELTA_COMPILE=1`: every run is a full compile that uses the per-model caches |
|`v2_compile_strict`| dbt v2 `compile --static-analysis strict`, which checks columns like SQLBuild does |
108
-
|`core_parse` / `core_compile`| the same `parse` / `compile` commands with dbt Core |
109
-
110
-
Both tools use their defaults otherwise: SQLBuild's compile cache, reuse and delta compile, and
111
-
dbt's partial parsing. `DBT_SEND_ANONYMOUS_USAGE_STATS=false` is set. The DuckDB profile sets
112
-
`threads: 4`.
125
+
|`sqb_norules`| SQLBuild default (compile with column and type checks): `sqb --no-color compile` in `sqlbuild-norules/`|
126
+
|`sqb_norules_full`|`sqb_norules` with `SQLBUILD_DISABLE_COMPILE_REUSE=1 SQLBUILD_DISABLE_DELTA_COMPILE=1`: every run is a full compile with per-model caches |
127
+
|`sqb_rules`| SQLBuild with every built-in rule and a custom Python rule: `sqb --no-color compile` in `sqlbuild/`|
128
+
|`sqb_rules_full`|`sqb_rules` as a full compile with per-model caches, as for `sqb_norules_full`|
129
+
|`v2_compile_cached`| dbt v2 with parse cache (opt-in; compiles the whole project): `compile --partial-parse --partial-load`|
130
+
|`v2_compile_strict_cached`| the same with `--static-analysis strict`|
131
+
|`v2_compile_dirty`| dbt v2 `--dirty` (changed models and descendants only). Cold: `compile --partial-parse --partial-load`; then `compile --partial-load --dirty`|
132
+
|`v2_compile_strict_dirty`| the same with `--static-analysis strict` on every run |
133
+
|`v2_compile`| dbt v2 default: `compile`|
134
+
|`v2_compile_strict`| dbt v2 default with `--static-analysis strict`, which binds columns and types; contract and nullability checks differ |
135
+
|`v2_compile_strict_remote`|`v2_compile_strict` in `dbt-remote/`: source schemas downloaded from the warehouse on every run. Default mode, because the parse cache doesn't skip the download either, so the difference from `v2_compile_strict` is the introspection cost |
136
+
|`v2_parse`| dbt v2 `parse` (not run by default) |
137
+
|`core_parse` / `core_compile`| dbt Core `parse` / `compile` (not run by default) |
138
+
139
+
SQLBuild uses its defaults: the compile cache, replaying an unchanged compile, and compiling only
140
+
changed files after an edit. In SQLBuild's default, configurable rules are off (`sqb_norules`). dbt Core uses its default partial parsing.
141
+
`DBT_SEND_ANONYMOUS_USAGE_STATS=false` is set. The DuckDB profile sets `threads: 4`.
142
+
143
+
### dbt v2 modes
144
+
145
+
dbt v2 2.0.9 has three ways to run `compile`. The benchmark times each, with and without
146
+
`--static-analysis strict`, in this order:
147
+
148
+
-**With parse cache** (`*_cached`): `--partial-parse` keeps a parse cache under
149
+
`target/private/metadata/` and `--partial-load` loads it faster. The cache is opt-in, and
150
+
`--partial-parse` is hidden from `dbt --help` in 2.0.9. The cold run builds the cache. Every later
151
+
run still compiles the whole project.
152
+
-**`--dirty`** (`*_dirty`): selects only models whose files changed, by modification time, since the
153
+
last `--partial-parse` run, plus all their downstream dependents. It needs that cache, so the cold
154
+
run is a `--partial-parse --partial-load` compile and is timed as cold. With nothing changed it
155
+
reports `NoNodesSelected` ("Nothing to do"). Each run refreshes the cache, so each edit is compared
156
+
with the run before it. Dense models form long `ref` chains, so one edited model can select dozens
157
+
of descendants.
158
+
- In both cached modes, when more than 100 files changed since the cache was written, dbt 2.0.9 logs
159
+
`[partial-parse] ... falling back to full parse`. The summary marks those runs "full parse", and
0 commit comments