Skip to content

ci: benchmark dbt v2 partial parse and dirty compiles - #1189

Merged
kvlonge merged 7 commits into
mainfrom
ci/bench-dbt-v2-cache
Oct 21, 2026
Merged

kvlonge merged 7 commits into
mainfrom
ci/bench-dbt-v2-cache

Conversation

@kvlonge

@kvlonge kvlonge commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Why

dbt v2 can cache between runs when asked (--partial-parse, hidden in 2.0.9; --partial-load; --dirty compiles only changed models and descendants), so timing only its default is unfair. Strict analysis is only in the dbt distribution and can fall back silently, so the workflow must prove it ran. dbt Core is a reference only.

Changes

  • bench.sh: dbt v2 cached, --dirty and default variants, each also strict. v2_compile_strict_remote reads source schemas from the warehouse; sqb_norules_full added. .cmd per run.
  • write_dbt_dense.py: sources declare schema_origin: local (like SQLBuild); dbt-remote/ keeps dbt's default. warehouse.duckdb holds every source and model relation, empty, typed, matching materialization.
  • check_strict.py, run by the workflow after correctness (before the benchmark) and after it: the seeded unknown column fails strict with dbt0227 in default, remote, cached and --dirty modes; plain compile passes it; clean passes strict; every timed strict run exits 0; no strict output matches the listed fallback patterns.
  • summarize.py: labelled rows (SQLBuild default first; dbt v2 cached, --dirty, default, strict alongside), "warm (replay)", full-parse fallback and skipped analysis noted.
  • Defaults SQLBuild and dbt v2 only; dbt Core installed only for core_*.
  • README: warehouse, sources, v2 modes, strict proof patterns and equivalence notes.

Verification

  • 50 models: correctness matrix and check_strict.py correctness pass; a fake fallback line or a passing strict run fails it.
  • 100 models: all eleven default variants exit 0; check_strict.py runs passes; local sources download no schemas.
  • sqb compile output identical without the warehouse.
  • actionlint, ruff, hygiene scan pass.

@kvlonge
kvlonge enabled auto-merge (squash) October 21, 2026 03:06
@kvlonge
kvlonge disabled auto-merge October 21, 2026 03:37
@kvlonge
kvlonge enabled auto-merge (squash) October 21, 2026 03:47
@kvlonge
kvlonge merged commit ee12488 into main Oct 21, 2026
26 checks passed
@kvlonge
kvlonge deleted the ci/bench-dbt-v2-cache branch October 21, 2026 04:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant