Skip to content

fix: faster Snowflake table-type conversion and accurate schema diffs - #789

Merged
kvlonge merged 5 commits into
mainfrom
fix/snowflake-plan-noise-and-table-type
Sep 23, 2026
Merged

kvlonge merged 5 commits into
mainfrom
fix/snowflake-plan-noise-and-table-type

Conversation

@kvlonge

@kvlonge kvlonge commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

Why

On large Snowflake projects sqb build spent a long time before any model ran, and plans were noisy:

  • Table-type conversion (for example transient → permanent) ran one table at a time on a single connection before the build started, with two metadata checks and a full data copy per table, including tables the same run was about to rebuild. It also recursed once per table.
  • The planner compared declared and warehouse column types without the adapter dialect, so Snowflake aliases (VARCHAR vs VARCHAR(16777216), INTEGER vs NUMBER(38,0), TIMESTAMP vs TIMESTAMP_NTZ) became type-mismatch warnings and false "Schema changed" entries. Upper-case declared names were reported as removed plus added, because the Snowflake adapter lowercases warehouse column names.
  • The table-type and warnings plan sections printed every item.

Changes

  • Schema comparison uses the dialect recorded on the planner warehouse snapshot, and matches names case-insensitively on Snowflake.
  • No conversion for tables the build always recreates with the planned type (table models, non-microbatch full rebuilds). Other drifted tables convert on their worker connection just before materialization; a failed conversion fails only that model before any DML. Microbatch full refreshes still convert because they can finish without recreating the table.
  • Table-type and warnings sections use the standard display cap.

Verification

  • New unit tests for alias and case handling and the recreation rule; a new DuckDB CLI build integration test proves routing, ordering and failure isolation (fails on main).
  • Real Snowflake runs on a private project: false warnings and schema changes gone with identical planned actions; deliberate drift converted only the incremental table, and the rebuilt table came out permanent with rows intact.
  • make test, make check-ci, one review cycle (finding fixed and re-verified).

@kvlonge
kvlonge enabled auto-merge (squash) September 23, 2026 12:50
@kvlonge
kvlonge merged commit a7a7e08 into main Sep 23, 2026
28 checks passed
@kvlonge
kvlonge deleted the fix/snowflake-plan-noise-and-table-type branch September 23, 2026 13:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant