Repository navigation
feat!: move to DuckDB 2.0 - #1260
Merged
Merged
Conversation
Raise the duckdb requirement from 1.3.2 to 2.0.0 and lock 2.0.0. Fresh installs already resolved 2.0.0 because the requirement had no upper bound; this makes 2.0 the tested and supported version. DuckDB 2.0 writes new database files in storage version v2.0.0+, which DuckDB 1.x cannot open. SQLBuild still leaves the warehouse file at DuckDB's default, so the storage tests now expect v2.0.0+, and the DuckDB adapter page explains how to keep a new file readable by 1.x. BREAKING CHANGE: SQLBuild now requires duckdb>=2.0.0. Database files SQLBuild creates use DuckDB 2.0 storage and cannot be opened by DuckDB 1.x; existing files keep their storage version.
DuckDB 2.0 lets ALTER TABLE ... RENAME rename a view, where 1.x refused. Promoting a table over a model that was a view swapped the view into the staging name, then failed to drop it as a table, so a model changed from view to table failed in promotion. The DuckDB swap now refuses a view on its left before renaming anything, so promotion takes its existing path for a view destination: drop the view and rename the staged table into place.
DuckDB 2.0 refuses connection calls other than execute() while a result from execute() is not read to its end, and a fetchone() never reads to the end. Registering the VARIANT comparison function after a fetchone() therefore failed every local scenario that compares VARIANT or JSON output, and a Python UDF registered after such a read would fail the same way. Both registrations now go through one DuckDB adapter helper that finishes the open result before calling create_function.
…rrupted Local replay relied on DuckDB stopping a statement when Ctrl-C raised KeyboardInterrupt in the main thread. DuckDB 2.0 no longer does, so an interrupted local replay kept running its statement until it finished. Local replay now registers its run-scoped connection with the running scenarios, like target runs, so Ctrl-C interrupts the statement.
DuckDB 2.0 disables the single-arrow lambda syntax by default, so the scenario fixture that checks lambda parameters are not mock columns now uses lambda x: syntax.
DuckDB 2.0 stores a view's join in parentheses, so the clone test expects that text. Relations no longer have fetchone(), so the CTE-name helper reads json_serialize_sql through execute().
…uild A build-aside full refresh moves the live relation to __sqb_prev__ with ALTER TABLE ... RENAME. On DuckDB 2.0 that also renames a view, so a model changed from a view to a snapshot or microbatch model left a __sqb_prev__ view behind, and the next full refresh failed with R008 when it dropped that name as a table. DuckDB rename now refuses a view, as swap already does, with a dedicated error. The full-refresh promotion catches it, drops the live view and renames the rebuild into place; a view has no data to keep as the previous generation.
DuckDB 2.0 stores a view's SQL without its own catalog prefix, so a view at a database-qualified location never matched the three-part SELECT SQLBuild renders for it. Old-name compatibility views were then recreated on every refresh, and the janitor stopped recognising them. When the plain comparison fails, the DuckDB adapter drops the view's own catalog qualifier from both texts and compares again.
MotherDuck's extension refuses to load on DuckDB 2.0 and supports DuckDB up to 1.5.6. Connecting now reports that MotherDuck does not support the installed DuckDB yet and that MotherDuck users should stay on the last SQLBuild release that supports DuckDB 1.5, instead of only the extension's initialisation error. Other connection errors are raised unchanged. The MotherDuck adapter page says the same.
The DuckDB adapter page now covers arrow lambdas, partial ~ matching, MERGE with duplicate source rows, untyped NULL columns, integer division and modulo by zero, and dlt's dataset reads. The rename page's lambda example uses the lambda x: syntax DuckDB 2.0 accepts.
Local replay transpiles captured Snowflake SQL to DuckDB, and DuckDB 2.0 rejects the arrow lambda the transpiler wrote. This E2E test replays a Snowflake FILTER(..., x -> ...) model locally. It passes once the pinned polyglot-sql-chio release writes DuckDB lambdas as LAMBDA x : body.
…rename Guarding DuckDB rename added a catalog read to every rename, so a janitor run archiving many tables issued one extra query per table. Only the build-aside full refresh renames a target that may be a view, so it now reads the live relation type once, through the relation metadata cache, on DuckDB and MotherDuck, and replaces a live view instead of moving it aside. DuckDB rename no longer reads the catalog.
…dlt's whole-table Arrow reads The MotherDuck connection error no longer repeats its help text in the message, the dlt note says only whole-table Arrow reads fail, and the local lambda replay test is an expected failure until the polyglot wheel that writes DuckDB lambdas as LAMBDA x : body is published.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
duckdb>=1.3.2had no upper bound, so fresh installs already get DuckDB 2.0.0 while CI tested 1.5.5. On 2.0, table and view switches failed to promote. Local replay also broke: VARIANT and JSON comparisons, Ctrl-C and transpiled arrow lambdas failed. Database-qualified compatibility views also stopped matching.Changes
duckdb>=2.0.0and lock 2.0.0. New database files use 2.0 storage, which DuckDB 1.x cannot open.ALTER TABLE, which DuckDB 2.0 now allows.FILTER(arr, x -> ...)model locally. It is an expected failure until the polyglot-sql-chio wheel that writes DuckDB lambdas asLAMBDA x : bodyis published; a follow-up bumps the pin.lambda x:.BREAKING CHANGE: SQLBuild requires DuckDB 2.0, and MotherDuck cannot be used until it supports 2.0. Database files SQLBuild creates cannot be opened by DuckDB 1.x.
Verification
make test: 12037 passed.make compiler-differential: 390 projects, no golden changes.make check-cipasses afteruv sync --all-extras.