feat!: move to DuckDB 2.0

Makefile

@@ -155,6 +155,7 @@ E2E_DUCKDB_CLI_DATA_PATHS := \
 	tests/e2e/src/sqlbuild/cli/commands/main/scenario/test_helper_ctes.py \
 	tests/e2e/src/sqlbuild/cli/commands/main/scenario/test_fixture_columns.py \
 	tests/e2e/src/sqlbuild/cli/commands/main/scenario/test_local_variant.py \
+	tests/e2e/src/sqlbuild/cli/commands/main/scenario/test_local_lambda.py \
 	tests/e2e/src/sqlbuild/cli/commands/main/scenario/test_snapshot_staleness.py \
 	tests/e2e/src/sqlbuild/cli/commands/main/scenario/test_unmocked_inputs.py
 

pyproject.toml

@@ -12,7 +12,7 @@ description = "Typed, test-first SQL pipelines with local E2E testing"
 readme = "README.md"
 requires-python = ">=3.12,<3.15"
 dependencies = [
-  "duckdb>=1.3.2",
+  "duckdb>=2.0.0",
   "filelock>=3.20.3",
   "orjson>=3.11.7",
   "polyglot-sql-chio==0.13.9",

src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/rename.md

@@ -132,7 +132,7 @@ Renaming a column rewrites the model that produces it and every query that reads
 - the producing projection, as `amount AS revenue`, or the existing alias;
 - every downstream reference, including filters, joins, grouping, and ordering;
 - column declarations, including the model's own `MODEL (columns (...))` entries with or without commas between them, and column-valued config such as `unique_key`, `cursor`, and `partition_by`, and `cursor_inputs` and `relationships (field ...)` entries that point at the column;
-- string arguments of audits attached in `MODEL` headers, such as `where_condition "is_flagged = TRUE"` or `expression "amount > 0"`, when the generic audit's SQL definition splices the argument in as SQL with `@where_condition`. These arguments are parsed as SQL expressions, so only references to the audited model's column change: string literals stay as written, and a subquery reading another model, such as `customer_id IN (SELECT customer_id FROM __ref("vip_customers"))`, keeps that model's column. You are asked to edit an argument by hand when it cannot be parsed, when it reads a relation whose columns SQLBuild does not know, when the audit quotes it as a string with `@'prefix'`, when it joins on the column with `USING` or `NATURAL`, or when no SQL definition of the audit is found. A lambda parameter named like the column, as in `list_filter(xs, customer_id -> customer_id > 0)`, keeps its name;
+- string arguments of audits attached in `MODEL` headers, such as `where_condition "is_flagged = TRUE"` or `expression "amount > 0"`, when the generic audit's SQL definition splices the argument in as SQL with `@where_condition`. These arguments are parsed as SQL expressions, so only references to the audited model's column change: string literals stay as written, and a subquery reading another model, such as `customer_id IN (SELECT customer_id FROM __ref("vip_customers"))`, keeps that model's column. You are asked to edit an argument by hand when it cannot be parsed, when it reads a relation whose columns SQLBuild does not know, when the audit quotes it as a string with `@'prefix'`, when it joins on the column with `USING` or `NATURAL`, or when no SQL definition of the audit is found. A lambda parameter named like the column, as in `list_filter(xs, lambda customer_id: customer_id > 0)`, keeps its name;
 - fixture columns of `__ref__<model>` and `__expected__<model>` CTEs.
 
 By default a column rename is one step: downstream models keep their output names, so `SELECT o.amount` becomes `SELECT o.revenue AS amount`. With `--cascade`, a downstream model that passes the column through unchanged, as a bare column, a same-named alias, or `SELECT *`, renames its output too, and the rename continues through its own consumers.

src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/duckdb.md

@@ -8,6 +8,32 @@ Online: https://sqlbuild.com/docs/concepts/adapters/duckdb/
 
 DuckDB is included as a core dependency. No extra installation needed.
 
+SQLBuild requires DuckDB 2.0 or later. DuckDB 2.0 writes new database files in a storage format
+that DuckDB 1.x cannot open, so a database file SQLBuild creates can only be read by DuckDB 2.0
+clients. A file that already exists keeps its storage version, so a file created by DuckDB 1.x
+stays readable by 1.x. To create a new file that 1.x can read, create it before the first build
+with an explicit storage version, for example
+`duckdb my_project.duckdb -storage-version v1.5.0 -c "SELECT 1"`.
+
+## SQL that behaves differently on DuckDB 2.0
+
+DuckDB 2.0 changes how some SQL runs. SQLBuild runs model SQL as written, so check models for
+these before upgrading:
+
+- Arrow lambdas such as `list_filter(xs, x -> x > 0)` are rejected by default. Write
+  `list_filter(xs, lambda x: x > 0)` instead. DuckDB 2.1 removes the arrow syntax.
+- `~` and `!~` match a regular expression anywhere in the string, as in PostgreSQL, so
+  `'abc' ~ 'b'` is now true. Use `regexp_full_match` for a match of the whole string.
+- `MERGE` fails when one target row matches several source rows, instead of picking one. Keep
+  an incremental model's new rows unique on its `unique_key`.
+- An untyped `NULL`, as in `SELECT NULL AS discount`, creates a column of type `"NULL"` that
+  rejects later non-null values. Cast it, for example `CAST(NULL AS DECIMAL(18, 2)) AS discount`.
+- Integer division `//` and modulo `%` by zero raise an error instead of returning `NULL`. Wrap
+  the expression in `TRY(...)` to get `NULL`.
+
+dlt's dataset read API calls a DuckDB method that DuckDB 2.0 removed when it reads a loaded table
+back as one whole Arrow table, so that read fails. Dataframe and chunked Arrow reads still work. Loading data with dlt is not affected.
+
 ## Connection config
 
 ```toml

src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/adapters/motherduck.md

@@ -8,6 +8,13 @@ Online: https://sqlbuild.com/docs/concepts/adapters/motherduck/
 
 MotherDuck uses DuckDB's built-in `md:` connection support. No extra installation needed beyond the core DuckDB dependency.
 
+## DuckDB 2.0 support
+
+SQLBuild requires DuckDB 2.0, and MotherDuck does not support DuckDB 2.0 yet: its extension
+supports DuckDB up to 1.5.6 and refuses to load on 2.0. Connecting reports
+`MotherDuck does not support DuckDB 2.0.0 yet`. If you use MotherDuck, stay on SQLBuild 0.182.0,
+the last release that supports DuckDB 1.5, until MotherDuck supports DuckDB 2.0.
+
 ## Connection config
 
 ```toml

src/sqlbuild/adapter/contract/classes/duckdb_backed_adapter.py

@@ -5,6 +5,7 @@
 import importlib.util
 import json
 import logging
+import re
 from datetime import date, datetime
 from decimal import Decimal
 from importlib.machinery import ModuleSpec
@@ -47,7 +48,7 @@
     LATEST_DUCKDB_STORAGE_VERSION,
     QUALIFIED_NAME_SEPARATOR,
 )
-from sqlbuild.adapter.contract.exceptions import AdapterUserError
+from sqlbuild.adapter.contract.exceptions import AdapterUserError, ViewRenameRefusedError
 from sqlbuild.adapter.contract.main.run_relation_read_probe import run_relation_read_probe
 from sqlbuild.adapter.contract.main.same_view_definition import same_view_definition
 from sqlbuild.adapter.contract.main.unsupported_relation_usage import unsupported_relation_usage
@@ -1478,7 +1479,27 @@ def create_function(
             f"RETURNS {returns}"
         )
         statement_recorder.record(registration_sql)
-        connection.create_function(function_name, callable_function, parameter_types, returns)
+        self.register_python_function(
+            connection=connection,
+            name=function_name,
+            function=callable_function,
+            parameter_types=parameter_types,
+            returns=returns,
+        )
+
+    def register_python_function(
+        self,
+        *,
+        connection: Any,
+        name: str,
+        function: Any,
+        parameter_types: list[str],
+        returns: str,
+    ) -> None:
+        """Register a Python function after finishing a partly read result, which DuckDB 2 needs."""
+
+        connection.complete()
+        connection.create_function(name, function, parameter_types, returns)
 
     def _load_python_udf_callable(self, *, source_file_path: Path, entry_point: str) -> Any:
         module_name: str = "sqlbuild_python_udf_" + "_".join(
@@ -1645,13 +1666,32 @@ def swap(
         right: str,
         statement_recorder: StatementRecorder,
     ) -> None:
+        if self._is_view(connection=connection, relation=left):
+            raise ViewRenameRefusedError(message=f"DuckDB cannot swap the view {left} with a table")
         statements: tuple[str, ...] = self.render_swap(left=left, right=right)
         statement_recorder.record_many(statements)
         with self.transaction(connection):
             stmt: str
             for stmt in statements:
                 self.execute(connection=connection, sql=stmt)
 
+    def _is_view(self, *, connection: Any, relation: str) -> bool:
+        """Whether a relation is a view; DuckDB 2 lets ALTER TABLE rename a view, so swap checks."""
+
+        *qualifiers, name = [_unquote_duckdb_identifier(part) for part in relation.split(".")]
+        schema_sql: str = (
+            _duckdb_string_literal(qualifiers[-1]) if qualifiers else "current_schema()"
+        )
+        database_sql: str = (
+            _duckdb_string_literal(qualifiers[-2]) if len(qualifiers) > 1 else "current_database()"
+        )
+        query: str = (
+            "SELECT 1 FROM duckdb_views() WHERE NOT internal "
+            f"AND view_name = {_duckdb_string_literal(name)} "
+            f"AND schema_name = {schema_sql} AND database_name = {database_sql}"
+        )
+        return self.execute(connection=connection, sql=query).fetchone() is not None
+
     def render_clone(
         self,
         *,
@@ -1774,7 +1814,19 @@ def view_definition_matches(
         definition: str | None = self.read_view_definition(
             connection=connection, database=database, schema=schema, name=name
         )
-        return definition is not None and same_view_definition(definition=definition, sql=sql)
+        if definition is None:
+            return False
+        if same_view_definition(definition=definition, sql=sql):
+            return True
+        catalog: str = database if database is not None else self._current_database(connection)
+        return same_view_definition(
+            definition=_without_catalog_qualifier(text=definition, catalog=catalog),
+            sql=_without_catalog_qualifier(text=sql, catalog=catalog),
+        )
+
+    def _current_database(self, connection: Any) -> str:
+        row: Any = self.execute(connection=connection, sql="SELECT current_database()").fetchone()
+        return str(row[0])
 
     def render_relation_revokes(
         self, *, grants: tuple[RelationGrant, ...], destination: str
@@ -2714,6 +2766,17 @@ def _duckdb_string_literal(self, value: str) -> str:
         return value.replace("'", "''")
 
 
+def _without_catalog_qualifier(*, text: str, catalog: str) -> str:
+    """Drop a view's own catalog prefix, which DuckDB 2 leaves out of stored view SQL."""
+
+    quoted_catalog: str = re.escape('"' + catalog.replace('"', '""') + '"')
+    pattern: re.Pattern[str] = re.compile(
+        rf"""(?<![\w."])(?:{quoted_catalog}|{re.escape(catalog)})\.(?=[\w"])""",
+        re.IGNORECASE,
+    )
+    return pattern.sub("", text)
+
+
 def _duckdb_string_literal(value: str) -> str:
     return "'" + value.replace("'", "''") + "'"
 

src/sqlbuild/adapter/contract/exceptions.py

@@ -17,6 +17,10 @@ def __init__(self, message: str, *, code: str | None = None, help: str | None =
         self.help = help
 
 
+class ViewRenameRefusedError(AdapterUserError):
+    """Raised before renaming a view with table DDL, which DuckDB 2 would otherwise accept."""
+
+
 class SqlStatementExecutionError(RuntimeError):
     """Warehouse error attributed to the exact SQL statement that raised it."""
 

src/sqlbuild/adapters/motherduck/classes/motherduck_adapter.py

@@ -10,9 +10,12 @@
 
 from sqlbuild.adapter.contract.classes.duckdb_backed_adapter import DuckDbBackedAdapter
 from sqlbuild.adapter.contract.classes.microbatch import MicrobatchMixin
+from sqlbuild.adapter.contract.exceptions import AdapterUserError
 from sqlbuild.adapter.contract.types import BuiltinAdapter, LoaderLogicalType
 from sqlbuild.adapters.motherduck.constants import (
     EMPTY_CONNECTION_VALUE,
+    MOTHERDUCK_UNSUPPORTED_DUCKDB_HELP,
+    MOTHERDUCK_UNSUPPORTED_DUCKDB_MARKER,
     URI_ADDITIONAL_PARAMETER_SEPARATOR,
     URI_QUERY_SEPARATOR,
 )
@@ -154,7 +157,15 @@ def connect(self, config: dict[str, Any]) -> Any:
         import duckdb
 
         database: str = str(duckdb_config.get("database", ":memory:"))
-        connection: duckdb.DuckDBPyConnection = duckdb.connect(database=database)
+        try:
+            connection: duckdb.DuckDBPyConnection = duckdb.connect(database=database)
+        except duckdb.Error as error:
+            if MOTHERDUCK_UNSUPPORTED_DUCKDB_MARKER not in str(error):
+                raise
+            raise AdapterUserError(
+                message=f"MotherDuck does not support DuckDB {duckdb.__version__} yet.",
+                help=MOTHERDUCK_UNSUPPORTED_DUCKDB_HELP,
+            ) from error
 
         extensions: list[str] | tuple[str, ...] = duckdb_config.get("extensions", ())
         extension_name: str

src/sqlbuild/adapters/motherduck/constants.py

@@ -3,3 +3,8 @@
 EMPTY_CONNECTION_VALUE: str = ""
 URI_QUERY_SEPARATOR: str = "?"
 URI_ADDITIONAL_PARAMETER_SEPARATOR: str = "&"
+MOTHERDUCK_UNSUPPORTED_DUCKDB_MARKER: str = "is not yet supported by MotherDuck"
+MOTHERDUCK_UNSUPPORTED_DUCKDB_HELP: str = (
+    "This SQLBuild release requires DuckDB 2.0. To use MotherDuck, stay on SQLBuild 0.182.0, "
+    "the last release that supports DuckDB 1.5, until MotherDuck supports DuckDB 2.0."
+)

src/sqlbuild/executor/pipeline/_helpers/scenario.py

@@ -293,6 +293,7 @@ def run_scenario_local_test_pipeline(
     promotion_mode: TablePromotionMode = resolve_promotion_mode(
         settings=pipeline_result.project.settings, adapter=adapter
     )
+    running: RunningScenarios = RunningScenarios(adapter=adapter)
 
     def execute(scenario_plan: ScenarioExecutionPlan) -> ScenarioRunResult:
         return execute_local_scenario_load_only_run(
@@ -303,6 +304,7 @@ def execute(scenario_plan: ScenarioExecutionPlan) -> ScenarioRunResult:
             promotion_mode=promotion_mode,
             capture_adapter=replay_source.capture_adapter,
             capture_dialect=replay_source.capture_dialect,
+            running=running,
         )
 
     def failed(*, scenario_name: str, exc: Exception) -> ScenarioRunResult:
@@ -326,6 +328,11 @@ def failed(*, scenario_name: str, exc: Exception) -> ScenarioRunResult:
         failed=failed,
         on_scenario_start=on_scenario_start,
         on_scenario_complete=on_scenario_complete,
+        scheduling=_ScenarioScheduling(
+            interrupts=ScenarioInterrupts(
+                cancel_statements=partial(running.interrupt, close_uncancellable=False)
+            )
+        ),
     )
 
 

src/sqlbuild/executor/run/_helpers/materializations/full_refresh.py

@@ -7,8 +7,9 @@
 
 from sqlbuild.adapter.contract.classes.base_adapter import BaseAdapter
 from sqlbuild.adapter.contract.classes.statement_recorder import StatementRecorder
-from sqlbuild.adapter.contract.types import BuiltinAdapter
+from sqlbuild.adapter.contract.types import BuiltinAdapter, RelationType
 from sqlbuild.adapter.relations.main.cached_relation_exists import cached_relation_exists
+from sqlbuild.adapter.relations.main.cached_relation_type import cached_relation_type
 from sqlbuild.adapter.relations.main.resolve_qualified_name_parts import (
     resolve_qualified_name_parts,
 )
@@ -22,6 +23,9 @@
 )
 
 _PREVIOUS_PREFIX: str = "__sqb_prev__"
+_TABLE_RENAME_MOVES_VIEWS: frozenset[str] = frozenset(
+    {BuiltinAdapter.DUCKDB, BuiltinAdapter.MOTHERDUCK}
+)
 _REBUILD_PREFIX: str = "__sqb_rebuild__"
 
 
@@ -50,6 +54,8 @@ def resolve_full_refresh_relations(
         target_qualified=_qualified(
             adapter=adapter, database=database, schema=schema, name=target_name
         ),
+        database=database,
+        schema=schema,
         rebuild_name=rebuild_name,
         rebuild_qualified=_qualified(
             adapter=adapter, database=database, schema=schema, name=rebuild_name
@@ -107,19 +113,69 @@ def promote_full_refresh_rebuild(
                     if_exists=True,
                     statement_recorder=statement_recorder,
                 )
-            promote_staged_relation(
-                adapter=adapter,
-                connection=connection,
-                target_qualified=relations.target_qualified,
-                staged_qualified=relations.rebuild_qualified,
-                displaced_qualified=relations.previous_qualified,
-                target_exists=target_exists,
-                statement_recorder=statement_recorder,
-            )
+            if (
+                target_exists
+                and adapter.adapter_name in _TABLE_RENAME_MOVES_VIEWS
+                and _live_view(adapter=adapter, connection=connection, relations=relations)
+            ):
+                _replace_live_view(
+                    adapter=adapter,
+                    connection=connection,
+                    relations=relations,
+                    statement_recorder=statement_recorder,
+                )
+            else:
+                promote_staged_relation(
+                    adapter=adapter,
+                    connection=connection,
+                    target_qualified=relations.target_qualified,
+                    staged_qualified=relations.rebuild_qualified,
+                    displaced_qualified=relations.previous_qualified,
+                    target_exists=target_exists,
+                    statement_recorder=statement_recorder,
+                )
             _ = guard.rebind()
         lifecycle.completed(metadata={"changed_count": 1})
 
 
+def _live_view(*, adapter: BaseAdapter, connection: Any, relations: FullRefreshRelations) -> bool:
+    """Whether the live relation is a view, which table renames must not move aside."""
+
+    return (
+        cached_relation_type(
+            adapter=adapter,
+            connection=connection,
+            database=relations.database,
+            schema=relations.schema,
+            name=relations.target_name,
+        )
+        == RelationType.VIEW
+    )
+
+
+def _replace_live_view(
+    *,
+    adapter: BaseAdapter,
+    connection: Any,
+    relations: FullRefreshRelations,
+    statement_recorder: StatementRecorder,
+) -> None:
+    """Drop a live view the rebuild replaces; a view keeps no data to retain as prev."""
+
+    adapter.drop_view(
+        connection=connection,
+        destination=relations.target_qualified,
+        if_exists=True,
+        statement_recorder=statement_recorder,
+    )
+    adapter.rename(
+        connection=connection,
+        origin=relations.rebuild_qualified,
+        destination=relations.target_qualified,
+        statement_recorder=statement_recorder,
+    )
+
+
 def promote_staged_relation(
     *,
     adapter: BaseAdapter,

src/sqlbuild/executor/run/models.py

@@ -600,6 +600,8 @@ class FullRefreshRelations:
 
     target_name: str
     target_qualified: str
+    database: str | None
+    schema: str | None
     rebuild_name: str
     rebuild_qualified: str
     previous_name: str

src/sqlbuild/executor/scenario/_helpers/lifecycle/variant_comparison.py

@@ -66,4 +66,10 @@ def _register_canonical_json(*, adapter: DuckDbBackedAdapter, connection: Any) -
         ),
     ).fetchone()
     if registered is not None and registered[0] == 0:
-        connection.create_function(CANONICAL_JSON_FUNCTION, canonical_json, ["VARCHAR"], "VARCHAR")
+        adapter.register_python_function(
+            connection=connection,
+            name=CANONICAL_JSON_FUNCTION,
+            function=canonical_json,
+            parameter_types=["VARCHAR"],
+            returns="VARCHAR",
+        )