Repository navigation
feat(motherduck): connect through a separate DuckDB 1.5 process (install sqlload[motherduck]), as MotherDuck doesn't accept DuckDB 2.0 clients yet - #182
Merged
Conversation
…all sqlload[motherduck]), as MotherDuck doesn't accept DuckDB 2.0 clients yet A motherduck target now runs its SQL on Python's duckdb package (1.4.1 to 1.5.6, which MotherDuck accepts) in a process of its own, so the two DuckDB versions never share a process and every other target stays on the bundled DuckDB 2.0. SQLLoad sends it newline-delimited JSON requests over a pipe, one cursor per table connection, and hands it each batch as a temporary Parquet file it inserts with read_parquet inside the batch's transaction. The token goes over the pipe, never in the command line, and is redacted from everything the process prints. The sqlload package gains a motherduck extra (duckdb>=1.4.1,<=1.5.6) and runs the process on its own Python; sqlload-cli and other setups name one with SQLLOAD_MOTHERDUCK_PYTHON. Everything specific to it sits behind the DuckDB adapter's Bridge connection and adapters/motherduck_bridge; the MotherDuck page lists the steps to remove it once MotherDuck accepts DuckDB 2.0. Tested offline through a stand-in for MotherDuck (every write strategy and portable type compared with DuckDB, an incremental rerun, the process ending mid-load, a rejected value, setup errors) and live against a MotherDuck account (test_motherduck_live.py, run with SQLLOAD_TEST_MOTHERDUCK_TOKEN).
Deploying sqlload with
|
| Latest commit: |
1fdbba0
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://cb366125.sqlload.pages.dev |
| Branch Preview URL: | https://chi-1237-motherduck-bridge.sqlload.pages.dev |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
MotherDuck accepts DuckDB clients 1.4.1 to 1.5.6, and SQLLoad bundles DuckDB 2.0, so a
motherducktarget couldn't connect. It now runs its SQL on Python'sduckdbpackage in a separate process; every other target stays on the bundled DuckDB 2.0, and the two versions never share a process.How it works
crates/sqlload/src/adapters/motherduck_bridge/: a small Python program (bridge.py, run withpython -c) and the Rust side that talks to it over newline-delimited JSON on stdin/stdout. One DuckDB cursor per table connection, run in order on its own thread. Batches go over as temporary Parquet files inserted withread_parquet(?)inside the batch's transaction.DuckDbConnection::{Local, Bridge}; each entry inadapters/duckdb/mainhas aBridgebranch. Local DuckDB is unchanged.sqlload[motherduck]addsduckdb>=1.4.1,<=1.5.6; thesqlloadcommand runs the process on its own Python.SQLLOAD_MOTHERDUCK_PYTHONnames another (needed withsqlload-cli).Tests
packages/sqlload/tests/test_motherduck.py, inmake test-python): a stand-in for MotherDuck keeps eachmd:database in a local file. Covers every write strategy and portable type compared with the same load into DuckDB, an incremental rerun with a new column, the process ending mid-load (nothing committed, the next load recovers, the token is redacted), a value MotherDuck rejects, and setup errors (no Python, no duckdb package, refused version, refused token).packages/sqlload/tests/test_motherduck_live.py, skipped unlessSQLLOAD_TEST_MOTHERDUCK_TOKENis set): run against a real MotherDuck account on duckdb 1.5.6. It creates a prefixed database, runsdebug, loads all four strategies and every portable type, does an incremental load with a new column, reruns, and drops the database. Passed twice; the account is left with only its original databases.