by SQLStack
Up to 2.3× faster than ingestr and 150× faster than dlt. 10 million Parquet rows into DuckDB in 3.2 seconds. Benchmarks
Website ·
Docs ·
Quickstart ·
Benchmarks
Load Postgres, files, Snowflake, BigQuery, Google Sheets and Python sources into your warehouse. SQLLoad is Rust and Arrow end to end, and when your warehouse reads the files itself, rows never pass through SQLLoad, so there's no loader bottleneck as your data grows.
uv tool install sqlload # or: pip install sqlloadsqlload plan shows the tables it will create from Postgres, each column's DuckDB type and how
each table loads. sqlload load then creates and fills them.
sqlload plan # every table, column type and route, before anything is created
sqlload apply # create the tables
sqlload load # load themA sqlload.toml names your sources and targets; the
quickstart builds one. To load one table once with no
configuration file:
sqlload copy orders/ duckdb://shop.duckdb#raw.ordersThe same data loaded by each tool on a 4-vCPU machine, median of 5 runs:
| Scenario | SQLLoad | ingestr | dlt |
|---|---|---|---|
| Parquet folder into DuckDB, 1M rows | 0.92 s | 2.12 s | 138 s |
| Parquet folder into DuckDB, 10M rows | 3.20 s | 5.85 s | over 20 min (one run) |
| Postgres (40 columns) into DuckDB, 1M rows | 6.84 s | 8.29 s | 27.2 s |
| Postgres into Snowflake, 1M rows | 14.56 s | 16.38 s | 29.4 s |
See the benchmarks for every scenario, the method, the versions and how to run them yourself.
- See the plan before any data moves.
sqlload planshows every table, column type, write strategy and route before anything is created or loaded, so there are no surprises in your warehouse. - Crash-safe loads. Every commit lands whole, even if a load is killed halfway. A full refresh swaps in one go, so your dashboards see the old table or the new one, never a mix.
- Postgres change data capture. A
postgres_cdcsource copies your tables once, then applies every insert, update and delete from a logical replication slot, in whole source transactions.sqlload load --followkeeps streaming. See Postgres CDC.
| System | Source | Target |
|---|---|---|
| Postgres | ✓ (incl. CDC) | ✓ |
| Snowflake | ✓ | ✓ |
| BigQuery | ✓ | preview |
| Databricks | ✓ | |
| ClickHouse | preview | |
| DuckDB | ✓ | |
| MotherDuck | beta² | |
| Files | ✓ Parquet, CSV, JSON (S3, local folder, SFTP) | ✓ Parquet, CSV (S3, local folder) |
| Google Sheets | ✓ | |
| HTTP APIs and Python | ✓ | |
| ODBC databases¹ | ✓ |
¹ In builds with the odbc feature.
² From 0.34.0, through a separate DuckDB 1.5 process: install sqlload[motherduck].
See Supported systems for every combination and write strategy.
Built in Rust. Free and open source. Apache 2.0. Contributing · Changelog
Part of SQLStack.
