Skip to content
sqlstack-ioPublic

About

No description, website, or topics provided.

Resources

Contributing

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

SQLLoad
by SQLStack

The fastest open-source data loader.

Up to 2.3× faster than ingestr and 150× faster than dlt. 10 million Parquet rows into DuckDB in 3.2 seconds. Benchmarks
Website · Docs · Quickstart · Benchmarks

Load Postgres, files, Snowflake, BigQuery, Google Sheets and Python sources into your warehouse. SQLLoad is Rust and Arrow end to end, and when your warehouse reads the files itself, rows never pass through SQLLoad, so there's no loader bottleneck as your data grows.

uv tool install sqlload        # or: pip install sqlload

sqlload plan lists the two tables it will create from Postgres, their column types in DuckDB, and how each loads; sqlload load then creates them and loads 1,050 rows

sqlload plan shows the tables it will create from Postgres, each column's DuckDB type and how each table loads. sqlload load then creates and fills them.

Quick start

sqlload plan     # every table, column type and route, before anything is created
sqlload apply    # create the tables
sqlload load     # load them

A sqlload.toml names your sources and targets; the quickstart builds one. To load one table once with no configuration file:

sqlload copy orders/ duckdb://shop.duckdb#raw.orders

Speed

The same data loaded by each tool on a 4-vCPU machine, median of 5 runs:

Scenario SQLLoad ingestr dlt
Parquet folder into DuckDB, 1M rows 0.92 s 2.12 s 138 s
Parquet folder into DuckDB, 10M rows 3.20 s 5.85 s over 20 min (one run)
Postgres (40 columns) into DuckDB, 1M rows 6.84 s 8.29 s 27.2 s
Postgres into Snowflake, 1M rows 14.56 s 16.38 s 29.4 s

See the benchmarks for every scenario, the method, the versions and how to run them yourself.

Why SQLLoad

  • See the plan before any data moves. sqlload plan shows every table, column type, write strategy and route before anything is created or loaded, so there are no surprises in your warehouse.
  • Crash-safe loads. Every commit lands whole, even if a load is killed halfway. A full refresh swaps in one go, so your dashboards see the old table or the new one, never a mix.
  • Postgres change data capture. A postgres_cdc source copies your tables once, then applies every insert, update and delete from a logical replication slot, in whole source transactions. sqlload load --follow keeps streaming. See Postgres CDC.

What it loads

System Source Target
Postgres ✓ (incl. CDC) ✓
Snowflake ✓ ✓
BigQuery ✓ preview
Databricks ✓
ClickHouse preview
DuckDB ✓
MotherDuck beta²
Files ✓ Parquet, CSV, JSON (S3, local folder, SFTP) ✓ Parquet, CSV (S3, local folder)
Google Sheets ✓
HTTP APIs and Python ✓
ODBC databases¹ ✓

¹ In builds with the odbc feature.

² From 0.34.0, through a separate DuckDB 1.5 process: install sqlload[motherduck].

See Supported systems for every combination and write strategy.


Built in Rust. Free and open source. Apache 2.0. Contributing · Changelog

Part of SQLStack.

About

No description, website, or topics provided.

Resources

Contributing

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages