DuckDB spawns worker threads on module import rather than connection creation

What happens?

Importing the duckdb Python module (import duckdb) immediately spawns ~100 worker threads (appears to be ncpus), even without creating any connections. This causes unexpected resource consumption in applications that import the module but may not immediately use it.

Expected Behavior

Worker threads should only be created when:

An actual database connection is established via duckdb.connect()
Or when explicitly configured to do so

The module import itself should be lightweight and not spawn background threads.

Proposed Solution

Consider lazy initialization of the thread pool:

Defer TaskScheduler/thread pool creation until first connect() call
Or provide an environment variable/config option to control thread spawning behavior on import
Similar to how other database drivers handle connection pooling

Workaround

Currently requires monkey-patching or lazy imports to avoid the thread spawning:

# Must be done before any code imports duckdb
import sys
import importlib.util

def lazy_import_duckdb():
    spec = importlib.util.find_spec("duckdb")
    module = importlib.util.module_from_spec(spec)
    sys.modules["duckdb"] = module
    # Don't execute the module yet
    return module

To Reproduce

import duckdb  # This alone spawns ~100 threads

When attaching gdb to a process that has imported duckdb:

thread apply all bt shows 100+ threads with duckdb in their stack traces
All threads show ExecuteForever in their call stacks
Base stack frame is clone3 syscall

OS:

Ubuntu 22.04 x86_64

DuckDB Version:

0.6.1

DuckDB Client:

Python

Hardware:

128 core CPU

Full Name:

Nova DasSarma

Affiliation:

Anthropic, PBC

What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

I have tested with a stable release

Did you include all relevant data sets for reproducing the issue?

Not applicable - the reproduction does not require a data set

Did you include all code required to reproduce the issue?

Yes, I have

Did you include all relevant configuration (e.g., CPU architecture, Python version, Linux distribution) to reproduce the issue?

Yes, I have

@l1n

Hi @l1n, just double-checking some information: your report states that you are using DuckDB 0.6.1. This version was released back in 2022 and is no longer supported. Are you indeed using this version?

I can reproduce this on currently supported Python packages, and the regression starts at 1.4.0.

Environment: Linux x86_64, Python 3.12, 128 logical CPUs. Thread counts are from /proc/self/task after a short settle:

package	before import	after `import duckdb`	after `duckdb.connect()`	after close
1.3.0	1	1	128	1
1.4.0	1	128	255	128

The import-time behavior is also present in 1.5.5 (1 → 128 threads). With taskset -c 0,1, sched_getaffinity(0) reports two CPUs but importing 1.5.5 still creates 128 threads.

I traced the 1.4.0 regression to the DB-API type objects added at module scope in duckdb/__init__.py:

typing.DuckDBPyType("BIGNUM")
typing.DuckDBPyType("DECIMAL")

In later 1.4.x/current sources this code lives in duckdb/_dbapi_type_object.py, but the behavior is the same. Constructing a DuckDBPyType from a string with no connection calls DuckDBPyConnection::DefaultConnection(), which creates a :memory: database and starts its scheduler during module import. A subsequent duckdb.connect() creates a second uncached in-memory database, accounting for the second worker pool.

There is a partial post-import workaround for the global/default connection:

import duckdb
duckdb.sql("SET threads=1")

On 1.5.5 this reduces the process from 128 threads back to 1. duckdb.connect(config={"threads": "1"}) only configures the new connection and does not shrink the import-created default instance. This workaround also cannot prevent the transient import-time spike, especially when DuckDB is imported indirectly.

The current preview build makes the symptom larger on this host: 1.6.0.dev291 imports with 384 threads, with threads=128 and async_threads=256. SET threads=1 leaves 257 threads; setting async_threads=1 as well reduces the process to 2 threads.

A narrow fix may be to avoid connection-dependent string parsing while constructing the DB-API NUMBER sentinel—for example, expose parser-free BIGNUM and default DECIMAL(18,3) type objects (or add a no-connection fast path for these primitive names). A broader scheduler-lazy-initialization change could then be considered separately.

Would you prefer to keep tracking this here, or should I open a dedicated bug in duckdb/duckdb-python with the full reproducer and version matrix?

Still happening on 1.5.6, and the 2.0 dev build is worse (60 threads on a 12-core box: 12 workers plus 48 async).

We hit this with a child process that caps its memory with RLIMIT_DATA. Linux counts thread stacks toward that limit, so the threads started by the import use up about cores × stack size of the cap before we open a connection. Our test passed on a laptop and failed on an 8-core CI runner.

Quick check (Linux, python:3.14-slim, 12 cores):

import resource

def status():
    with open("/proc/self/status") as f:
        return " ".join(l.split()[0] + l.split()[1] for l in f if l.startswith(("Threads", "VmData")))

print("before:", status())
import duckdb
print("after import:", status())
duckdb.default_connection().close()
print("after close:", status())

                         before                after import             after close
1.5.6,   8 MiB stacks    VmData:4936 Threads:1  VmData:110560 Threads:12   VmData:58764 Threads:1
1.5.6,  64 MiB stacks    VmData:4936 Threads:1  VmData:741344 Threads:12   VmData:25996 Threads:1
2.0 dev, 8 MiB stacks    VmData:4936 Threads:1  VmData:506508 Threads:60   VmData:65720 Threads:1
2.0 dev, 64 MiB stacks   VmData:4936 Threads:1  VmData:3889804 Threads:60  VmData:34144 Threads:1

If the cap is set before the import, the import itself can die. With 1.5.6, 8 MiB stacks and a 64 MiB RLIMIT_DATA, import duckdb aborts every time with cannot allocate memory for thread-local data: ABORT (or segfaults), before any query runs. The 2.0 dev build does the same at 256 MiB. Restricting CPU affinity (--cpuset-cpus 0-1) doesn't help on 1.5.6; it still starts 12 threads.

I can confirm the cause traced above. Replacing the two DuckDBPyType("BIGNUM") / DuckDBPyType("DECIMAL") sentinels in _dbapi_type_object.py with prebuilt sqltypes constants (just to test, not as a fix) keeps import duckdb at one thread. Those lines are still on main and the 1.5/2.0 branches.

Workarounds we've found: closing default_connection() right after import, which doesn't help if the import is what dies, and setting SLURM_CPUS_ON_NODE=1 before import, which works on Linux but isn't something I'd want to rely on.

It would be great if the default connection were created on first use rather than at import, with a test that import duckdb doesn't start threads, since this already regressed once in 1.4.0. Failing that, a documented env var to turn off or size the default connection would do.

Keep import duckdb from opening a database and fix the default connec…

Thanks all, and thanks for tracing it to the DB API type objects. The fix is merged into v2.0-cyanoptera in #680 and will ship in v2.0. The default database now opens on first use, so explicit threads and async_threads settings will apply the limits:

import duckdb
con = duckdb.connect(config={"threads": 2, "async_threads": 0})
duckdb.set_default_connection(con)

Navigation Menu

Repository navigation

DuckDB spawns worker threads on module import rather than connection creation #612

What happens?

Expected Behavior

Proposed Solution

Workaround

To Reproduce

OS:

DuckDB Version:

DuckDB Client:

Hardware:

Full Name:

Affiliation:

What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

Did you include all relevant data sets for reproducing the issue?

Did you include all code required to reproduce the issue?

Did you include all relevant configuration (e.g., CPU architecture, Python version, Linux distribution) to reproduce the issue?

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

DuckDB spawns worker threads on module import rather than connection creation #612

Description

What happens?

Expected Behavior

Proposed Solution

Workaround

To Reproduce

OS:

DuckDB Version:

DuckDB Client:

Hardware:

Full Name:

Affiliation:

What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

Did you include all relevant data sets for reproducing the issue?

Did you include all code required to reproduce the issue?

Did you include all relevant configuration (e.g., CPU architecture, Python version, Linux distribution) to reproduce the issue?

Activity

szarnyasg commented on Aug 19, 2025

bowser-bot commented on Jul 23, 2026

willfrey commented on Oct 14, 2026

evertlammerts commented on Oct 19, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions