Free-threaded 3.14.10 and 3.15.2 keep memory freed by other threads: 10 GB peaks since the mimalloc 2.5.2 update #161420

Bug report

I've been looking into the performance of streaming decompression with compression.zstd (archive extraction in a package manager), first on 3.14 and 3.15, then on the free-threaded builds. On 3.14.10t, extracting an archive with 1.1 GiB of files peaked at 1.3 GB, against 170 MB on 3.14.

The extraction decompresses on worker threads and hands the decompressed bytes to one thread that writes them out. Boiled down: worker threads make large bytes objects, the main thread consumes and drops them, with a bounded number in flight. That needs about 150 MiB at any time, but on 3.14.10t, 3.15.2t and current main the peak grows to 9–10 GB:

import resource
import sys
from collections import deque
from concurrent.futures import ThreadPoolExecutor

BLOCK, COUNT, THREADS = 16 << 20, 1536, 8      # 24 GiB in total
pattern = bytes(range(256)) * (BLOCK // 256)

def produce(i):
    return memoryview(pattern).tobytes()         # a new bytes object, as decompress() returns

with ThreadPoolExecutor(THREADS) as pool:
    pending = deque()
    for i in range(COUNT):
        pending.append(pool.submit(produce, i))
        if len(pending) >= THREADS:
            pending.popleft().result()           # freed here, on the main thread
    for future in pending:
        future.result()

peak = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
peak = peak if sys.platform == "darwin" else peak * 1024
print(f"{sys.version.split()[0]}{'' if sys._is_gil_enabled() else 't'}: peak RSS {peak >> 20} MiB")

macOS 26 (arm64), python-build-standalone builds, and main built from source:

Linux (aarch64) and Windows (x64) show the same: 9.2–10.5 GiB on 3.14.10t and 3.15.2t. Builds with the GIL are affected under PYTHONMALLOC=mimalloc; free-threaded builds always use mimalloc. The memory is not leaked: the peak levels off (see below).

Bisected to the mimalloc 2.5.2 update

3.14.10 and 3.15.2 carry the vendored mimalloc 2.5.2 from gh-159111 and its backports; 3.13.18 and 3.16.0a1 still have 2.1.2. Free-threaded builds of main with the script above, macOS:

Cause

_mi_malloc_generic() in Objects/mimalloc/page.c used to process the heap's delayed frees (blocks other threads freed back to it) on every call. Since 2.5.2 it does so only every 100 calls:

  // do administrative tasks every N generic mallocs
  if mi_unlikely(++heap->generic_count >= 100) {
    ...
    // free delayed frees from other threads (but skip contended ones)
    _mi_heap_delayed_free_partial(heap);

Every block over MI_MEDIUM_OBJ_SIZE_MAX (64 KiB in 2.5.2) gets a page of its own, and 2.5.2 moves such a page to the full queue right away. A free of that block from another thread goes onto the owning heap's delayed-free list, and allocating such a block always takes this slow path. So each allocating thread keeps up to about 100 freed blocks out of reach. The peak follows threads × 100 × block size (free-threaded main at aff7a2e, macOS, 24 GiB in total):

That also explains why MIMALLOC_GENERIC_COLLECT=1 doesn't help: it's only read inside the same branch. MIMALLOC_PURGE_DELAY=0 doesn't either: the blocks aren't freed until that list is processed, so there is nothing to purge. MIMALLOC_TARGET_SEGMENTS_PER_THREAD=2 works around it (4 GiB down to about 600 MiB).

The batching came with upstream commit 58d13f6a ("collect every N generic calls", in mimalloc 2.2.2), which added the periodic heap collection (generic_collect) and moved the per-call housekeeping into the same every-100 block. That works when generic mallocs are frequent, as for small objects, but every allocation over 64 KiB is one, and 100 of those can be gigabytes. Upstream's main and dev branches still have it, and microsoft/mimalloc#1117 reports the same symptom (large blocks freed by other threads that aren't collected), without a cause so far; I'll add this there.

Possible fixes

I'm not sure what the right fix is. The every-100 batching probably exists for a good reason: it amortizes the housekeeping over workloads with many small allocations, which I haven't measured beyond a quick benchmark. Two options came to mind, and I'm happy to help with either:

1. A small, targeted revert for main, 3.14 and 3.15: process the delayed frees on every generic malloc again, as 2.1.2 did, and keep the deferred-free callback and the periodic collection batched. It costs an atomic load per generic malloc when the list is empty, which I couldn't measure. It would be one more CPython-only change to the vendored mimalloc (Mimalloc differences from upstream #113141). Patch and measurements below.
2. A better strategy that keeps the batching for small objects but bounds what large blocks can hold back, for example a byte budget next to the call count (housekeeping every 100 generic mallocs or every few MiB allocated through the generic path). That needs larger changes, a new heap field and a tuning constant, and is probably better discussed with upstream mimalloc, where it would apply too.

diff --git a/Objects/mimalloc/page.c b/Objects/mimalloc/page.c
--- a/Objects/mimalloc/page.c
+++ b/Objects/mimalloc/page.c
@@ -1071,6 +1071,12 @@ void* _mi_malloc_generic(mi_heap_t* heap, size_t size, bool zero, size_t huge_al
   }
   mi_assert_internal(mi_heap_is_initialized(heap));
 
+  // free delayed frees from other threads (but skip contended ones)
+  // CPython: on every generic malloc, as before mimalloc 2.5: a block of a singleton page (any
+  // block over MI_MEDIUM_OBJ_SIZE_MAX) freed by another thread is a delayed free, and up to 100
+  // generic mallocs of such blocks per heap kept that many freed blocks out of reach.
+  _mi_heap_delayed_free_partial(heap);
+
   // do administrative tasks every N generic mallocs
   if mi_unlikely(++heap->generic_count >= 100) {
     heap->generic_collect_count += heap->generic_count;
@@ -1078,9 +1084,6 @@ void* _mi_malloc_generic(mi_heap_t* heap, size_t size, bool zero, size_t huge_al
     // call potential deferred free routines
     _mi_deferred_free(heap, false);
 
-    // free delayed frees from other threads (but skip contended ones)
-    _mi_heap_delayed_free_partial(heap);
-
     // collect every once in a while (10000 by default)
     const long generic_collect = mi_option_get_clamp(mi_option_generic_collect, 1, 1000000L);
     if (heap->generic_collect_count >= generic_collect) {

With option 1, the peak is back at 2.1.2 levels on macOS, Linux (GCC 8.5) and Windows (MSVC), and the full test suite passes on a free-threaded build of main (macOS). I couldn't measure a cost: in a quick allocation benchmark (small bytes, dicts of lists, 4 MiB blocks) the patched and unpatched builds of main are within run-to-run noise of each other.

A middle ground I also tried: process the delayed frees only right before a large allocation in mi_find_page(). That keeps the batching for small objects and also fixes this, but it was less steady: in some runs (16 threads × 8 MiB, 8 threads × 16 MiB) it peaked at about 700 MiB, where option 1 stayed at 300–350 MiB.

Related

• Free-threaded build never returns freed memory to the OS on macOS, even with gc.collect() and MIMALLOC_PURGE_DELAY=0 #158972 (memory not returned to the OS on macOS) is a different problem: small objects, pages waiting for QSBR, and it predates 2.5.2. Its fix in progress, gh-158972: Drain each thread's page list during GC #160150, doesn't change this.
• gh-113141: Upgrade mimalloc to upstream v3 #159718 (mimalloc v3) makes this worse: the script keeps nearly everything, because v3 does the same housekeeping every 1,000 generic mallocs and the PR sets page_full_retain to -1.

CPython versions tested on:

3.13, 3.14, 3.15, CPython main branch

Operating systems tested on:

Windows, macOS, Linux