Repository navigation
perf: stop mimalloc asking for transparent huge pages - #1391
Merged
Merged
Conversation
mimalloc madvises its 1 GiB arenas for transparent huge pages, so on Linux hosts with THP in madvise mode (the Ubuntu default) every partly used 2 MiB page stayed resident in full. A cold 10,000-model compile held about 470 MiB of huge pages at its peak. Building mimalloc with no_thp keeps those arenas on normal pages and lowers that compile's peak memory by about 85 MB, with unchanged wall and CPU time. Other platforms are unaffected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The 10,000-model fresh-process compile guard (2.5 GiB peak RSS) fails on main itself, so it can block every pull request. mimalloc madvises its 1 GiB arenas for transparent huge pages, and with THP in
madvisemode (the Ubuntu and CI default) each partly used 2 MiB page stays resident in full. At the compile's peak,smapsshows three arenas flaggedhgholding about 470 MiB of huge pages.Changes
Build mimalloc with its
no_thpfeature (definesMI_NO_THPon Linux only), so its arenas stay on normal pages. The process THP setting and other platforms are unchanged. The budget is unchanged.Verification
On one orb, release builds made by
uv sync --locked --all-groups --all-extras, running CI's exactsystemd-run … MemoryMax=4294967296 … make test-e2e-cold-compile-performance SQLBUILD_BENCHMARK_MODELS=10000. Runs alternated main 43bc9f7 and this branch, 3 each, at load 3 to 4.3 on 8 cores.Median peak RSS is 85.5 MB lower (−3.2%) and 3.5% under the 2,684,354,560 budget. Compile CPU (user+sys) has a median of 49.6 s on this branch against 50.9 s on main. A
smapssnapshot of this build shows no huge-page regions.make check-cipasses. DuckShop: not applicable, because this is an allocator build setting with no project shape or data value.