Skip to content

fix: avoid slow compile for large std::array in array_caster - #6258

Open
henryiii wants to merge 3 commits into
pybind:masterfrom
henryiii:fix/large-std-array-compile
Open

henryiii wants to merge 3 commits into
pybind:masterfrom
henryiii:fix/large-std-array-compile

Conversation

@henryiii

Copy link
Copy Markdown
Collaborator

🤖 AI text below 🤖

Description

Fixes #5765.

array_caster loaded fixed-size arrays into a temporary std::vector, then built the array with an index_sequence of Size elements. That expansion is very slow to compile for large arrays: the example from the issue (std::array<int, 102400>) took 135 s with Apple clang 21 at -O1.

If the element type is default constructible and move assignable, the caster now fills the array in place. Now the example compiles in 1.5 s. Other element types still use the old path. The array is assigned only after all elements load.

This removes the old comment that said one generic path compiles faster than specializations, because this issue shows the opposite for large Size.

Suggested changelog entry:

  • Fixed very slow compilation when binding large std::array types with pybind11/stl.h.

@rwgk

rwgk commented Nov 21, 2026

Copy link
Copy Markdown
Collaborator

— codex on behalf of Ralf —

Keep arrays of explicitly default-constructible elements compiling. The optimized branch accepts element types with an explicit default constructor because std::is_default_constructible is true. new ArrayType{} then copy-list-initializes the omitted aggregate elements, where an explicit default constructor is ill-formed. A binding taking const std::array<ExplicitDefault, 2>& builds on the base and fails on this head with Clang 18 in C++17, using both with-GIL and free-threaded CPython 3.15.2 headers. The optimized branch could be limited to types with trivial default construction and assignment, or the old construction path could remain available for these elements. Fast path and aggregate initialization

When resolving the conflict with master, please also carry forward the single-length indexed iteration from #6278 into the new loop. New loop

Generated by Codex. Ralf has yet to review these findings.

@henryiii

Copy link
Copy Markdown
Collaborator Author

🤖 AI text below 🤖

Both points are valid.

Explicit default constructor: Confirmed. new ArrayType{} does not compile with Clang (Apple clang 21, LLVM 23; C++11 to C++23) if the element type has an explicit default constructor. GCC 16 accepts it. Fixed in 8c058ee: the fast path now uses new ArrayType(). This compiles with all the compilers above and keeps the fast path for these types. The commit adds a test that binds const std::array<ExplicitDefault, 2> &. The test did not compile before the fix.

Indexed iteration from #6278: Merged master in bb21b98. After the merge, test_array_caster_len_changes[*-array_cast_sequence] failed, because the new loop still iterated over the sequence. Fixed in 0804cdb: the new loop reads the length one time and uses l[i], the same as the other two overloads.

Fill fixed-size arrays in place when the element type is default
constructible and move assignable, instead of going through an
index_sequence expansion of Size elements. A std::array<int, 102400>
member took 135 s to compile with clang; it now takes 1.5 s.

Fixes pybind#5765.

Assisted-by: ClaudeCode:claude-opus-5-5
Assisted-by: ClaudeCode:claude-fable-5-2
@henryiii
henryiii force-pushed the fix/large-std-array-compile branch from 0804cdb to f36aef9 Compare November 21, 2026 18:35

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: slow compile if very long std::array being used with clang-18

2 participants