Skip to content

fix(cmake): don't link SHARED modules against libpython on Apple platforms in FindPython mode - #6140

Open
henryiii wants to merge 3 commits into
masterfrom
fix-shared-libpython-link
Open

henryiii wants to merge 3 commits into
masterfrom
fix-shared-libpython-link

Conversation

@henryiii

@henryiii henryiii commented Aug 7, 2026 •

Copy link
Copy Markdown
Collaborator

Our own test suite fails if using python-build-standalone (or any other statically linked Python). This is one option for a fix.

🤖 AI text below 🤖

Description

In FindPython mode, pybind11_add_module(... SHARED ...) routes through CMake's Python_add_library, which hard-links every non-MODULE library against Python::Python. An extension module must not link libpython on macOS: with a statically linked interpreter (uv / python-build-standalone), the linked libpython3.x.dylib loads as a second, uninitialized copy of the Python runtime, two-level namespace binds the module's C-API calls to it, and the import aborts with:

Fatal Python error: PyInterpreterState_Get: the function must be called with the GIL
held, ... but the GIL is released (the current Python thread state is NULL)

This is exactly what the installed_function case of test_cmake_build (the only case that uses SHARED) hits when the repo's documented cmake --workflow venv flow (uv-created venv) runs on macOS. Linux hides the same duplicate-runtime setup through ELF symbol interposition, and macOS CI uses dynamically linked python.org builds, so CI never sees it.

Fix, in pybind11NewTools.cmake. SHARED skips Python_add_library and links pybind11::module instead of pybind11::embed in two cases:

  • on Apple platforms, and
  • when FindPython did not find Development.Embed (this configuration was an error before).

On other platforms with Development.Embed, SHARED is unchanged and still links libpython, so an executable can link the library alone. STATIC is unchanged.

Verified locally on macOS arm64 with a uv-managed CPython: the built module no longer links libpython, the previously-failing installed_function case passes, and all other test_cmake_build cases still pass.

Behavior notes:

  • On Apple platforms, a SHARED library no longer records a libpython dependency. An executable that links it must also link pybind11::embed (already the case in classic mode). This is now in the docs, and subdirectory_function tests it.
  • SHARED + WITH_SOABI is accepted and has no effect, as before.

Suggested changelog entry:

  • pybind11_add_module in FindPython mode no longer links SHARED extension modules against libpython on Apple platforms; this fixes import crashes with statically linked interpreters (e.g. uv / python-build-standalone). An executable that links such a library must link pybind11::embed. SHARED also no longer requires Development.Embed.

@henryiii
henryiii marked this pull request as ready for review August 7, 2026 12:45
… mode

Python_add_library() hard-links non-MODULE libraries against
Python::Python. On macOS with a statically linked interpreter (e.g.
uv / python-build-standalone), importing such a module loads a second,
uninitialized copy of the Python runtime and aborts with a fatal
PyInterpreterState_Get error. Bypass python_add_library for SHARED and
link pybind11::module instead of pybind11::embed, matching the classic
pybind11Tools.cmake behavior.

Assisted-by: ClaudeCode:claude-fable-5
Assisted-by: ClaudeCode:claude-opus-5-5
@henryiii
henryiii force-pushed the fix-shared-libpython-link branch from c538993 to 41db6b8 Compare November 12, 2026 15:02
@rwgk

rwgk commented Nov 21, 2026

Copy link
Copy Markdown
Collaborator

— codex on behalf of Ralf —

Preserve or document the executable-linking contract for SHARED. On Linux with CMake 3.28.3 and CPython 3.14.10, a pybind11_add_module(helper SHARED ...) library exports a C wrapper around Py_GetVersion(). An executable linking only helper links on the parent and fails on this PR head with undefined reference to Py_GetVersion. The switch to pybind11::module drops the libpython dependency, even though the docs advertise SHARED for use as a regular dynamic library and the reported crash is on macOS. Could the fix preserve existing linkage where it is needed? If the change is intentional across platforms, please document how downstreams should link pybind11::embed for this use. Link target selection · Documented SHARED use

Generated by Codex. Ralf has yet to review these findings.

…forms

An executable that links only a SHARED pybind11 library failed to link
on Linux, because the library no longer recorded its libpython
dependency. Limit the change to Apple platforms, where the libpython
link breaks import from a statically linked interpreter, and to the case
where Development.Embed is not available. Document when an executable
must link pybind11::embed itself.

Assisted-by: ClaudeCode:claude-fable-5-2
@henryiii

Copy link
Copy Markdown
Collaborator Author

🤖 AI text below 🤖

The point is valid. Fixed in 48f8ca5.

  • Check: I could not run Linux here. On macOS, an executable that links only the SHARED library aborted at load on the previous head (dyld: symbol not found in flat namespace '_PyBaseObject_Type'); the library had no libpython dependency. This is the same cause as the Linux link error.
  • Linkage: the change now applies only on Apple platforms, where the libpython link breaks import from a statically linked interpreter. On other platforms SHARED links libpython as on master, so the Linux case links again. If FindPython did not find Development.Embed (an error on master), SHARED links as a module.
  • Docs: docs/compiling.rst now says when a SHARED library does not link libpython (Apple platforms, classic mode, no Development.Embed) and that the executable must then link pybind11::embed.
  • Test: subdirectory_function builds and runs an executable that links the SHARED library (FindPython mode, not Windows). It links pybind11::embed only on Apple platforms, so on Linux it fails to link without the fix. The non-Apple branch was run locally only by forcing the condition on macOS; CI must confirm Linux.

The PR description and changelog entry are updated.

@henryiii
henryiii marked this pull request as draft November 21, 2026 18:29
@henryiii

Copy link
Copy Markdown
Collaborator Author

🤖 AI text below 🤖

Linux results for 48f8ca5. Image: official python:3.14 (Debian 13, aarch64), CPython 3.14.10 with shared libpython, GCC 14.2.0, CMake 4.4.5, Ninja, PYBIND11_FINDPYTHON=ON.

  • Before the fix (48f8ca5 with tools/pybind11NewTools.cmake from 41db6b8): test_build_subdirectory_function fails. The executable that links only the SHARED library does not link: ld: libtest_subdirectory_function_shared.so: undefined reference to 'PyTuple_SetItem' (and other Py* symbols). readelf -d shows no libpython in NEEDED.
  • After the fix (48f8ca5): all of test_cmake_build passes, including subdirectory_function and installed_function. The SHARED library has NEEDED libpython3.14.so.1.0. The executable links only the SHARED library and runs.

@henryiii henryiii changed the title fix(cmake): don't link SHARED modules against libpython in FindPython mode fix(cmake): don't link SHARED modules against libpython on Apple platforms in FindPython mode Nov 21, 2026
@henryiii
henryiii marked this pull request as ready for review November 21, 2026 18:45

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants