Skip to content

fix: add smart_holder keep_alive patient before publishing the instance - #6264

Open
henryiii wants to merge 2 commits into
pybind:masterfrom
henryiii:fix/smart-holder-keep-alive-race
Open

henryiii wants to merge 2 commits into
pybind:masterfrom
henryiii:fix/smart-holder-keep-alive-race

Conversation

@henryiii

Copy link
Copy Markdown
Collaborator

🤖 AI text below 🤖

Description

Part of #6214, follow-up to #6215.

smart_holder_from_unique_ptr and smart_holder_from_shared_ptr called keep_alive_impl() after init_instance(). init_instance() registers the instance, so on free-threaded builds another thread could find it and read its flags while has_patients was written. This PR moves the keep_alive_impl() call before init_instance(). The generic cast path already uses this order.

test_concurrent_cast_of_shared_cpp_object has a new case for the reference_internal policy. The race is only visible under ThreadSanitizer, which CI does not run. I checked it locally with a TSan build of CPython 3.14.7t: without the fix TSan reports the race (write in add_patient, read in the value_and_holder constructor) in 3 of 3 runs, and with the fix it reports nothing in 5 of 5 runs.

This PR does not close #6214. Two races remain:

  • Custom holders still register the instance before the holder is constructed.
  • py::keep_alive<> on an instance that is already shared writes has_patients while other threads read the same bitfield byte.

Suggested changelog entry:

  • Fixed a data race on free-threaded builds where a smart_holder instance returned with return_value_policy::reference_internal was visible to other threads before its keep-alive flag was set.

smart_holder_from_unique_ptr and smart_holder_from_shared_ptr called
keep_alive_impl() after init_instance(). On free-threaded builds another
thread could find the registered instance and read its flags while
has_patients was written. The generic cast path already uses this order.

Part of pybind#6214.

Assisted-by: ClaudeCode:claude-fable-5-2
@rwgk

rwgk commented Nov 21, 2026

Copy link
Copy Markdown
Collaborator

— codex on behalf of Ralf —

Preserve holder ownership if keep_alive_impl throws. In both the unique-pointer and shared-pointer branches, keep_alive_impl can now throw after the Python wrapper has received the raw pointer and before its holder is constructed. The temporary smart holder still owns the pointer, and the wrapper’s unconstructed-holder cleanup also calls operator delete. A missing parent reproduces this with py::cast(std::make_unique<Item>(), py::return_value_policy::reference_internal) or a zero-argument reference_internal binding returning const std::shared_ptr<Item>&. Please preserve the holder-aware cleanup on this path while keeping the patient insertion ahead of instance publication. Unique-pointer branch · Shared-pointer branch

I tested both reproductions on CPython 3.15.2 with the GIL enabled and with a free-threaded build. The parent commit raised Could not activate keep_alive! in both cases. The PR head aborted with double-free or invalid-free diagnostics in both cases.

Generated by Codex. Ralf has yet to review these findings.

…inter

keep_alive_impl() can throw (no parent). It ran after the wrapper had
owned = true and the raw pointer, but before the holder was constructed,
so the wrapper cleanup and the temporary smart_holder both deleted the
object. Call it immediately after make_new_instance() instead.

Assisted-by: ClaudeCode:claude-fable-5-2
@henryiii

Copy link
Copy Markdown
Collaborator Author

🤖 AI text below 🤖

The finding is correct. keep_alive_impl ran after the wrapper had owned = true and the raw pointer, but before the holder was constructed, so a throw caused a double delete.

Fixed in the latest commit: in both branches, keep_alive_impl now runs immediately after make_new_instance, before the wrapper gets the pointer. If it throws, the wrapper is empty and the caller's unique_ptr / shared_ptr still owns the object. The patient is still added before init_instance.

test_reference_internal_no_parent in tests/test_class_sh_basic.py covers both reproductions. It crashed before the fix and now raises Could not activate keep_alive!. Tested locally on CPython 3.13 with the GIL only.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: Data race in class_::init_instance() on free-threaded builds

2 participants