Skip to content

fix: reset gil_safe_call_once_and_store cache on interpreter finalize - #6162

Open
henryiii wants to merge 3 commits into
pybind:masterfrom
henryiii:fix/gil-safe-call-once-restart
Open

henryiii wants to merge 3 commits into
pybind:masterfrom
henryiii:fix/gil-safe-call-once-restart

Conversation

@henryiii

@henryiii henryiii commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

🤖 AI text below 🤖

Description

When subinterpreter support is enabled, gil_safe_call_once_and_store keeps the stored value in the
state dict of the interpreter, and caches a pointer to it plus an "initialized" flag in the module.
Nothing cleared that cache when the interpreter was finalized. After
finalize_interpreter(); initialize_interpreter(); the fast path gave freed memory back to the
caller. py::register_exception uses this class, thus re-importing an embedded module which
registers an exception used the freed exception object.

The per-interpreter storage now knows its owner and resets the cached pointer and the flag from its
destructor. The interpreter destroys the storage during finalization, thus the fast path becomes
invalid at the correct time and the next call stores the value again.

New Catch2 tests in tests/test_with_catch/test_interpreter.cpp restart the interpreter and then
use the stored value. Before the fix, the callable was not called again for the new interpreter, and
the re-imported exception module had no exception attribute.

Suggested changelog entry:

  • Fixed a use-after-free in py::gil_safe_call_once_and_store (and therefore in
    py::register_exception) when an embedded interpreter is finalized and then initialized again.

The old reference was released into the new interpreter's allocator, which
crashed on Python 3.12. This was also the cause of the enum test skip.
@henryiii
henryiii force-pushed the fix/gil-safe-call-once-restart branch from 67b0007 to 695f1a1 Compare November 12, 2026 19:40
@rwgk

rwgk commented Nov 21, 2026

Copy link
Copy Markdown
Collaborator

— codex on behalf of Ralf —

Avoid writes to owner from concurrent get_stored() calls. Once a subinterpreter has existed, two threads in the same interpreter can both enter the slow path and write the same non-atomic value->owner. On free-threaded CPython 3.15.2, ThreadSanitizer reported conflicting writes in three of three runs at this head and zero of three on the original code. call_once_and_store_result() already assigns the owner in the std::call_once initializer. Keeping the assignment there would avoid the race. Slow path and owner assignment

Generated by Codex. Ralf has yet to review these findings.

Threads of one interpreter can run the slow path of get_stored() at the
same time when a subinterpreter has existed. The write to the non-atomic
owner was a data race under free-threading. The owner is already set in
the std::call_once initializer.

Assisted-by: ClaudeCode:claude-fable-5-2
@henryiii

Copy link
Copy Markdown
Collaborator Author

🤖 AI text below 🤖

Writes to owner from concurrent get_stored() calls: confirmed and fixed in 2563686.

  • I added a Catch test (gil_safe_call_once_and_store concurrent get_stored after subinterpreter): four threads call get_stored() after a subinterpreter existed.
  • With free-threaded CPython 3.14 and ThreadSanitizer, the test reported the conflicting writes at the value->owner = this line in get_stored() before the fix. After the fix, three of three runs are clean, and the full test_with_catch run has no ThreadSanitizer reports.
  • The fix removes the assignment from get_stored(). The std::call_once initializer in call_once_and_store_result() sets owner for each storage that holds a value, so the second assignment was not necessary.

CI does not use ThreadSanitizer, so the new test only detects this race in a local sanitizer build.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants