Skip to content

Draft: Support arbitrary type in item accessor's subscript operator [with more tests] - #4007

Closed
cielavenir wants to merge 11 commits into
pybind:masterfrom
cielavenir:addTests
Closed

cielavenir wants to merge 11 commits into
pybind:masterfrom
cielavenir:addTests

Conversation

@cielavenir

Copy link
Copy Markdown
Contributor

(to run ci)

#4004

Comment thread include/pybind11/pytypes.h Resolved
Comment thread tests/test_pytypes.cpp Outdated
py::list l1 = py::list();
l1.append(1);
l1.append(2);
py::object l2 = l1;

@Skylion007 Skylion007 Jun 17, 2022 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This appears to be causing clang-tidy errors and a segfault when using python debug mode.

LIkely because the underlying list holding the elements is deleted and it's using the generic object accessor instead of the specialized list accelerator. This making l[1] still be attached the list object which causes issues.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually super odd that only Python built in debug mode causes the segfault. Thoughts @rwgk @henryiii ?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, but I looked a little bit only.
Sorry, I don't have the free bandwidth to help debugging this.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rwgk No prob, just wondering if you have seen this behavior before.

@Skylion007 Skylion007 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nits: should make clang-tidy happy and are better style more general (allows reference stealing).

Comment thread tests/test_pytypes.cpp Resolved
Comment thread tests/test_pytypes.cpp Resolved
Comment thread tests/test_pytypes.cpp Resolved
@cielavenir

Copy link
Copy Markdown
Contributor Author

by the way, L62 is access_list, which is this:

    m.def("access_list", []() {
        py::list l1 = py::list();
        l1.append(1);
        l1.append(2);
        return l1[1];
    });

why this one segfaults?

@cielavenir

Copy link
Copy Markdown
Contributor Author

@rwgk @Skylion007 do you have ideas about segfault in debug build?

@Skylion007

Copy link
Copy Markdown
Collaborator

@rwgk @Skylion007 do you have ideas about segfault in debug build?

Yeah, unfortunately and I am not sure it's an easy fix (at least if it's really the method after (access_list_as_object) that is segfaulting.

It's because depending on the type of the object, the accessor either borrows or steals the reference to the underlying object. What's happening here is that the list is deleted which is the only remaining reference to the object since it's not properly borrowed by the item accessor. The GC then frees the item since the ref_count is 0. However, we still have a pointer to that object (returned from the function), but it's now pointing to free memory. Hence, the segfault.

Not sure these is an easy solution here as the type of the accessor that needs to be called depends on the object. You may be able to get away with this through virtual inheritance / overloading, but that itself seems bugprone and currently our class inheritance method is non-virtual and pybind11 even supports compilation without RTTI (which this solution may break).

We could also try to return strong reference for the accessors by seeing if we can use different CPython APIs, but this may both have performance implications and could introduce ref_count based memory leaks if we are not careful.

Not sure what a good solution is here other than ensure that object is properly downcasted or that the argument is a proper Python object. We already pushed a PR that disable implicit conversions of int to handles so that should fix some of the issues that this PR brought up.

@cielavenir

Copy link
Copy Markdown
Contributor Author

@henryiii Do you mind explaining why you closed this pull request? I pushed https://github.com/cielavenir/pybind11/tree/addTests again but the CI does not run anymore.

Or is the issue fixed upstream actually?

@cielavenir

Copy link
Copy Markdown
Contributor Author

or do we need to use customized version of pybind11?

@henryiii

Copy link
Copy Markdown
Collaborator

I was closing all duplicate PRs, I assumed 4004 was the main one. This said "to run ci", so I thought it was a CI-only duplicate of #4004, which is still open. If it's better to keep this one open instead, let me know and I can switch them.

@henryiii

Copy link
Copy Markdown
Collaborator

I opened #6286, which I think solves these cleanly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants