Skip to content

feat: accept any castable type as a key in operator[] - #6286

Open
henryiii wants to merge 1 commit into
pybind:masterfrom
henryiii:feat/item-accessor-any-key
Open

henryiii wants to merge 1 commit into
pybind:masterfrom
henryiii:feat/item-accessor-any-key

Conversation

@henryiii

@henryiii henryiii commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

The key difference here is that it's a constrained template, so that the old fast-paths can keep working. Every one of the three current specialization does something useful: const char * avoids alocation and copy, py::handle can't be done generally (would take/drop refcounts), and even py::object && avoids a borrow and move.

I didn't do any of the other accessors, but getattr and maybe setattr for consistency seems reasonable too.

This fixes a bug: if you pass obj[0], before this PR that 0 matched const char * as nullptr (common C++ annoyance).

🤖 AI text below 🤖

Description

Closes #1309. Supersedes and closes #4004 (and #4007). Thanks to @woodychow and @cielavenir for the original proposal and tests.

obj[key] now accepts any C++ type that py::cast can convert, for example d[1], d[2.5], d[std::string("s")], d[std::make_pair(1, 2)] and o[1][2].

The change adds one constrained template overload of object_api::operator[]. The handle, object && and const char * overloads are unchanged, so existing code resolves to the same overload at the same cost. #4004 replaced all three with a catch-all T &&. That sends string literals through py::cast, which builds a temporary std::string, and it loses the implicit conversions to handle (for example from a type with operator PyObject *()).

py::tuple, py::list and py::sequence declare their own operator[], which hides the new overload. t[1] on those types still uses the size_t accessors.

Behaviour change: obj[0] compiled before, because the literal 0 converted to a null const char * and went to py::str(nullptr). It is now the integer key 0.

The segfault seen in #4007 was in the tests. The lambdas returned l1[1] with a deduced return type, which is a list_accessor. An accessor holds a non-owning handle to the container and reads the item lazily, so the read occurred after the list was destroyed. This also occurs on master.

Suggested changelog entry:

  • operator[] on py::object, py::dict and accessors now accepts any key type that py::cast can convert (for example d[1]). The literal 0 is now the integer key 0; before, it was a null const char *.

Add a constrained template overload of object_api::operator[] that
converts the key with pybind11::cast. The handle, object&& and
const char* overloads are unchanged.

The literal 0 is now the integer key 0. Before, it converted to a null
const char*.

Closes pybind#1309.

Assisted-by: ClaudeCode:claude-fable-5-2
@henryiii

henryiii commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator Author

Here's ideas for a followup. I do wonder if this should not close 1309 until getattr is also supported...

🤖 AI text below 🤖

These are the entry points that still accept only Python objects (or const char *), sorted by how useful a cast would be. I read the declarations in pytypes.h but did not prototype any of these.

Entry point Today Would allow Notes
getattr(obj, name, default_) handle default_ py::getattr(o, "x", 42) Named in #1309. Small and low risk.
setattr(obj, name, value) handle value py::setattr(o, "x", 42) o.attr("x") = 42 already casts, so this is only for consistency.
Arithmetic and bitwise operators (+ - * / | & ^ << >> and the = forms) object_api const &other o + 1, o * 2.0 Most useful after operator[]. Higher risk: 18 operators, and a template operand can cause ambiguous overloads in user code.
equal, not_equal, <, <=, >, >= object_api const &other o.equal(1), o < 2 Same risk as arithmetic. == and != must stay as pointer identity on handle.
attr(name), getattr, hasattr, delattr, setattr name argument handle or const char * o.attr(std::string(...)), std::string_view Only strings are valid names, so a std::string overload is sufficient. Low value.
py::len, py::repr, py::hash, py::iter, isinstance handle py::repr(42) Little practical use. I would not change these.

These already cast, so they need no change: contains() on object_api, dict and set; list::append and list::insert; set::add; accessor assignment (d[k] = 42, o.attr("x") = 42); call arguments; make_tuple; print.

operator[] on py::tuple, py::list and py::sequence takes size_t, so l[-1] wraps to a very large index instead of counting from the end. This is a related gap, but it is an index problem, not a cast problem.

I recommend that #6286 stays as it is. The getattr default and setattr value are a good small follow-up. The operators need their own PR because of the overload risk.

@rwgk rwgk left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My main concern is that obj[key] can hide a fair amount of work for a small convenience.

With the STL caster, a vector used as an index becomes a fresh Python list on every access, with every element converted. That happens as soon as the accessor is created, even though the lookup is lazy. I’d prefer to see that conversion at the call site. Conversion path

There’s also a less obvious identity issue. In a small reproducer with a bound C++ key that uses Python’s default identity hashing, d[key] = 42 followed by d[key] raised KeyError. The two casts created distinct Python objects. Casting once and reusing the Python key worked.

I see the consistency with dict::contains and dict::get. For operator[], could we start with integral keys? obj[1] is useful, and obj[0] fixes the null C-string trap. I’d keep py::cast(key) explicit for compound and user-defined keys, where the cost and identity of the converted key can matter.

@henryiii

Copy link
Copy Markdown
Collaborator Author

I'm trying an int-only version, in #6291. But I don't follow:

In a small reproducer with a bound C++ key that uses Python’s default identity hashing

But keys of dictionaries are supposed to be hashable. object() shouldn't be used as the key of a dictionary. For any well formed key, this should be fine. For a non-well formed key, this isn't much different than expecting dict[object()] to be repeatable.

Most places in pybind11 autocast, I think, so I still think this is fairly sensible.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implicit conversion of ints (and char const*) to handles?

2 participants