This is about an extension type created via PyType_FromSpec that overrides tp_setattro (minimal example attached). In this case cpython does not let me grab and use the __setattr__ function "manually". Example:

>>> import demo
>>> mytype_setattr = demo.MyType.__setattr__
>>> i = demo.MyType()
>>> mytype_setattr(i, "foo", "bar")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: can't apply this __setattr__ to object object

I suspect this is related to the loop at the beginning of typeobject.c / hackcheck() that skips over types with Py_TPFLAGS_HEAPOBJECT. (Though removing the loop breaks things like the enum module).

I ran into this, too. I agree that the "hackcheck" loop on heap types is probably wrong.

It was written at a time (Py2.3?) when (practically) only Python implemented types were heap types, not extension types. I think what it tried to do was to find the builtin base type of a Python type, and check that no-one played tricks on the C slot functions of that C implemented type. With extension types implemented as heap types, having a different slot function in there is a perfectly valid thing.

I'll call in a couple of people since I'm also not sure how to fix the "hackcheck".

I'll also leave Py3.7 in the list of affected Python versions since we still have a short week before its final non-secfix-release. :)

See also test_carloverre() in test_descr.py. I don't recall exactly what this was about, but that test is definitely relevant.

This SO answer by Martijn Pieters explains the background:

https://stackoverflow.com/a/44854477

and links to the original python-dev discussion:

https://mail.python.org/pipermail/python-dev/2003-April/034535.html

The current implementation checks that the function being called is the one defined in the first non-heap type up the hierarchy. As long as heap types are only Python types, this is the first builtin/native/C-implemented (super)type. With extension types as heap types, however, it's no longer necessarily the type that defines the correct slot function.

IMHO, Greg Ewing gives the right direction here:

https://mail.python.org/pipermail/python-dev/2003-April/034569.html

Providing some way for objects to prevent superclass
methods from being called on them when they're not looking

So, I think we should do something like walking up the hierarchy to find the C function in it that is currently being called, and then check if it's the one we would expect or if a subclass defines a different one. The current check does not bother to search and just assumes that it knows which (super)type defines the right function to call.

That sounds like the right thing to do. You should be able to recognize pure Python __setattr__ implementations by the presence of slot_tp_setattro in the tp_setattro slot. (Likewise for __delattr__.)

It's been really long since I looked at this -- do we still only support single inheritance at the C level? Because otherwise there would be another backdoor.

There's also tp_setattr (no 'o') but I think that's basically unused, and we ignored that in 2003 so we should be able to ignore it now. :-)

do we still only support single inheritance at the C level?

Not any more. Heap types may have multiple bases, and they can be created at the C level (since Python 3.2, PEP-384).

So, I think we should do something like walking up the hierarchy to find the C function in it that is currently being called, and then check if it's the one we would expect or if a subclass defines a different one. The current check does not bother to search and just assumes that it knows which (super)type defines the right function to call.

Should we be bold and skip the check for heap types?
That would mean that when/if str is converted to a heap type, you could do object.__delattr__(str, "lower"). That would break your Python, but only at the Python level (similar to things like import builtins; del builtins.__build_class__).
Heap types are not shared between interpreters, so that reason is gone. But Guido's [2003 mail] suggests there are other reasons to prevent changing built-in types. What are they?

[2003 mail] https://mail.python.org/pipermail/python-dev/2003-April/034535.html

I chose to go through the MRO, which takes multiple inheritance into account.

I think we missed the train for fixing 3.7 (which was questionable anyway), but I added a test, so it's ready for merging into 3.8+ (minus merge conflicts for the test in 3.8, probably).

New changeset 148f329 by scoder in branch 'master':
bpo-39960: Allow heap types in the "Carlo Verre" hack check that override "tp_setattro()" (GH-21092)
148f329

Stefan can you do the 3.8 backport?

New changeset bfec674 by Miss Islington (bot) in branch '3.9':
bpo-39960: Allow heap types in the "Carlo Verre" hack check that override "tp_setattro()" (GH-21092)
bfec674

New changeset 8912c18 by scoder in branch '3.8':
bpo-39960: Allow heap types in the "Carlo Verre" hack check that override "tp_setattro()" (GH-21092) (GH-21339)
8912c18

Fixed in 3.8+. Closing.
Thanks for the feedback.

You're welcome. Hope this helps Cpython users.

Reopening because there appears to be a problem with the fix (see PR 21473).

@Stefan Do you agree that the fix proposed in PR 21473 needs to be made?

The problem in the test added in PR 21473 is that there is an intermediate base type "object" in the hierarchy:

    class A(type):
        def __setattr__(cls, key, value):
            type.__setattr__(cls, key, value)

    class B:
        pass

    class C(B, A):
        pass

>>> [c for c in C.__mro__]
[<class '__main__.C'>, <class '__main__.B'>, <class '__main__.A'>, <class 'type'>, <class 'object'>]

>>> [c.__setattr__ for c in C.__mro__]
[<function A.__setattr__ at 0x7f3ca4308378>, <slot wrapper '__setattr__' of 'object' objects>, <function A.__setattr__ at 0x7f3ca4308378>, <slot wrapper '__setattr__' of 'type' objects>, <slot wrapper '__setattr__' of 'object' objects>]

I think the change to the algorithm there is too broad, it disables much of what the check was written for (or against).

Given Guido's second (negative) test case, I think it would also not be correct to add "object.__setattr__" to the list of allowed (intermediate) slot methods:

    class A(type):
        def __setattr__(cls, key, value):
            object.__setattr__(cls, key, value)   # this should fail!

    class B:
        pass

    class C(B, A):
        pass

It's difficult to turn this into an algorithm. Is the MRO really the place to look? For "A", we're only really interested in the C-level bases, aren't we?

intermediate base type "object" in the hierarchy

Sorry, I meant an intermediate base type "B", which inherits its setattr from "object".

I pushed PR 21528 with a new proposal. See bpo-41295.

Closing again since #65727 has been merged in bpo-41295.