This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: ctypes: fail to create a _ctypes._SimpleCData subclass using a closure like calling super() without arguments
Type: behavior Stage: needs patch
Components: ctypes Versions: Python 3.11, Python 3.10, Python 3.9
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Asdger Gdsfe, Dave Jones, eryksun, hakril, ncoghlan, serhiy.storchaka, vstinner
Priority: normal Keywords: patch

Created on 2017-01-13 22:10 by Dave Jones, last changed 2022-04-11 14:58 by admin.

File name Uploaded Description Edit
issue_29270_01.patch eryksun, 2017-01-14 02:15 review
Messages (14)
msg285444 - (view) Author: Dave Jones (Dave Jones) * Date: 2017-01-13 22:10
While investigating a bug report in one of my libraries ( I've come across a behaviour that appears in Python 3.6 but not prior versions. Specifically, calling super() in a sub-class of a ctypes scalar type appears to fail at the class definition stage. A minimal test case is as follows:

    import ctypes as ct

    class SuperTest(ct.c_uint32):
        def __repr__(self):
            return super().__repr__()

This works happily under python 3.2, 3.4, and 3.5 (that I've tested), and also under 2.7 (with the appropriate modification to super's arguments). However, under 3.6 it elicits the following exception:

    Traceback (most recent call last):
      File "", line 3, in <module>
        class SuperTest(ct.c_uint32):
    TypeError: __class__ set to <class '__main__.SuperTest'> defining 'SuperTest' as <class '__main__.SuperTest'>

Reading through the "What's New" list in 3.6, I thought this might be something to do with the PEP-487 implementation (given it modified class construction), but having read through the PEP and associated patches I'm not so sure as I can't see anything that affects the setting of the "__class__" attribute (but don't rule it out on that basis; I'm no expert!).

I'll admit that sub-classing one of ctypes' scalar types is a little odd, but given this works in prior versions and there doesn't appear to be anything in the documentation banning the practice (that I've found?) this might constitute a bug?
msg285450 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2017-01-13 23:53
In 3.6, type_new in Objects/typeobject.c sets the __classcell__ in the dict if it's a cell object. It happens that CreateSwappedType in Modules/_ctypes/_ctypes.c re-uses the dict to create the swapped type (e.g. big endian), which in turn updates the __classcell__. Thus in builtin___build_class__ in Python/bltinmodule.c, the check `cell_cls != cls` ends up being true, which leads to the observed TypeError. CreateSwappedType should be able to avoid this by either deleting "__classcell__" from the dict or creating a copy without it before passing it to type_new.
msg285454 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2017-01-14 02:15
Here's a patch that deletes __classcell__ from the dict before calling type_new.
msg285469 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-01-14 08:20
Eryk's diagnosis sounds right to me, and the suggested patch will get this back to working as well as it did in Python 3.5.

However, it's worth noting that that state itself was already broken when it comes to zero-argument super() support on the type definition with reversed endianness:

>>> import ctypes as ct
>>> class SuperText(ct.c_uint32):
...     def __repr__(self):
...         return super().__repr__()
>>> SuperText.__ctype_le__(1)
<SuperText object at 0x7fde526daea0>
>>> SuperText.__ctype_be__(1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in __repr__
TypeError: super(type, obj): obj must be an instance or subtype of type

The apparently nonsensical error message comes from both the original type and the type with swapped endianness having the same representation:

>>> SuperText.__ctype_le__
<class '__main__.SuperText'>
>>> SuperText.__ctype_be__
<class '__main__.SuperText'>
msg285473 - (view) Author: Dave Jones (Dave Jones) * Date: 2017-01-14 11:20
I confess I'm going to have to read a bit more about Python internals before I can understand Eryk's analysis (this is my first encounter with "cell objects"), but many thanks for the rapid analysis and patch!

I'm not too concerned about the state being broken with reversed endianness; I don't think that's going to affect any of my use-cases in the near future, but it's certainly useful to know in case it does come up.
msg285474 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2017-01-14 11:35
OK, this is completely broken and needs a more thoughtful solution than my simpleminded hack. Here's a practical example of the problem, tested in 3.5.2:

    class MyInt(ctypes.c_int):
        def __repr__(self):
            return super().__repr__()

    class Struct(ctypes.BigEndianStructure):
        _fields_ = (('i', MyInt),)

    >>> s = Struct()
    >>> s.i
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "<stdin>", line 3, in __repr__
    TypeError: super(type, obj): obj must be an instance or subtype of type
msg285475 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-01-14 12:04
Yeah, re-using Python-level method objects in different types is genuinely invalid when combined with __class__ or zero-argument super(), as there's no way to make the __class__ closure refer to two different classes at runtime - it will always refer back to the original defining class.

And while ctypes could be updated to do the right thing for functions, classmethod, staticmethod, abstractmethod, property, etc, there's nothing systematic it can do that would work for arbitrary descriptors (any of which may have a zero-argument-super-using method definition hidden inside their internal state).
msg285480 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2017-01-14 13:28
Resolving this would be straightforward if we could use a subclass for the swapped type, but ctypes simple types behave differently for subclasses. A simple subclass doesn't automatically call the getfunc to get a converted value when returned as a function result, struct field, or array index.
msg286287 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-01-25 23:19
See also similar issue with scrapy:
msg286310 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-01-26 12:40
The scrapy case looks to just be the new metaclass constraint that's already covered in the "Porting to Python 3.6" guide:

The ctypes case is more complicated, as its actually *reusing* the same class namespace to define two different classes, which is fundamentally incompatible with the way zero-argument super works.
msg313985 - (view) Author: Asdger Gdsfe (Asdger Gdsfe) Date: 2018-03-17 01:22
Hey 3.6 is pretty old now so can we get this patch merged I'd really like this code to start working again, appreciate all your hard work!

class Something(ctypes.c_ulong):
  def __repr__(self):
    return super(Something, self).value

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: __class__ set to <class '__main__.Something'> defining 'Something' as <class '__main__.Something'>
msg355368 - (view) Author: Clement Rouault (hakril) * Date: 2019-10-25 13:33

I have a Python2 project that relies heavily on ctypes and I cannot migrate this project to Python3 due to this bug. (I target 3.6)
I have some experience with CPython and submitting patchs and I would like to know what I can do to help moving this issue forward.

Thanks !
msg360252 - (view) Author: Clement Rouault (hakril) * Date: 2020-01-18 22:14

As this issue may never be fixed for python3.6. I wanted to post a solution to bypass the bug. It may be useful for the next person stumbling on this as I have.

The __class__ closure is only created if a function use the word super(). This closure allow to call super() without argument.

By using another name than super() the closure is not created and your code can work. Only downside is that you need to call super in its explicit form super(Cls, self). But it is better that not working at all (and it is compatible python2).

Here is a sample:

super_bypass_issue29270 = super

class Something(ctypes.c_ulong):
  def __repr__(self):
    return "BYPASS: " + super_bypass_issue29270(Something, self).__repr__()

s = Something(42)

BYPASS: <Something object at 0x00000134C9BA6848>
msg416219 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-03-28 22:12
CreateSwappedType() is an helper function used by the _ctypes.PyCSimpleType type. Python script reproducing this ctypes bug:
class PyCSimpleType(type):
    def __new__(cls, name, bases, dct):
        print(f"PyCSimpleType: create {name} class")
        cell = dct.get('__classcell__', None)
        # type.__new__() sets __classcell__
        x = super().__new__(cls, name, bases, dct)
        if cell is not None:
            print(f"PyCSimpleType: after first type.__new__() call: __classcell__={cell.cell_contents}")

        name2 = name + "_Swapped"
        dct2 = dict(dct, __qualname__=name2)
        # Calling type.__new__() again with the same cell object overrides
        # __classcell__
        x.__ctype_be__ = super().__new__(cls, name2, bases, dct2)
        if cell is not None:
            print(f"PyCSimpleType: after second type.__new__() call: __classcell__={cell.cell_contents}")

        return x

class BaseItem:

class Item(BaseItem, metaclass=PyCSimpleType):
    def get_class(self):
        # get '__class__' to create a closure
        return __class__

    # Alternative to create a closure:
    #def __repr__(self):
    #    return super().__repr__()

PyCSimpleType: create Item class
PyCSimpleType: after first type.__new__() call: __classcell__=<class '__main__.Item'>
PyCSimpleType: after second type.__new__() call: __classcell__=<class '__main__.Item_Swapped'>
Traceback (most recent call last):
  File "", line 23, in <module>
    class Item(BaseItem, metaclass=PyCSimpleType):
TypeError: __class__ set to <class '__main__.Item_Swapped'> defining 'Item' as <class '__main__.Item'>

It's not a bug in Python types, but a bug specific to the _ctypes.PyCSimpleType type which prevents creating subclasses which use closures ("__class__", "super()", etc.).
Date User Action Args
2022-04-11 14:58:42adminsetgithub: 73456
2022-03-28 22:12:40vstinnersetnosy: + vstinner
title: super call in ctypes subclass fails -> ctypes: fail to create a _ctypes._SimpleCData subclass using a closure like calling super() without arguments
messages: + msg416219

versions: + Python 3.11, - Python 3.8
2021-03-19 05:09:30eryksunsettitle: super call in ctypes sub-class fails in 3.6 -> super call in ctypes subclass fails
stage: patch review -> needs patch
versions: + Python 3.8, Python 3.9, Python 3.10, - Python 3.6, Python 3.7
2020-01-18 22:14:51hakrilsetmessages: + msg360252
2019-10-25 13:33:52hakrilsetnosy: + hakril
messages: + msg355368
2018-03-17 01:22:01Asdger Gdsfesetnosy: + Asdger Gdsfe
messages: + msg313985
2017-01-26 12:40:55ncoghlansetmessages: + msg286310
2017-01-25 23:19:32serhiy.storchakasetmessages: + msg286287
2017-01-14 13:28:51eryksunsetmessages: + msg285480
2017-01-14 12:04:52ncoghlansetmessages: + msg285475
2017-01-14 11:35:41eryksunsetmessages: + msg285474
2017-01-14 11:20:43Dave Jonessetmessages: + msg285473
2017-01-14 08:20:58ncoghlansetmessages: + msg285469
2017-01-14 02:18:46eryksunsetstage: patch review
2017-01-14 02:15:13eryksunsetfiles: + issue_29270_01.patch
keywords: + patch
messages: + msg285454
2017-01-13 23:53:52eryksunsetnosy: + eryksun
messages: + msg285450
2017-01-13 22:36:52serhiy.storchakasetnosy: + ncoghlan, serhiy.storchaka

versions: + Python 3.7
2017-01-13 22:10:22Dave Jonescreate