classification
Title: __init_subclass__ causes TypeError when used with standard library metaclasses (such as ABCMeta)
Type: behavior Stage: backport needed
Components: Versions: Python 3.7, Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Kevin Shweh, Martin.Teichmann, Nate Soares, levkivskyi, lukasz.langa, ncoghlan
Priority: normal Keywords:

Created on 2017-02-16 18:54 by Nate Soares, last changed 2017-04-25 16:19 by So8res.

Pull Requests
URL Status Linked Edit
PR 527 merged python-dev, 2017-03-06 23:03
PR 1282 open So8res, 2017-04-25 16:19
Messages (7)
msg287966 - (view) Author: Nate Soares (Nate Soares) Date: 2017-02-16 18:54
I believe I've found a bug (or, at least, critical shortcoming) in the way that python 3.6's __init_subclass__ interacts with abc.ABCMeta (and, presumably, most other metaclasses in the standard library). In short, if a class subclasses both an abstract class and a class-that-uses-__init_subclass__, and the __init_subclass__ uses keyword arguments, then this will often lead to TypeErrors (because the  metaclass gets confused by the keyword arguments to __new__ that were meant for __init_subclass__).

Here's an example of the failure. This code:

from abc import ABCMeta
class Initifier:
    def __init_subclass__(cls, x=None, **kwargs):
        super().__init_subclass__(**kwargs)
        print('got x', x)

class Abstracted(metaclass=ABCMeta):
    pass

class Thingy(Abstracted, Initifier, x=1):
    pass

thingy = Thingy()

raises this TypeError when run:

Traceback (most recent call last):
  File "<filename>", line 10, in <module>
    class Thingy(Abstracted, Initifier, x=1):
TypeError: __new__() got an unexpected keyword argument 'x'

See http://stackoverflow.com/questions/42281697/typeerror-when-combining-abcmeta-with-init-subclass-in-python-3-6 for further discussion.
msg288119 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-02-19 08:06
This is going to be the case anytime an attempt is made to combine parent classes with incompatible constructor signatures. "type" is unusual in that it goes to great lengths to present an adaptive signature that aligns with whatever the class definition does.

The main relevant trick is to filter the extra arguments out from those passed to metaclasses such that only init_subclass sees them:

================
def ignore_extra_args(base):
    base_meta = type(base)
    class _FilteredMeta(base_meta):
        def __new__(*args, **kwds):
            return base_meta.__new__(*args)
        def __init__(*args, **kwds):
            return base_meta.__init__(*args)
    class _Filtered(base, metaclass=_FilteredMeta):
        pass
    return _Filtered

class InitX():
    def __init_subclass__(cls, x=None):
        print('x')

from abc import ABCMeta
class Abstract(metaclass=ABCMeta):
    pass

class AbstractWithInit(ignore_extra_args(Abstract), InitX, x=1):
    pass

AbstractWithInit()
================

If folks were to iterate on "ignore_extra_args" variants outside the standard library with 3.6, then it would be something we could sensibly standardise for 3.7.
msg288255 - (view) Author: Kevin Shweh (Kevin Shweh) Date: 2017-02-21 03:05
Doesn't that ignore_extra_args thing prevent InitX.__init_subclass__ from receiving the x argument it wanted? It doesn't seem like a solution.
msg288450 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-02-23 15:25
No, the filtering is only applied to the __new__ and __init__ calls on the metaclass, not to the __init_subclass__ call.

Showing the last part of an interactive session where I ran the above commands:

===========
>>> class AbstractWithInit(ignore_extra_args(Abstract), InitX, x=1):
...     pass
... 
x
>>> AbstractWithInit()
<__main__.AbstractWithInit object at 0x7f9086694a58>
>>> 
===========
msg288451 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-02-23 15:29
Oops, and now I see the typo in the example code, it seems you're right. I was misremembering an earlier more decorator-like variant of the design where we didn't rely on passing the __init_subclass__ arguments through the metaclass constructor.
msg288453 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-02-23 15:38
Note that the publisher side workaround for this is relatively straightforward: __init_subclass__ implementations that want to be compatible with arbitrary metaclasses will need to take any additional parameters as class attributes (which they may delete), rather than as class header keyword arguments.

For 3.7 though, it probably makes sense to update abc.ABCMeta to pass along arbitrary keyword arguments based on the same rationale as used in PEP 487 to justify making this change for type itself: by default, type.__init_subclass__ will still complain about it, but if someone overrides __init_subclass__ to accept additional keyword arguments, doing so will just work.
msg290179 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2017-03-24 22:18
New changeset bd583ef9857d99f9145ad0bb2c4424cc0baa63fc by Łukasz Langa (Nate) in branch 'master':
bpo-29581: Make ABCMeta.__new__ pass **kwargs to type.__new__ (#527)
https://github.com/python/cpython/commit/bd583ef9857d99f9145ad0bb2c4424cc0baa63fc
History
Date User Action Args
2017-04-25 16:19:26So8ressetpull_requests: + pull_request1390
2017-04-24 21:14:58Mariattasetstage: backport needed
2017-03-28 04:57:53xiang.zhanglinkissue29923 superseder
2017-03-24 22:18:28lukasz.langasetnosy: + lukasz.langa
messages: + msg290179
2017-03-06 23:03:43python-devsetpull_requests: + pull_request437
2017-02-23 15:38:23ncoghlansetmessages: + msg288453
versions: + Python 3.7
2017-02-23 15:30:17ncoghlansetnosy: + Martin.Teichmann
2017-02-23 15:29:23ncoghlansetmessages: + msg288451
2017-02-23 15:25:36ncoghlansetmessages: + msg288450
2017-02-21 03:05:42Kevin Shwehsetnosy: + Kevin Shweh
messages: + msg288255
2017-02-19 08:06:13ncoghlansetnosy: + ncoghlan
messages: + msg288119
2017-02-17 17:28:25levkivskyisetnosy: + levkivskyi
2017-02-16 18:54:07Nate Soarescreate