classification
Title: PEP487: Simpler customization of class creation
Type: enhancement Stage: resolved
Components: Interpreter Core Versions: Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: ncoghlan Nosy List: Martin.Teichmann, abarry, berker.peksag, ncoghlan, python-dev
Priority: normal Keywords: patch

Created on 2016-06-22 07:19 by Martin.Teichmann, last changed 2016-08-20 00:38 by python-dev. This issue is now closed.

Files
File name Uploaded Description Edit
pep487.patch Martin.Teichmann, 2016-07-22 20:35 The patch for PEP 487 review
pep487.patch Martin.Teichmann, 2016-07-24 09:04 Changes proposed by Nick review
pep487.patch Martin.Teichmann, 2016-07-25 15:37 The patch for PEP 487 review
issue27366_tweaks.diff berker.peksag, 2016-07-30 09:46 review
Messages (20)
msg269050 - (view) Author: Martin Teichmann (Martin.Teichmann) * Date: 2016-06-22 07:19
This is the implementation of PEP 487.

It adds a metaclass to types that calls a method on a class
once it is subclassed. This way one can customize the creation
of classes without the need to write an own metaclass.

As a second functionality, it calls a method on each descriptor
in a class, such that descriptors know their name.
msg269723 - (view) Author: Martin Teichmann (Martin.Teichmann) * Date: 2016-07-02 17:35
This is a C implementation of PEP 487, directly in the Python core
msg271129 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2016-07-24 05:15
I started reviewing Martin's patch, and I initially thought I had found a problem with the way __init_subclass__ is currently defined. It turned out I was wrong about it actually being broken, but I *do* now think it's inherently confusing, and we may be able to do something different that's more obviously correct (or at least easier to document - it was proposing revisions to the documentation that got me thinking along this path).

Specifically, I was thinking using super() in either the zero argument form or the explicit form could create an infinite loop due to the way we're currently proposing to interact with the MRO. Consider:

    class BaseClass:
        @classmethod
        def __init_subclass__(cls):
            super(cls, BaseClass).__init_subclass__()

    class SubClass(BaseClass):
        pass

If the initial call made by type.__new__() is effectively "SubClass.mro()[1].__init_subclass__()", then the super() call is going to call BaseClass.__init_subclass__ again.

However, it turned out I was wrong, as that's not what happens: the call made by the type machinery is instead "super(SubClass, SubClass).__init_subclass__", which gets it to the right place in the MRO and causes further super() calls to do the right thing.

However, the "more obviously correct" signature that occurred to me was to do this instead:

    class BaseClass:
        @classmethod
        def __init_subclass__(cls, subcls):
            super(cls, BaseClass).__init_subclass__(subcls)

    class SubClass(BaseClass):
        pass

Then the invocation from type.__new__ could be defined more simply as:

    SubClass.mro()[1].__init_subclass__(SubClass)

In all cases then (regardless of where you were in the MRO), "cls" would refer to "the class first in the MRO after the class being defined" and "subcls" would refer to "the class currently being defined".

If you consider the plugin example in the PEP, with the revised signature, it would look like:

    class PluginBase:
        subclasses = []

        def __init_subclass__(cls, subcls, **kwargs):
            super().__init_subclass__(**kwargs)
            cls.subclasses.append(subcls)

And *even if the subclass being defined shadowed the "subclasses" attribute*, this initialisation would still work. (You can still get yourself in trouble if a subclass somewhere else in the MRO shadows the attribute, but that's life in complex type hierarchies)

In the version in the PEP, the fact that "cls" is actually a subclass, and we're relying on the MRO to find "subclasses" is a really subtle implementation detail, while having two parameters makes it clear that "the class defining __init_subclass__" is distinct from the "new subclass being defined".
msg271132 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2016-07-24 05:33
Scratch that, my revised idea is fundamentally broken, as this example illustrates:

>>> class BaseClass: pass
... 
>>> class OtherClass: pass
... 
>>> class SubClass(OtherClass, BaseClass): pass
... 
>>> SubClass.mro()
[<class '__main__.SubClass'>, <class '__main__.OtherClass'>, <class '__main__.BaseClass'>, <class 'object'>]


The PEP as written handles this correctly, while the alternative signature would fail miserably ("OtherClass" wouldn't chain up to "BaseClass" properly). So I'll review the rest of the patch, and we can figure out the documentation problem later.
msg271137 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2016-07-24 08:29
Martin's current implementation basically looks good to me - I've mainly suggested some changes to the documentation and additional test cases that help stress test some of the more complex inheritance hierarchies described above, although there are a few other other suggestions as well.
msg271141 - (view) Author: Martin Teichmann (Martin.Teichmann) * Date: 2016-07-24 09:04
Thanks for the nice review!

I made the proposed changes, see attached patch.

I am still waiting for a decision whether type.__setattr__ should call __set_name__ from python-dev, once that's sorted out I'll implement and test the one or the other behavior.
msg271163 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2016-07-24 15:21
That's a no from me.

On Sunday, July 24, 2016, Martin Teichmann <report@bugs.python.org> wrote:

>
> Martin Teichmann added the comment:
>
> Thanks for the nice review!
>
> I made the proposed changes, see attached patch.
>
> I am still waiting for a decision whether type.__setattr__ should call
> __set_name__ from python-dev, once that's sorted out I'll implement and
> test the one or the other behavior.
>
> ----------
> Added file: http://bugs.python.org/file43854/pep487.patch
>
> _______________________________________
> Python tracker <report@bugs.python.org <javascript:;>>
> <http://bugs.python.org/issue27366>
> _______________________________________
>
msg271287 - (view) Author: Martin Teichmann (Martin.Teichmann) * Date: 2016-07-25 15:37
I looked over the patch once more and found some places where
I didn't follow normal coding standards. I changed that, namely
now the code returns -1 meaning an exception happened and 0 on
success, which is what many functions in typeobject.c do.

So I think this patch should be ready.
msg271665 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2016-07-30 05:05
Martin's latest patch looks good to me, so I'm applying it now and will push it once a local run of the test suite gives it the thumbs up :)
msg271667 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-07-30 06:26
New changeset ecc7bff738e0 by Nick Coghlan in branch 'default':
Issue #27366: Implement PEP 487
https://hg.python.org/cpython/rev/ecc7bff738e0
msg271668 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2016-07-30 06:41
Thanks once again Martin, especially for your patience with the long process in getting this proposal all the way through to resolution :)

I mostly applied your patch as-is, but tweaked a few aspects of the documentation before committing it (mainly expanding the What's New entry, and showing a few more of the moving parts in the  __init_subclass__ example).
msg271672 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2016-07-30 09:46
Sorry, I'm a bit late to the party. Here are some tweaks to ecc7bff738e0. I've added versionadded directives, updated the tests to use new style classes and removed a duplicate sentence from the __init_subclass__ docstring.
msg271673 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2016-07-30 09:50
Good catches Berker - go ahead and apply the improvements whenever's convenient :)
msg271682 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-07-30 11:05
New changeset 8747e3455ecb by Berker Peksag in branch 'default':
Issue #27366: Tweak PEP 487 documentation
https://hg.python.org/cpython/rev/8747e3455ecb
msg271683 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2016-07-30 11:08
Thanks for the review, Nick! (and also thanks to Martin for the great PEP!)
msg271717 - (view) Author: Anilyka Barry (abarry) * (Python triager) Date: 2016-07-31 01:35
Hello Martin, and thank you for your patch! However, this patch removed the ability to pass keyword arguments to `type`, and it's not documented anywhere. I believe it should be documented at least in e.g. the "Changes in the Python API" of the What's New in Python 3.6 document. Anyone cares to submit a patch?
msg271718 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2016-07-31 02:28
D'oh, another good catch Emanuel - and I'm even the one that raised the need to mention that in the Porting notes during the python-dev discussion :P

I'll post an update to the What's New shortly.
msg271720 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-07-31 02:43
New changeset 313e8fdb0d0c by Nick Coghlan in branch 'default':
Issue 27366: PEP 487 docs updates
https://hg.python.org/cpython/rev/313e8fdb0d0c
msg271721 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2016-07-31 02:46
The new porting note doesn't quite capture all the subtleties of the situation, but should be sufficient for folks to get any affected custom metaclasses working again (since the key is to avoid passing those parameters up to type.__new__).

I also added a note about the fact that __init_subclass__ implementations don't have access to the metaclass hint, but can retrieve the actual metaclass based on the passed in class object.
msg273174 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-08-20 00:38
New changeset 545bfa4c20eb by Victor Stinner in branch 'default':
Issue #27366: Fix init_subclass()
https://hg.python.org/cpython/rev/545bfa4c20eb
History
Date User Action Args
2016-08-20 00:38:51python-devsetmessages: + msg273174
2016-07-31 02:46:45ncoghlansetstatus: open -> closed
resolution: fixed
messages: + msg271721

stage: needs patch -> resolved
2016-07-31 02:43:06python-devsetmessages: + msg271720
2016-07-31 02:28:11ncoghlansetmessages: + msg271718
stage: resolved -> needs patch
2016-07-31 01:35:10abarrysetstatus: closed -> open

nosy: + abarry
messages: + msg271717

resolution: fixed -> (no value)
2016-07-30 11:08:12berker.peksagsetmessages: + msg271683
2016-07-30 11:05:42python-devsetmessages: + msg271682
2016-07-30 09:50:38ncoghlansetmessages: + msg271673
2016-07-30 09:46:08berker.peksagsetfiles: + issue27366_tweaks.diff

messages: + msg271672
2016-07-30 06:41:30ncoghlansetstatus: open -> closed
resolution: fixed
messages: + msg271668

stage: patch review -> resolved
2016-07-30 06:26:27python-devsetnosy: + python-dev
messages: + msg271667
2016-07-30 05:05:25ncoghlansetmessages: + msg271665
2016-07-25 15:40:05gvanrossumsetnosy: - gvanrossum
2016-07-25 15:37:20Martin.Teichmannsetfiles: + pep487.patch

messages: + msg271287
2016-07-24 15:21:04gvanrossumsetmessages: + msg271163
2016-07-24 09:04:05Martin.Teichmannsetfiles: + pep487.patch

messages: + msg271141
2016-07-24 08:29:11ncoghlansetmessages: + msg271137
2016-07-24 05:34:31ncoghlansetassignee: ncoghlan
2016-07-24 05:33:49ncoghlansetmessages: + msg271132
2016-07-24 05:15:02ncoghlansetnosy: + gvanrossum
messages: + msg271129
2016-07-24 04:03:49ncoghlansetnosy: + ncoghlan
2016-07-22 20:35:34Martin.Teichmannsetfiles: + pep487.patch
2016-07-22 20:35:07Martin.Teichmannsetfiles: - pep487.patch
2016-07-22 20:22:28berker.peksagsetnosy: + berker.peksag

components: + Interpreter Core, - Library (Lib)
stage: patch review
2016-07-22 19:58:16Martin.Teichmannsetfiles: + pep487.patch
2016-07-22 19:57:37Martin.Teichmannsetfiles: - pep487a.patch
2016-07-13 14:12:07Martin.Teichmannsetfiles: - pep487.patch
2016-07-02 17:35:25Martin.Teichmannsetfiles: + pep487a.patch

messages: + msg269723
2016-06-22 07:19:27Martin.Teichmanncreate