classification
Title: Code, test, and doc review for PEP-0435 Enum
Type: enhancement Stage: resolved
Components: Documentation, Library (Lib), Tests Versions: Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: ethan.furman Nosy List: alex, barry, benjamin.peterson, docs@python, dstufft, eli.bendersky, ethan.furman, ezio.melotti, gvanrossum, haypo, isoschiz, ncoghlan, pconnell, python-dev, zach.ware
Priority: normal Keywords: patch

Created on 2013-05-10 06:50 by ethan.furman, last changed 2013-06-15 00:57 by ncoghlan. This issue is now closed.

Files
File name Uploaded Description Edit
pep-0435.01.patch ethan.furman, 2013-05-10 07:09 review
pep-0435.02.patch ethan.furman, 2013-05-10 15:21 review
pep-0435.03.patch ethan.furman, 2013-05-10 17:54 review
pep-0435.04.patch ethan.furman, 2013-05-10 21:04 review
pep-0435.05.patch ethan.furman, 2013-05-12 14:24 review
pep-0435.06.patch ethan.furman, 2013-05-15 02:28 latest code with appropriate changes review
pep-0435.07.eliben.patch eli.bendersky, 2013-05-26 16:04 review
pep-0435.09.stoneleaf.patch ethan.furman, 2013-06-08 08:41 review
pep-0435.10.stoneleaf.patch ethan.furman, 2013-06-09 04:45 review
pep-0435.11.stoneleaf.patch ethan.furman, 2013-06-09 06:20 review
pep-0435.12.stoneleaf.patch ethan.furman, 2013-06-11 18:40
Messages (65)
msg188812 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2013-05-10 06:49
PEP-0435 has been approved!

Now for lots of code review.
msg188814 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-05-10 07:01
Can you upload the patch as a diff against the CPython repo?
msg188815 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2013-05-10 07:09
Here it is (I hope ;) .
msg188833 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-05-10 13:21
I don't see docs in the patch, but perhaps that should be a separate patch to keep reviewing easier.

Also, Ethan, number the patch files in some way (like pep-435.1.patch, pep-435.N.patch) as they go through rounds of reviews.
msg188837 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-05-10 14:03
OK, I sent another batch of reviews through the code review system - Ethan you should've gotten an email about it.
msg188839 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-05-10 14:05
Regarding module vs. module_name for the extra param in the functional API, Guido rightly points out in issue 17941 that __module__ is the class attribute, so "module" is a consistent choice.
msg188841 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2013-05-10 15:21
Incorporated comments.
msg188856 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2013-05-10 17:54
More adjustments due to review.
msg188881 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2013-05-10 21:04
Round 4.  I wonder if I should have used two digits to number the patches.  ;)
msg188891 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-05-11 03:03
By the way, if anyone is willing to take on the documentation, that could be of great help. It shouldn't be very hard since PEP 435 is extremely detailed. The stdlib .rst doc would be the descriptions from PEP 435 plus a reference of the exposed classes which is pretty minimal.

We'll need the documentation patch available by the time the code is done reviewing.
msg188893 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-05-11 03:09
Two requests:

1. Lose the frame hack. With the explicit "module=__name__" API available, the implicit fragile magic isn't necessary and leads to Enum instances that may or may not support pickling depending on the implementation. Better to say "if you use the functional API without passing the module name, your Enum instances won't support pickling".

2. Mike Bayer (SQL Alchemy author) has requested the ability to derive a custom metaclass that turns off the Enums-with-members-are-final subclassing restriction (relaxing the rule that members of an enum are instances of that enum to "members of an enum are instances of that enum, or one of its parent enums", but otherwise keeping the same behaviour as the standard enums).

Barry would probably appreciate this capability, too ;)

We don't need to explicitly support this for the initial patch (besides, as I read the current code, you can get something like this already just by overriding EnumMeta._get_mixins), but we may want to revisit it as a supported subclassing API at some point in the future.
msg188894 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-05-11 03:26
On Fri, May 10, 2013 at 8:09 PM, Nick Coghlan <report@bugs.python.org>wrote:

>
> Nick Coghlan added the comment:
>
> Two requests:
>
> 1. Lose the frame hack. With the explicit "module=__name__" API available,
> the implicit fragile magic isn't necessary and leads to Enum instances that
> may or may not support pickling depending on the implementation. Better to
> say "if you use the functional API without passing the module name, your
> Enum instances won't support pickling".
>
>
Strong -1 from me on this. The PEP has been accepted. Feel free to raise
this for discussion if you want to change the decision. With all due
respect to IronPython, most users can write simpler code and have pickling
work just fine.
_______
msg188895 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-05-11 03:28
> 2. Mike Bayer (SQL Alchemy author) has requested the ability to derive a
> custom metaclass that turns off the Enums-with-members-are-final
> subclassing restriction (relaxing the rule that members of an enum are
> instances of that enum to "members of an enum are instances of that enum,
> or one of its parent enums", but otherwise keeping the same behaviour as
> the standard enums).
>

Nick, after you read the whole code, please consider whether it's not
already complex enough even without catering to every other need out there.
msg188921 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-05-11 13:58
The accepted PEP states that the frame hack should be removed: "To support pickling of these enums, the module name can be specified using the module keyword-only argument."

Ergo, if you don't specify it, they cannot be pickled. Explicit is better than implicit, and this hack should not propagate beyond namedtuple (it should actually be deprecated in namedtuple as well).

As far as the second point goes, I had already reviewed the PEP implementation before making the comment (that's why I am reasonably sure you can already do it just by overriding _get_mixins). I see it as similar to the changes that were already made to support autonumbered subtypes.

However, also note that I said we should wait before doing anything about providing a supported mechanism for that customisation. I've now created issue 17954 to cover a possible refactoring and documentation of that part of the implementation.
msg188933 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2013-05-11 17:10
Sorry everyone, the frame hack needs to stay.  This is not negotiable.
msg188941 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-05-11 18:38
Why we need two ways to do it? If "module=__name__" is available, what's the rationale for providing the option of leaving it out when pickling support is required? (The only times the frame hack will work to enable pickling, the explicit fix will also work).
msg188947 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-05-11 19:10
Ethan, something is wrong with _StealthProperty. Well, two things. First, it's way too general and can be cut to just what Enum needs. Second, this is enough to make the tests pass:

class _StealthProperty():
    """
    Returns the value in the instance, or the virtual attribute on the class.

    A virtual attribute is one that is looked up by __getattr__, as opposed to
    one that lives in __class__.__dict__

    """

    def __init__(self, fget):
        self.fget = fget

    def __get__(self, obj, objtype=None):
        return self.fget(obj)

    def __set__(self, obj, value):
        raise AttributeError("can't set attribute")

    def __delete__(self, obj):
        raise AttributeError("can't delete attribute")

Now  this is fishy because __get__ gets obj=None when called on a class and therefore self.fget(obj) raises an exception. But this gets caught somewhere in your code or tests and ignored. However, the right thing is still returned. I didn't have more time to investigate, but this must be cleaned out.
msg188948 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-05-11 19:11
Also, your test must run within the regrtest framework. Currently I get:

./python -mtest.regrtest test_enum
[1/1] test_enum
test test_enum failed -- Traceback (most recent call last):
  File "/home/eliben/python-src/default/Lib/test/test_enum.py", line 245, in test_pickle_enum_function_with_module
    self.assertIs(Question.who, loads(dumps(Question.who)))
_pickle.PicklingError: Can't pickle <enum 'Question'>: attribute lookup __main__.Question failed

1 test failed:
    test_enum
msg188975 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2013-05-12 05:25
Eli,

The original _StealthProperty checked to see if it was being called on instance or class, and if it was the class it invoked __getattr__ to attempt a lookup for an enum member.  Your version does not check, but, ironically, the exception raised is AttributeError, and so Python is calling __getattr__ anyway and so finds the virtual enum member.

While this is cool, and it works, I think it's much less mysterious, magical, and downright confusing to keep the original behavior and call __getattr__ from _StealthProperty.  On the other hand, it might make somebody think, and that's always good, so I'm happy to leave it your way.
msg188977 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2013-05-12 05:39
regrtest framework now supported  (had to change the module name I was passing in from '__main__' to __name__).
msg188979 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-05-12 06:25
Guido has promised an explanation for why he wants to keep the frame hack once he is back at work next week. To help him target that reply appropriately for my concrete objections to retaining it, I'd like to explain a bit more about why I think it's fundamentally impossible to create a robust stack inspection mechanism for implicit pickling support. Even if a dedicated mechanism for it is added, the information the interpreter needs to make a sensible reliable decision simply isn't there.

Consider the following:

    # In utils.py
    def enum_helper(name, prefix, fields):
        if isinstance(fields, str): fields = fields.split()
        e = Enum(name, (prefix + field for field in fields))
        return e

    # In some other module
    from utils import enum_helper
    MyEnum = enum_helper(MyEnum, "st_", "mtime atime ctime")

This breaks the frame hack, but would work correctly if enum_helper was redesigned to accept an explicit module name, or if we adopted something like the "name binding protocol" discussed on Python ideas.

If we adopted a simplistic rule of ignoring function scopes and only look at module and class scopes, then we break the semantics of the global keyword.

I consider the frame hack fundamentally broken, since there's currently no way the interpreter can distinguish between an incidental assignment (like the "e = Enum..." call in util.py) and a destination assignment (like the "MyEnum = enum_helper..." call), and if we *do* add a different syntax for "this supports pickling" assignments (like the def name = expr syntax on Python ideas), then a name binding protocol makes more sense than implicit contextual magic.
msg188982 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-05-12 06:52
I also created issue 17959 to track a legitimate concern with the current aliasing design, without impacting incorporation of the accepted PEP.
msg189014 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-05-12 10:59
Another post-incorporation proposal (issue 17961) relating to the values used for the functional API.
msg189020 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2013-05-12 11:24
After more thought, I think we should leave the shortened version of _StealthProperty as is.

The reasoning being that if _StealthProperty does the __getattr__ lookup itself, and fails, it will return an AttributeError... and then Python will use __getattr__ again to try and find the attribute.

So I'll add a comment into the shortened version and leave it at that.
msg189021 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-05-12 12:17
Nick, could you open a separate issue for the frame hack discussion, like you did for the other things? You can make this one depend on it, if you feel it's a "blocker", but let's please not intermix more discussion here.
msg189027 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-05-12 12:54
Ethan, I pasted the minimized version to point out the problem; I fully agree it should do the getattr lookup because otherwise it's very difficult to understand what's going on. Let's keep exceptions for actual exceptional situations and explicitly code the normal path.

Also, _MemberOrProperty is probably a more descriptive name.
msg189029 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-05-12 13:09
Eli: done, created as #17963
msg189036 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2013-05-12 14:24
Here's the latest patch.

Note that the functional API portion is broken, and I'll get back to that this evening.  Please only comment on the working code.  ;)

I'll make that _MemborOrProperty name change then as well.
msg189160 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2013-05-13 18:32
Here's the promised explanation why I want to keep the getframe hack.  I'm sure it won't satisfy everyone, but this will have to do.

There are two parts to my argument.  TL;DR: (a) by implementing this hack, we will maximize user happiness; (b) I expect that all Python implementations can provide the functionality needed.

So why does this maximize user happiness?  First of all, enums are such simple objects that it's a disgrace to have an enum that isn't picklable.  But most users are lazy and lack imagination, so if they find they have to do extra work to ensure that something is picklable, they will make a judgement call -- is the extra work to make it picklable worth it?  Unfortunately they will try to make this judgment call in a fraction of a second, and they limited imagination they may well say "I can't imagine ever pickling this", save themselves the work, and move on.  If you think more than a second about the decision, you've wasted more time than it takes to type "module=__name__", so it's going to be a split-second decision.  But nevertheless, having to think about it and typing it is a distraction, and when you're "in the zone" you may prefer to spend your time thinking about the perfect names for your enum class and values rather than the red tape of making in picklable.

So, in my mind, it's a given that there will be many enums that are missing the module=__name__ keyword argument.  Moreover, it's likely that most of the time this will be fine -- you can get through most days just fine without ever reading or writing a pickle.  (For example, I very much doubt that I've ever pickled one of the flag values used by the socket module to define e.g. the address family, socket type, or protocol.)

But now you enter a different phase of your project, or one of your collaborators does, or perhaps you've released your code on PyPI and one of your users does.  So someone tries to pickle some class instance that happens to contain an unpicklable enum.  That's not a great experience.  Pickling and unpickling errors are often remarkably hard to debug.  (Especially the latter, so I have privately admonished Ethan to ensure that if the getframe hack doesn't work, the pickle failure should happen at pickling time, not at unpickle time.)  Once you've tracked down the source, you have to figure out the fix -- hopefully just typing the error message into Google will link back to a StackOverflow answer explaining the need to say "module=__name__".  But the damage is done, especially if the person encountering the pickling error is not the original author of the code defining the enum.  (Often the person downloading and using a package from PyPI has less advanced Python knowledge than the package author, so they may have a hard time debugging the situation.)

You can see how having the getframe hack in place makes life more pleasant for many people -- the package user won't have to debug the pickling problem, and the package author won't have to deal with the bug report and fix.

But what about other Python implementations?  Well, TBH, they have plenty of other issues.  The reality is that you can't take a large package that hasn't been tested on Jython, or IronPython, or PyPy, and expect it to just work on any of those.  Sure, things are getting better.  But there are still tons of differences between the various Python implementations (as there are between different versions of CPython), and whether you like it or not, CPython is still the Python version of choice for most people.  The long and short of it is that porting any significant package to another implementation is a bit of work, and keeping the port working probably requires adding some set of additional line items to the style guide used by its developers -- don't use feature X, don't depend on behavior Y, always use pattern Z...


However, I don't expect that "always pass module=__name__ when using the enum convenience API" won't have to be added to that list.  sys._getframe() is way more powerful than what's needed.  Even on platforms where sys._getframe() is unimplementable (or disabled by default in favor of a 10% speedup), it should still be possible to provide an API that *just* gets the module name of the caller, at least in case the call site is top-level module code (and for anything else, the getframe hack doesn't work anyway).  After all, we're talking about a full Python implementation, right?  That's a dynamic language with lots of introspection APIs, and any implementation worth its salt will have to deal with that, even if full-fledged sys._getframe() is explicitly excluded.

So I propose that we use sys._getframe() for the time being, and the authors of Jython, IronPython and PyPy can get together and figure out how to implement something like sys.get_calling_module_name().  They have plenty of time -- at least until they pledge support for Python 3.4.  To make it easy for them we should probably add that API to CPython 3.4.  And it's fine with me if the function only works if the caller is top-level code in a module.

Which reminds me.  Nick offered another use case where using sys._getframe() breaks down: a wrapper function that constructs an enum for its caller.  First of all, I think this is a pretty rare use case.  But additionally, that wrapper could just use sys.get_calling_module_name(), and everything would be fine.

PS. Whoever adds sys.get_calling_module_name() to CPython, please pick a shorter name. :-)
msg189164 - (view) Author: Alex Gaynor (alex) * (Python committer) Date: 2013-05-13 18:40
From PyPy's perspective we don't really care what you name this particular bikeshed, and it's probably not that important to us (in this particular case).

As far as I know IronPython is the only Python VM that doesn't have _getframe() support by default (you need a CLI flag at startup, IIRC). In PyPy calling _getframe (right now) has a negative performance impact, but it's local to wherever you're calling it, so unless you create Enum in frequent loops (as opposed to just once at module level), this doesn't really matter to us.

If someone wants to invent a more narrow, tailored API for stack introspection, logging is a much better target (right now each call to logging.{info,error,etc.} calls _getframe(), which does have a real performance impact. (We'll eventually invent enough engineering to fix this ourselves, but in the meantime if somewhere were to pain this bikeshed, we certainly wouldn't object ;)).
msg189175 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-05-13 20:17
I've come across something in the implementation here that I'd like some clarification on.  What is the purpose of overriding __dir__ in Enum and EnumMeta?  It doesn't change any behavior that I'm aware of, just makes things look a little nicer when someone calls dir() on their Enum.  And, in fact, it can make things a little confusing.  For example:

>>> class Test(enum.Enum):
...     foo = 1
...     bar = 2
...     baz = 3
...
>>> dir(Test)
['__class__', '__doc__', '__members__', 'bar', 'baz', 'foo']
>>> Test.mro
<built-in method mro of EnumMeta object at 0x01D94D20>

This brings up another interesting case:

>>> class Test2(enum.Enum):
...     mro = 1
...     _create = 2
...
>>> dir(Test2)
['__class__', '__doc__', '__members__', '_create', 'mro']
>>> Test2.__members__
mappingproxy(OrderedDict([('mro', <Test2.mro: 1>), ('_create', <Test2._create: 2>)]))
>>> Test2['mro']
<Test2.mro: 1>
>>> Test2.mro
<built-in method mro of EnumMeta object at 0x01D90210>
>>> Test2._create
<bound method type._create of <class 'enum.EnumMeta'>>
>>>

From using "mro" or "_create", I would have expected either ValueError or for them to work properly.  I don't know whether this should be fixed (one way or the other), documented, or just left alone; those kind of names really shouldn't ever be used anyway.  It's something I stumbled across, though, and I just wanted to make sure that those who do have opinions that matter are aware of it :)
msg189188 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-05-13 23:21
Thanks Guido, now that I fully understand your reasoning, I can accept that
this is a valid "practicality beats purity" situation.
msg189263 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2013-05-15 02:28
Got the pickle issues worked out.  Added super to the metaclass' __new__.  Checking for illegal names of members and raising ValueError if any are found (I know, I know, safety checks!  But such an enum is broken from the getgo so I see no reason to allow those names through).

Thanks everyone for the excellent feed back.  I really appreciate it.  Hopefully we're almost done!  :)
msg189446 - (view) Author: Donald Stufft (dstufft) * (Python committer) Date: 2013-05-17 11:19
Small nitpick, weakref is imported but not used in the latest patch.
msg190033 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-05-25 23:06
Ethan, just a reminder to write that documentation...

It's basically a stripped down version of PEP 435 (leave all the philosophy and history out), with a few concrete "reference" sections explaining the API precisely.
msg190095 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-05-26 16:04
I tweaked the code a bit (no functionality changes, mostly cleanups and a bit of refactoring). Figured it will be easier to just send an updated patch than another review. The diff from patch 06 can be seen via the Rietveld interface.

Also, after reading the code more carefully, I think we're doing a mistake by over-complicating it for the sake of custom enum metaclasses and over-customization (like auto numbering). The original point Guido raised against auto-numbering was too much magic in the implementation. Well, we already have that in Lib/enum.py - the code is so complex it seems fragile because of the tight coupling with many class and metaclass related protocols. Just defining a wholly new enum implementation that does something very specific seems simpler than customizing the existing one.

I'd suggest we stick to the existing Enum + IntEnum, giving up the more complex customizations for now. It can always be added in the future if we see it's very important.
msg190099 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2013-05-26 16:52
Wow.  I definitely felt like an apprentice after reading the changes.  Thanks, Eli, that looks worlds better!
msg190115 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-05-26 20:56
Thanks Ethan :)

From my point of view this is LGTM, as long as:

* There's ReST documentation
* You remove the code to support extensions and customizations not mandated by PEP 435. As I mentioned before, this seems to be a YAGNI that complicates the code needlessly. It's find to keep the changes in some external repo and at a later point discuss their gradual addition similarly to the way Nick has been pushing enhancements through additional issues.

We can always add more capabilities to Enum. We can "never" take them away once added, and this complicated code will remain with us forever even if no one ends up using it.
msg190118 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-05-26 22:31
Supporting extensions was one of the things that got Ethan's version
through review. So -1 on going back on our promise to support those
variants. They have been reviewed and tested just as thoroughly as the rest
of the design.
msg190119 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-05-26 22:33
Also, we know for a fact that people plan to use the customisation features
- it was making their code work that drove the current extension design.
msg190122 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-05-27 01:12
I'm not sure which promises you're referring to Nick, and to whom they were made; the only formal promise we made is PEP 435 - and it doesn't mention this extensibility.

I won't argue beyond this comment, since I know I'm part of the minority opinion here. However, I still think this is a mistake.

The most important original goal of Enum (as discussed during the language summit) was to replace all the custom enum implementations by one that is standard. A far fledged extension mechanism will just make it so we'll have a fleet of bastardized "extended enums", each with its own capabilities, each different from the others. With one standard Enum, when you're reading someone's code and you see:

class Foo(Enum):
  ...

You know very well what Foo is. Restricted extensions like IntEnum and even your @enum.unique are still tolerable because they're explicit:

# enum.unique is standard and says what it is explicitly
@enum.unique
class Foo(Enum):
  ...

But if we open the gates on customization, we'll have:

class Foo(AutoEnum):
  Red, White, Black

And:

class Bar(SomeOtherAutoEnum):
  Red = ...
  White = ...
  Black = ...

And:

class Baz(SomeEvenOtherMagicEnum):
  ... # whatever goes here

And we're back to square 1, because these Enums are not standard, and each framework will have its own clever customization one will need to understand in order to read code with Enums.

Exposing and documenting the metaclass and customizations of __new__ is a whole coffin for the "there is only one way to do it" decision of stdlib's Enum. It might have been better to just define AutoNumberedEnum, BitfieldEnum and Magic42Enum as part of the enum package in stdlib and be over with it; but this was strongly rejected by others and particularly Guido during the summit and later. Now we're just creating a back-door to get into the same situation.
msg190123 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2013-05-27 01:40
Eli, what's wrong with having a backdoor?  Python is literally *full* of backdoors.  I have a feeling that somehow you are trying to build an Enum class that is unpythonic in its desire to enforce some kind of "ideal enum" behavior.
msg190126 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-05-27 02:32
Guido, IMHO back-doors are fine in many cases, just not this one. The way I see it, our main goal here is to collect a bunch of custom implementations of enums under a single umbrella. This is not very different from what was done with OrderedDict and namedtuple at some point. There were probably a bunch of custom implementations, along with more and less commonly used recipes. At some point a single implementation was added to the stdlib, without (AFAICS) major back-doors.

Yes, the Enum case is vastly more complex than either OrderedDict or namedtuple, and there is a multitude of different behaviors that can be anticipated (as the lengthy discussions leading to the acceptance of PEP 435 demonstrated). And yet, I was also hoping to have a single canonical implementation, so that people eventually accept it as "the one". Stdlib modules tend to win over in the long run.

The other point is that I think the implementation could be much simpler without having these back doors. As it stands now, the code is complex and hence brittle. Any change will be difficult to do because we're locked down very strictly by a set of intrusive and deep, yet externally "promised" interfaces. The same can be said, again, about OrderedDict and namedtuple, the code of which is very straightforward.

Maybe I'm blowing this out of proportions, maybe not. I'm not sure. As I said, I don't want to strongly argue about this. If both you and Nick are OK with keeping the customization mechanisms in, I defer to your judgment.
msg190130 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-05-27 04:06
Eli, remember that TOOWTDI stands for "There's one *obvious* way to do it" rather than "There's *only* one way to do it". The latter interpretation leads to insanely complex APIs that attempt to solve everyone's problems, while the former favours 80% solutions that cover most use cases, with extension hooks that let people handle the other 20% as they see fit.

The point of the enum standardisation is to have a conventional way that enums *behave*, and then allow variations on that theme for those cases where the stdlib implementation is "close, but not quite what I need or want".

The whole metaclass machinery is built around this concept of letting people create domain specific behaviour, that is still somewhat unified due to conventions like the descriptor protocol. You can do a *lot* with just descriptors, so if you don't need a custom metaclass, you shouldn't use one.

PEP 422's class initialisation hook is aimed specifically at certain cases that currently need a metaclass and providing a simpler way to do them that lets you just use "type" as the metaclass instead.

It's the same with enums - if you don't need to customise the metaclass, you shouldn't. But there are some use cases (such as syncing the Python level enum definition with a database level one) where additional customisation will be needed. We also want to give people the freedom they need to experiment with different forms of definition time syntactic sugar to see if they can come up with one we like enough to add to the standard library in 3.5.

Does documenting these definition time extension points constrain what we're allowed to do in the future? Yes, it does. But, at the same time, it takes a lot of pressure off us to add more features to the standard enum type over time - if people have niche use cases that aren't handled well by the standard solution (and we already know they do), we can point them at the supported extension interface and say "go for it". For the majority of users though, the standard enum type will work fine, just as ordinary classes are adequate for the vast majority of object oriented code.
msg190131 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-05-27 04:10
Somewhat related, I *know* you've read type.__new__. Compared to that, enum.EnumMeta.__new__ is still pretty straightforward ;)
msg190312 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2013-05-29 15:12
Working on documentation...
msg190795 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2013-06-08 08:36
Hopefully the final bit of code, plus docs.

Code changes:  

  _names_ are reserved

  
Doc changes (different from the PEP):

  examples of AutoEnum, UniqueEnum, and OrderedEnum
msg190796 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2013-06-08 08:41
Apologies for the noise -- was having trouble getting the correct patch attached.  :/
msg190803 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2013-06-08 15:55
So, which is better?

To have a @unique class decorator as part of the module, or to have a UniqueEnum recipe in the docs?

A decorator is immediately usable, but requires remembering an extra line of code.

An example requires being put into a local utility module, but also serves as a guide to customising Enums.  (Took be about five tries to get it right. :/ )
msg190804 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-06-08 16:00
Nick prudently moved the unique discussion to its own issue - 18042. Let's get the initial implementation & docs committed first (without unique in the implementation, although it's fine to have it as an example in the docs for now), close this issue, and then discuss in 18042 whether unique should be part of the stdlib-provided API or not.
msg190805 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2013-06-08 16:05
Good idea, thanks.
msg190806 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-06-08 16:20
I sent a fresh review - nothing major; it's very near commit readiness now. Additional changes can be done after the initial commit. We have time until 3.4 beta (November 2013) to tweak stuff (and the documentation whenever...)
msg190842 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2013-06-09 04:45
Doc updates are in.

I removed the 'unique, constant' from the first line of the intro, as neither of those things are necessarily true.
msg190844 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2013-06-09 05:25
Hmm -- I was confusing member names with member values;  I'll put 'unique' back in.
msg190847 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2013-06-09 06:20
Hopefully the last update.  :)
msg190862 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-06-09 16:26
LGTM.

I suggest you wait for a couple of days to see if others have any critical comments and then commit.
msg190870 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-06-09 19:06
enum.rst will need to be added to a table of contents page somewhere, I would guess possibly Development Tools (Doc/library/development.rst) or maybe Data Types (Doc/library/datatypes.rst).  I would trust almost anybody else's opinion over mine on where it should go, though :)
msg190980 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2013-06-11 18:40
Final (hopefully! ;) patch.

I stuck the toc reference in datatypes.

If no more edits are necessary I'll commit on Friday (three days from now).
msg191103 - (view) Author: Roundup Robot (python-dev) Date: 2013-06-14 07:31
New changeset fae92309c3be by Ethan Furman in branch 'default':
Closes issue 17947.  Adds PEP-0435 (Enum, IntEnum) to the stdlib.
http://hg.python.org/cpython/rev/fae92309c3be
msg191104 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-06-14 07:44
That commit looks just a touch incomplete...
msg191133 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-06-14 14:29
Ethan, did you forget to "hg add" ?

On Fri, Jun 14, 2013 at 12:44 AM, Nick Coghlan <report@bugs.python.org>wrote:

>
> Nick Coghlan added the comment:
>
> That commit looks just a touch incomplete...
>
> ----------
> resolution: fixed ->
> stage: committed/rejected -> commit review
> status: closed -> open
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue17947>
> _______________________________________
>
msg191137 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2013-06-14 15:56
Well, that made me laugh first thing in the morning!

I had nuked and redone my clone, and yeah, forgot to re-add the files.  :/

Trying again...

Commit message was okay?
msg191172 - (view) Author: Roundup Robot (python-dev) Date: 2013-06-14 23:56
New changeset e7a01c7f69fe by Ethan Furman in branch 'default':
Closes issue 17947.  Adds PEP-0435 (Adding an Enum type to the Python standard library).
http://hg.python.org/cpython/rev/e7a01c7f69fe
msg191173 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-06-14 23:57
> New changeset e7a01c7f69fe by Ethan Furman in branch 'default':
> Closes issue 17947.  Adds PEP-0435 (Adding an Enum type to the Python standard library).
> http://hg.python.org/cpython/rev/e7a01c7f69fe

Great job :-)
msg191181 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-06-15 00:57
Nicely done - you can also mark the PEP as Final now :)
History
Date User Action Args
2013-06-15 00:57:57ncoghlansetmessages: + msg191181
2013-06-14 23:57:42hayposetnosy: + haypo
messages: + msg191173
2013-06-14 23:56:10python-devsetstatus: open -> closed
resolution: fixed
messages: + msg191172

stage: commit review -> resolved
2013-06-14 15:56:15ethan.furmansetmessages: + msg191137
2013-06-14 14:29:19eli.benderskysetmessages: + msg191133
2013-06-14 07:44:40ncoghlansetstatus: closed -> open
resolution: fixed -> (no value)
messages: + msg191104

stage: resolved -> commit review
2013-06-14 07:31:13python-devsetstatus: open -> closed
resolution: fixed
messages: + msg191103

stage: patch review -> resolved
2013-06-11 18:40:20ethan.furmansetfiles: + pep-0435.12.stoneleaf.patch

messages: + msg190980
2013-06-09 19:06:51zach.waresetmessages: + msg190870
2013-06-09 16:26:50eli.benderskysetmessages: + msg190862
2013-06-09 06:20:42ethan.furmansetfiles: + pep-0435.11.stoneleaf.patch

messages: + msg190847
2013-06-09 05:25:16ethan.furmansetmessages: + msg190844
2013-06-09 04:46:00ethan.furmansetfiles: + pep-0435.10.stoneleaf.patch

messages: + msg190842
2013-06-08 16:20:11eli.benderskysetmessages: + msg190806
2013-06-08 16:05:17ethan.furmansetmessages: + msg190805
2013-06-08 16:00:19eli.benderskysetmessages: + msg190804
2013-06-08 15:55:42ethan.furmansetmessages: + msg190803
2013-06-08 08:41:08ethan.furmansetfiles: + pep-0435.09.stoneleaf.patch

messages: + msg190796
2013-06-08 08:39:34ethan.furmansetfiles: - pep-0435.08.stoneleaf.patch
2013-06-08 08:38:44ethan.furmansetfiles: + pep-0435.08.stoneleaf.patch
2013-06-08 08:38:01ethan.furmansetfiles: - pep-0435.08.stoneleaf.patch
2013-06-08 08:36:13ethan.furmansetfiles: + pep-0435.08.stoneleaf.patch

messages: + msg190795
2013-05-29 15:12:06ethan.furmansetmessages: + msg190312
2013-05-27 04:10:05ncoghlansetmessages: + msg190131
2013-05-27 04:06:11ncoghlansetmessages: + msg190130
2013-05-27 02:32:40eli.benderskysetmessages: + msg190126
2013-05-27 01:40:44gvanrossumsetmessages: + msg190123
2013-05-27 01:12:50eli.benderskysetmessages: + msg190122
2013-05-26 22:33:16ncoghlansetmessages: + msg190119
2013-05-26 22:31:41ncoghlansetmessages: + msg190118
2013-05-26 20:56:09eli.benderskysetmessages: + msg190115
2013-05-26 16:52:27ethan.furmansetmessages: + msg190099
2013-05-26 16:04:20eli.benderskysetfiles: + pep-0435.07.eliben.patch

messages: + msg190095
2013-05-25 23:06:04eli.benderskysetmessages: + msg190033
2013-05-23 08:56:55ncoghlanlinkissue18042 dependencies
2013-05-17 11:19:39dstufftsetmessages: + msg189446
2013-05-17 10:49:03dstufftsetnosy: + dstufft
2013-05-15 02:28:27ethan.furmansetfiles: + pep-0435.06.patch
assignee: ethan.furman
messages: + msg189263
2013-05-13 23:21:25ncoghlansetmessages: + msg189188
2013-05-13 20:17:26zach.waresetmessages: + msg189175
2013-05-13 18:40:23alexsetmessages: + msg189164
2013-05-13 18:32:50gvanrossumsetmessages: + msg189160
2013-05-13 17:05:36pconnellsetnosy: + pconnell, isoschiz
2013-05-13 00:06:17ncoghlanunlinkissue17959 dependencies
2013-05-12 14:24:59ethan.furmansetfiles: + pep-0435.05.patch

messages: + msg189036
2013-05-12 13:32:06ncoghlanunlinkissue17954 dependencies
2013-05-12 13:09:36ncoghlansetmessages: + msg189029
2013-05-12 12:54:50eli.benderskysetmessages: + msg189027
2013-05-12 12:17:46eli.benderskysetmessages: + msg189021
2013-05-12 11:24:44ethan.furmansetmessages: + msg189020
2013-05-12 10:59:53ncoghlansetmessages: + msg189014
2013-05-12 10:57:35ncoghlanlinkissue17961 dependencies
2013-05-12 06:52:47ncoghlansetmessages: + msg188982
2013-05-12 06:51:58ncoghlanlinkissue17959 dependencies
2013-05-12 06:25:55ncoghlansetmessages: + msg188979
2013-05-12 05:39:20ethan.furmansetmessages: + msg188977
2013-05-12 05:25:20ethan.furmansetmessages: + msg188975
2013-05-11 21:41:32benjamin.petersonsetmessages: - msg188956
2013-05-11 21:41:25benjamin.petersonsetmessages: - msg188955
2013-05-11 21:41:08benjamin.petersonsetmessages: - msg188954
2013-05-11 21:38:04eli.benderskysetmessages: + msg188956
2013-05-11 21:24:25benjamin.petersonsetnosy: + benjamin.peterson
messages: + msg188955
2013-05-11 21:11:04python-devsetnosy: + python-dev
messages: + msg188954
2013-05-11 19:11:30eli.benderskysetmessages: + msg188948
2013-05-11 19:10:46eli.benderskysetmessages: + msg188947
2013-05-11 18:38:20ncoghlansetmessages: + msg188941
2013-05-11 17:10:42gvanrossumsetnosy: + gvanrossum
messages: + msg188933
2013-05-11 13:58:44ncoghlansetmessages: + msg188921
2013-05-11 13:55:45ncoghlanlinkissue17954 dependencies
2013-05-11 03:28:32eli.benderskysetmessages: + msg188895
2013-05-11 03:26:11eli.benderskysetmessages: + msg188894
2013-05-11 03:09:06ncoghlansetnosy: + ncoghlan
messages: + msg188893
2013-05-11 03:03:35eli.benderskysetmessages: + msg188891
2013-05-10 21:04:08ethan.furmansetfiles: + pep-0435.04.patch

messages: + msg188881
2013-05-10 17:54:59ethan.furmansetfiles: + pep-0435.03.patch

messages: + msg188856
2013-05-10 15:33:11zach.waresetnosy: + zach.ware
2013-05-10 15:24:58ethan.furmansethgrepos: - hgrepo189
2013-05-10 15:21:23ethan.furmansetfiles: + pep-0435.02.patch

messages: + msg188841
2013-05-10 14:42:08alexsetnosy: + alex
2013-05-10 14:05:21eli.benderskysetmessages: + msg188839
2013-05-10 14:03:17eli.benderskysetmessages: + msg188837
2013-05-10 13:21:38eli.benderskysetassignee: docs@python -> (no value)
messages: + msg188833
2013-05-10 13:19:21eli.benderskysetfiles: - ref435.py
2013-05-10 07:09:03ethan.furmansetfiles: + pep-0435.01.patch
keywords: + patch
messages: + msg188815
2013-05-10 07:01:10ezio.melottisetmessages: + msg188814
2013-05-10 06:58:11ezio.melottisetnosy: + ezio.melotti

type: enhancement
stage: patch review
2013-05-10 06:50:00ethan.furmancreate