Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add AutoNumberedEnum to stdlib #71175

Closed
johnthagen mannequin opened this issue May 10, 2016 · 62 comments
Closed

Add AutoNumberedEnum to stdlib #71175

johnthagen mannequin opened this issue May 10, 2016 · 62 comments
Assignees
Labels
stdlib Python modules in the Lib dir type-feature A feature request or enhancement

Comments

@johnthagen
Copy link
Mannequin

johnthagen mannequin commented May 10, 2016

BPO 26988
Nosy @warsaw, @rhettinger, @vstinner, @ethanfurman, @vedgar, @Vgr255, @johnthagen, @kennethreitz
Files
  • issue26988.stoneleaf.01.patch
  • issue26988.stoneleaf.02.patch
  • issue26988.stoneleaf.03.patch
  • issue26988.stoneleaf.05.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/ethanfurman'
    closed_at = <Date 2016-08-20.07:02:02.617>
    created_at = <Date 2016-05-10.01:34:31.690>
    labels = ['type-feature', 'library']
    title = 'Add AutoNumberedEnum to stdlib'
    updated_at = <Date 2016-08-20.07:02:02.616>
    user = 'https://github.com/johnthagen'

    bugs.python.org fields:

    activity = <Date 2016-08-20.07:02:02.616>
    actor = 'ethan.furman'
    assignee = 'ethan.furman'
    closed = True
    closed_date = <Date 2016-08-20.07:02:02.617>
    closer = 'ethan.furman'
    components = ['Library (Lib)']
    creation = <Date 2016-05-10.01:34:31.690>
    creator = 'John Hagen'
    dependencies = []
    files = ['43694', '43743', '43972', '43986']
    hgrepos = []
    issue_num = 26988
    keywords = ['patch']
    message_count = 62.0
    messages = ['265214', '265215', '265216', '265218', '265237', '269476', '269521', '270039', '270048', '270049', '270051', '270054', '270056', '270071', '270088', '270107', '270117', '270137', '270142', '270148', '270151', '270199', '270201', '270212', '270219', '270238', '270523', '270524', '271811', '271855', '271868', '272071', '272874', '272876', '272885', '272901', '272932', '272941', '272951', '272952', '272971', '272982', '273006', '273039', '273040', '273064', '273065', '273076', '273084', '273092', '273103', '273105', '273109', '273118', '273125', '273126', '273127', '273131', '273134', '273145', '273154', '273188']
    nosy_count = 11.0
    nosy_names = ['barry', 'rhettinger', 'vstinner', 'eli.bendersky', 'ethan.furman', 'python-dev', 'veky', 'abarry', 'John Hagen', 'David Hagen', 'kennethreitz']
    pr_nums = []
    priority = 'normal'
    resolution = 'rejected'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue26988'
    versions = ['Python 3.6']

    @johnthagen
    Copy link
    Mannequin Author

    johnthagen mannequin commented May 10, 2016

    I suggest that the AutoNumberedEnum be added to the standard library for the following reasons:

    1. Provides a fundamental tool for defining a unique, abstract set of coupled members
    2. Avoids boilerplate @enum.unique for a very common use case of enumerations
    3. The code already exists in the Python documentation, so it has been vetted at some level

    The AutoNumberedEnum also allows the developer to make a clearer distinction
    between enumerations whose values have special meaning and those that do not.

    Consider:

    @enum.unique
    class Color(enum.Enum):
        red = 1
        blue = 2
        green = 3
    
    @enum.unique
    class Shape(enum.Enum):
        """Member values denote number of sides."""
        circle = 1
        triangle = 3
        square = 4

    With AutoNumberedEnum it's possible to better express the intent that
    the value of Color members does not hold special meaning, while
    Shape members do:

    class Color(enum.AutoNumberedEnum):
        red = ()
        blue = ()
        green = ()
    
    @enum.unique
    class Shape(enum.Enum):
        """Member values denote number of sides."""
        circle = 1
        triangle = 3
        square = 4

    For enumerations with many members (10s), there becomes a maintenance
    issue when inserting new enumerations into the list:

    @enum.unique
    class Color(enum.Enum):
        aquamarine = 1
        blue = 2
        fushia = 3
        # inserting a member here (perhaps because it's clearest to keep these in alphabetic order)
        # results in having to increment all following members
        ...
        green = 40
        red = 41

    Most other languages have support for naming enumerations
    without explicitly declaring their values (albeit with
    specialized syntax that makes it cleaner):

    C++: http://en.cppreference.com/w/cpp/language/enum
    C#: https://msdn.microsoft.com/en-us/library/sbbt4032.aspx
    Java: https://docs.oracle.com/javase/tutorial/java/javaOO/enum.html
    Rust: https://doc.rust-lang.org/book/enums.html

    Currently, a developer can copy the code from the Python docs into
    his or her project, or add a dependency on aenum. I would argue
    that it belongs in the standard library.

    If there are objections to it being too magical, I would argue
    it's already spelled out in the docs, so it's being prescribed
    as a solution already. I think it should be very clear when you
    derive from "AutoNumberedEnum" what is going on.

    The code is very simple, so I would
    hope maintenance would not be difficult.

    @johnthagen johnthagen mannequin added stdlib Python modules in the Lib dir type-feature A feature request or enhancement labels May 10, 2016
    @ethanfurman
    Copy link
    Member

    On the other hand, if you use aenum you can do:

    class Color(AutoNumber):
        red
        green
        blue

    And isn't that better*? ;)

    • For those in danger of swallowing their tongue over the magic involved: The magic is turned off as soon as any descriptor is defined (property, function, etc.).

    @warsaw
    Copy link
    Member

    warsaw commented May 10, 2016

    Wow Ethan, that's quite interesting. I'm not sure if I like it or not. :)

    It's better than having to assign a dummy value to the enum items, and if we did require that, I'd suggest just throwing away the values for autonumbering.

    But not having to specify any values at all is probably better, although it does look weird!

    /me goes to look at the magic you're using...

    @ethanfurman
    Copy link
    Member

    It's in _EnumDict.__getitem__; there's some duplication in __setitem__ for supporting Python 2 (although with 2 you have to use either the empty tuple, or some other handy thing that may go in the __doc__ attribute, for example).

    @johnthagen
    Copy link
    Mannequin Author

    johnthagen mannequin commented May 10, 2016

    @Ethan, I hadn't tried to use the aenum AutoNumberEnum that way, but I agree with Barry that I like it. To me that is the ideal case we should shoot for as I think it's the best* long term and deviate only if practical concerns prevent it.

    So I am +1 for empty member assignment and if that is rejected, +1 for = () assignment as at least it is a big step forward. I feel both solutions already have some "magic", so would lean toward the one that leads to the least amount of boilerplate.

    As for the empty assignment, I have played around with something similar before and will throw out one con for it: static analyzers get really confused. PyCharm, for example, thinks this is has unresolved references in it:

    class Color(AutoNumberEnum):
        red
        green
        blue

    But the counter point is that if this is in the stdlib, static analyzer authors are much more likely to add a special case for it than if in a "non-official" third party package (PyCharm example: https://youtrack.jetbrains.com/issue/PY-19150).

    Another piece of evidence to support inclusion is that Python already provides specialized Enum subclasses (like IntEnum). I've written a lot of Python code that uses Enums and haven't personally needed IntEnum yet, but would have used an AutoEnum many, many times.

    • I am assuming here that a true "enum" keyword is out of the question at this point for Python, which would probably be even better.

    @johnthagen
    Copy link
    Mannequin Author

    johnthagen mannequin commented Jun 29, 2016

    @Ethan/Barry what needs to be done now to accept or reject this for Python 3.6? Should I propose it onto python-dev? If accepted, would be nice to get it in before alpha 3 (~2 weeks). What's nice about this proposal is Ethan has already written the code, so it's just a matter of getting consensus to add it.

    @warsaw
    Copy link
    Member

    warsaw commented Jun 29, 2016

    I don't really think much more needs to be done other than agree on this tracker issue that it's something we want, and that Ethan's implementation is good enough. It's been a while since I looked that the code but it seemed good to me before, so if Ethan is happy with it going into the stdlib, so am I. As for the feature, it *is* a little magical, but not so much that it would be confusing or ambiguous, so I'm +1 on it.

    I'll defer to Ethan for the final decision, but if we have tests and documentation, I'd be very happy to see this in 3.6.

    @johnthagen
    Copy link
    Mannequin Author

    johnthagen mannequin commented Jul 9, 2016

    Is this something we want to get in before the next alpha in two days? Just wanted to bring up the deadline since this may be a feature people want to play around with during the alpha phase.

    Ethan, I'm happy to help with documentation or anything else.

    @ethanfurman
    Copy link
    Member

    There is one wrinkle with auto-numbering. Consider the following code:

    class Color(AutoNumberEnum):
      red
      green
      blue
      @property
      def cap_name(self):
        return self.name.upper()

    What happens with property?

    • property is looked up in the class namespace
    • it's not found, so is given the number 4
    • it's then called to handle cap_name
    • an exception is raised

    As far as I can tell there is only one "good" way around this:

    • store any names you don't want auto-numbered into an _ignore_
      attribute (this only needs to be global and built-in names that
      are used before the first method is defined)

    another option is to build a proxy around any found global/built-in objects and decide what to do based on whether those objects are immediately called, but that fails when the object is simply assigned for later use.

    So, barring any other ideas to handle this problem, the above example should look like this:

    class Color(AutoNumberEnum):
      _ignore_ = 'property'
      red
      green
      blue
      @property
      def cap_name(self):
        return self.name.upper()

    Another advantage of using ignore is the ability to have temporary variables automatically discarded:

      class Period(timedelta, Enum):
        '''
        different lengths of time
        '''
        _ignore_ = 'Period i'
        Period = vars()
        for i in range(31):
          Period['day_%d' % i] = i, 'day'
        for i in range(15):
          Period['week_%d' % i] = i*7, 'week'
        for i in range(12):
          Period['month_%d' % i] = i*30, 'month'
        OneDay = day_1
        OneWeek = week_1
        def __new__(self, value, period):
          ...

    and the final enumeration does not have the temp variables Period nor i.

    Thoughts?

    @warsaw
    Copy link
    Member

    warsaw commented Jul 9, 2016

    On Jul 09, 2016, at 03:20 PM, Ethan Furman wrote:

    As far as I can tell there is only one "good" way around this:

    • store any names you don't want auto-numbered into an _ignore_
      attribute (this only needs to be global and built-in names that
      are used before the first method is defined)

    Seems reasonable, though why not a dunder, e.g. __ignore__?

    _ignore_ = 'Period i'

    Why space separated names inside a string instead of a sequence of strings?

    @ethanfurman
    Copy link
    Member

    A standard feature of Enum is that either space separated strings or a list of strings is accepted any where either is.

    _sunder_ names are the ones reserved for Enum use (such as _value_, _name_, and soon _order_ (for py2/py3 compatible code).

    @johnthagen
    Copy link
    Mannequin Author

    johnthagen mannequin commented Jul 9, 2016

    What happens with property?

    • property is looked up in the class namespace

    Perhaps you've already considered this, I'm not intimately familiar with how classes are parsed and constructed but is it possible to determine if the object is a decorator? It already determines to stop auto-numbering when it hits the first method, could it stop when it hits the first decorator or method?

    Being able to use temporaries is an interesting side effect, but I feel like that would be used less often than a @staticmethod, @Property, or @classmethod over a method, in which case it becomes a little more complex.

    That being said, I think either solution is valid.

    @ethanfurman
    Copy link
    Member

    The problem with testing the type of object a name refers to outside the class is it then becomes more difficult to make that an Enum member:

    class AddressType(Enum):
        pobox
        mailbox  # third-party po box
        property

    Having to assign a value to property pretty much negates the value of the magic.

    I'll go with _ignore_.

    @ethanfurman
    Copy link
    Member

    I need names. aenum already has an AutoNumberEnum (the one from the docs, no magic) so I hate to use the same name for the stdlib version with different behavior.

    So I either need a different name for the stdlib version, or a different name for the aenum version.

    Any ideas?

    Hmmm... maybe SequentialEnum for the aenum version...

    @johnthagen
    Copy link
    Mannequin Author

    johnthagen mannequin commented Jul 10, 2016

    Some ideas for the new stdlib class:

    BasicEnum - This helps emphasize that it is a simpler version of Enum that doesn't support all of the Enum features (like assigning values). It also helps communicate that if you don't need values this is a better fit.

    AutoEnum - This new version (compared with AutoNumberEnum in the docs) does more than just auto number, since it does even the value assignment. Auto helps communicate that this is automatically creating much of the class internals for you.

    @ethanfurman
    Copy link
    Member

    I like AutoEnum.

    Another auto-related thought: one of the more common Enum questions on StackOverflow is how to automatically have the value be the stringified version of the name:

    class Huh(Enum):
      this
      that
      those

    Huh.this.name == Huh.this.value
    # True

    So the question then becomes: is there a way to easily support both auto-number and auto-string values?

    While I don't have the auto-string feature yet in aenum, the system I am using to specify optional settings looks like this:

    ------

    class Color(Enum, settings=AutoNumber):
      red
      ...
    ------
    
    ------
    class Color(IntEnum, settings=AutoNumber):
      red
      ...
    ------
    
    ------
    class Color(Enum, settings=AutoName):
      red
      ...

    The other option, of course, is to just stick with the prebuilt method of doing things:

    class Color(AutoEnum):
      ...
    
    class Color(AutoEnum, IntEnum):
      ...
    
    class Color(AutoNameEnum):
      ...

    @warsaw
    Copy link
    Member

    warsaw commented Jul 10, 2016

    On Jul 10, 2016, at 05:42 PM, Ethan Furman wrote:

    class Color(Enum, settings=AutoNumber):
    [...]
    class Color(Enum, settings=AutoName):

    I guess settings would take an AutoType enum. But that can't also be
    autonumbered or it would be autos all the way down. :)

    @johnthagen
    Copy link
    Mannequin Author

    johnthagen mannequin commented Jul 10, 2016

    To me, class Color(AutoEnum) and class Color(AutoEnum, IntEnum) is a little more straightforward. It makes usage of AutoEnum similar to IntEnum, and I would expect it to be at least as popular as it.

    A enum-related side question, since the plan is for this to go into the stdlib, would it also go into enum34 since that is the official back port of the stdlib Enum?

    @ethanfurman
    Copy link
    Member

    Yeah, I think the public interface will just be the AutoEnum and AutoNameEnum style.

    ---

    Will these features go into enum34?

    Not sure. At this point I have the stdlib enum, enum34 enum, and aenum enum.

    In terms of capability, aenum is the most advanced, followed by the stdlib enum, and finally enum34 (really the only difference between stdlib and enum34 is the automatic definition order).

    The only advantage of enum34 over aenum is if it works in enum34 it will definitely work in the stdlib, whilst aenum has features not in the stdlib (speaking from a user point of view).

    So I haven't decided, but at this moment I'm not excited about the prospect.
    :(

    What I'll probably do is put enum34 in bug-fix only mode.

    @warsaw
    Copy link
    Member

    warsaw commented Jul 11, 2016

    On Jul 11, 2016, at 12:27 AM, Ethan Furman wrote:

    Not sure. At this point I have the stdlib enum, enum34 enum, and aenum enum.

    In terms of capability, aenum is the most advanced, followed by the stdlib
    enum, and finally enum34 (really the only difference between stdlib and
    enum34 is the automatic definition order).

    The only advantage of enum34 over aenum is if it works in enum34 it will
    definitely work in the stdlib, whilst aenum has features not in the stdlib
    (speaking from a user point of view).

    So I haven't decided, but at this moment I'm not excited about the prospect.
    :(

    What I'll probably do is put enum34 in bug-fix only mode.

    It's been useful to have a standalone version of the stdlib module, and in
    fact, I maintain the enum34 package in Debian. However, we only support that
    for Python 2 since we don't have to worry about any Python 3 versions before
    3.4 (and even there, 3.5 is the default for Stretch and Ubuntu 16.04 LTS).

    We do have reverse dependencies for python-enum34, but given that we *really*
    want people to port to Python 3, I'm not sure I really care too much any more
    about enum34 in Debian.

    @ethanfurman
    Copy link
    Member

    That brings up a really good point -- this feature requires the __prepare__ method of the metaclass, so it won't work in Python 2 any way.

    So, yeah, bug-fix-mostly mode for enum34. :)

    @ethanfurman
    Copy link
    Member

    If you are constructing your own base Enum type, which would you rather do?

    class BaseZeroEnum(Enum):
       "initial integer is 0"
       _start_ = 0
       ...

    or

    class BaseZeroEnum(Enum, start=0):
       "initial integer is 0"
       ...

    ? Oh, and yes if you specify a starting number you also activate the AutoNumber feature.

    @warsaw
    Copy link
    Member

    warsaw commented Jul 11, 2016

    On Jul 11, 2016, at 07:05 PM, Ethan Furman wrote:

    class BaseZeroEnum(Enum, start=0):
    "initial integer is 0"
    ...

    ? Oh, and yes if you specify a starting number you also activate the
    AutoNumber feature.

    I like the way this one looks.

    @johnthagen
    Copy link
    Mannequin Author

    johnthagen mannequin commented Jul 11, 2016

    class BaseZeroEnum(Enum, start=0):
    "initial integer is 0"
    ...

    I also think this looks better.

    @ethanfurman
    Copy link
    Member

    Here's the code. I'll do the doc changes next.

    @johnthagen
    Copy link
    Mannequin Author

    johnthagen mannequin commented Jul 12, 2016

    I like the addition of UniqueEnum. It's the default use case often.

    @ethanfurman
    Copy link
    Member

    Okay, here's an updated patch with the doc changes.

    Once the main patch is committed I'm going to reorganize the docs a bit, but that's later.

    @ethanfurman ethanfurman self-assigned this Jul 16, 2016
    @rhettinger
    Copy link
    Contributor

    The use of a bare identifier as a self-assigning statement is unprecedented in the world of Python. PyFlakes flags it as an error (because prior to now, a bare identifier in a class statement was almost always an error). I suspect this will cause issues with other static analysis and refactoring tools as well:

    --- tmp_enum_example.py \-----
    
        from enum import AutoEnum
    
        class Color(AutoEnum):
            red
            green = 5
            blue
    
        print(list(Color))
    --- bash session \------------
    
        $ py -m pyflakes tmp_enum_example.py
        tmp_enum_example.py:4: undefined name 'red'
        tmp_enum_example.py:6: undefined name 'blue'
        tmp_enum_example.py:11: undefined name 'yellow'

    Also, the usual technique of commenting out blocks with triple quotes introduces unexpected breakage:

    --- tmp_enum_example2.py \-----
    
        from enum import AutoEnum
    
        class Color(AutoEnum):
            red
            green = 5
            blue
            ''' XXX temporarily comment these out
            brown
            orange
            '''
            yellow
    
        print(list(Color))
    --- bash session \------------
    
        $ py -m tmp_enum_example.py
        [<Color.red: 1>, <Color.green: 5>, <Color.blue: 6>, <Color.yellow: 7>]
        /Users/raymond/cpython/python.exe: Error while finding spec for
        'tmp_enum_example.py' (AttributeError: module 'tmp_enum_example'
        has no attribute '__path__')

    I worry that this proposal is worse than just being non-idiomatic Python. In a way, it undermines pillars of the language and conflict everyone's mental model of how the language works. Historically, a bare variable name raised a NameError if undefined and would otherwise act as a expression who's result was discarded. However, as used here, it fiats an attribute into existence and assigns it a value. That to my eyes looks like a new language that isn't Python. This is really weird and undermines my expectations.

    The need for the "ignore" parameter for the "shielded set" is a hint that the patch is working against the grain of the language and is not in harmony with Python as a coherent whole. It is a harbinger of problems to come.

    Lastly, I question whether there is any real problem being solved. You already have "Color = IntEnum('Animal', 'red green blue')" that works well enough, doesn't mess with language norms, that works nicely with triple quotes for multiline entries, and that extends easily to hundreds of constants.

    It seems to me that too much magic and unidiomatic weirdness are being leveled at too small of a problem. Plus we already have one way to do it.

    In teaching people to make effective use of the language, a key learning point is learning the portents of trouble to come and recognizing that that not everything that can be made to work should actually be done.

    Please reconsider whether you really want to open this Pandora's box. Right now, it's not too late. No doubt that you will find some people who like this (it reminds them of C), but you will also find some very experienced developers who are made queasy by the bare identifier transforming from an expression into an assigning statement. This more than an "if you don't like it, don't use it" decision, I think an important and invisible line is being crossed that we will regret.

    P.S. A lesson I learned from maintaining the itertools module is that adding more variants of a single idea tends to make the overall toolkit harder to learn and impairs usability. Users suffer when given too many choices for closely related tasks. The "one way to do it" line in the Zen of Python is there for a reason.

    @johnthagen
    Copy link
    Mannequin Author

    johnthagen mannequin commented Aug 17, 2016

    @Raymond, you raise valid concerns to be sure. Hoping we can work something out.

    @Ethan, what are your thoughts?

    It's not just C that has enums where you can define a unique group of names and omit the values for clarity when they are not significant:

    C++: http://en.cppreference.com/w/cpp/language/enum
    C#: https://msdn.microsoft.com/en-us/library/sbbt4032.aspx
    Java: https://docs.oracle.com/javase/tutorial/java/javaOO/enum.html
    Rust: https://doc.rust-lang.org/book/enums.html

    In my experience this is the most common and simple use case for enums.

    Raymond, what are your thoughts about the version of AutoEnum that requires that a bare tuple be used as the value. It has been in the Python docs since 3.4 and was actually the original request of this issue: https://docs.python.org/library/enum.html#autonumber

    It avoids many of the concerns that you've raised while still providing a way to create Enums in the normal class declaration method users would expect with a minimum amount of boilerplate. Note that normally you want to use @enum.unique with a normal Enum, but AutoEnum also allows you to omit that boilerplate as you can't accidentally alias the values.

    @enum.unique
    class Color(enum.Enum):
        aquamarine = 1
        blue = 2
        fushia = 3
        # inserting a member here (perhaps because it's clearest to keep these in alphabetic order)
        # results in having to increment all following members
        ...
        green = 40
        red = 41

    vs.

    class Color(enum.AutoEnum):
        aquamarine = ()
        blue = ()
        fushia = ()
        # inserting a member here (perhaps because it's clearest to keep these in alphabetic order)
        # results in no refactoring
        ... (30+ more)
        green = ()
        red = ()

    A big advantage of the class based Enums compared to the functional API is that you can clearly document an Enum and its members in way Sphinx can take advantage of and developers would be used to.

    # Assuming tuple assignment version for this example.
    class ClientOperationMode(enum.AutoEnum):
        """Modes of operations of the network client."""
    
        push = ()
        """The client pushes data to the server automatically."""
    
        pull = ()
        """The client pulls commands from the server."""
    
        hybrid = ()
        """The client both pushes data and pulls for commands from the server."""

    Sphinx will document this AutoEnum like a normal class, pulling in the class docstring, and attribute docstrings.

    I don't see an obvious way to do this in the functional API docs: https://docs.python.org/library/enum.html#functional-api

    @Vgr255
    Copy link
    Mannequin

    Vgr255 mannequin commented Aug 17, 2016

    I tend to like all things magic, but the more I think about it, and the less I like it being a part of the standard library. I've had a use for this feature before, and when I did, I cooked my own 12-lines subclass of EnumMeta and _EnumDict. Raymond's points are pretty spot-on, and I also think that this shouldn't go in the stdlib. There's still time.

    While this particular flavour of magic sounds too, well, magic, for Python or even myself, I think a less magical approach can be taken. Namely, something like AutoNumberedEnum which requires values to be empty tuples is being explicit about what you want and not allowing typos in the class definition. Raymond's point about (temporarily) commenting out enum members breaking the order highlights this, and while this approach doesn't solve it, it makes it obvious that there is *something* that changed.

    Another approach, which doesn't exclude the above, is to make EnumMeta and _EnumDict public and documented classes (!), thus allowing people like me to create their own blueberry-flavoured magic enumerations without any impact on the people who don't use my code. Or if I don't feel like reinventing the wheel, I can just pip install the module and use that instead.

    I think that the whole idea of making enums in Python work like they do in C is looking at the problem from the wrong angle. It's true that Python takes some of its design from C, but naively trying to replicate C-like behaviour with C-like syntax doesn't work all the time. How often do beginners think 'x ^ y' means 'x to the power of y', and are then surprised by the behaviour?

    I think this version of Enum raises the barrier to entry for new programmers. Enum is a nice feature, and it helps new and old programmers alike write clean(er) code for various purposes. When a programmer sees this use, they won't think "oh this must call __getitem__ and then assign an automatic value", they'll wonder "why is this code even running?". And then it's up to the Raymonds of this world to explain that Enum uses a metaclass (which, I'm sure, is not a topic they'll want to tackle by the time these programmers reach Enum) and that the mapping overloads __getitem__.

    All in all, this magical approach is just too magical for Python. I understand that magic is fun to have and play with, but the standard libary isn't where you should keep your toys. I use a throwaway repo for all my magic this-is-not-a-good-idea-but-I-wanna-do-it-anyway ideas, and this is where I think such magic goes. It definitely doesn't belong in the standard library, within an arm's reach of the first curious programmer to wander there.

    @vstinner
    Copy link
    Member

    It's not just C that has enums where you can define a unique group of names and omit the values ...

    Yes, Python 3.4 too: Animal = Enum('Animal', 'ant bee cat dog')

    https://docs.python.org/dev/library/enum.html#functional-api

    @vstinner
    Copy link
    Member

    Raymond, what are your thoughts about the version of AutoEnum that requires that a bare tuple be used as the value. It has been in the Python docs since 3.4 and was actually the original request of this issue: https://docs.python.org/library/enum.html#autonumber

    Well, I suggest to keep it as a recipe in the doc ;-)

    @ethanfurman
    Copy link
    Member

    Raymond, I appreciate your review and your poll. I am open to removing AutoEnum, but would like to give it a couple more weeks of review. (I'll post on py-dev.)

    The only point you made that I will firmly refute is the "unexpected breakage": you ran your test script with the -m "run module as script" flag, which is what caused the problem.

    @rhettinger
    Copy link
    Contributor

    you ran your test script with the -m "run module as script" flag

    Right, that was a mistest, so it looks like triple quotes do work.

    I did notice that there's also an issue if one line reads, "red='hello'".

    But really, the big issue is using a bare-identifier to fiat an attribute into existence. That's a door that really shouldn't be opened.

    Secondarily, the doesn't seem to be any use case that can't be readily covered by the existing classes. There is no real need for the witchcraft and the departure from Python norms.

    @DavidHagen
    Copy link
    Mannequin

    DavidHagen mannequin commented Aug 18, 2016

    Secondarily, the doesn't seem to be any use case that can't be readily covered by the existing classes.

    The use case that doesn't have a clean interface in 3.5 at the moment is the most common use case of enums: just a collection of named objects of given type; I don't care about the values because they don't really have values apart from their object identities.

    When writing enums in Rust, Swift, C#, etc., the bare identifier not only saves typing--it allows the developer to explicitly indicate that the underlying value has no meaning. (It may be better to use "object()" rather than an integer on AutoEnum, but that is not very important.)

    It was said that Python has this feature already:

    Yes, Python 3.4 too: Animal = Enum('Animal', 'ant bee cat dog')

    I will concede that this can do what I want. I hope others will concede that this is not a clean interface. The class name is duplicated and the members are regexed out of a space-delimited string. This same argument could be made to deprecate the unnecessary "class" keyword in favor of the "type" function.

    I will also concede that there is some deep magic going on in AutoEnum and that magic should be avoided when it obscures. I personally think the only people who will be truly surprised will be those who already know Python at a deep enough level to know that deep magic must be required here. Everyone else will see "Enum" and a list of bare identifiers, and correctly conclude that this is your basic enum from everyone other language.

    Perhaps an ideal solution would be an enum keyword:

    enum PrimaryColor:
    red
    blue
    green

    But that's not happening ever.

    The next best solution is the current implementation:

    class PrimaryColor(AutoEnum):
        red
        blue
        green

    But because of the magic, it only barely beats out what I think is the other great solution already mentioned here:

    class PrimaryColor(AutoEnum):
        red = ()
        blue = ()
        green = ()

    These two solutions are isomorphic. Both save the developer from having to provide a (possibly meaningless) value. Both let docstrings be added. Both provide the ability to reorganize without renumbering. The last one trades magic for boilerplate.

    I'll keep using them from the aenum package if they don't make it into 3.6, but I think this is a fundamental enough construct that it belongs in the standard library. It is hard to convince tool maintainers to fully support these until they are blessed here.

    @rhettinger
    Copy link
    Contributor

    Temporarily, marking this as open so that more people can see the new comments.

    For John, I'm not sure what I can say that will help you. The goal of the standard libraries modules is to provide tools the use language we have, not to invent a new language the isn't really Python or Pythonic. In the absence of an "enum" keyword, you have a number of ways to work within the language.

    In this case (wanting auto numbering but not caring what the values are), you already have several ways to do it. I your issue is not where the use case is met; instead, you just don't like how their spelled (because it isn't exactly how it it looks in C).

    This already works: Animal = Enum('Animal', ['ant', 'bee', 'cat', 'dog']). This is very flexible and lets you read the constants from many possible sources.

    If you're attracted to multiline input, that is already possible as well:

        Animal = Enum('Animal', '''
    	ant
    	bee
    	cat
    	dog
        ''')

    It is clear that you're bugged by writing Animal twice, but that is how Python normally works and it is a very minor nuisance (it only occurs once when the enum is defined). Note, you already are rewriting "Animal" every time you use the enum value (presumably many times): board_ark(Animal.ant, Animal.bee)

    This whole feature request boils down to wanting a currently existing feature to be spelled a little differently, in a way that doesn't look like normal Python. Elsewhere, we resisted the temptation to alter the language look-and-feel to accommodate small spelling tweaks for various modules (i.e. we pass in normal strings to the regex module even though that sometimes requires double escaping, we pass in the class name to namedtuples even though that uses the class name twice, the xpath notation in passed into XML tools as strings even though parts of it look like regular langauge, we don't abuse the | operator to link together chains of itertools, etc)

    Since the enum module already provides one way to do it, I recommend that we stick with that one way. Creating too many variants does not help users. Using the core language in odd ways also does not help users.

    @rhettinger rhettinger reopened this Aug 18, 2016
    @warsaw
    Copy link
    Member

    warsaw commented Aug 18, 2016

    Ethan, the suggestion has come up several times about using a dummy value such as the empty tuple to do autonumbering, thus looking more Pythonic. I'm not a huge fan of the empty tuple, and I'm still not sure whether we need this, but I wonder if it would be possible to not have a new base class, but to put the smarts in the value to which the enums were assigned. E.g. is this possible (a separate question than whether it's good :):

    from enum import Enum, auto
    
    class Color(Enum):
        red = auto
        green = auto
        blue = auto

    Apologies if this has already been suggested; this tracker thread is too long to read the whole thing. :(

    @ethanfurman
    Copy link
    Member

    Thank you, Raymond, David, Barry, John, etc. for your feedback.

    While I really like AutoEnum I can see that that much magic doesn't need to exist in the stdlib.

    Unfortunately, the alternatives aren't very pretty, so I'll leave the AutoNumberEnum as a recipe in the docs, and not worry about an 'auto' special value.

    For those that love AutoEnum, it will be in the aenum third-party package.

    @johnthagen
    Copy link
    Mannequin Author

    johnthagen mannequin commented Aug 18, 2016

    Ethan, thank you so much for all of your work. Looking forward to any future collaboration.

    @vedgar
    Copy link
    Mannequin

    vedgar mannequin commented Aug 19, 2016

    I don't think the problem is with (Auto)Enum at all. Problem is in a different place, and this is just a symptom.

    Problem is that people make too many assumptions about class suite. Yes, type is default metaclass. But not every class should be constructed by type's rules - Python provides a very detailed and powerful mechanism for specifying a different set of rules, should you need it. This is what metaclass keyword argument should be all about. And that is the true reason why metaclass conflict is an issue: every metaclass is a DSL specification, and the same suite cannot be interpreted according to two different DSLs at the same time.

    But since there's a widespread myth that users don't like to type, implementors use a trick, "class Spam(metaclass=SpamMeta): pass", and say to their users: "there, just inherit from Spam like you inherit from ordinary classes". That way, we spare them a few keystrokes, sparing them also of opportunity to learn that metaclasses _do_ change the semantics of what they see indented after the colon.

    I wonder what Raymond's poll result would be if a normal way to write such code would expose the metaclass

        class Color(metaclass=AutoEnum):
            blue
            yellow

    ? Raymond has many valid objections, but they all apply to "ordinary" classes, instances of type. Color is not an instance of type, or at least conceptually it isn't. type (as a metaclass in Python) means a blueprint, a way to construct new instances via __call__, and every instance of it has that behavior, some of which even customize it by defining __new__ and/or __init__. type is also special because its power of creation is transitive: its instances know how to produce their instances in almost the same way it does.

    But nothing of it is mandatory, and in fact it just stands in the way when we try to define Enum (or singletons, or database models, or... whatever that is not really a Python type). We do inherit from type because it's easier, and then usurp and override its __call__ behavior to do something almost entirely different. (Don't you feel something is wrong when your __new__ method doesn't construct a new object at all?:) It's completely possible, but it's not the only way.

    Maybe class is just too tainted a keyword. There was a thread on python-ideas, that we should have a new keyword, "make". "make m n(...)" would be a clearer way to write what's currently written "class n(..., metaclass=m)", with a much more prominent position for the metaclass, obviating the need for "just inherit from Spam" trick, and dissociating in people's minds the connections of arbitrary metaobjects with type. ("class" keyword could be just a shortcut for "make type" in that syntax.) But of course, a new keyword is probably too much to ask. (A mild effect can be gained by reusing keywords "class" and "from", but it looks much less nice.)

    However, there is _another_ underused way metaclasses can communicate with the external world of users of their instances: keyword arguments. I wonder if Raymond's objections would be as strong if the autovivification was explicitly documented?

        class Color(Enum, style='declarative'):
            blue
            yellow

    Of course we can bikeshed about exact keyword argument names and values, but my point is that there is an unused communication channel for converting typical Python user's surprise "why does this code even work" into "hey, what does that style='declarative' mean?"

    @kennethreitz
    Copy link

    This addition to Python, specifically the bare assignment-free red syntax would be a huge mistake for the language. It looks like a syntax error to me, and completely destroys my mental model for how Python code either does work or should work.

    A simple assignment to None, (), or imported static AUTO_ENUM (or similar, akin to subprocess.PIPE) would be much preferred.

    @vstinner
    Copy link
    Member

    Would it be possible to use a auto or AUTO_ENUM constant from the enum in
    the Enum class? (And drop AutoEnum class.)

    @vedgar
    Copy link
    Mannequin

    vedgar mannequin commented Aug 19, 2016

    I must say I never understood how exactly is assigning a constant better in this regard. To me, after

        a = SOMETHING
        b = SOMETHING

    a and b should be the same, right? _Especially_ when SOMETHING is None or (), but also when it's some bare identifier like AUTO_ENUM.

    Mental models are already broken. We are already "lying" according to semantics of type metaclass. There is no reason then to have to sprinkle some magic dust over our code, just so it looks like an assignment is the most important thing here, when it really isn't.

    Either we should embrace Python's power in full and communicate to users that there's something weird going on, or we shouldn't have AutoEnum. To me, green = None is in no way better, in fact it seems more dishonest, than simply green. Again, if we communicate (e.g. via style='declarative') that the semantics is not ordinary. (Alternatively, we might just write green = uuid4() and be done with it.:)

    @kennethreitz
    Copy link

    Explicit is better than implicit :)

    @vedgar
    Copy link
    Mannequin

    vedgar mannequin commented Aug 19, 2016

    Absolutely. How the hell is green = None explicit?!

    On the other hand, `class Color(Enum, style='declarative'):` is explicit. People must learn something. Why then don't they learn the right thing instead of "hey, assigning None inside enums magically does the right thing" - except when it doesn't.

    Just a hint of a nightmare scenario: you write a method decorator, but you forget to return a value from it (happened to me more times than I care to admit). Ooops, your method is now next member of your enum. Good luck debugging that. :-O

    In fact, what _is_ explicit, is this:

        class Color(metaclass=Enum):
            green = object()
            yellow = object()

    I could live with it. :-)

    @vstinner
    Copy link
    Member

    2016-08-19 14:19 GMT+02:00 Vedran Čačić <report@bugs.python.org>:

    In fact, what _is_ explicit, is this:

    class Color(metaclass=Enum):
        green = object()
        yellow = object()
    

    I could live with it. :-)

    For me, it's legit to use a singleton in an enum. I mean that I expect
    that green != yellow. Some remark for an empty tuple.

    So I woud really prefer a contant from enum like AUTO_ENUM.

    @vedgar
    Copy link
    Mannequin

    vedgar mannequin commented Aug 19, 2016

    For me, it's legit to use a singleton in an enum.

    But you cannot explain _why_, right? Like Raymond says, it's not working like anything else in Python works.

    To me, this looks almost the same as the gripes of those lazy people who want "myfile.close" expression to close myfile, or worse, "quit" to exit the Python interpreter. An irrational phobia of parentheses. Why not call your object, if you expect some code to be executed? [At least a dotted name like myfile.close _could_ be made to work using descriptors, but still I think everyone agrees it's not a good idea.]

    To me, we either want to stay with default type metaclass (style='imperative':), and write something like

        class Color(Enum):
            green = member()
            yellow = member()

    or we acknowledge that style='declarative' has its place under the sun (not only for Enums, of course... database models, namedtuples, even ABCs could profit from it), and embrace its full power, realizing it's not Python we usually see, but it's still Python.

    Magic is not something to be afraid of, if you understand it. We are talking Py3.6 here... aren't formatted strings a totally insane magic? Yet they ended up in the language, because they are immensely better than the alternatives. Here the gain is much smaller, but the threshold is much lower too: we don't need new syntax at all.

    @ethanfurman
    Copy link
    Member

    Vedran, you have some very interesting points. However, if you really want to champion this you'll need to take it to Python Ideas.

    @warsaw
    Copy link
    Member

    warsaw commented Aug 19, 2016

    Hey, I just realized that you can get pretty darn close with just a little bit
    of extra typing, and existing stdlib:

        from enum import Enum
        from itertools import count
    
        auto = count()
    
        class Color(Enum):
            red = next(auto)
            green = next(auto)
            blue = next(auto)

    Look ma, no magic!

    @ethanfurman
    Copy link
    Member

    No magic, but a whole heap of extra boiler-plate. :(

    @kennethreitz
    Copy link

    There's a difference between boiler-plate and "code".

    On Aug 19, 2016, at 10:53 AM, Ethan Furman <report@bugs.python.org> wrote:

    Ethan Furman added the comment:

    No magic, but a whole heap of extra boiler-plate. :(

    ----------


    Python tracker <report@bugs.python.org>
    <http://bugs.python.org/issue26988\>


    @DavidHagen
    Copy link
    Mannequin

    DavidHagen mannequin commented Aug 19, 2016

    One solution similar to one proposed by Vedran works with the current Enum:

        class Color(Enum):
            red = object()
            green = object()
            blue= object()

    I tested this in PyCharm and it is perfectly happy with the autocomplete and everything. The main disadvantage is the boilerplate, of course. And perhaps "object()" does not show the clearest intent, but it depends on your perspective. The repr also looks kind of funny:

        >>>  repr(Color.red)
        <Color.red: <object object at 0x7fb2f353a0d0>>

    One possibility would be to add an auto() function to enum as a wrapper around object(), providing a more explicit name and a cleaner repr:

        from enum import Enum, auto
    
        class Color(Enum):
            red = auto()
            blue = auto()
            green = auto()
    
        repr(Color.red)
        <Color.red>
        # auto() means it has no (meaningful) value, so show nothing

    @vedgar
    Copy link
    Mannequin

    vedgar mannequin commented Aug 19, 2016

    # Just wait until I put the keys to the time machine in their usual place... :-)

    Ok, while we're talking about whether declarative style is a good idea, Python has already got the initial seed of that style. With Guido's blessing! PEP-526 enables us to mix declarative and imperative style in "ordinary" code, so we can write

        @Enum
        class Color:
            green: member
            yellow: member

    without any additional syntax. I think it satisfies everyone: there are no parentheses, and there are no assignments. [And there is no misleading analogy with existing syntax, because this is a new syntax.:] There are just declarations, and the decorator instantiates them.

    Decorator is needed because formally we need to exclude the type checking semantics, and the only official way currently is through a decorator. But in fact we can use the forward references to _actually_ annotate the members with their real type:

        class Color(Enum):
            green: 'Color'
            yellow: 'Color'

    And once the forward references get a nicer syntax, and the unpacking issues are solved, we'll be able to write

        class Color(Enum):
            green, yellow: Color

    And I think finally everyone will be happy. :-)

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Aug 20, 2016

    New changeset 2e243f78720e by Ethan Furman in branch 'default':
    bpo-26988: remove AutoEnum
    https://hg.python.org/cpython/rev/2e243f78720e

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    stdlib Python modules in the Lib dir type-feature A feature request or enhancement
    Projects
    None yet
    Development

    No branches or pull requests

    5 participants