Here is preliminary implementation of IntFlags (no docs). This is an int subclass purposed to represent a set of integer flags. It supports named constants (as IntEnum), supports bitwise operations (&, |, ~) and has funny str and repr. See discussion on Python-Ideas [1].

The patch includes tests and few examples of using IntFlags in the stdlib.

[1] http://comments.gmane.org/gmane.comp.python.ideas/32267

It would make more sense and be more consistent if the str() and repr() used one’s complement in all cases, i.e.:

self.assertEqual(str(Perm(~0)), "~0")

Also, the repr() seems to be doing a bad attempt at Python pseudo code. Instead of

<Perm.R|W: 3>

maybe it could be something like these:

<Perm.R|Perm.W: 3>  # Mirroring str() -> "Perm.R|Perm.W"
<Perm R|W: 3>
<Perm: R|W = 3>

I wonder if the addition (+) operator should also be overridden; I haven’t looked, but I suspect some people may do FLAG1 + FLAG2 instead of FLAG1 | FLAG2.

The current patch is more along the lines of a proof-of-concept.

The final IntFlag type (if there is one) would be quite a bit more extensive since part of the reason for its existence is to not lose type -- so pretty much every __op__ would have to be overridden.

Agreed about the repr() and str(), though.

I choose repr() so that for single flag it looks as for IntEnum instance. Other variants except <Perm.R|Perm.W: 3> have different format. <Perm.R|Perm.W: 3> is too verbose, it repeats the same class name multiple times.

But I don't like current repr.

Other operators are not overridden for purpose. IntFlags is int and can be used in all cases when int is used, but not in all cases the result should preserve the type. FLAG1 + FLAG2 works as if flags would be ints, but it should be rewritten to FLAG1 | FLAG2 to got the benefit from using IntFlags.

However I forgot to override the exclusive-or operator. It should be overridden.

Updated patch changes str and repr, override the exclusive-or operator and adds some documentation.

Why have you rejected this issue Ethan? I think this is useful feature and must be in Python, and other core developers agreed with this. Only minor implementation details are discussable.

I agree.  Closing this does not follow our normal development workflow, since there has been a consensus in favor and none for rejection.  If you think the API needs wider discussion a new thread can be started on python-ideas.

Ethan: to clarify...based on my years of watching this tracker, the way we normally handle an issue like this is to leave the issue open until it is resolved, sometimes with people proposing competing patches.  (There might be reasons to change that tradition, but if so the forum for that discussion would be python-workflow.)

My experience is that a module maintainer, or somebody claiming to speak for the module maintainer, can close any issue in their area at any time regardless of the number of core devs in favor of a change.

Whatever.  I'll leave this open and write up a spec of what IntFlags should look like in case somebody wants to write the patch for it.

What should be done for landing this in 3.6?

Just look briefly through your patches, and they look pretty good.  I'll take a more in-depth look in the next couple weeks.  (Feel free to ping again if you don't see any activity from me, and thanks for your patience.)

Ping again.

Reviewing...

I don't think I'll have this in before the next alpha (today? tomorrow?) but I'll get it merged in the next couple weeks (need to do some integration work with the other Enum enhancements).

Flags -> int-backed, but not ints; useful for pure Python flags
IntFlags -> ints (like IntEnum); useful for interfacing with other systems

Are there better names?

The name IntFlags is inspired by the Flags attribute in C# (this is the most close concept). The "Int" prefix is added for parallel with IntEnum and because the class behaves as int in some contexts (supports operators |, &, ^, ~ and can be passed to functions that require int). I don't know whether there is a need in Flags class that is not a subclass of int.

Other languages provide something like EnumSet - a specialized set of enums. But it has different interface.

1) If there are enum members FOO and BAR, their union is {FOO, BAR}, not FOO|BAR.
2) The union member itself is not a set.
3) A set can contain only predefined set of values (IntFlag allows non-defined bits be set).

If you thing that non-int flags could be useful, you can consider adding EnumSet for general enums and IntFlags (or just Flags) for int-like enums. But I think that new API can just use a set or a tuple of enums. IntFlags is needed for compatibility with old API or C API.

The idea behind Flags is that they are easily combined enums.  The advantage they have over IntFlags is they do not interact with integers, and they cannot have non-existing values used.  By keeping the same API as IntFlags we only have two APIs instead of three (Enum and Flag instead of Enum, EnumSet, and IntFlag), and also make life easier for changing code from one to the other (Flag <-> IntFlag).

Due to dynamic typing EnumSet actually is not needed in Python. You can use sets, tuples or lists for combining enums in new API. IntFlags is needed for old API that already uses ints. I think introducing non-int Flags is overengineering, can't imagine somebody will use it in new code.

There is no much time left before feature freeze. If general IntFlags will not added, I'm going to open separate issues for adding specialized class for every particular case (re, os, etc).

IntFlags will be in before feature freeze.

I tried to implement this once. The biggest problem I encountered is  inconsistency with Enum identity: Enum's main idea is that its objects are pre-created and __new__ just return one of them, so equality is simply identity.

If you try to do this with flags, you quickly hit the exponential blowup and with most flags applications, it's utterly impractical to construct all the combinations at the start. So you have MyFlags.a|MyFlags.b is not MyFlags.a|MyFlags.b, and that is very weird and unexpected (especially if MyFlags.a is MyFlags.a, which it is if this case just delegates to Enum.__new__).

How does this patch propose to solve this problem?

Currently, the patch has the main values pre-created (so identity works as expected), and combinations are created on the fly (like normal class instances) -- in which case identity cannot be relied upon.

Once I have the basic work done, I'll go back and add a cache so each unique value is only created once;  at some point I'll make those cached values weakref'ed so they disappear when no longer used.

I don't think this is a problem. For now os.O_WRONLY is os.O_WRONLY, but os.O_WRONLY|os.O_CREAT|os.O_TRUNC is not os.O_WRONLY|os.O_CREAT|os.O_TRUNC. There is an exponential number of combinations (actually infinite number if count not named bits) and you shouldn't use identity check for flags.

It's true, but it seems that with Enums, we're trying to retrain people to use identity testing (see https://docs.python.org/3/library/enum.html#comparisons). It would be unfortunate if we had to reuntrain them. :-/

Ethan's proposal (of caching weakrefs) is fine, if he manages to do it correctly (I didn't). It's much more complicated than it seems at first, since it has to work during the class definition itself, too.

Ok, second problem I encountered: zeros. Imagine

    class MyFlags(enum.Flags):
        NONE = 0
        FIRST = 1 << 0
        SECOND = 1 << 1

What is MyFlags.FIRST & MyFlags.SECOND? 0 or MyFlags.NONE? Or even False? :-) I would almost be for the third option, if not for the fact that & is used not only for "membership" testing, but also (with ~) for clearing flags. Imitating Go, we might want to introduce another operator, &~, for clearing flags, but that would probably be a too great disruption.

MyFlags.NONE seems cool, but then we have an enum member that is false, another thing we have tried to avoid with current Enum design (that's why functional API starts values at 1, for example). And what about the case when MyFlags doesn't define NONE, only FIRST and SECOND?

If you opt for 0, you open another can of worms: first, return type depends on argument values, and Guido hates that. :-) Also, that 0 is still somehow tied to MyFlags: it would be a type error to write (MyFlags.FIRST & MyFlags.SECOND) | AnotherFlags.SOMETHING, right?

Maybe the correct solution is to _always_ have a special value in every Flags class for such cases, but then what is it's ~__NONE__? :-) Fortunately, once you add __NONE__ and __ALL__, you get a Boolean closure, and you can meaningfully define &, |, ^ and ~ in a type-safe way. But that's probably overkill.

And so on... there are a lot of problems, and I've been through them. I'd like you to succeed where I didn't, but I have a bad feeling.

You still can use identity testing for named instances of IntFlags. But since the purpose of IntFlags is replacing int flags, tested values can be int (e.g. when read from files as ints). For unknown values you should use either equality testing or wrap them in IntFlags.

In your example MyFlags.FIRST & MyFlags.SECOND is MyFlags.NONE. If MyFlags.NONE not exists, the result is MyFlags(0). You can apply the patch and experiment with it.

Yes, IntFlags sidesteps a lot of these issues (though not all: inconsistency with a lot of principles of IntEnum is still jarring). But I thought we were talking about Flags too (it is not in the patch, as far as I see).

But now I see that Flags was kinda abandoned, and we're going forward only with IntFlags, right? Maybe the title should be changed?

Serhiy's patch is only for IntFlags, and my patch hasn't yet fully incorporated his (many thanks for decompose!)

For IntFlags I do not expect to have very many instances alive at once, especially since not-bitwise operators will lose the IntFlag class and become plain ints.

Flags are closed.  If the zero value is not specified the repr and str are:

>>> Hah(0)
<Hah: 0>
>>> str(Hah(0))
'Hah.0'

(An RGB class might be:  <RGB.Black: 0>)

A question I have about IntFlags:

If a third-party lib specifies that certain bits are reserved and should always be zero (or at least not changed), do we want to add some easy support for that?

So, in fact, your Flags are simply an overlayed namespace over int (the way to give some ints sticky names), and any int is accessible from any Flags, no matter whether it has a name or not? I must say that to me it seems radically different than (Int)Enum philosophy.

    class MyIntEnum(IntEnum):
        A = 1

    >>> MyIntEnum(0)
    ValueError: 0 is not a valid MyIntEnum

So, flags are not enums, nor they share the same principles (identity, exclusiveness). Why are they in the enum module at all?

The zero value is special in that it represents a flag with nothing set.  Excluding it has its own problems, but if you have an argument for that behavior I'm listening.

Any other non-Flag value is not allowed, and will raise an error (IntFlags won't share this behavior).

I think I have written about all the options in detail in http://bugs.python.org/msg272732. If I see it right, your ".0" corresponds to what I called ".__NONE__". It is a "right way" to do it, but for it to be complete, you also have to add ".__ALL__", representing -1, as its complement. If you do both of these, I'm fine with it, and we can discuss further problems... :-)

... like inverting representation. If you have non-powers of 2 named, how do you know which exact flags a value consists of?

    class Weird(Flags):
        AB = 3
        C = 4
        BC = 6
        A = 1

what is Weird(7)? Weird.(AB|C) or Weird.(A|BC)? What if you don't have C (or A) as a name? Are you going to employ a minimal exact cover algorithm? :-O

'.0' does not have a name unless the user defines it; similarly, '.-1' does not have a name unless the user defines it.

For Flags, negative numbers are supported in order to specify which flags are desired, but the final representation will be zero or positive:

>>> class Hah(enum.Flags):
...   this, that, these, those, thuse
... 
>>> Hah(0)
<Hah: 0>
>>> Hah(-1)
<Hah.this|that|these|those|thuse: 31>

The algorithm is simple: start with the biggest Flag and mask off matching bits until all bits are are matched.  If any unmatched bits remain an error is raised.

If a user does horrible things like your Weird class then any breakage is on them.

As it stands, Weird(7) would be <Weird.A|BC: 7>, and if A was not defined an error would be raised.

I suppose you'll also forbid adding new members to Flags, just like Enum does, right? (https://docs.python.org/3/library/enum.html#restricted-subclassing-of-enumerations) Otherwise cropping -1 at a fixed number of bits might be very counterintuitive.

But still, your "simple algorithm" seems too simple. You might easily produce values you won't be able to interpret.

    class ThirdBitMustBeSet(Flags):
        FIRST = 5
        SECOND = 6

    >>> ThirdBitMustBeSet.FIRST | ThirdBitMustBeSet.SECOND

produces error when printing, but otherwise works completely fine. A debugging nightmare. :-/ I'd at least ensure each bit has a separate name if you're going to use that scheme.

The values used in Flags are an implementation detail.  The expected, and supported, use is naming the flags:

class Color(Flags):
    Red, Green, Blue

even custom names can be set using this method:

>>> class Color(enum.Flags):
...   red, green, blue
...   black = red & green & blue
... 
>>> list(Color)
[<Color.red: 1>, <Color.green: 2>, <Color.blue: 4>, <Color.black: 0>]

>>> class Color(enum.Flags):
...   red, green, blue
...   purple = red | blue
... 
>>> list(Color)
[<Color.red: 1>, <Color.green: 2>, <Color.blue: 4>, <Color.purple: 5>]
>>> Color(5)
<Color.purple: 5>
>>> Color(3)
<Color.red|green: 3>

I'll have to think about ensuring each bit is named -- it's only a (potential) problem if the values are specified manually.

I really like the idea of IntFlags.

I expected better repr for such code:

>>> socket.SOCK_STREAM | socket.SOCK_CLOEXEC
524289
>>> os.O_RDONLY | os.O_APPEND
1024

The repr is better -- which patch did you test?

> The repr is better -- which patch did you test?

Sorry, I wasn't clear. I didn't test any patch :-) I expect (not
expected) better repr when changes will be applied, it's just a
remark...

Patch includes tests, but not docs.

From a previous comment:
-----------------------
> As it stands, Weird(7) would be <Weird.A|BC: 7>, and if A was not
> defined an error would be raised.

Actually, it would be <Weird.BC|A: 7> since BC has a higher value than A.

[Ethan]
> My experience is that a module maintainer, or somebody claiming to speak
> for the module maintainer, can close any issue in their area at any 
> time regardless of the number of core devs in favor of a change.

Ethan, it is still up to you and the other module maintainers to decide whether this goes in or whether to defer it for a release cycle.  If you have any concerns about having too many enum variants and think this is at odds with your goals for the module, say so here and we can resolve it at the sprints.  On the other hand, if you're happy with it, we should get it applied as soon as possible.

[Serhiy]
> If general IntFlags will not added, I'm going to open separate issues
> for adding specialized class for every particular case (re, os, etc).

When enum went in, I thought Guido had said and everyone agreed to refrain sweeping through the standard lib and applying enums broadly to long standing and stable APIs.  Perhaps something has changed since them but there was a clear intention to show restraint, let enums mature, respect stable APIs, and to wait for demonstrated need (i.e. actual rather than imagined user difficulties).

Raymond, thanks for your support.

I am happy to have a flags variant.  The advice to wait and let Enum mature instead of adding the Flag variant at the beginning was sound, and we have a better product to show for it now.

The four base Enum-type classes will be:
- Enum
- IntEnum
- Flags
- IntFlags

I see no reason to add any more than these four to the stdlib -- further specializations can live in recipes, third-party modules, or private libraries.

While Enum and Flags are the heart of the enum module, IntEnum and IntFlags are present for two reasons:
- commonly needed when interfacing with other systems
- use in the stdlib itself

Pollution (otherwise known as enums escaping into other code where they don't make sense) is controlled by:
- IntEnums losing their enum status as soon as they are combined with
  any other number; and
- IntFlags losing their enum status as soon as a non-bitwise operator is
  used on them.

I'll commit as soon as the test suite is finished (and any failures are not attributable to these changes), and work on the docs later.

Serhiy, I'm not going to include your changes to re, os, etc., as they are not part of adding Flags/IntFlags.  Please open new issues for them.  I am not opposed to those changes, as

>>> re.I
<Flags.IGNORECASE: 2>

is more useful than

>>> re.I
2

The weirdest thing is that it already works pretty well in output of re.compile.

    >>> re.compile('', re.I | re.M)
    re.compile('', re.IGNORECASE|re.MULTILINE)
    >>> _.flags  # re.UNICODE == 32 added automatically
    42

So the only thing we should enhance is the output of .flags, and it seems that all the necessary code is already in the source code of __repr__ of SRE_Pattern objects.

Also, I suppose that means you've given up on the autocreation (since the values _are_ semantical here), and I suppose you'll require all the declared values to be powers of 2. With those conditions, I think this is a good enhancement of Python.

Since we're using re as the sample, here's where re.I is defined:

Lib/re.py:
---------
I = IGNORECASE = sre_compile.SRE_FLAG_IGNORECASE # ignore case

As already mentioned, re.I results in 2, and a re.compile object is not usable as a re flag.

> Also, I suppose that means you've given up on the autocreation

We decided auto-created values were too magical for the stdlib (see issue26988).  They will exist in my aenum package, though.

> (since the values _are_ semantical here),

The values have meaning because the underlying library gave them meaning; so assuming a type of Flags are used, they will be IntFlags.

> and I suppose you'll require all the declared values to be powers of 2.

Nope.  You are welcome to give more meaningful names to different combinations of powers of two.

> With those conditions, I think this is a good enhancement of Python.

Hopefully you still think so. ;)

New changeset 39661e2ff030 by Ethan Furman in branch 'default':
issue23591: add Flags, IntFlags, and tests
https://hg.python.org/cpython/rev/39661e2ff030

Still needs docs.

As noticed by veky on the review, _high_bit() is slow and can be optimized using int.bit_length(). Attached bit_length.patch implements this.

_high_bit(0) returns -1. Maybe an exception must be raised if the argument is < 1? (also fail for negative number)

Sure, errors should never pass silently. This function shouldn't be called with nonpositive arguments. And there is no highbit of 0 (and there are infinitely many of negative numbers;).

ValueError is probably best - though IndexError can also be argued for, since this is quite analogous to pop from empty list. :-)

New changeset adbc7eec97f1 by Ethan Furman in branch 'default':
issue23591: add docs; code cleanup; more tests
https://hg.python.org/cpython/rev/adbc7eec97f1

> Nope.  You are welcome to give more meaningful names to different combinations of powers of two.

Yes, you're correct here, but what about output? Do all relevant bits have to be named separately? And will the output always talk about separate bits? We have talked about this. If I give meaningful name to 1, 3, 4 and 6, but not to 2, is it supported? And is the repr of 7 an implementation detail, or is it specified?

One more question: in the docs, you say 

> otherwise, all members evaluate as :data:`True`.

I would like you to reconsider the case of `.0` (.NONE in my terminology). Many other things in the interface (e.g. `in` operator) are modelled as if flags instance were just a container (set, in fact) of bits. In that case, empty set _should_ be false. Otherwise, this will be a weird exception to the usual container semantics.

> Yes, you're correct here, but what about output? Do all relevant bits have
> to be named separately?

No.

> And will the output always talk about separate
> bits?

No.

> We have talked about this.

Indeed.  Here's my previous reply. ;)

>> The algorithm is simple: start with the biggest Flag and mask off
>> matching bits until all bits are are matched.

When I said "biggest flag" I meant the flag with the highest value (so a flag of 5 would get matched before a flag of four).

>> If any unmatched bits remain an error is raised.

This part is specifically for Flag; IntFlag will just show the values of any unmatched bits.

> If I give meaningful name to 1, 3, 4 and
> 6, but not to 2, is it supported?

Depends.  Not having 2 named has different consequences for Flag vs IntFlag (although neither is an error):

- Flag: no combination of flags will ever have the 2 bit set
- IntFlag: '2' will show in the name portion if the 2 bit is set

> And is the repr of 7 an implementation detail, or is it specified?

I should specify it in the docs, thanks.

> One more question: in the docs, you say 
>>
>> otherwise, all members evaluate as :data:`True`.
>
>I would like you to reconsider the case of `.0` (.NONE in my
> terminology). Many other things in the interface (e.g. `in` operator)
> are modelled as if flags instance were just a container (set, in fact)
> of bits. In that case, empty set _should_ be false. Otherwise, this will
> be a weird exception to the usual container semantics.

Very good point.  Unless I can think of a good reason not to, I'll make that change in the next couple days.

Thanks.

New changeset 31586a2f01b6 by Ethan Furman in branch 'default':
issue23591: optimize _high_bit()
https://hg.python.org/cpython/rev/31586a2f01b6

New changeset f33fc2117bb2 by Ethan Furman in branch 'default':
issue23591: bool(empty_flags) == False; more docs & tests
https://hg.python.org/cpython/rev/f33fc2117bb2

Hmm... so if I read you right:

1) IntFlags is not simply the meet of int and Flags (like IntEnum is the meet of int and Enum, https://docs.python.org/3.5/library/enum.html#others)? It seems a very different class.

2) (more important) If I give names to 1, 3, 4, and 6, you're in fact saying that MyFlags(7) == MyFlags(5) (since 2 is never set)? Seems like succumbing to temptation to guess. ;-/ But maybe it's not bad in this case, since there is really no ambiguity.

> 1) IntFlags is not simply the meet of int and Flags (like IntEnum is the
> meet of int and Enum,
> https://docs.python.org/3.5/library/enum.html#others)?
> It seems a very different class.

No, I think it is.  The differences between Enum and IntEnum are entirely because IntEnum members are also ints, so they can do anything an int can do (but they lose their IntEnum-ness when non-Enum operations are performed on them).  Similarly, the differences between Flag and IntFlag are entirely because IntFlag members are also ints, so can do whatever ints can do (and, similarly, will lose their IntFlag-ness when non-Flag operations are performed on them).

> 2) (more important) If I give names to 1, 3, 4, and 6, you're in fact
> saying that MyFlags(7) == MyFlags(5) (since 2 is never set)?

(I think you messed up your example, as 4 | 1 == 5 and 6 | 1 == 7 -- 2 is never a factor.)

Absolutely not.  5 != 7, so the two will never be equal no matter which of Flag | IntFlag you are using.

Moreover, if MyFlags is a Flag then MyFlags(impossible_combination) will raise an exception.  If MyFlags is an IntFlag then MyFlags(impossible_combination) will have the value of <impossible_combination> and the repr will show all the non-specified bits as base-2 numbers and the specified bits as names.

Ok, I believe you that you have the interface for IntFlags right (I always did, and I apologize if I didn't make it clear from the start). All my questions pertain to Flags.

You said what to me seem like two contradictory things:

> Not having 2 named has different consequences for Flag vs IntFlag (although *neither is an error*): Flag: no combination of flags will ever have the 2 bit set

> if MyFlags is a Flag then MyFlags(impossible_combination) *will raise an exception.*

Are you saying that after I write

    class MyFlags(Flags):
        b001 = 1
        b011 = 3
        b100 = 4
        b110 = 6

this is _not_ an error, but if after that I call

    print(MyFlags(7))

it _is_ an error? It doesn't seem so bad now I think about it, but I'd like the error to be reported before ("3 has bit of value 2 set, but that bit is not a member" or something like that).

> All my questions pertain to Flags.

Ah, okay.

> You said what to me seem like two contradictory things:
> 
>> Not having 2 named has different consequences for Flag vs IntFlag
>> (although *neither is an error*): Flag: no combination of flags will
>> ever have the 2 bit set [...]

and

>> if MyFlags is a Flag then MyFlags(impossible_combination) *will raise an exception.*

First, let me state the intended use of Flags:  primary flags (single-bit flags) will have values of powers of 2:

--> class Perm(Flag):
...     R = 4
...     W = 2
...     X = 1
...
--> Perm.R | Perm.W | Perm. X
<Perm.R|W|X: 7>
-->

If nicer names are desired for certain combinations (aka secondary flags, or multi-bit), then name them:

--> class Perm(Flag):
...     ...
...     RWX = 7
...
--> Perm.R | Perm.W | Perm.X
<Perm.RWX: 7>

If, for whatever strange reason, you don't give a certain power of 2 a name, Flag doesn't care:

--> class Perm(Flag):
...     R = 8
...     W = 4
...     X = 1
...
--> Perm.R | Perm.W | Perm. X
<Perm.R|W|X: 13>
-->

But trying to use that missing value will be an error:

--> Perm(6)
Traceback (most recent call last):
  ...
ValueError: 6 is not a valid MyFlags

This is what I was referring to as "not naming a bit is not an error, but using an impossible value is".  What I missed in your example was that, although you hadn't named 2, you still had multi-bit values that included the 2 bit.  In other words, in my example there will never be a combination that has the 2 bit set, while in yours (because of your weird values) it is possible.


> Are you saying that after I write
> 
>      class MyFlags(Flags):
>          b001 = 1
>          b011 = 3
>          b100 = 4
>          b110 = 6
> 
> this is _not_ an error, but if after that I call
> 
>      print(MyFlags(7))
> 
> it _is_ an error? 

No, that's not what I'm saying, and hopefully my explanation above clears that up.

As a side note, I suspect there would be much less confusion with Flag if we had an AutoEnum.

C'est la vie.

New changeset 8e9d3a5d47d5 by Ethan Furman in branch 'default':
issue23591: more docs; slight change to repr
https://hg.python.org/cpython/rev/8e9d3a5d47d5

> As a side note, I suspect there would be much less confusion with Flag if we had an AutoEnum.

No, there would be _different_ confusion. :-P

Anyway, let's get back to the discussion. I am quite aware about what's the intended use, but you can't just assume people will know about it. In my view, you must do one of two things:

1) (at least in documentation, and preferably in the code by raising Exceptions at class definition time) forbid the use of Flags whose values are not either a) powers of two, or b) bitwise or of some already declared values

-or-

2) Specify and implement a robust algorithm for finding the "cover of bits" for a given numeric value. If the intended algorithm is really "pick the largest member value smaller than given value, subtract and repeat until 0 remains", then it should be said so in the documentation, and preferably some reasons given for the usage of that exact algorithm. ("it was easiest to implement" does not rank very high on my list.)

In other words, MyFlags(7) should either be illegal, or I should be able to interpret what it will be by reading the documentation. Leaving it unspecified is not acceptable IMO.

(In case it isn't clear: I'm for option 1. I _don't_ intend to write MyFlags ever in Python. But if I happen to write it unintentionally (e.g. if I forget to declare 2), I would like Python to tell me I'm doing something wrong.)

(But if you really want option 2 for some reason, that's ok too. I'm just saying I would like it specified, with a rationale if it's not too much of a problem.)

> I am quite aware about what's the intended use, but you can't just
> assume people will know about it.

Fair enough.

> In my view, you must do one of two things:
> 
> 1) (at least in documentation, and preferably in the code by raising Exceptions at class definition time) forbid the use of Flags whose values are not either a) powers of two, or b) bitwise or of some already declared values

The closest I would get to forbidding something would be to forbid non-integers as values -- and I'm not going to do that.  In general, Python doesn't take such steps (one notable exception being sum refusing to work with strings because it was such a common trap).

> -or-
> 
> 2) Specify and implement a robust algorithm for finding the "cover of
> bits" for a given numeric value. If the intended algorithm is really
> "pick the largest member value smaller than given value, subtract and
> repeat until 0 remains", then it should be said so in the
> documentation, and preferably some reasons given for the usage of that
> exact algorithm. ("it was easiest to implement" does not rank very high
> on my list.)

How about "it works for me"?  Seriously, if every bit is named then the exact repr used is irrelevant.  If you don't like the standard method then write your own __repr__ method, because that's what we're talking about -- not the actual value (which isn't going to be affected by the algorithm for finding the "cover of bits", but the display of the names).

> In other words, MyFlags(7) should either be illegal, or I should be
> able to interpret what it will be by reading the documentation. Leaving
> it unspecified is not acceptable IMO.

Thankfully, docs can still change during beta.

> (In case it isn't clear: I'm for option 1. I _don't_ intend to write
> MyFlags ever in Python. But if I happen to write it unintentionally
> (e.g. if I forget to declare 2), I would like Python to tell me I'm
> doing something wrong.)

We could certainly add a decorator that double-checks that, just like we have the `unique` decorator to ensure no duplicates in an Enum.  Any idea what to call it?  complete?  named?  allbits?

> (But if you really want option 2 for some reason, that's ok too. I'm
> just saying I would like it specified, with a rationale if it's not
> too much of a problem.)

(1) isn't going to happen, except possibly as a decorator.  If you would like to contribute code and/or docs for (2) and/or a decorator for (1) I'd be happy to review.

I would like to add a function that can be used when creating a Flag (and Enum, but that's less important) that will generate the next value.

This is important because in Flag the values are an important internal detail, but are largely irrelevant to the user (if they weren't then an IntFlag is probably more appropriate).

Actually, this function already exists and is used to supply the values when using the Functional API: _generate_next_value_ .  But that's a bit clunky for use during class definition:

class Color(Flag):
    red = _generate_next_value_()
    blue = _generate_next_value_()
    green = _generate_next_value_()

What would be a better name for class use?

- _next_value_
- _value_
- _flag_
- _next_flag_

Any others?

I think we had that discussion in the other thread, and concluded that auto() (or _auto_() if you insist) is quite fine. I think it's important to emphasize it's automatically generated, not that it's really "next" in any particular order.

I like it!  (Although I have no idea which other thread you are talking about.)

And yes, it needs the leading and trailing underscore so as not to clash with member names.

Any problems with:

class Color(Enum):  # or Color(Flag)
  red = _auto_
  green = _auto_
  blue = _auto_

In other words:
- is the missing parenthesis an issue?
- are linters/checkers/IDEs going to have (or report) problems with that?

For me, it is an issue. But you probably already know that.
(http://bugs.python.org/issue26988#msg273125) (That's "the other thread".)

Try to explain: how exactly is that different than wanting "file.close" to close the file, or "quit" to quit the REPL? And I surely hope we're not going in that direction.

> For me, it is an issue. But you probably already know that.
>(http://bugs.python.org/issue26988#msg273125) (That's "the other thread".)

Ah, thanks.  I had forgotten about that one.

For what it's worth, I largely agree with you there.

> Try to explain: how exactly is that different than wanting "file.close"
> to close the file, or "quit" to quit the REPL?

Enum is already full of magic:
- class is iterable
- class is indexable
- member attributes are instances of class
- members are all created when class is created
- etc.

Flag, especially, needs a way to specify "give me a working value" even when the user doesn't care what that value is (as your, um, interesting examples have shown ;).

Whatever we choose as the placeholder (since it has been decided that no placeholder is too magical) will still have magic involved, so I don't want parenthesis there disguising that.

I'd also be happy with _member_ instead of _auto_ -- maybe that makes even more sense.

> And I surely hope we're not going in that direction [dropping parens from function calls].

No, we're not.

_member_() is fine with me, though I prefer _auto_(). _auto_member_() is probably too much. :-)

The argument "we already do magic, so let's do also this bit of non-connected magic" seems very weak to me. But in fact those other things are not _magic_ in the same sense as this one (except repatching __new__, I agree that's weird, but Guido started that game when he declared bool unsubclassable:). They are just added behaviors.

This is something fundamental: it is breaking the promise that class body is a suite of commands, where Python statements (such as assignment) have their usual semantics. If I ever see, in any suite,

    a = x
    b = x

and after that `a is not b`, I feel cheated. Even more strongly: I _don't know how to think about the code_ anymore. Those look like assignment statements, but they are not. What are they? Is this Python?

And what if you really do want to make aliases? Are you saying that

    a = x
    b = a

should be fundamentally different than the above? Oh, please let's not go there. :-)

> Any problems with:
> 
> class Color(Enum):  # or Color(Flag)
>  red = _auto_
>  green = _auto_
>  blue = _auto_

As long as _auto_ has been defined somewhere (i.e. from enum import _auto_), it is normal Python and doesn't fight with the rest of language or its toolchains.

The idea does seem to be at odds with the enum module itself.  Heretofore, the rule for the module is that if the same object is assigned to multiple variable names, those names become synonyms.

    >>> from enum import Enum
    >>> class Color(Enum):
	red = 1
	magenta = 1
	orange = 2
	ochre = 2

    >>> Color.orange is Color.ochre
    True

Also, I still question the need, you already have multiple ways to do it:

    Color = Enum('Color', ['red', 'green', 'blue'])

    Color = Enum('Color', '''
        red
        green
        blue
    ''')

Vedran commented:
> This is something fundamental: it is breaking the promise that class body
> is a suite of commands, where Python statements (such as assignment) have
> their usual semantics.

I find it curious that you're okay with

>>> class Color(Enum):
...     red = 1
...
>>> Color.red == 1
False

But you find this completely incomprehensible

>>> class Color(Enum):
...    red = _auto_
...    blue = _auto_
...
>>> Color.red == Color.blue
False

Yes, I didn't explain well. I'm ok with postprocessing. After all, since we got PEP 520, it's quite obvious that "the namespace class body is executed in" is not the same as "the final __dict__ of a class".

But even "the namespace class body is executed in" _should_ be a Python namespace, with all its rules. If objects are the same, then after postprocessing they should also be same. _Especially_ if we currently do have semantics for reassignment: aliasing. "Name is not a part of object" is a fundamental tenet of Python. And `_auto_ is _auto_`.

Since you like examples, what do you say about

    class MyEnum(Enum):
        red = some_function()
        blue = red

Now, is MyEnum.blue the same as MyEnum.red (watch: not "equal", but "same")? Well, it depends on what some_function returns, right? If it returns _auto_, they are not the same, but in all the other cases, blue is just an alias for red. So object identity depends on some value that could be external to the class. To me that's obviously unacceptable.

Just so we're clear:  I'm going to go with "auto()" and (mostly) use Python's standard routines (with only a little bit of metaclass magic).

Vedran opined:
> Since you like examples, what do you say about

I love examples!  Examples are like pictures, which are worth a thousand words.  ;)

>    class MyEnum(Enum):
>        red = some_function()
>        blue = red
>
> Now, is MyEnum.blue the same as MyEnum.red (watch: not "equal", but
> "same")?

Yes.

> Well, it depends on what some_function returns, right?

Nope.

> If it returns _auto_, they are not the same, but in all the other
> cases, blue is just an alias for red. So object identity depends on
> some value that could be external to the class. To me that's
> obviously unacceptable.

That would be unacceptable.  However, it would also be impossible*, because "_auto_" would be injected into the class namespace during "__prepare__()" and be unavailable for an external function to return.

* as impossible as anything is in Python -- so really, really difficult

> But even "the namespace class body is executed in" _should_ be a Python
> namespace, with all its rules.

One of those rules is:  the namespace is dict-like.

Which means it has methods such as __getitem__, __setitem__, etc., which means those methods can implement whatever is needed to give the namespace the desired semantics (within Python syntax).

Which, to some extent, is what will be used to transform an instance of "auto" into an appropriate integer.  In other words:

>>> class Color(Enum):
...     red = auto()
...     print(red)
...
1
>>> Color.red
<Color.red: 1>

New changeset aad7443e62be by Ethan Furman in branch 'default':
issue23591: add auto() for auto-generating Enum member values
https://hg.python.org/cpython/rev/aad7443e62be

> Which means it has methods such as __getitem__, __setitem__, etc., which means those methods can implement whatever is needed to give the namespace the desired semantics (within Python syntax).

Ah, _that_'s what you had in mind. (All this time, I thought _auto_ was just a shorter name for _generate_next_value_.:) I'm ok with that. But aren't we then back to square one? (Using magic of the same kind as the "declarative style" that Raymond pronounced not Pythonic enough.)

Even Raymond says
> As long as _auto_ has been defined somewhere (i.e. from enum import _auto_), it is normal Python

I'm sure he didn't think of the loophole you're currently exploiting. :-D

But anyway, I'm happy. Parentheses are there, so the intuition of calling a function to execute code is respected, and if you get out free with this hack, it opens a door a bit wider for complete declarative solution in the future.

>> Which means it has methods such as __getitem__, __setitem__, etc.,
>> which means those methods can implement whatever is needed to give
>> the namespace the desired semantics (within Python syntax).

> Ah, _that_'s what you had in mind. (All this time, I thought _auto_
> was just a shorter name for _generate_next_value_.:) I'm ok with that.
> But aren't we then back to square one? (Using magic of the same kind
> as the "declarative style" that Raymond pronounced not Pythonic enough.)

Not at all.  The concern with that proposal (issue26988: AutoEnum via completely omitting values of any kind) was the transformation of a NameError into a value.  The current method is taking a unique object and transforming that into a value, which is entirely analagous to what Enum already does.

Besides being convenient, it's also necessary to present a cohesive, non-leaky abstraction to the user where Flag is concerned: using ints as the flag values is entirely an implementation detail, but without auto the user would be forced to manage that implementation detail when creating the Flag, which negates much of Flag's value.

Many thanks to everyone for their input and help.

Now that I actually had the chance to play with the implementation, I see most of my worst fears were justified. :-( Look:

    >>> import enum
    >>> class B(enum.Flag):
            b = 3
            c = 4
            d = 6
    >>> B.b | B.c
    Traceback (most recent call last): ...
    ValueError: 7 is not a valid B
    >>> t = B.b | B.c
    >>> t
    <B.d|1: 7>
    >>> B.b | B.c
    >>> B.b | B.c
    <B.d|1: 7>
    >>> ~B.d
    <B.0: 0>

Do you really find this behavior acceptable?? I see at least three bugs here.

At least you did say in the documentation

> Individual flags should have values that are powers of two (1, 2, 4, 8, ...)

but it seems to me it's not prominent enough.

---

Also, auto, despite parentheses, works exactly as it shouldn't.

    >>> class C(enum.Enum):
	a = b = enum.auto()	
    >>> C.a
    <C.a: 1>
    >>> C.b
    <C.b: 2>

    >>> def f():
            return enum.auto()
    >>> class E(enum.Enum):
            a = b = f()
            c = d = 2
    >>> E.a is E.b
    False
    >>> E.c is E.d
    True
    >>> E.b is E.c
    True

In my opinion, this is simply horrible. Even _you_ said (http://bugs.python.org/issue23591#msg275093) this would be unacceptable. I'm really, really sad.

Vedran:

1) Thank you for the bug report, it's appreciated.

2) I would appreciate it even more if you just kept it to a bug report
   without the, for lack of a better word, drama.

-----------------------------------------------

>    >>> import enum
>    >>> class B(enum.Flag):
>            b = 3
>            c = 4
>            d = 6
>    >>> B.b | B.c
>    Traceback (most recent call last): ...
>    ValueError: 7 is not a valid B

This part is correct.  If the programmer wants to use actual values instead of auto() then it is on them to do it correctly.  I'd be happy to add a decorator to help ensure all bits have names, but I'm not sure what to call it.

-----------------------------------------------

>    >>> t = B.b | B.c
>    >>> t
>    <B.d|1: 7>

>    >>> ~B.d
>    <B.0: 0>

These parts are not, and will be fixed.

-----------------------------------------------

>    >>> class C(enum.Enum):
>	a = b = enum.auto()	
>    >>> C.a
>    <C.a: 1>
>    >>> C.b
>    <C.b: 2>

Bug, will be fixed.

>    >>> def f():
>            return enum.auto()
>    >>> class E(enum.Enum):
>            a = b = f()
>            c = d = 2
>    >>> E.a is E.b
>    False

Same as above bug, will be fixed

>    >>> E.c is E.d
>    True
>    >>> E.b is E.c
>    True

Same as above bug, will be fixed.  (Had "a" and "b" been given different "auto()"s that last True would be correct as "b" would then have had a value of 2 from auto(), and "c" and "d" also were given values of 2, so all three would have been the same.)

About drama: I don't know how to approach the problem. I tell you about various concerns of mine, you seem to agree, and still you do exactly what I was warning you about. It could be that you just consider me a boring nag, but then saying so would be more useful than this situation.

---

Yes, you solved some problems I did have, and it's great, but still many of them remain. This thing is inherently complicated. You are trying to construct a discrete Boolean algebra, but without any guarantee of atoms. Imagine if the language was different:

    class MySets(DiscreteSets):
        a = {0, 1}
        b = {2}
        c = {1, 2}

You'd naturally view {0}, {1} and {2} as atoms, of different "sort" than {0,1} and such. That's what's missing here: a clear distinction between atoms and "molecules". It could be that the intended usage is for atoms to _always_ be constructed with auto(), and for molecules to be constructed from previous atoms, but then the documentation ought to say so.

So far, the only thing you say is "then it is on them to do it correctly", but you never really tell what "correctly" means. I understand your desire not to hard forbid anything, but the docs should tell how your class is _intended_ to be used, _AND_ the class should make every reasonable effort to preserve what it can of its behavior even in the face of not-quite-intended usage. Look at Raymond's collections.Counter's treatment of non-integer values for example.

BTW I'm not aware of any stdlib object whose repr raises ValueError. It surely makes debugging a lot harder.

I'll respond to the bulk of your comment later.  For now let me assure you your feedback has been invaluable.

New changeset b56290a80ff7 by Ethan Furman in branch '3.6':
issue23591: fix flag decomposition and repr
https://hg.python.org/cpython/rev/b56290a80ff7

New changeset 7372c042e9a1 by Ethan Furman in branch 'default':
issue23591: fix flag decomposition and repr
https://hg.python.org/cpython/rev/7372c042e9a1

The "What’s New In Python 3.6" page should really get an update to mention Flag, IntFlag and auto as enhancements to the enum module. Right now, the ssl module update docs mentions (but does not properly link for some reason) IntFlag, but otherwise, there is no mention of the new feature in either the "What's New" docs or even in the theoretically comprehensive changelog; if you don't think to look at the enum module itself, you'd never know there were new features to use.

New changeset f404a59d834e by Ethan Furman in branch '3.6':
closes issue23591: add NEWS entry
https://hg.python.org/cpython/rev/f404a59d834e