New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
enum.IntFlag regression: missing values cause TypeError #88408
Comments
With the change introduced in https://bugs.python.org/issue38250 7aaeb2a I observe a regression in behavior of enum.IntFlag with missing values. Consider this code (from pyproj): from enum import IntFlag
class GeodIntermediateFlag(IntFlag):
DEFAULT = 0x0
NPTS_MASK = 0xF
NPTS_ROUND = 0x0
NPTS_CEIL = 0x1
NPTS_TRUNC = 0x2
DEL_S_MASK = 0xF0
DEL_S_RECALC = 0x00
DEL_S_NO_RECALC = 0x10
AZIS_MASK = 0xF00
AZIS_DISCARD = 0x000
AZIS_KEEP = 0x100 This is a valid code in Python 3.9, however produces a TypeError in Python 3.10.0b1: Traceback (most recent call last):
File "intflag.py", line 3, in <module>
class GeodIntermediateFlag(IntFlag):
File "/usr/lib64/python3.10/enum.py", line 544, in __new__
raise TypeError(
TypeError: invalid Flag 'GeodIntermediateFlag' -- missing values: 4, 8, 32, 64, 128, 512, 1024, 2048 Since I don't see this behavior mentioned in https://docs.python.org/3.10/library/enum.html or https://docs.python.org/3.10/whatsnew/3.10.html or https://docs.python.org/3.10/whatsnew/changelog.html I believe this is a regression. |
We are already blocked in Python 3.10 beta 2, Ethan, could you give a look at this so we can introduce the fix when we release the next beta? |
That is an intentional change. The cause is that the masks include bits that are not named in the Flag. The user-side fix is to add a class GeodIntermediateFlag(IntFlag, boundary=KEEP) The enum library fix could be one of two things:
I'm inclined to go with option 2, since |
To clarify, it's caused by these mask entries in the enum: NPTS_MASK = 0xF
DEL_S_MASK = 0xF0
AZIS_MASK = 0xF00 Since the masks are not an aggregation of individual bits defined in the enum, it's an error.
I think that would cause various problems in the API and implementation. For example, without underlying individual bits, repr() may be nonsensical. |
I wonder if CONFORM could tolerate these, since it's ultimately going to discard invalid bits. And then perhaps CONFORM is the default. |
Actually, thinking about that a little bit more, KEEP was added for exactly this situation, as some stdlib flags exhibit the same behavior. So the real question is what should happen with, for example, GeodIntermediateFlag(0x80) ? The idea behind boundary is what should happen when values are created that don't have names in the Enum/Flag? The options for boundary are: STRICT -> an error is raised (default for Enum) So KEEP is currently doing double-duty -- this reinforces my desire to go with option 2 and return KEEP to single-duty status. |
Those are good points -- the difficulty is knowing which behavior the user wants. And if the desired run-time behavior doesn't match the boundary flag the user is stuck. |
For example, if the default is CONFORM or KEEP, but the user wants an error if 0x80 comes up, they would have to explicitly check for that value since the Flag would happily return it instead of raising. |
Rather than make such masks containing unknown bits, this would be best practice if you want to use STRICT, correct? NPTS_ROUND = 0x0
NPTS_CEIL = 0x1
NPTS_TRUNC = 0x2
NPTS_MASK = NPTS_ROUND | NPTS_CEIL | NPTS_TRUNC Otherwise, if your input may have unknown bits, use CONFORM. |
Yes, that would be best practice. However, if the user is interfacing with other software/hardware, they may not have a choice on which bits make up the mask. |
I think it's the same as saying "unknown bits may come on the input". An important use case is for forward compatibility, where new bits may be added but you don't want to break existing code. That is what CONFORM is for. The cited code should either declare the masks as I've shown if STRICT is desired, or use CONFORM if forward compatibility is desired. |
That could be, and the user can set the boundary to whatever works best for their use-case, so long as the boundary they want to use does not conflict with the initial creation checks. Do you agree that simply removing the unnamed member check that takes place at Flag creation is a good way forward? |
It's uncomfortable, because then STRICT is not actually strict, and it's going to be show raw values in str/repr.
This behavior of CONFORM is a little dubious. I expect it to conform new values after the class is constructed. But the class members themselves should not have that transform applied, and raise an error on invalid bits. |
I see your point about the str/repr.
But I'm not sure I understand that. Either you are agreeing with me that we should lose the creation time check (not apply the transform), or we should still have the check (raise an error on invalid bits) which would still leave us in this situation. |
I'll be offline for a couple hours, but I'll check back. |
That is only one option (which undesirable consequences are already identified). We should consider a few options carefully. From the start, if default Enum declarations are going to be subject to new runtime exceptions, it's certain that they'll be hit in some existing code. Anything other than the default being legacy-compatible, or defining the new stuff as Enum2, or deferring all the changes to Python 4, is going to lead to bug reports and disgruntled users. I don't think that suggests we should remove the runtime exceptions-- they are what supports the new API and implementation. Perhaps KEEP should be renamed to LEGACY, that should be the default, and there should be guidance on how to get the KEEP use cases you've found migrated to one of the other modes. (And changing the default could be considered for Python 4.)
It's an important point that should be fixed in the code. CONFORM should be regarding transforming external values into valid member instances. The enum author's declaration itself should be coherent and not rely on such transforms. Users will be surprised when they define a "FOO = 0xFF" and they read the value back as 0x1F or something. The runtime exception lets the enum author know about the issue in the declaration, and normalize it. |
I'm very much of the practicality beats purity philosophy, so I want to support the OP's flag without making them jump through hoops. On the flip side, I also appreciate that there are folks that want the extra security... So here's my plan: remove the creation time check (which will remove the error for the OP), and then add a decorator to the enum module that does that creation time check -- just like the Now I just need a good name for that decorator:
|
This change also affects PyQt6: Python 3.10.0b2+ (heads/3.10:d0991e2db3, Jun 1 2021, 11:42:08) [GCC 11.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import PyQt6.QtCore
>>> PyQt6.QtCore.PYQT_VERSION_STR
'6.1.1.dev2105301600'
>>> PyQt6.QtCore.Qt.Key
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/tcaswell/.pybuild/bleeding/lib/python3.10/enum.py", line 615, in __call__
return cls._create_(
File "/home/tcaswell/.pybuild/bleeding/lib/python3.10/enum.py", line 762, in _create_
return metacls.__new__(metacls, class_name, bases, classdict, boundary=boundary)
File "/home/tcaswell/.pybuild/bleeding/lib/python3.10/enum.py", line 544, in __new__
raise TypeError(
TypeError: invalid Flag 'DropAction' -- missing values: 8, 16, 32, 64, 128, 32768
>>>
I am very supportive of this plan! Could this be labeled as a higher priority / release blocker? |
Raising this to a release blocker for 3.10.0 beta 3 as it breaks PyQt6 and other projects. |
Thank you everyone for your help. |
Also changing error reporting to be less susceptible to DOS attacks. |
With the followup patch merged, can this be closed now? |
Yup, just had to get back from the weekend. :-) |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: