Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

enum with inherited type won't pickle #88508

Closed
TomBrown mannequin opened this issue Jun 8, 2021 · 10 comments
Closed

enum with inherited type won't pickle #88508

TomBrown mannequin opened this issue Jun 8, 2021 · 10 comments
Assignees
@ethanfurman
Labels
3.8 only security fixes 3.9 only security fixes 3.10 only security fixes 3.11 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@TomBrown
Copy link
Mannequin

TomBrown mannequin commented Jun 8, 2021

BPO 44342
Nosy @ethanfurman, @miss-islington
PRs
  • bpo-44342: [Enum] change pickling from by-value to by-name #26658
  • [3.10] bpo-44342: [Enum] changed pickling from by-value to by-name (GH-26658) #26660
  • [3.9] bpo-44342: [Enum] improve pickle support #26666
  • bpo-44342: [Enum] fix data type search #26667
  • bpo-44342: [Enum] improve test, add andrei kulakov to ACKS #26726
  • [3.10] bpo-44342: [Enum] fix data type search (GH-26667) #26746
  • [3.10] bpo-44342: [Enum] improve test, add andrei kulakov to ACKS (GH-26726) #26747
  • bpo-44342: [Enum] sync current docs to 3.10 #26750
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/ethanfurman'
closed_at = <Date 2021-06-19.21:53:03.719>
created_at = <Date 2021-06-08.00:56:53.414>
labels = ['type-bug', '3.8', '3.9', '3.10', '3.11', 'library']
title = "enum with inherited type won't pickle"
updated_at = <Date 2021-06-19.21:53:03.718>
user = 'https://bugs.python.org/TomBrown'

    bugs.python.org fields:

    activity = <Date 2021-06-19.21:53:03.718>
actor = 'ethan.furman'
assignee = 'ethan.furman'
closed = True
closed_date = <Date 2021-06-19.21:53:03.719>
closer = 'ethan.furman'
components = ['Library (Lib)']
creation = <Date 2021-06-08.00:56:53.414>
creator = 'Tom.Brown'
dependencies = []
files = []
hgrepos = []
issue_num = 44342
keywords = ['patch']
message_count = 10.0
messages = ['395300', '395578', '395585', '395589', '395609', '395613', '395615', '395898', '395902', '395903']
nosy_count = 3.0
nosy_names = ['ethan.furman', 'Tom.Brown', 'miss-islington']
pr_nums = ['26658', '26660', '26666', '26667', '26726', '26746', '26747', '26750']
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue44342'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10', 'Python 3.11']
    @TomBrown
    Copy link
    Mannequin Author

    TomBrown mannequin commented Jun 8, 2021

    The following script runs without error in 3.8.5 and raises an error in 3.8.6, 3.9.5 and 3.10.0b1.

    Source:

    import enum, pickle

class MyInt(int):
    pass
    # work-around: __reduce_ex__ = int.__reduce_ex__

class MyEnum(MyInt, enum.Enum):
    A = 1

pickle.dumps(MyEnum.A)

    Error (same in 3.8.6, 3.9.5 and 3.10.0b1):

    Traceback (most recent call last):
  File "/home/thecap/projects/covid-data-model/./enum-pickle.py", line 12, in <module>
    pickle.dumps(MyEnum.A)
  File "/home/thecap/.pyenv/versions/3.10.0b1/lib/python3.10/enum.py", line 83, in _break_on_call_reduce
    raise TypeError('%r cannot be pickled' % self)
TypeError: MyEnum.A cannot be pickled

    Like https://bugs.python.org/issue41889 this seems to be related to the fix for https://bugs.python.org/issue39587 which changes member_type from int to MyInt. A work-around is in the comment above.

    @TomBrown TomBrown mannequin added 3.10 only security fixes 3.8 only security fixes 3.9 only security fixes stdlib Python modules in the Lib dir labels Jun 8, 2021
    @ethanfurman ethanfurman added 3.11 only security fixes labels Jun 10, 2021
    @ethanfurman
    Copy link
    Member

    Looking into this I think the root of the problem is the way reduce is handled -- currently, Enum's __reduce_ex__ works by returning the class, and the value to use to lookup the member. Because that lookup can fail with complex enums, EnumType will sabotage reduce if it can't find support in the new enum class.

    However, if __reduce_ex__ working by returning

    `getattr, (self.__class, self._name_)`

    then we should be fine, as that should never fail.

    @ethanfurman
    Copy link
    Member

    New changeset 62f1d2b by Ethan Furman in branch 'main':
    bpo-44342: [Enum] changed pickling from by-value to by-name (GH-26658)
    62f1d2b

    @ethanfurman
    Copy link
    Member

    New changeset b613132 by Miss Islington (bot) in branch '3.10':
    bpo-44342: [Enum] changed pickling from by-value to by-name (GH-26658) (GH-26660)
    b613132

    @ethanfurman
    Copy link
    Member

    Changing __reduce_ex__ is too big a change for 3.9, so making the search for pickle support more robust there.

    @ethanfurman
    Copy link
    Member

    New changeset 3a7cccf by Ethan Furman in branch 'main':
    bpo-44342: [Enum] fix data type search (GH-26667)
    3a7cccf

    @ethanfurman
    Copy link
    Member

    Also fixing the search for the data type so that in

    A -> B -> C

    the data type is C and not B.

    @ethanfurman
    Copy link
    Member

    New changeset 0f99324 by Miss Islington (bot) in branch '3.10':
    bpo-44342: [Enum] fix data type search (GH-26667)
    0f99324

    @ethanfurman
    Copy link
    Member

    New changeset 41c2a4a by Ethan Furman in branch '3.10':
    [3.10] bpo-44342: [Enum] improve test, add andrei kulakov to ACKS (GH-26726)
    41c2a4a

    @ethanfurman
    Copy link
    Member

    New changeset 741b8ae by Ethan Furman in branch 'main':
    bpo-44342: [Enum] sync current docs to 3.10 (GH-26750)
    741b8ae

    @ethanfurman ethanfurman added the type-bug An unexpected behavior, bug, or error label Jun 19, 2021
    @ethanfurman ethanfurman added the type-bug An unexpected behavior, bug, or error label Jun 19, 2021
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees

    @ethanfurman ethanfurman

    Labels
    3.8 only security fixes 3.9 only security fixes 3.10 only security fixes 3.11 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    1 participant
    @ethanfurman