Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve pickling of typing types #77833

Closed
serhiy-storchaka opened this issue May 26, 2018 · 8 comments
Closed

Improve pickling of typing types #77833

serhiy-storchaka opened this issue May 26, 2018 · 8 comments
Labels
3.7 (EOL) end of life 3.8 only security fixes stdlib Python modules in the Lib dir

Comments

@serhiy-storchaka
Copy link
Member

BPO 33652
Nosy @gvanrossum, @ned-deily, @serhiy-storchaka, @ilevkivskyi, @miss-islington
PRs
  • bpo-33652: Improve pickle support in the typing module. #7123
  • [3.7] bpo-33652: Improve pickle support in the typing module. (GH-7123) #7132
  • bpo-33652: Remove __getstate__ and __setstate__ methods in typing. #7144
  • [3.7] bpo-33652: Remove __getstate__ and __setstate__ methods in typing. (GH-7144) #7148
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2018-05-28.12:57:46.782>
created_at = <Date 2018-05-26.07:10:15.083>
labels = ['3.8', '3.7', 'library']
title = 'Improve pickling of typing types'
updated_at = <Date 2018-05-28.12:57:46.781>
user = 'https://github.com/serhiy-storchaka'

    bugs.python.org fields:

    activity = <Date 2018-05-28.12:57:46.781>
actor = 'levkivskyi'
assignee = 'none'
closed = True
closed_date = <Date 2018-05-28.12:57:46.782>
closer = 'levkivskyi'
components = ['Library (Lib)']
creation = <Date 2018-05-26.07:10:15.083>
creator = 'serhiy.storchaka'
dependencies = []
files = []
hgrepos = []
issue_num = 33652
keywords = ['patch']
message_count = 8.0
messages = ['317727', '317752', '317753', '317766', '317768', '317837', '317839', '317846']
nosy_count = 5.0
nosy_names = ['gvanrossum', 'ned.deily', 'serhiy.storchaka', 'levkivskyi', 'miss-islington']
pr_nums = ['7123', '7132', '7144', '7148']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue33652'
versions = ['Python 3.7', 'Python 3.8']
    @serhiy-storchaka
    Copy link
    Member Author

    The following PR makes pickles for typing types more portable. Type variables no longer use _find_name() and can be unpickled in 3.6. Subscripted generics no longer expose internals and can be unpickled in 3.6 and future versions with changed internal implementation.

    Before this PR:

    >>> import pickle, pickletools, typing                                            
>>> pickletools.dis(pickletools.optimize(pickle.dumps(typing.T, 4)))              
    0: \x80 PROTO      4
    2: \x95 FRAME      30
   11: \x8c SHORT_BINUNICODE 'typing'
   19: \x94 MEMOIZE    (as 0)
   20: \x8c SHORT_BINUNICODE '_find_name'
   32: \x93 STACK_GLOBAL
   33: h    BINGET     0
   35: \x8c SHORT_BINUNICODE 'T'
   38: \x86 TUPLE2
   39: R    REDUCE
   40: .    STOP
highest protocol among opcodes = 4
>>> pickletools.dis(pickletools.optimize(pickle.dumps(typing.Union[int, str], 4)))
    0: \x80 PROTO      4
    2: \x95 FRAME      198
   11: \x8c SHORT_BINUNICODE 'copyreg'
   20: \x8c SHORT_BINUNICODE '_reconstructor'
   36: \x93 STACK_GLOBAL
   37: \x8c SHORT_BINUNICODE 'typing'
   45: \x94 MEMOIZE    (as 0)
   46: \x8c SHORT_BINUNICODE '_GenericAlias'
   61: \x93 STACK_GLOBAL
   62: \x8c SHORT_BINUNICODE 'builtins'
   72: \x94 MEMOIZE    (as 1)
   73: \x8c SHORT_BINUNICODE 'object'
   81: \x93 STACK_GLOBAL
   82: N    NONE
   83: \x87 TUPLE3
   84: R    REDUCE
   85: }    EMPTY_DICT
   86: (    MARK
   87: \x8c     SHORT_BINUNICODE '_inst'
   94: \x88     NEWTRUE
   95: \x8c     SHORT_BINUNICODE '_special'
  105: \x89     NEWFALSE
  106: \x8c     SHORT_BINUNICODE '_name'
  113: N        NONE
  114: \x8c     SHORT_BINUNICODE '__origin__'
  126: h        BINGET     0
  128: \x8c     SHORT_BINUNICODE 'Union'
  135: \x93     STACK_GLOBAL
  136: \x8c     SHORT_BINUNICODE '__args__'
  146: h        BINGET     1
  148: \x8c     SHORT_BINUNICODE 'int'
  153: \x93     STACK_GLOBAL
  154: h        BINGET     1
  156: \x8c     SHORT_BINUNICODE 'str'
  161: \x93     STACK_GLOBAL
  162: \x86     TUPLE2
  163: \x8c     SHORT_BINUNICODE '__parameters__'
  179: )        EMPTY_TUPLE
  180: \x8c     SHORT_BINUNICODE '__slots__'
  191: N        NONE
  192: \x8c     SHORT_BINUNICODE '__module__'
  204: h        BINGET     0
  206: u        SETITEMS   (MARK at 86)
  207: b    BUILD
  208: .    STOP
highest protocol among opcodes = 4

    With this PR:

    >>> import pickle, pickletools, typing                                                              
>>> pickletools.dis(pickletools.optimize(pickle.dumps(typing.T, 4)))                       
    0: \x80 PROTO      4
    2: \x95 FRAME      13
   11: \x8c SHORT_BINUNICODE 'typing'
   19: \x8c SHORT_BINUNICODE 'T'
   22: \x93 STACK_GLOBAL
   23: .    STOP
highest protocol among opcodes = 4
>>> pickletools.dis(pickletools.optimize(pickle.dumps(typing.Union[int, str], 4)))       
    0: \x80 PROTO      4
    2: \x95 FRAME      66
   11: \x8c SHORT_BINUNICODE '_operator'
   22: \x8c SHORT_BINUNICODE 'getitem'
   31: \x93 STACK_GLOBAL
   32: \x8c SHORT_BINUNICODE 'typing'
   40: \x8c SHORT_BINUNICODE 'Union'
   47: \x93 STACK_GLOBAL
   48: \x8c SHORT_BINUNICODE 'builtins'
   58: \x94 MEMOIZE    (as 0)
   59: \x8c SHORT_BINUNICODE 'int'
   64: \x93 STACK_GLOBAL
   65: h    BINGET     0
   67: \x8c SHORT_BINUNICODE 'str'
   72: \x93 STACK_GLOBAL
   73: \x86 TUPLE2
   74: \x86 TUPLE2
   75: R    REDUCE
   76: .    STOP
highest protocol among opcodes = 4

    If there is a chance it would be nice to merge these changes into 3.7. Otherwise either pickles created in 3.7.0 will be incompatible not only with older Python versions, but with future Python versions too, or we will need to add complex code for supporting specific 3.7.0 pickles in future versions.

    @serhiy-storchaka serhiy-storchaka added 3.7 (EOL) end of life 3.8 only security fixes stdlib Python modules in the Lib dir labels May 26, 2018
    @ilevkivskyi
    Copy link
    Member

    New changeset 09f3221 by Ivan Levkivskyi (Serhiy Storchaka) in branch 'master':
    bpo-33652: Improve pickle support in the typing module. (GH-7123)
    09f3221

    @miss-islington
    Copy link
    Contributor

    New changeset d498625 by Miss Islington (bot) in branch '3.7':
    bpo-33652: Improve pickle support in the typing module. (GH-7123)
    d498625

    @serhiy-storchaka
    Copy link
    Member Author

    There is a question -- what to do with all these __getstate__ and __setstate__ methods? They are part of the pickle protocol, but they are not used when define __reduce__. And they are not needed for supporting compatibility with older versions, because in 3.6 pickleable generic types were pickled by name. I think it is better to remove these methods.

    @ilevkivskyi
    Copy link
    Member

    Yes, these are just legacy from times when TypeVars were serialized by value, not by identity like now. I think it should be safe to remove them. Would you like to make a PR?

    @ilevkivskyi
    Copy link
    Member

    New changeset 97b523d by Ivan Levkivskyi (Serhiy Storchaka) in branch 'master':
    bpo-33652: Remove __getstate__ and __setstate__ methods in typing. (GH-7144)
    97b523d

    @miss-islington
    Copy link
    Contributor

    New changeset 98b089e by Miss Islington (bot) in branch '3.7':
    bpo-33652: Remove __getstate__ and __setstate__ methods in typing. (GH-7144)
    98b089e

    @ilevkivskyi
    Copy link
    Member

    I think this can be closed now.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.7 (EOL) end of life 3.8 only security fixes stdlib Python modules in the Lib dir
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    3 participants
    @serhiy-storchaka @ilevkivskyi @miss-islington