Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Opt out serialization/deserialization for heap type #85224

Closed
corona10 opened this issue Jun 20, 2020 · 10 comments
Closed

Opt out serialization/deserialization for heap type #85224

corona10 opened this issue Jun 20, 2020 · 10 comments
Labels
3.9 only security fixes 3.10 only security fixes topic-C-API type-bug An unexpected behavior, bug, or error

Comments

@corona10
Copy link
Member

BPO 41052
Nosy @pitrou, @vstinner, @serhiy-storchaka, @corona10, @shihai1991
PRs
  • bpo-41052: Opt out serialization/deserialization for _random.Random #21002
  • [3.9] bpo-41052: Opt out serialization/deserialization for _random.Random (GH-21002). #21030
  • bpo-41052: Optout serialization/deserialization for blake2s/b #22189
  • bpo-41052: Fix pickling heap types implemented in C with protocols 0 and 1 #22870
  • [3.9] bpo-41052: Fix pickling heap types implemented in C with protocols 0 and 1 (GH-22870). #22963
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2020-10-25.16:09:30.699>
created_at = <Date 2020-06-20.13:17:29.949>
labels = ['expert-C-API', 'type-bug', '3.9', '3.10']
title = 'Opt out serialization/deserialization for heap type'
updated_at = <Date 2020-10-27.04:22:53.436>
user = 'https://github.com/corona10'

    bugs.python.org fields:

    activity = <Date 2020-10-27.04:22:53.436>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2020-10-25.16:09:30.699>
closer = 'serhiy.storchaka'
components = ['C API']
creation = <Date 2020-06-20.13:17:29.949>
creator = 'corona10'
dependencies = []
files = []
hgrepos = []
issue_num = 41052
keywords = ['patch']
message_count = 10.0
messages = ['371937', '371992', '371993', '376951', '379214', '379223', '379251', '379532', '379585', '379735']
nosy_count = 5.0
nosy_names = ['pitrou', 'vstinner', 'serhiy.storchaka', 'corona10', 'shihai1991']
pr_nums = ['21002', '21030', '22189', '22870', '22963']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue41052'
versions = ['Python 3.9', 'Python 3.10']
    @corona10
    Copy link
    Member Author

    See https://bugs.python.org/issue40077#msg371813

    We noticed that heap type has different behavior about serialization/deserialization.

    Basically it can occur the regression issues.

    Two things needed.

    1. opt out serialization/deserialization for converted modules.
    2. Add unit tests to check whether their serialization is blocked.
    3. If the module is already ported to 3.9 backport patch is needed.
      Long term
    • Add the object.reduce() and/or update pickle can be smarter

    @corona10 corona10 added 3.9 only security fixes 3.10 only security fixes topic-C-API type-bug An unexpected behavior, bug, or error labels Jun 20, 2020
    @corona10
    Copy link
    Member Author

    New changeset 6989af0 by Dong-hee Na in branch 'master':
    bpo-41052: Opt out serialization/deserialization for _random.Random (GH-21002)
    6989af0

    @corona10
    Copy link
    Member Author

    New changeset 814b07b by Dong-hee Na in branch '3.9':
    [3.9] bpo-41052: Opt out serialization/deserialization for _random.Random (GH-21002). (GH-21030)
    814b07b

    @serhiy-storchaka
    Copy link
    Member

    There are other heap types implemented in C in the stdlib and third-party libraries for which pickling with protocols 0 and 1 works incorrectly. It would be better to fix copyreg._reduce_ex() instead of disabling pickling for these types one by one.

    From PEP-307:

    Let D be the class on the object to be pickled. First, find the nearest base class that is implemented in C (either as a built-in type or as a type defined by an extension class). Call this base class B, and the class of the object to be pickled D. Unless B is the class 'object', instances of class B must be picklable, either by having built-in support (as defined in the above three bullet points), or by having a non-default __reduce__ implementation. B must not be the same class as D (if it were, it would mean that D is not implemented in Python).

    The problem is with determining which class is implemented in C. The current code implies that heap types are implemented in Python, and static types are implemented in C. It is not always true, because some heap types can be implemented in C.

    @corona10
    Copy link
    Member Author

    It would be better to fix copyreg._reduce_ex() instead of disabling pickling for these types one by one

    I agree :)

    @pitrou
    Copy link
    Member

    pitrou commented Oct 21, 2020

    At some point, pickle protocols 0 and 1 should be deprecated and slated for removal. Protocol 2 is 17 years ago already, and protocol 3 has become the default in Python 3.

    That would probably be a better use of contributor time than trying to make heap types compatible with those obsolete protocols.

    @serhiy-storchaka
    Copy link
    Member

    I came to the same conclusion after trying to fix it. But we cannot just do it now. We have to fix bugs in Python 3.9 and support protocols 0 and 1 for some deprecation period.

    Also protocols 0 and 1 are used in some third-party implementations for other programming languages, so they can be used for interoperability with other languages.

    The proposed patch adds additional check. It uses the fact that the __new__ attribute for classes implemented in C is a bultin function with __self__ pointing to this class (it is created in tp_new_wrapper()).

    @serhiy-storchaka
    Copy link
    Member

    New changeset 8cd1dba by Serhiy Storchaka in branch 'master':
    bpo-41052: Fix pickling heap types implemented in C with protocols 0 and 1 (GH-22870)
    8cd1dba

    @serhiy-storchaka
    Copy link
    Member

    New changeset 0aaecb3 by Serhiy Storchaka in branch '3.9':
    [3.9] bpo-41052: Fix pickling heap types implemented in C with protocols 0 and 1 (GH-22870). (GH-22963)
    0aaecb3

    @vstinner
    Copy link
    Member

    Thanks for the fix Serhiy!

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    @vstinner vstinner mentioned this issue Jun 22, 2022
    Closed
    This was referenced May 7, 2023
    Merged
    Closed
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.9 only security fixes 3.10 only security fixes topic-C-API type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    4 participants
    @vstinner @pitrou @serhiy-storchaka @corona10