Title: Opt out serialization/deserialization for heap type
Type: behavior Stage: patch review
Components: C API Versions: Python 3.10, Python 3.9
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: corona10, serhiy.storchaka, shihai1991, vstinner
Priority: normal Keywords: patch

Created on 2020-06-20 13:17 by corona10, last changed 2020-09-15 20:10 by serhiy.storchaka.

Pull Requests
URL Status Linked Edit
PR 21002 merged corona10, 2020-06-20 13:47
PR 21030 merged corona10, 2020-06-21 09:58
PR 22189 open corona10, 2020-09-10 14:48
Messages (4)
msg371937 - (view) Author: Dong-hee Na (corona10) * (Python committer) Date: 2020-06-20 13:17

We noticed that heap type has different behavior about serialization/deserialization.

Basically it can occur the regression issues.

Two things needed.
1. opt out serialization/deserialization for converted modules.
2. Add unit tests to check whether their serialization is blocked.
3. If the module is already ported to 3.9 backport patch is needed.
Long term
- Add the object.reduce() and/or update pickle can be smarter
msg371992 - (view) Author: Dong-hee Na (corona10) * (Python committer) Date: 2020-06-21 09:45
New changeset 6989af0bc7ea1e9a1acea16794e6f723d7b44110 by Dong-hee Na in branch 'master':
bpo-41052: Opt out serialization/deserialization for _random.Random (GH-21002)
msg371993 - (view) Author: Dong-hee Na (corona10) * (Python committer) Date: 2020-06-21 10:33
New changeset 814b07bf814a804f60b897d18f1dbb5578b2c7fd by Dong-hee Na in branch '3.9':
[3.9] bpo-41052: Opt out serialization/deserialization for _random.Random (GH-21002). (GH-21030)
msg376951 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-09-15 20:10
There are other heap types implemented in C in the stdlib and third-party libraries for which pickling with protocols 0 and 1 works incorrectly. It would be better to fix copyreg._reduce_ex() instead of disabling pickling for these types one by one.

From PEP 307:

> Let D be the class on the object to be pickled. First, find the nearest base class that is implemented in C (either as a built-in type or as a type defined by an extension class). Call this base class B, and the class of the object to be pickled D. Unless B is the class 'object', instances of class B must be picklable, either by having built-in support (as defined in the above three bullet points), or by having a non-default __reduce__ implementation. B must not be the same class as D (if it were, it would mean that D is not implemented in Python).

The problem is with determining which class is implemented in C. The current code implies that heap types are implemented in Python, and static types are implemented in C. It is not always true, because some heap types can be implemented in C.
Date User Action Args
2020-09-15 20:10:45serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg376951
2020-09-10 14:48:38corona10setpull_requests: + pull_request21250
2020-06-21 10:33:14corona10setmessages: + msg371993
2020-06-21 09:58:01corona10setpull_requests: + pull_request20201
2020-06-21 09:45:05corona10setmessages: + msg371992
2020-06-20 13:47:45corona10setkeywords: + patch
stage: patch review
pull_requests: + pull_request20177
2020-06-20 13:45:04corona10setnosy: + shihai1991
2020-06-20 13:17:29corona10create