Title: Add support of pickling very large bytes and str objects with protocol < 4
Type: enhancement Stage: patch review
Components: Extension Modules, Library (Lib) Versions: Python 3.6
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: alexandre.vassalotti, i3v, pitrou, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2015-10-10 20:03 by serhiy.storchaka, last changed 2018-04-08 16:55 by pitrou.

File name Uploaded Description Edit
pickle_large_strings_2.patch serhiy.storchaka, 2015-10-14 10:14 review
Messages (4)
msg252745 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-10-10 20:03
Pickle protocols < 4 allow to pickle bytes and str object with 32-bit size. One of features of protocol 4 was the support of 64-bit sizes.

It is possible to add support of very large bytes and str objects with protocols 2 and 3. The idea is that the long string is splitted on chunks with 32-bit sizes and the resulting string is reconstructed with calling ''.join(). The main part of the feature can be implemented in Python as helpers in copyreg.

The question is: is it worthwhile to implement this feature? I already implemented two other features of protocol 4 for protocols 2 and 3: nested objects and __new__ with keyword arguments.
msg252956 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-10-13 19:44
Here is preliminary patch. It doesn't include changes to tests. Some bigmem tests need to be changed.
msg252983 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-10-14 10:14
Changed tests and fixed bugs.

Could anyone please test the patch with bigmem pickle tests? Tests need at least 32GB of memory, I have no such much memory.
msg315087 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2018-04-08 16:55
Serhiy, now that protocol 4 is the default, this isn't needed anymore, right?
Date User Action Args
2018-04-08 16:55:06pitrousetmessages: + msg315087
2018-04-08 16:53:05i3vsetnosy: + i3v
2015-10-14 10:14:25serhiy.storchakasetfiles: - pickle_large_strings.patch
2015-10-14 10:14:13serhiy.storchakasetfiles: + pickle_large_strings_2.patch

messages: + msg252983
2015-10-13 19:44:06serhiy.storchakasetfiles: + pickle_large_strings.patch
keywords: + patch
messages: + msg252956

stage: patch review
2015-10-10 20:03:10serhiy.storchakacreate