Author serhiy.storchaka
Recipients Olivier.Grisel, artxyz, davin, mrjbq7, neologix, pitrou, sbt, serhiy.storchaka
Date 2017-03-13.19:04:58
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1489431898.57.0.702253307343.issue17560@psf.upfronthosting.co.za>
In-reply-to
Content
Pickle currently handle byte strings and unicode strings larger than 4GB only with protocol 4. But multiprocessing currently uses the default protocol which currently equals 3. There was suggestions to change the default pickle protocol (issue23403), the pickle protocol for multiprocessing (issue26507) or customize the serialization method for multiprocessing (issue28053). There is also a patch that implements the support of byte strings and unicode strings larger than 4GB with all protocols (issue25370).

Beside this I think that using some kind of shared memory is better way for transferring large data between subprocesses.
History
Date User Action Args
2017-03-13 19:04:58serhiy.storchakasetrecipients: + serhiy.storchaka, pitrou, mrjbq7, neologix, sbt, Olivier.Grisel, davin, artxyz
2017-03-13 19:04:58serhiy.storchakasetmessageid: <1489431898.57.0.702253307343.issue17560@psf.upfronthosting.co.za>
2017-03-13 19:04:58serhiy.storchakalinkissue17560 messages
2017-03-13 19:04:58serhiy.storchakacreate