Author sbt
Recipients mrjbq7, pitrou, sbt
Date 2013-03-27.17:11:43
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1364404303.34.0.316220484918.issue17560@psf.upfronthosting.co.za>
In-reply-to
Content
> I *think* we need to keep compatibility with the wire format, but perhaps 
> we could use a special length value (-1?) to introduce a longer (64-bit) 
> length value.

Yes we could, although that would not help on Windows pipe connections (where byte oriented messages are used instead).  Also, does pickle currently handle byte strings larger than 4GB?

But I can't help feeling that multigigabyte arrays should be transferred using shared mmaps rather than serialization.  numpy.frombuffer() could be used to recreate the array from the mmap.

multiprocessing currently only allows sharing of such shared arrays using inheritance.  Perhaps we need a picklable mmap type which can be sent over pipes and queues.  (On Unix this would probably require fd passing.)
History
Date User Action Args
2013-03-27 17:11:43sbtsetrecipients: + sbt, pitrou, mrjbq7
2013-03-27 17:11:43sbtsetmessageid: <1364404303.34.0.316220484918.issue17560@psf.upfronthosting.co.za>
2013-03-27 17:11:43sbtlinkissue17560 messages
2013-03-27 17:11:43sbtcreate