Author neologix
Recipients mrjbq7, neologix, pitrou, sbt
Date 2013-03-27.17:47:25
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1364406446.23.0.751253038666.issue17560@psf.upfronthosting.co.za>
In-reply-to
Content
> Also, does pickle currently handle byte strings larger than 4GB?

The 2.7 failure is indeed a pickle limitation, which should now be fixed by issue #13555.

> On a machine with 256GB of RAM, it makes more sense to send arrays
> of this size than say on a laptop...

Richard was saying that you shouldn't serialize such a large array, that's just a huge performance bottleneck. The right way would be to use a shared memory.

> multiprocessing currently only allows sharing of such shared arrays
> using inheritance.

You mean through fork() COW?

>  Perhaps we need a picklable mmap type which can be sent over pipes
> and queues.  (On Unix this would probably require fd passing.)

If you use POSIX semaphores, you could pass the semaphore path and use sem_open in the other process (but that would mean you can't unlink it right after open).
History
Date User Action Args
2013-03-27 17:47:26neologixsetrecipients: + neologix, pitrou, mrjbq7, sbt
2013-03-27 17:47:26neologixsetmessageid: <1364406446.23.0.751253038666.issue17560@psf.upfronthosting.co.za>
2013-03-27 17:47:26neologixlinkissue17560 messages
2013-03-27 17:47:25neologixcreate