> Also, does pickle currently handle byte strings larger than 4GB?

The 2.7 failure is indeed a pickle limitation, which should now be fixed by issue #13555.

> On a machine with 256GB of RAM, it makes more sense to send arrays
> of this size than say on a laptop...

Richard was saying that you shouldn't serialize such a large array, that's just a huge performance bottleneck. The right way would be to use a shared memory.

> multiprocessing currently only allows sharing of such shared arrays
> using inheritance.

You mean through fork() COW?

>  Perhaps we need a picklable mmap type which can be sent over pipes
> and queues.  (On Unix this would probably require fd passing.)

If you use POSIX semaphores, you could pass the semaphore path and use sem_open in the other process (but that would mean you can't unlink it right after open).