Author dtasev
Recipients davin, dtasev, gdr@garethrees.org, sbt, serhiy.storchaka
Date 2017-07-14.08:50:58
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1500022259.0.0.0311394295533.issue30919@psf.upfronthosting.co.za>
In-reply-to
Content
If I understand correctly, there is no way to force the old behaviour in Python 3.5, i.e. to use an anonymous memory mapping in multiprocessing.heap.Arena so that the memory can be shared between the processes instead of writing to a shared file?

The data sizes usually are, on average, 20 GB, and writing it out to a file is not desirable. As I understand from Gareth Rees' comment, ftruncate() will speed up initialisation, however any processing afterwards would be IO capped.

To shed more light to the processing going on, the data is handled as a 3D array, so each process gets a 2D array to operate on, and no information needs to be shared between processes.

If the anonymous memory mapping cannot be forced, then the multiprocessing module with a shared array becomes unusable for me. Are you aware of any way to use the multiprocessing module to run execution in parallel, that somehow doesn't use a shared array?
History
Date User Action Args
2017-07-14 08:50:59dtasevsetrecipients: + dtasev, sbt, gdr@garethrees.org, serhiy.storchaka, davin
2017-07-14 08:50:59dtasevsetmessageid: <1500022259.0.0.0311394295533.issue30919@psf.upfronthosting.co.za>
2017-07-14 08:50:58dtasevlinkissue30919 messages
2017-07-14 08:50:58dtasevcreate