Author John Jones
Recipients John Jones, alex, benjamin.peterson, dhduvall, gennad, gregory.p.smith, neologix
Date 2017-03-17.18:49:37
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1489776577.7.0.530513240047.issue20104@psf.upfronthosting.co.za>
In-reply-to
Content
I agree with everything you're saying Gregory, however I don't think the significance of the memory doubling is as inconsequential as you might first think. For example, i have on my 64bit Linux system 128Gb of RAM, and a numpy table that's around 70Gb. Spawning a subprocess, even though memory is doubled for a very short period of time, is enough to raise a MemoryError, despite the subprocess i'm spawning using only 2 or 3Mb after the exec().

I do appreciate that for most Python users however, they will not see much benefit from what I imagine is quite a lot of development work.

FWIW, I did try the posix_spawn module, but i couldn't figure out how to write data to the stdin of a posix_spawn subprocess, and gave up in place of the commonly recommended solution to this problem (via StackExchange) of spawning lots of subprocesses before you put stuff in memory. Fortunately, for my problem, this was a possible solution. For others I think they're going to have to use posix_spawn, or an entirely different programming language if that doesn't work.
History
Date User Action Args
2017-03-17 18:49:37John Jonessetrecipients: + John Jones, gregory.p.smith, benjamin.peterson, alex, dhduvall, neologix, gennad
2017-03-17 18:49:37John Jonessetmessageid: <1489776577.7.0.530513240047.issue20104@psf.upfronthosting.co.za>
2017-03-17 18:49:37John Joneslinkissue20104 messages
2017-03-17 18:49:37John Jonescreate