Author oesteban
Recipients oesteban
Date 2018-11-13.21:29:55
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <>
## Context

We are developers of nipype ( which is a workflow engine for neuroimaging software. We are experiencing problems that gave rise to the addition of ``os.posix_spawn`` to Python 3.8, and particularly, this -

Our software runs command line subprocesses that can be quite memory-hungry and in some cases, in the order of tens of thousands processes. Therefore, we frequently see the OOM killing some of the processes.

## Status

We have successfully leveraged the ``forkserver`` context (in addition to a low number of `maxtasksperchild`) of multiprocessing to ease the load. However, the fork_exec memory allocation is still problematic on systems that do not allow overcommitting virtual memory. Waiting for os.posix_spawn to be rolled out might not be an option for us, as the problem is hitting badly right now.

## Proposed solution

I'd like to page experts on Lib/multiprocessing and Lib/subprocess to give their opinions about the following: is it possible to write an extension to `multiprocessing.util.Popen` such that it has the API of `subprocess.Popen` but the fork happens via the forkserver?

My naive intuition is that we would need to create a new type of Process, make sure that it then calls os.exec*e() --possibly around here, and finally handle communication with the subprocess.

Please let me know if that is even possible.
Date User Action Args
2018-11-13 21:29:55oestebansetrecipients: + oesteban
2018-11-13 21:29:55oestebansetmessageid: <>
2018-11-13 21:29:55oestebanlinkissue35238 messages
2018-11-13 21:29:55oestebancreate