Author oesteban
Recipients oesteban
Date 2018-11-13.21:29:55
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1542144595.84.0.788709270274.issue35238@psf.upfronthosting.co.za>
In-reply-to
Content
## Context

We are developers of nipype (https://github.com/nipype) which is a workflow engine for neuroimaging software. We are experiencing problems that gave rise to the addition of ``os.posix_spawn`` to Python 3.8, and particularly, this - https://bugs.python.org/issue20104#msg222570

Our software runs command line subprocesses that can be quite memory-hungry and in some cases, in the order of tens of thousands processes. Therefore, we frequently see the OOM killing some of the processes.

## Status

We have successfully leveraged the ``forkserver`` context (in addition to a low number of `maxtasksperchild`) of multiprocessing to ease the load. However, the fork_exec memory allocation is still problematic on systems that do not allow overcommitting virtual memory. Waiting for os.posix_spawn to be rolled out might not be an option for us, as the problem is hitting badly right now.

## Proposed solution

I'd like to page experts on Lib/multiprocessing and Lib/subprocess to give their opinions about the following: is it possible to write an extension to `multiprocessing.util.Popen` such that it has the API of `subprocess.Popen` but the fork happens via the forkserver?

My naive intuition is that we would need to create a new type of Process, make sure that it then calls os.exec*e() --possibly around here https://github.com/python/cpython/blob/f966e5397ed8f5c42c185223fc9b4d750a678d02/Lib/multiprocessing/popen_forkserver.py#L51--, and finally handle communication with the subprocess.

Please let me know if that is even possible.
History
Date User Action Args
2018-11-13 21:29:55oestebansetrecipients: + oesteban
2018-11-13 21:29:55oestebansetmessageid: <1542144595.84.0.788709270274.issue35238@psf.upfronthosting.co.za>
2018-11-13 21:29:55oestebanlinkissue35238 messages
2018-11-13 21:29:55oestebancreate