Message329868
## Context
We are developers of nipype (https://github.com/nipype) which is a workflow engine for neuroimaging software. We are experiencing problems that gave rise to the addition of ``os.posix_spawn`` to Python 3.8, and particularly, this - https://bugs.python.org/issue20104#msg222570
Our software runs command line subprocesses that can be quite memory-hungry and in some cases, in the order of tens of thousands processes. Therefore, we frequently see the OOM killing some of the processes.
## Status
We have successfully leveraged the ``forkserver`` context (in addition to a low number of `maxtasksperchild`) of multiprocessing to ease the load. However, the fork_exec memory allocation is still problematic on systems that do not allow overcommitting virtual memory. Waiting for os.posix_spawn to be rolled out might not be an option for us, as the problem is hitting badly right now.
## Proposed solution
I'd like to page experts on Lib/multiprocessing and Lib/subprocess to give their opinions about the following: is it possible to write an extension to `multiprocessing.util.Popen` such that it has the API of `subprocess.Popen` but the fork happens via the forkserver?
My naive intuition is that we would need to create a new type of Process, make sure that it then calls os.exec*e() --possibly around here https://github.com/python/cpython/blob/f966e5397ed8f5c42c185223fc9b4d750a678d02/Lib/multiprocessing/popen_forkserver.py#L51--, and finally handle communication with the subprocess.
Please let me know if that is even possible. |
|
Date |
User |
Action |
Args |
2018-11-13 21:29:55 | oesteban | set | recipients:
+ oesteban |
2018-11-13 21:29:55 | oesteban | set | messageid: <1542144595.84.0.788709270274.issue35238@psf.upfronthosting.co.za> |
2018-11-13 21:29:55 | oesteban | link | issue35238 messages |
2018-11-13 21:29:55 | oesteban | create | |
|