classification
Title: multiprocessing's default start method of fork()-without-exec() is broken
Type: behavior Stage:
Components: Versions: Python 3.9, Python 3.8, Python 3.7, Python 3.6, Python 3.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Julian, davin, itamarst, itamarst2, pitrou
Priority: normal Keywords:

Created on 2020-04-24 18:22 by itamarst, last changed 2020-11-06 22:02 by itamarst2.

Messages (4)
msg367210 - (view) Author: Itamar Turner-Trauring (itamarst) Date: 2020-04-24 18:22
By default, multiprocessing uses fork() without exec() on POSIX. For a variety of reasons this can lead to inconsistent state in subprocesses: module-level globals are copied, which can mess up logging, threads don't survive fork(), etc..

The end results vary, but quite often are silent lockups.

In real world usage, this results in users getting mysterious hangs they do not have the knowledge to debug.

The fix for these people is to use "spawn" by default, which is the default on Windows.

Just a small sample:

1. Today I talked to a scientist who spent two weeks stuck, until she found my article on the subject (https://codewithoutrules.com/2018/09/04/python-multiprocessing/). Basically multiprocessing locked up, doing nothing forever. Switching to "spawn" fixed it.
2. https://github.com/dask/dask/issues/3759#issuecomment-476743555 is someone who had issues fixed by "spawn".
3. https://github.com/numpy/numpy/issues/15973 is a NumPy issue which apparently impacted scikit-learn.


I suggest changing the default on POSIX to match Windows.
msg367211 - (view) Author: Itamar Turner-Trauring (itamarst) Date: 2020-04-24 18:31
Looks like as of 3.8 this only impacts Linux/non-macOS-POSIX, so I'll amend the above to say this will also make it consistent with macOS.
msg368173 - (view) Author: Itamar Turner-Trauring (itamarst) Date: 2020-05-05 15:35
Just got an email from someone for whom switching to "spawn" fixed a problem. Earlier this week someone tweeted about this fixing things. This keeps hitting people in the real world.
msg380478 - (view) Author: Itamar Turner-Trauring (itamarst2) Date: 2020-11-06 22:02
Another person with the same issue: https://twitter.com/volcan01010/status/1324764531139248128
History
Date User Action Args
2020-11-06 22:02:52itamarst2setnosy: + itamarst2
messages: + msg380478
2020-06-24 22:12:37Juliansetnosy: + Julian
2020-05-06 01:33:18ned.deilysetnosy: + pitrou, davin
2020-05-05 15:35:22itamarstsetmessages: + msg368173
2020-04-24 18:31:05itamarstsetmessages: + msg367211
2020-04-24 18:22:23itamarstcreate