classification
Title: segfault running code in jupyter on macOS 10.14.5 - crashed on child side of fork pre-exec
Type: crash Stage: resolved
Components: macOS Versions: Python 3.7
process
Status: closed Resolution: third party
Dependencies: Superseder:
Assigned To: Nosy List: ned.deily, ronaldoussoren, vstinner, wesinator
Priority: normal Keywords:

Created on 2019-07-16 16:56 by wesinator, last changed 2019-11-02 14:45 by ned.deily. This issue is now closed.

Files
File name Uploaded Description Edit
Python_2019-07-16-123850_MacBook-Pro.crash wesinator, 2019-07-16 16:56
Messages (5)
msg348032 - (view) Author: wesinator (wesinator) * Date: 2019-07-16 16:56
python 3.7.4 (homebrew) macOS 10.14.5
16 GB DDR4

segfault encountered when running test code in jupyter. 

Application Specific Information:
crashed on child side of fork pre-exec

I'm not in a position to share the code unfortunately, but I can say it was running multiple iterations of subprocess.Popen() within a multiprocessing Pool using os.cpu_count() / 2 number of processes (in this case 12/2 = 6). 


macOS .crash dump file attached
msg348033 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-07-16 17:01
See bpo-33725 and multiprocessing documentation:
https://docs.python.org/dev/library/multiprocessing.html#contexts-and-start-methods

"Changed in version 3.8: On macOS, the spawn start method is now the default. The fork start method should be considered unsafe as it can lead to crashes of the subprocess. See bpo-33725."

Change the start method to spawn.
msg355790 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-11-01 04:57
Looking at the macOS crash report you supplied (thanks!), the crash is occurring below libcurl which is called by the third-party pycurl module, so not in Python Standard Library code. And the crash appears to another of macOS's attempts to catch doing unsafe calls to system frameorks when forking withing an exec: note the "crashed on child side of fork pre-exec" message.  This is a well-know pitfall particularly on macOS which is why we changed the default for the multiprocessing module on macOS as Victor noted above.  There are various articles out there about the general problem, like https://www.evanjones.ca/fork-is-dangerous.html.  If you are still seeing the problem, you may need to restructure your application to avoid fork without exec.  Good luck!
msg355860 - (view) Author: wesinator (wesinator) * Date: 2019-11-02 04:01
Thanks ned - should I report issue to libcurl ?
msg355871 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-11-02 14:45
> should I report issue to libcurl

It wouldn't hurt although they probably are aware of the problem.
History
Date User Action Args
2019-11-02 14:45:15ned.deilysetmessages: + msg355871
2019-11-02 04:01:37wesinatorsetmessages: + msg355860
2019-11-01 04:57:17ned.deilysetstatus: open -> closed
resolution: third party
messages: + msg355790

stage: resolved
2019-07-16 17:01:17vstinnersetnosy: + vstinner
messages: + msg348033
2019-07-16 16:56:43wesinatorcreate