classification
Title: macOS 10.14 Mojave crashes in multiprocessing
Type: Stage: resolved
Components: Versions: Python 3.8, Python 3.7, Python 3.6
process
Status: closed Resolution: duplicate
Dependencies: Superseder: Python crashes on macOS after fork with no exec
View: 33725
Assigned To: Nosy List: barry, davin, ned.deily, pablogsal, ronaldoussoren
Priority: normal Keywords:

Created on 2018-11-12 17:57 by barry, last changed 2018-11-14 20:56 by barry. This issue is now closed.

Messages (14)
msg329747 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2018-11-12 17:57
As we're beginning to roll out macOS 10.14 High Sierra, we're seeing an increase in core files inside /cores.  This can consume a ton of disk space if you have coredumps enabled.  I don't have a short reproducer yet, but we believe this is related to changes in macOS behavior around fork().  Here are some clues:

https://github.com/ansible/ansible/issues/32499#issuecomment-341578864

https://stackoverflow.com/questions/50168647/multiprocessing-causes-python-to-crash-and-gives-an-error-may-have-been-in-progr/52230415#52230415

http://sealiesoftware.com/blog/archive/2017/6/5/Objective-C_and_fork_in_macOS_1013.html
msg329752 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2018-11-12 21:27
That should of course be 10.14 Mojave.
msg329753 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2018-11-12 21:34
Apparently setting the env variable OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES should fix some fork-related changes. Have you tried already doing this? Does it change the behaviour of the errors in any way?
msg329754 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2018-11-12 21:36
On Nov 12, 2018, at 13:34, Pablo Galindo Salgado <report@bugs.python.org> wrote:

> Apparently setting the env variable OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES should fix some fork-related changes. Have you tried already doing this? Does it change the behaviour of the errors in any way?

Yes, sorry I got distracted before I could add a comment about this.  I have indeed tested this and it does prevent the core files.
msg329757 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2018-11-12 21:42
Apparently, GitLab have a more permanent solution around this by enabling the Objective-C runtime before any forking happens: 

https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/16649/diffs
msg329758 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2018-11-12 21:45
This can a duplicate of https://bugs.python.org/issue33725
msg329760 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2018-11-12 21:50
I'm not sure whether it's a duplicate or not, since in my case it doesn't hang, but instead core dumps.  It's seems like the reasoning given in the Ruby bug is relevant to Python too, so maybe we should adopt the same workaround.  For our internal projects, I'm probably just going to set the environment variable for now.

I'd love to know what Ned and Ronald thing, thus cc'ing them.
msg329780 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2018-11-13 00:30
I've done a fair bit of testing and it seems rather inconsistent as to whether either of these work when added right before an explicit call to `os.fork()`:

os.environ['OBJC_DISABLE_INITIALIZE_FORK_SAFETY'] = 'YES'

ctyles.cdll.LoadLibrary('/System/Library/Frameworks/Foundation.framework/Foundation')

Neither of these seems to reliably prevent the __NSPlaceholderDate warning, nor prevent the core dumps.  The best I've been able to do is to prevent them by setting the environment variable *before* the parent process starts (i.e. outside of the Python code of the process).
msg329784 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2018-11-13 01:20
Based on my testing, the environment variable has to be set before the parent process starts.  Neither os.environ nor os.putenv seem to do the trick.
msg329795 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2018-11-13 02:00
I'm still testing this solution, but it looks like if you set the environment variable, and then double fork, the granchild won't crash.  Roughly:

os.putenv('OBJC_DISABLE_INITIALIZE_FORK_SAFETY', 'YES')
pid = os.fork()
if pid == 0:
    subpid = os.fork()
    if subpid == 0:
        # I'm in a safe grandchild
msg329796 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2018-11-13 02:07
Nope, actually double fork doesn't work.  It's misleading because in my testing, the first invocation of the process causes the core dump, but subsequent runs do not.
msg329869 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2018-11-13 21:49
I am going to dupe this to 33725 after all.
msg329928 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2018-11-14 19:47
Resolution is marked dupe but status is still open.  Are we closing this one or is there a more specific remedy for this situation (as opposed to what issue33725 discusses) that would be helpful to document?
msg329935 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2018-11-14 20:56
Closing in favor of issue33725
History
Date User Action Args
2018-11-14 20:56:16barrysetstatus: open -> closed

messages: + msg329935
stage: resolved
2018-11-14 19:47:15davinsetnosy: + davin
messages: + msg329928
2018-11-13 21:49:01barrysetsuperseder: Python crashes on macOS after fork with no exec
resolution: duplicate
messages: + msg329869
2018-11-13 02:07:54barrysetmessages: + msg329796
2018-11-13 02:00:10barrysetmessages: + msg329795
2018-11-13 01:20:06barrysetmessages: + msg329784
2018-11-13 00:30:11barrysetmessages: + msg329780
2018-11-12 21:50:02barrysetnosy: + ronaldoussoren, ned.deily
messages: + msg329760
2018-11-12 21:45:15pablogsalsetmessages: + msg329758
2018-11-12 21:42:47pablogsalsetmessages: + msg329757
2018-11-12 21:36:07barrysetmessages: + msg329754
2018-11-12 21:34:22pablogsalsetnosy: + pablogsal
messages: + msg329753
2018-11-12 21:27:14barrysetmessages: + msg329752
2018-11-12 21:27:00barrysettitle: macOS 10.14 High Sierra crashes in multiprocessing -> macOS 10.14 Mojave crashes in multiprocessing
2018-11-12 17:57:32barrycreate