As we're beginning to roll out macOS 10.14 High Sierra, we're seeing an increase in core files inside /cores. This can consume a ton of disk space if you have coredumps enabled. I don't have a short reproducer yet, but we believe this is related to changes in macOS behavior around fork(). Here are some clues:

ansible/ansible#32499 (comment)

https://stackoverflow.com/questions/50168647/multiprocessing-causes-python-to-crash-and-gives-an-error-may-have-been-in-progr/52230415#52230415

http://sealiesoftware.com/blog/archive/2017/6/5/Objective-C_and_fork_in_macOS_1013.html

That should of course be 10.14 Mojave.

Apparently setting the env variable OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES should fix some fork-related changes. Have you tried already doing this? Does it change the behaviour of the errors in any way?

On Nov 12, 2018, at 13:34, Pablo Galindo Salgado <report@bugs.python.org> wrote:

Apparently setting the env variable OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES should fix some fork-related changes. Have you tried already doing this? Does it change the behaviour of the errors in any way?

Yes, sorry I got distracted before I could add a comment about this. I have indeed tested this and it does prevent the core files.

Apparently, GitLab have a more permanent solution around this by enabling the Objective-C runtime before any forking happens:

https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/16649/diffs

This can a duplicate of https://bugs.python.org/issue33725

I'm not sure whether it's a duplicate or not, since in my case it doesn't hang, but instead core dumps. It's seems like the reasoning given in the Ruby bug is relevant to Python too, so maybe we should adopt the same workaround. For our internal projects, I'm probably just going to set the environment variable for now.

I'd love to know what Ned and Ronald thing, thus cc'ing them.

I've done a fair bit of testing and it seems rather inconsistent as to whether either of these work when added right before an explicit call to os.fork():

os.environ['OBJC_DISABLE_INITIALIZE_FORK_SAFETY'] = 'YES'

ctyles.cdll.LoadLibrary('/System/Library/Frameworks/Foundation.framework/Foundation')

Neither of these seems to reliably prevent the __NSPlaceholderDate warning, nor prevent the core dumps. The best I've been able to do is to prevent them by setting the environment variable *before* the parent process starts (i.e. outside of the Python code of the process).

Based on my testing, the environment variable has to be set before the parent process starts. Neither os.environ nor os.putenv seem to do the trick.

I'm still testing this solution, but it looks like if you set the environment variable, and then double fork, the granchild won't crash. Roughly:

os.putenv('OBJC_DISABLE_INITIALIZE_FORK_SAFETY', 'YES')
pid = os.fork()
if pid == 0:
    subpid = os.fork()
    if subpid == 0:
        # I'm in a safe grandchild

Nope, actually double fork doesn't work. It's misleading because in my testing, the first invocation of the process causes the core dump, but subsequent runs do not.

I am going to dupe this to 33725 after all.

Resolution is marked dupe but status is still open. Are we closing this one or is there a more specific remedy for this situation (as opposed to what bpo-33725 discusses) that would be helpful to document?

Closing in favor of bpo-33725