classification
Title: Python crashes on macOS after fork with no exec
Type: Stage: patch review
Components: macOS Versions: Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: barry, davin, gregory.p.smith, josh.r, kapilt, lukasz.langa, miss-islington, ned.deily, pablogsal, pitrou, ronaldoussoren, tdsmith, vstinner
Priority: critical Keywords: patch

Created on 2018-06-01 00:53 by kapilt, last changed 2019-06-17 10:19 by ned.deily.

Pull Requests
URL Status Linked Edit
PR 11043 merged ned.deily, 2018-12-09 06:30
PR 11044 merged miss-islington, 2018-12-09 06:50
PR 11045 merged miss-islington, 2018-12-09 06:50
PR 13603 merged vstinner, 2019-05-27 23:26
PR 13626 closed vstinner, 2019-05-28 14:13
PR 13841 merged vstinner, 2019-06-05 12:05
PR 13849 merged miss-islington, 2019-06-05 20:01
Messages (61)
msg318352 - (view) Author: Kapil Thangavelu (kapilt) Date: 2018-06-01 00:53
This issue seems to be reported a few times on various githubs projects. I've also reproduced using a brew install of python 2.7.15. I haven't been able to reproduce with python 3.6. Note this requires a framework build of python.

Background on the underlying issue cause due to a change in high Sierra 
http://sealiesoftware.com/blog/archive/2017/6/5/Objective-C_and_fork_in_macOS_1013.html
A ruby perspective on the same issue exhibiting for some apps
https://blog.phusion.nl/2017/10/13/why-ruby-app-servers-break-on-macos-high-sierra-and-what-can-be-done-about-it/


The work around seems to be setting an environment variable OBJC_DISABLE_INITIALIZE_FORK_SAFETY prior to executing python.

Other reports

https://bugs.python.org/issue30837
https://github.com/ansible/ansible/issues/32499
https://github.com/imWildCat/scylla/issues/22
https://github.com/elastic/beats-tester/pull/73
https://github.com/jhaals/ansible-vault/issues/60
msg318361 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2018-06-01 05:51
A better solution is to avoid using fork mode for multiprocessing. The spawn and fork server modes should work fine. 

The underlying problem is that macOS system frameworks (basically anything higher level than libc) are not save wrt fork(2) and fixing that appears to have no priority at all at Apple.
msg318396 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-06-01 10:57
(As a side note, the macOS Pythons provided by python.org installers should not behave differently on macOS 10.13 High Sierra since none of them are built with a 10.13 SDK.)
msg318397 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2018-06-01 11:10
I understand that Apple, with their limited resources, cannot spend expensive engineer manpower on improving POSIX support in macOS </snark>.

In any case, I'm unsure this bug can be fixed at the Python level.  If macOS APIs don't like fork(), they don't like fork(), point bar.  As Ronald says, on 3.x you should use "forkserver" (for multiple reasons, not only this issue).  On 2.7 you're stuck dealing with the issue by yourself.
msg318528 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2018-06-03 08:47
Antoine, the issue is not necessarily related to POSIX compliance, AFAIK strictly POSIX compliant code should work just fine. The problem is in higher-level APIs (CoreFoundation, Foundation, AppKit, ...), and appears to be related to using multi-threading in those libraries without spending effort on pre/post fork handlers to ensure that new processes are in a sane state after fork().  In older macOS versions this could result in hard to debug issues, in newer versions APIs seem to guard against this by aborting when the detect that the pid changed.

Anyways... I agree that we shouldn't try to work around this in CPython, there's bound to more problems that are hidden with the proposed workaround.

---

<http://www.sealiesoftware.com/blog/archive/2017/6/5/Objective-C_and_fork_in_macOS_1013.html> describes what the environment variable does, and this "just" changes behavior of the ObjC runtime, and doesn't make using macOS system frameworks after a fork saver.
msg318529 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2018-06-03 08:51
@Ned: In the long run the macOS installers should be build using the latest SDK, primarily to get full API coverage and access to all system APIs.

AFAIK building using the macOS 10.9 SDK still excludes a number of libSystem APIs that would be made available through the posix module when building with a newer SDK. 

That's something that would require some effort though to ensure that the resulting binary still works on older versions of macOS (basically similar to the work I've done in the post to weak link some other symbols in the posix module).
msg318708 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-06-04 21:39
{Note: this is not particularly relevant to the issue here.)

Ronald:
> In the long run the macOS installers should be build using the latest SDK [...] That's something that would require some effort though to ensure that the resulting binary still works on older versions of macOS

I agree that being able to build with the latest SDK would be nice but it's also true it would require effort on our part, both one-time and ongoing, at least for every new macOS SDK release and update to test with each older system.  It would also require that the third-party libraries we build for an installer also behave correctly.  And to make full use of it, third-party Python packages with extension modules would also need to behave correctly.  I see one of the primary use cases for the python.org macOS installers as being for Python app developers who want to provide apps that run on a range of macOS releases.  It seems to me that the safest and simplest way to guarantee that python.org macOS Pythons fulfill that need is to continue to always build them on the oldest supported system.  Yes, that means that users may miss out on a few features only supported on the more recent macOS releases but I think that's the right trade-off until we have the resources to truly investigate and decide to support weak linking from current systems.
msg329871 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2018-11-13 21:50
issue35219 is where I've run into this problem.  I'm still trying to figure out all the details in my own case, but I can confirm that setting the environment variable does not always help.
msg329880 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2018-11-14 01:14
Hoo boy.  I'm not sure I have the full picture, but things are starting to come into focus.  After much debugging, I've narrowed down at least one crash to urllib.request.getproxies().  On macOS (darwin), this ends up calling _scproxy.get_proxies() which calls into the SystemConfiguration framework.  I'll bet dollars to donuts that that calls into the ObjC runtime.  Thus it is unsafe to call between fork and exec.  This certainly seems to be the case even if the environment variable is set.

The problem is that I think requests.post() probably also ends up in here somehow (still untraced), because by removing our call to urllib.requests.getproxies(), we just crash later on when requests.post() is called.

I don't know what, if anything can be done in Python, except perhaps to document that anything that calls into the ObjC runtime between fork and exec can potentially crash the subprocess.
msg329885 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2018-11-14 01:36
A few other things I don't understand:

* Why does setting OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES only seem to work when it's set in the shell before the parent process executes?  AFAICT, it does *not* work if you set that in os.environ in the parent process before the os.fork().

* Why does it only crash on the first invocation of our app?  Does getproxies() cache the results somehow?  There's too much internal application code in the way to know if we're doing something that prevents getproxies() from getting called in subsequent calls.

* I can't seem to produce a smaller test case.
msg329919 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2018-11-14 17:52
FWIW, I suspect that setting the environment variable only helps if it's done before the process starts.  You cannot set it before the fork and have it affect the child.
msg329922 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2018-11-14 18:11
Barry's effort as well as comments in other links seem to all suggest that OBJC_DISABLE_INITIALIZE_FORK_SAFETY is not comprehensive in its ability to make other threads "safe" before forking.

"Objective-C classes defined by the OS frameworks remain fork-unsafe" (from @kapilt's first link) suggests we furthermore remain at risk using certain MacOS system libraries prior to any call to fork.

"To guarantee that forking is safe, the application must not be running any threads at the point of fork" (from @kapilt's second link) is an old truth that we continue to fight with even when we know very well that it's the truth.

For newly developed code, we have the alternative to employ spawn instead of fork to avoid these problems in Python, C, Ruby, etc.  For existing legacy code that employed fork and now surprises us by failing-fast on MacOS 10.13 and 10.14, it seems we are forced to face a technical debt incurred back when the choice was first made to spin up threads and afterwards to use fork.

If we didn't already have an "obvious" (zen of Python) way to avoid such problems with spawn versus fork, I would feel this was something to solve in Python.  As to helping the poor unfortunate souls who must fight the good fight with legacy code, I am not sure what to do to help though I would like to be able to help.
msg329923 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2018-11-14 18:16
Legacy code is easy to migrate as long as it uses Python 3.  Just call

  mp.set_start_method('forkserver')

at the top of your code and you're done.  Some use cases may fail (if sharing non-picklable types), but they're probably not very common.
msg329926 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-11-14 18:32
_scproxy has been known to be problematic for some time, see for instance Issue31818.  That issue also gives a simple workaround: setting urllib's "no_proxy" environment variable to "*" will prevent the calls to the System Configuration framework.
msg329927 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2018-11-14 19:42
Given the original post mentioned 2.7.15, I wonder if it is feasible to fork near the beginning of execution, then maintain and pass around a multiprocessing.Pool to be used when needed instead of dynamically forking?  Working with legacy code is almost always more interesting than you want it to be.
msg329933 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2018-11-14 20:43
On Nov 14, 2018, at 10:11, Davin Potts <report@bugs.python.org> wrote:
> 
> 
> Davin Potts <python@discontinuity.net> added the comment:
> 
> Barry's effort as well as comments in other links seem to all suggest that OBJC_DISABLE_INITIALIZE_FORK_SAFETY is not comprehensive in its ability to make other threads "safe" before forking.

Right.  Setting the env var will definitely not make it thread safe.  My understanding (please correct me if I’m wrong!) isn’t that this env var makes it safe, just that it prevents the ObjC runtime from core dumping.  So it’s still up to the developer to know whether threads are involved or not.  In our cases, these are single threaded applications.  I’ve read elsewhere that ObjC doesn’t care if threads have actually been spun up or not.

> "Objective-C classes defined by the OS frameworks remain fork-unsafe" (from @kapilt's first link) suggests we furthermore remain at risk using certain MacOS system libraries prior to any call to fork.

Actually, it’s unsafe to call anything between fork and exec.  Note that this doesn’t just affect Python; this is a pretty common idiom in other scripting languages too, from what I can tell.  It’s certainly very common in Python.

Note too that urllib.request.getproxies() will end up calling into the ObjC runtime via _scproxy, so you can’t even use requests after a fork but before exec.

What I am still experimenting with is to see if I can define a pthread_atfork handler that will initialize the ObjC runtime before fork is actually called.  I saw a Ruby approach like this, but it’s made more difficult in Python because pthread_atfork isn’t exposed to Python.  I’m trying to see if I can implement it in ctypes, before I write an extension.

> "To guarantee that forking is safe, the application must not be running any threads at the point of fork" (from @kapilt's second link) is an old truth that we continue to fight with even when we know very well that it's the truth.

True, but do realize this problem affects you even in single threaded applications.

> For newly developed code, we have the alternative to employ spawn instead of fork to avoid these problems in Python, C, Ruby, etc.  For existing legacy code that employed fork and now surprises us by failing-fast on MacOS 10.13 and 10.14, it seems we are forced to face a technical debt incurred back when the choice was first made to spin up threads and afterwards to use fork.

It’s tech debt you incur even if you don’t spin up threads.  Just fork and do some work in the child before calling exec.  If that work enters the ObjC runtime (as in the getproxies example), your child will coredump,

> If we didn't already have an "obvious" (zen of Python) way to avoid such problems with spawn versus fork, I would feel this was something to solve in Python.  As to helping the poor unfortunate souls who must fight the good fight with legacy code, I am not sure what to do to help though I would like to be able to help.

*If* we can provide a hook to initialize the ObjC runtime in pthread_atfork, I think that’s something we could expose in Python.  Then we can say legacy code can just invoke that, and at least you will avoid the worst outcome.
msg329941 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2018-11-15 03:22
I have a reliable way to call *something* in the pthread_atfork prepare handler, but I honestly don't know what to call to prevent the crash.

In the Ruby thread, it seemed to say that you could just dlopen /System/Library/Frameworks/Foundation.framework/Foundation but that does not work for me.  Neither does also loading the CoreFoundation and SystemConfiguration frameworks.

If anybody has something that will reliably initialize the runtime, I can post my approach (there are a few subtleties).  Short of that, I think there's nothing that can be done except ensure that exec is called right after fork.
msg331101 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2018-12-05 11:52
AFAIK there is nothing you can do between after calling fork(2) to "reinitialise" the ObjC runtime. And I don't think that's the issue anyway: I suspect that the actual problem is that Apple's system frameworks use multithreading (in particular libdispatch) and don't have code to ensure a sane state after calling fork. 

In Python 3 there is another workaround to avoid problems using multiprocessing: use multiprocessing.set_start_method() to switch away from the "fork" startup handler to "spawn" or "forkserver" (the latter only when calling set_start_method before calling any code that might call into Apple system frameworks.
msg331406 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-12-09 06:50
New changeset ac218bc5dbfabbd61c76ce8a17de088611e21981 by Ned Deily in branch 'master':
bpo-33725: skip test_multiprocessing_fork on macOS (GH-11043)
https://github.com/python/cpython/commit/ac218bc5dbfabbd61c76ce8a17de088611e21981
msg331407 - (view) Author: miss-islington (miss-islington) Date: 2018-12-09 07:06
New changeset d4bcf13e06d33b8ec66a68db20df34a029e66882 by Miss Islington (bot) in branch '3.7':
bpo-33725: skip test_multiprocessing_fork on macOS (GH-11043)
https://github.com/python/cpython/commit/d4bcf13e06d33b8ec66a68db20df34a029e66882
msg331409 - (view) Author: miss-islington (miss-islington) Date: 2018-12-09 07:11
New changeset df5d884defc8f1a94013ff9beb493f1428bd55b5 by Miss Islington (bot) in branch '3.6':
bpo-33725: skip test_multiprocessing_fork on macOS (GH-11043)
https://github.com/python/cpython/commit/df5d884defc8f1a94013ff9beb493f1428bd55b5
msg331411 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-12-09 07:33
Since it looks like multiprocessing_fork is not going to be fixable for macOS, the main issue remaining is how to help users avoid this trap (literally).  Should we add a check and issues a warning or error at run time?  Or is a doc change sufficient?

In the meantime, I've merged changes to disable running test_multiprocessing_fork which will sometimes (but not always) segfault on 10.14 Mojave.  I should apologize to Barry and others who have run into this.  I did notice the occasional segfault when testing with Mojave just prior to its release but it wasn't always reproducible and I didn't follow up on it.  Now that the change in 10.14 behavior makes this existing problem with fork no exec more obvious, it's clear that the test segfaults are another manifestation of this.
msg331435 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2018-12-09 15:13
Do we really need to disable the running of test_multiprocessing_fork entirely on MacOS?

My understanding so far is that not *all* of the system libraries on the mac are spinning up threads and so we should expect that there are situations where fork alone may be permissible, but of course we don't yet know what those are.  Pragmatically speaking, I have not yet seen a report of test_multiprocessing_fork tests triggering this problem but I would like to see/hear that when it is observed (that's my pitch for leaving the tests enabled).
msg331438 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2018-12-09 15:45
@ned.deily: Apologies, I misread what you wrote -- I would like to see the random segfaults that you were seeing on Mojave if you can still point me to a few.
msg331459 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2018-12-10 01:42
I think it make sense to disable this test; the only possible modification would be to only disable it for macOS <= 10.13.  AFAIK, that's the first version where core dumps were possible.  (Aside: I also saw these core dumps for a long time on 10.13 and never associated it with fork-without-exec in Python code.  Clearly, Apple has not done a good enough job of advertising this change.)

I think it is useful to help users on macOS avoid these problematic idioms, via documentation and defaults.  I think there's no way to predict when the core dumps will happen.  With internal cases, I've seen repeated invocations of the same code only core dump on the first run of the process, and not subsequent ones, for reasons I do not understand.  There seems to be a lot of mystery here, and without some explicit help from Apple, we're just doing our best to guess.
msg331610 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-12-11 11:12
Would it be safe to run the multiprocessing tests on recent macOS with the OBJC_DISABLE_INITIALIZE_FORK_SAFETY environment variable set?
msg331733 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-12-13 01:59
> Would it be safe to run the multiprocessing tests on recent macOS with the OBJC_DISABLE_INITIALIZE_FORK_SAFETY environment variable set?

See Ronald's reply above in msg331101. I believe his point is that there is nothing you can do to make this safe. And it's not a new problem with 10.14 or 10.13. What is new is that Apple is trying to more forcefully make you aware of the danger by causing the runtime to try to catch and crash these cases earlier rather than permit them to perhaps silently cause failures later.
msg331735 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2018-12-13 02:24
On Dec 12, 2018, at 17:59, Ned Deily <report@bugs.python.org> wrote:
> 
> Ned Deily <nad@python.org> added the comment:
> 
>> Would it be safe to run the multiprocessing tests on recent macOS with the OBJC_DISABLE_INITIALIZE_FORK_SAFETY environment variable set?
> 
> See Ronald's reply above in msg331101. I believe his point is that there is nothing you can do to make this safe. And it's not a new problem with 10.14 or 10.13. What is new is that Apple is trying to more forcefully make you aware of the danger by causing the runtime to try to catch and crash these cases earlier rather than permit them to perhaps silently cause failures later.

In my experiments at least, setting the env var *does* prevent the crash, but it doesn’t avoid the undefined semantics (i.e. what happens when the ObjC runtime is called at that point?) and I fully expect that Apple will remove that bandaid at some point.

The other key thing is that I don’t believe you can set the env var *in process* and have it take effect after the fork.  It must be set before the parent process starts.  So that probably makes it less useful for the multiprocessing tests by itself.
msg337587 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-03-10 01:49
What's the status of this issue?  As far as I know, we haven't come up with any workaround (see msg331011 above) other than disabling our tests.  If we don't have anything better, at the very least we should add a warning to the documentation to avoid fork mode on macOS, no?
msg337591 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2019-03-10 02:58
As best as I can see, there is no magic bullet to help mitigate this.

At a minimum, I am convinced we need to update the documentation to describe this behavior on MacOS and recommend alternatives.

I continue to give serious thought to the idea of changing the default start method on MacOS from fork to spawn.  This would be a breaking change though one could argue MacOS has already undergone a breaking change.  Is such a change warranted?

The alternative (which does not seem all that appealing) is that we start encouraging everyone to first consider the start method before attempting to use multiprocessing even for their first time.  Providing sensible defaults is to be preferred, but changing the default to reflect a non-trivial change in the underlying platform is still not to be taken lightly.
msg337733 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-03-12 11:07
Changing the default is trivial and safe. Let's do that. Additional doc wouldn't hurt.
msg338819 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2019-03-25 19:08
It's trivial but not safe in the sense that code that previously depended on some global state setup done in the master process right before fork will stop working. If this code is a library that is not in your control, you might not be able to "just revert" to fork mode easily.

And that change should be made for every platform so it will affect a much broader group of users than are affected by *this* issue.

Another thing is that some application packagers (at least Facebook's XAR but probably many others) don't work with "start" by default.

So don't take this lightly, just as Davin is saying. That being said, it's probably wise to change the default to "start" which is a better method. And if we *are* changing, doing it sooner rather than later makes the most sense.
msg338873 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-03-26 12:29
> And that change should be made for every platform so it will affect a much broader group of users than are affected by *this* issue.

Do you mean to stop using fork on Linux as well? Using fork is just fine on Linux, no?
msg341452 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2019-05-05 15:25
I believe we must change the default behavior on MacOS to use spawn instead of fork.  Encouraging people to use fork by default on MacOS is encouraging them to create something that effectively will not work.  Keeping fork as the default behavior when we have already turned off all of the tests of fork behavior on MacOS also makes no sense.  Existing Python code that depends upon the default behavior (fork) on MacOS has already been broken -- if we make this change, we are arguably not breaking anyone's working code.

Users can and will still be able to specify the start mechanism on MacOS, including fork.  This empowers users to continue to handle even the most esoteric use cases without loss of functionality from multiprocessing.  Though admittedly, without an ability to test the behavior of fork, this will need to be marked as deprecated.

I will supply a patch making this change and updating the docs shortly after PyCon.
msg341455 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2019-05-05 15:44
Victor raises an important question:  should the *default* start behavior be made consistent across platforms?  Assuming we change it on MacOS, the default start behavior on Windows and MacOS will be spawn but the default start behavior on Linux and FreeBSD (among others) will be fork.

Reasons to consider such a breaking change:
* This inconsistency in default start behavior on different platforms (Windows versus not) has historically been a significant source of confusion for many, many users.
* These days, the majority of users are not already familiar with the rule "fork-before-creating-threads" and so are surprised and confused when they fork a process that already has spun up multiple threads and bad things happen.
* We are changing the default on one platform (MacOS), which should prompt us to consider how are defaults are set elsewhere.

Reasons to reject such a breaking change:
* Though changing the default does not break everyone's code everywhere, it will require changes to any code that depends upon the default start method AND depends upon data/functions/stuff from the parent to also be present in the forked child process.
msg341475 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2019-05-05 21:41
On May 5, 2019, at 11:44, Davin Potts <report@bugs.python.org> wrote:
> 
> Victor raises an important question:  should the *default* start behavior be made consistent across platforms?

Yes, I think it should.  The pros of consistency and correctness outweigh the breaking change IMHO.
msg342042 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-05-10 02:15
I have no preference for the default on Linux, but please fix the default at least on macOS before Python 3.8 beta 1 ;-)
msg342071 - (view) Author: Josh Rosenberg (josh.r) * (Python triager) Date: 2019-05-10 11:07
I've seen far too many cases where Python code targeting Linux intentionally uses the COW benefits of fork for multiprocessing to think it would be a good idea to change the default start method there without *some* sort of deprecation period.
msg342412 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2019-05-14 01:39
On May 10, 2019, at 04:07, Josh Rosenberg <report@bugs.python.org> wrote:
> 
> I've seen far too many cases where Python code targeting Linux intentionally uses the COW benefits of fork for multiprocessing to think it would be a good idea to change the default start method there without *some* sort of deprecation period.

We need to resolve this for 3.8, and given that I think we have clear consensus to change the default on macOS to spawn to avoid the crashes, let’s do that.  We’ll need to update the documentation.

Then if we don’t have consensus to change the default on Linux, let’s issue a DeprecationWarning for the default ‘fork’ method in 3.8 and change it to ‘spawn' in 3.9.

Do we want to issue a warning on set_start_method(‘fork’) on macOS, given that it’s unlikely to be safe?
msg343704 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-05-27 23:28
I don't see a clear consensus to switch to spawn on *all* platforms, so I wrote PR 13603 which is the minimum fix: switch to spawn by default, but only on macOS.

If this PR is merged, I understand that it should be applied to 2.7 and 3.7 branches as well.
msg343773 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-05-28 12:54
As soon as the "fork" start method "just works" on Linux, I'm not sure why you want to use "spawn" by default on Linux.

I don't buy the "consistency" argument here. I'm not surprised that such "low level" module like multiprocessing behaves differently depending on the platform to get the "native" and most efficient way to spawn subprocesses.

In general, fork is bad, but it's also convenient and people rely on it to prepare data in a main process and then "duplicate" the process to inherit cooked data.

If someone wants to more away from fork by default, I would suggest to *first* enhance multiprocessing documentation to list issues caused by fork(), and maybe also described when fork is safe (never doesn't sound like a good answer, since so far, I'm not aware of tons of bug reports on Linux caused by fork :-)).
msg343779 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-05-28 14:02
New changeset 17a5588740b3d126d546ad1a13bdac4e028e6d50 by Victor Stinner in branch 'master':
bpo-33725: multiprocessing uses spawn by default on macOS (GH-13603)
https://github.com/python/cpython/commit/17a5588740b3d126d546ad1a13bdac4e028e6d50
msg343782 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-05-28 14:14
Ok, I pushed a minimum fix to make sure that it lands in Python 3.8 beta1. Please check if the documentation if properly updated :-)

We will still have time after beta1 to revisit the question of making spawn the default on more/all platforms.

I wrote PR 13626 to change also Python 3.7. Once this one will be merged, I will also write a PR for Python 2.7.
msg343807 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2019-05-28 17:07
On May 27, 2019, at 16:28, STINNER Victor <report@bugs.python.org> wrote:

> I don't see a clear consensus to switch to spawn on *all* platforms, so I wrote PR 13603 which is the minimum fix: switch to spawn by default, but only on macOS.

Fair enough.  Let’s fix what we have consensus on and go from there.  Thanks for working on this!
msg343826 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2019-05-28 23:18
There is a multiprocessing spawn method on Python 2.7.  And we should never do such a crazy huge feature backport.

IMNSHO - We should not change the default in 3.7 either.  That is a notable behavior change.  That decision is entirely up to Ned (RM).  If people are using an OS that changes behavior out from underneath their application between minor OS version upgrades, they should take that up with the OS vendor.  We cannot workaround this fundamental problem.

Applications and libraries will still need to support a slew of Python versions and OS versions, so they're all going to need to be modified to explicitly request the "spawn" method from multiprocessing on macOS anyways.  So there seems little point in changing the default within a patch release 3.7.4.
msg343828 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2019-05-28 23:20
FWIW I am in favor of "spawn" being the default on _all_ platforms in 3.8.  The safest option being the default just seems like the right thing to do.
msg343830 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-05-28 23:26
GPS beat me to it: this definitely should not be backported to either 3.7 or 2.7 since it is a major user behavior change.
msg343832 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-05-28 23:33
Ned Deily:
> GPS beat me to it: this definitely should not be backported to either 3.7 or 2.7 since it is a major user behavior change.

Hum ok, but test_multiprocessing_fork is now always skipped on macOS in Python 3.7.

Can we at least document that the default start method (fork) is now unsafe on macOS? Maybe also explain how to change the default start method on macOS.
msg343833 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-05-28 23:43
> Can we at least document that the default start method (fork) is now unsafe on macOS?

Thanks, I was just going to add that I would accept a doc change for 3.7.  But the wording should be a little clearer that fork has *always* been unsafe on macOS, i.e. this is not a new issue for 3.7 which is one of the reasons it should not be backported.
msg343838 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2019-05-29 00:06
On May 28, 2019, at 16:43, Ned Deily <report@bugs.python.org> wrote:

> Thanks, I was just going to add that I would accept a doc change for 3.7.  But the wording should be a little clearer that fork has *always* been unsafe on macOS, i.e. this is not a new issue for 3.7 which is one of the reasons it should not be backported.

To be clear, what is unsafe on macOS (as of 10.13, but even more so on 10.14) is calling into the Objective-C runtime between fork and exec.  The problem for Python is that it’s way too easy to do that implicitly, thus causing the macOS to abort the subprocess in surprising ways.
msg343841 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2019-05-29 00:17
Documentation of the issue for sure!

A RuntimeWarning is a maybe if it can be made conditional on the OS version (10.13 and higher?)... when the "fork" method would be used.  BUT be very cautious about warnings as they tend to crop up in front of users rather than developers when it isn't their fault.  So if there is any flaw in the logic of when to show it in a situation where the application isn't about to crash or lockup, it'll show up when it shouldn't and cause somebody some grief to deal with via new code to ignore that warning.
msg343842 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-05-29 00:21
> To be clear, what is unsafe on macOS (as of 10.13, but even more so on 10.14) is calling into the Objective-C runtime between fork and exec.  The problem for Python is that it’s way too easy to do that implicitly, thus causing the macOS to abort the subprocess in surprising ways.

Do only a few Python module use the Objective-C runtime? Or is it basically "everything"?

If it's just a few, would it be possible to emit a warning or even an exception if called in a child process after fork?
msg343844 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-05-29 00:39
> To be clear, what is unsafe on macOS (as of 10.13, but even more so on 10.14) is calling into the Objective-C runtime between fork and exec.

No, it has *always* been unsafe. What's new as of 10.13/14 is that macOS tries much harder at runtime to detect such cases and more predictably cause an error rather than let the process run on and possibly fail nondeterministically. 

> Do only a few Python module use the Objective-C runtime? Or is it basically "everything"?

I don't think we should try to second-guess this.  We now recognize that using fork like this on macOS has always been dangerous.  For some programs it will be fine, for others it won't.  People have had many macOS and Python releases to deal with this; if it works for their application, we shouldn't be changing the default for them.  But let's make it easier for new users to do the right thing - first by documenting the pitfall, then, in 3.8, changing the default.
msg343895 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2019-05-29 16:20
On May 28, 2019, at 17:21, STINNER Victor <report@bugs.python.org> wrote:
> 
> 
> STINNER Victor <vstinner@redhat.com> added the comment:
> 
>> To be clear, what is unsafe on macOS (as of 10.13, but even more so on 10.14) is calling into the Objective-C runtime between fork and exec.  The problem for Python is that it’s way too easy to do that implicitly, thus causing the macOS to abort the subprocess in surprising ways.
> 
> Do only a few Python module use the Objective-C runtime? Or is it basically "everything"?
> 
> If it's just a few, would it be possible to emit a warning or even an exception if called in a child process after fork?

I think it’s hard to know, but I found it through a path that lead from requests to _scproxy.c.  Here’s everything I know about the subject:

https://wefearchange.org/2018/11/forkmacos.rst.html

So yes, it’s theoretically possible to do *some* between fork and exec and not crash, and it’s of course perfectly safe to call exec pretty much right after fork.  It’s just hard to know for sure, and there are surprising ways to get into the Objective-C runtime.

I think we won’t be able to work around all of Apple’s choices here.  Documentation is the best way to handle it in <=3.7, and changing the default makes sense to me for 3.8.
msg343898 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2019-05-29 17:04
On May 28, 2019, at 17:38, Ned Deily <report@bugs.python.org> wrote:
> 
> Ned Deily <nad@python.org> added the comment:
> 
>> To be clear, what is unsafe on macOS (as of 10.13, but even more so on 10.14) is calling into the Objective-C runtime between fork and exec.
> 
> No, it has *always* been unsafe. What's new as of 10.13/14 is that macOS tries much harder at runtime to detect such cases and more predictably cause an error rather than letter than let the process run on and possibly fail nondeterministically.

Right, thanks for the additional nuance.  I think what changed is that in 10.13, Apple added a warning output when this condition occurred, and in 10.14 they actually abort the subprocess.
msg344590 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-04 15:11
Ned Deily:
> No, it has *always* been unsafe. What's new as of 10.13/14 is that macOS tries much harder at runtime to detect such cases and more predictably cause an error rather than letter than let the process run on and possibly fail nondeterministically.

Hum, in the doc, I wrote:

.. versionchanged:: 3.8

   On macOS, *spawn* start method is now the default: *fork* start method is no
   longer reliable on macOS, see :issue:`33725`.

Should we change this text? Any proposition?
msg344608 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2019-06-04 17:11
On Jun 4, 2019, at 08:11, STINNER Victor <report@bugs.python.org> wrote:
> Ned Deily:
>> No, it has *always* been unsafe. What's new as of 10.13/14 is that macOS tries much harder at runtime to detect such cases and more predictably cause an error rather than letter than let the process run on and possibly fail nondeterministically.
> 
> Hum, in the doc, I wrote:
> 
> .. versionchanged:: 3.8
> 
>   On macOS, *spawn* start method is now the default: *fork* start method is no
>   longer reliable on macOS, see :issue:`33725`.
> 
> Should we change this text? Any proposition?

Thanks Victor.  I don’t think “reliable” is strong enough, since this will definitely lead to core dumps under certain conditions.  What about:

   On macOS, the *spawn* start method is now the default.  The *fork* start method should
   be considered unsafe as it can lead to crashes of the subprocess.  See :issue:`33725`.
msg344710 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-05 12:05
> Thanks Victor.  I don’t think “reliable” is strong enough, since this will definitely lead to core dumps under certain conditions.  What about: (...)

That sounds better: I wrote PR 13841.
msg344762 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-05 19:59
New changeset 1e77ab0a35cf95318bb4893f7253a30f73201163 by Victor Stinner in branch 'master':
bpo-33725, multiprocessing doc: rephase warning against fork on macOS (GH-13841)
https://github.com/python/cpython/commit/1e77ab0a35cf95318bb4893f7253a30f73201163
msg344763 - (view) Author: miss-islington (miss-islington) Date: 2019-06-05 20:07
New changeset d74438b633184bbd8d775d7118d6f12f6f825a96 by Miss Islington (bot) in branch '3.8':
bpo-33725, multiprocessing doc: rephase warning against fork on macOS (GH-13841)
https://github.com/python/cpython/commit/d74438b633184bbd8d775d7118d6f12f6f825a96
msg345841 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-06-17 10:19
As far as I can tell, the only thing left to do for this issue is to add a documentation warning to the 3.7 documents similar to what was added to 3.8 but without the change in default.  A PR would be nice.
History
Date User Action Args
2019-06-17 10:19:06ned.deilysetmessages: + msg345841
2019-06-05 20:07:24miss-islingtonsetmessages: + msg344763
2019-06-05 20:01:23miss-islingtonsetpull_requests: + pull_request13726
2019-06-05 19:59:58vstinnersetmessages: + msg344762
2019-06-05 12:05:53vstinnersetmessages: + msg344710
2019-06-05 12:05:25vstinnersetpull_requests: + pull_request13718
2019-06-04 17:11:26barrysetmessages: + msg344608
2019-06-04 15:11:44vstinnersetmessages: + msg344590
2019-05-29 17:04:10barrysetmessages: + msg343898
2019-05-29 16:20:31barrysetmessages: + msg343895
2019-05-29 00:39:45ned.deilysetmessages: - msg343843
2019-05-29 00:39:36ned.deilysetmessages: + msg343844
2019-05-29 00:38:10ned.deilysetmessages: + msg343843
2019-05-29 00:21:33vstinnersetmessages: + msg343842
2019-05-29 00:17:40gregory.p.smithsetmessages: + msg343841
2019-05-29 00:06:50barrysetmessages: + msg343838
2019-05-28 23:43:35ned.deilysetmessages: + msg343833
2019-05-28 23:33:28vstinnersetmessages: + msg343832
2019-05-28 23:26:25ned.deilysetmessages: + msg343830
versions: - Python 2.7, Python 3.7
2019-05-28 23:20:08gregory.p.smithsetmessages: + msg343828
2019-05-28 23:18:24gregory.p.smithsetnosy: + gregory.p.smith
messages: + msg343826
2019-05-28 17:07:03barrysetmessages: + msg343807
2019-05-28 14:14:44vstinnersetmessages: + msg343782
2019-05-28 14:13:58vstinnersetpull_requests: + pull_request13525
2019-05-28 14:02:57vstinnersetmessages: + msg343779
2019-05-28 12:54:25vstinnersetmessages: + msg343773
2019-05-27 23:28:10vstinnersetmessages: + msg343704
2019-05-27 23:26:02vstinnersetpull_requests: + pull_request13509
2019-05-24 22:33:51tdsmithsetnosy: + tdsmith
2019-05-21 21:30:27ned.deilysetpull_requests: - pull_request13382
2019-05-21 20:39:08mbussonnsetpull_requests: + pull_request13382
2019-05-14 01:39:45barrysetmessages: + msg342412
2019-05-10 11:07:07josh.rsetnosy: + josh.r
messages: + msg342071
2019-05-10 02:15:53vstinnersetmessages: + msg342042
2019-05-05 21:41:59barrysetmessages: + msg341475
2019-05-05 15:44:41davinsetnosy: + pablogsal
messages: + msg341455
2019-05-05 15:26:00davinsetmessages: + msg341452
2019-03-26 12:29:20vstinnersetmessages: + msg338873
2019-03-25 19:08:25lukasz.langasetnosy: + lukasz.langa
messages: + msg338819
2019-03-12 11:07:17vstinnersetmessages: + msg337733
2019-03-10 02:58:29davinsetmessages: + msg337591
2019-03-10 01:49:04ned.deilysetpriority: normal -> critical

messages: + msg337587
versions: - Python 3.6
2019-03-10 01:43:51ned.deilysetpull_requests: - pull_request10552
2019-03-10 01:43:42ned.deilysetpull_requests: - pull_request10551
2018-12-25 03:12:34taleinatsetpull_requests: + pull_request10552
2018-12-25 03:12:16taleinatsetpull_requests: + pull_request10551
2018-12-13 02:24:44barrysetmessages: + msg331735
2018-12-13 01:59:53ned.deilysetmessages: + msg331733
2018-12-11 11:12:05vstinnersetnosy: + vstinner
messages: + msg331610
2018-12-10 01:42:49barrysetmessages: + msg331459
2018-12-09 15:45:38davinsetmessages: + msg331438
2018-12-09 15:13:53davinsetmessages: + msg331435
2018-12-09 07:33:31ned.deilysetmessages: + msg331411
2018-12-09 07:11:33miss-islingtonsetmessages: + msg331409
2018-12-09 07:06:56miss-islingtonsetnosy: + miss-islington
messages: + msg331407
2018-12-09 06:50:36miss-islingtonsetpull_requests: + pull_request10281
2018-12-09 06:50:28miss-islingtonsetpull_requests: + pull_request10280
2018-12-09 06:50:19ned.deilysetmessages: + msg331406
2018-12-09 06:30:19ned.deilysetkeywords: + patch
stage: patch review
pull_requests: + pull_request10279
2018-12-05 11:52:25ronaldoussorensetmessages: + msg331101
2018-11-15 03:22:35barrysetmessages: + msg329941
2018-11-14 20:43:14barrysetmessages: + msg329933
2018-11-14 19:42:27davinsetmessages: + msg329927
2018-11-14 18:33:00ned.deilysetmessages: + msg329926
2018-11-14 18:16:05pitrousetmessages: + msg329923
2018-11-14 18:11:26davinsetmessages: + msg329922
2018-11-14 17:52:37barrysetmessages: + msg329919
2018-11-14 01:36:15barrysetmessages: + msg329885
2018-11-14 01:14:28barrysetmessages: + msg329880
2018-11-14 01:07:44barrysettitle: Pytho crashes on macOS after fork with no exec -> Python crashes on macOS after fork with no exec
2018-11-13 21:51:14barrysettitle: macOS crashes after fork with no exec -> Pytho crashes on macOS after fork with no exec
2018-11-13 21:51:01barrysettitle: High Sierra hang when using multi-processing -> macOS crashes after fork with no exec
2018-11-13 21:50:33barrysetversions: + Python 3.6, Python 3.7, Python 3.8
2018-11-13 21:50:10barrysetmessages: + msg329871
2018-11-13 21:49:01barrylinkissue35219 superseder
2018-11-12 21:48:05barrysetnosy: + barry
2018-06-04 21:39:10ned.deilysetmessages: + msg318708
2018-06-03 08:51:31ronaldoussorensetmessages: + msg318529
2018-06-03 08:47:57ronaldoussorensetmessages: + msg318528
2018-06-01 11:10:22pitrousetmessages: + msg318397
2018-06-01 10:57:29ned.deilysetnosy: + pitrou, davin
messages: + msg318396
2018-06-01 05:51:16ronaldoussorensetmessages: + msg318361
2018-06-01 00:53:06kapiltcreate