On Nov 14, 2018, at 10:11, Davin Potts <report@bugs.python.org> wrote:
> 
> 
> Davin Potts <python@discontinuity.net> added the comment:
> 
> Barry's effort as well as comments in other links seem to all suggest that OBJC_DISABLE_INITIALIZE_FORK_SAFETY is not comprehensive in its ability to make other threads "safe" before forking.

Right.  Setting the env var will definitely not make it thread safe.  My understanding (please correct me if I’m wrong!) isn’t that this env var makes it safe, just that it prevents the ObjC runtime from core dumping.  So it’s still up to the developer to know whether threads are involved or not.  In our cases, these are single threaded applications.  I’ve read elsewhere that ObjC doesn’t care if threads have actually been spun up or not.

> "Objective-C classes defined by the OS frameworks remain fork-unsafe" (from @kapilt's first link) suggests we furthermore remain at risk using certain MacOS system libraries prior to any call to fork.

Actually, it’s unsafe to call anything between fork and exec.  Note that this doesn’t just affect Python; this is a pretty common idiom in other scripting languages too, from what I can tell.  It’s certainly very common in Python.

Note too that urllib.request.getproxies() will end up calling into the ObjC runtime via _scproxy, so you can’t even use requests after a fork but before exec.

What I am still experimenting with is to see if I can define a pthread_atfork handler that will initialize the ObjC runtime before fork is actually called.  I saw a Ruby approach like this, but it’s made more difficult in Python because pthread_atfork isn’t exposed to Python.  I’m trying to see if I can implement it in ctypes, before I write an extension.

> "To guarantee that forking is safe, the application must not be running any threads at the point of fork" (from @kapilt's second link) is an old truth that we continue to fight with even when we know very well that it's the truth.

True, but do realize this problem affects you even in single threaded applications.

> For newly developed code, we have the alternative to employ spawn instead of fork to avoid these problems in Python, C, Ruby, etc.  For existing legacy code that employed fork and now surprises us by failing-fast on MacOS 10.13 and 10.14, it seems we are forced to face a technical debt incurred back when the choice was first made to spin up threads and afterwards to use fork.

It’s tech debt you incur even if you don’t spin up threads.  Just fork and do some work in the child before calling exec.  If that work enters the ObjC runtime (as in the getproxies example), your child will coredump,

> If we didn't already have an "obvious" (zen of Python) way to avoid such problems with spawn versus fork, I would feel this was something to solve in Python.  As to helping the poor unfortunate souls who must fight the good fight with legacy code, I am not sure what to do to help though I would like to be able to help.

*If* we can provide a hook to initialize the ObjC runtime in pthread_atfork, I think that’s something we could expose in Python.  Then we can say legacy code can just invoke that, and at least you will avoid the worst outcome.