Author nirai
Recipients Giovanni.Bajo, avian, bobbyi, gregory.p.smith, neologix, nirai, pitrou, sdaoden, vstinner
Date 2011-07-28.08:26:09
SpamBayes Score 5.30909e-13
Marked as misclassified No
Message-id <1311841571.44.0.239068288852.issue6721@psf.upfronthosting.co.za>
In-reply-to
Content
Hi Gregory,

> Gregory P. Smith <greg@krypto.org> added the comment:
> No Python thread is ever fork safe because the Python interpreter itself can never be made fork safe. 
> Nor should anyone try to make the interpreter itself safe. It is too complex and effectively impossible to guarantee.

a) I think the term "Guarantee" is not meaningful here since the interpreter is probably too complex to guarantee it does not contain other serious problems.
b) If no Python thread is ever fork safe, can you illustrate how a trivial Python thread spinning endlessly might deadlock a child forked by another Python thread?

I was not able to find reports of deadlocks clearly related to multiprocessing worker threads so they could be practically safe already, to the point other Python-Dev developers would be inclined to bury this as a theoretical problem :)

Anyway, there exists at least the problem of forking from the pool worker thread and possibly other issues, so the code should be reviewed.
Another latent problem is multiprocessing logging which is disabled by default?


> There is no general solution to this, fork and threading is simply broken in POSIX and no amount of duct tape outside of the OS kernel can fix it. 

This is why we should "sanitize" the multithreading module and deprecate mixing of threading and multiprocessing. 
I bet most developers using Python are not even aware of this problem. 
We should make sure they are through documentation.

Here is another way to look at the current situation:

1) Don't use threading for concurrency because of the GIL.
2) Don't mix threading with multiprocessing because threading and forking don't mix well.
3) Don't use multiprocessing because it can deadlock.

We should make sure developers are aware of (2) and can use (3) safely***.


> My only desire is that we attempt to do the right thing when possible with the locks we know about within the standard library.

Right, with an atfork() mechanism.
History
Date User Action Args
2011-07-28 08:26:11niraisetrecipients: + nirai, gregory.p.smith, pitrou, vstinner, bobbyi, neologix, Giovanni.Bajo, sdaoden, avian
2011-07-28 08:26:11niraisetmessageid: <1311841571.44.0.239068288852.issue6721@psf.upfronthosting.co.za>
2011-07-28 08:26:10nirailinkissue6721 messages
2011-07-28 08:26:09niraicreate