This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: subprocess.Popen can hang in threaded applications in Python 2
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 2.7
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: gregory.p.smith Nosy List: Andrew.Lutomirski, gps, gregory.p.smith, martin.panter, r.david.murray, vstinner
Priority: normal Keywords:

Created on 2014-01-20 20:30 by Andrew.Lutomirski, last changed 2022-04-11 14:57 by admin. This issue is now closed.

File name Uploaded Description Edit Andrew.Lutomirski, 2014-01-20 20:30
Messages (7)
msg208577 - (view) Author: Andrew Lutomirski (Andrew.Lutomirski) Date: 2014-01-20 20:30
Running python -j4 often results in one of the threads failing to start until another thread finishes.

The bug appears to be that subprocess's pipe_cloexec function is racy: if another thread forks between os.pipe() and _set_cloexec_flag, then the resulting process could hang on to the write end of the pipe.  That will cause the Popen call that got rudely interrupted to wait until the whole resulting process tree dies.
msg208578 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-01-20 20:35
Hmm.  I thought someone had already reported this, but I can't find an issue.  I suspect it is fixed in 3.4 and it may not be practical to fix it in earlier versions.
msg208582 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2014-01-20 22:07
2.7 subprocess is fundamentally flawed and cannot be fixed. Install subprocess32 from PyPI. It's fixed in 3.2 and later.
msg208584 - (view) Author: Andrew Lutomirski (Andrew.Lutomirski) Date: 2014-01-20 22:19
Would it be worth adding something to the Python 2.7 subprocess docs indicating that subprocess is known to be broken?
msg208585 - (view) Author: Andrew Lutomirski (Andrew.Lutomirski) Date: 2014-01-20 22:26
FWIW, sticking a mutex in Popen.__init__ (wrapping the whole thing) seems to work around this issue (for programs that aren't using multiprocessing or fork, for example).  This might be a good-enough fix and be safe enough to stick in the standard library.
msg208606 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2014-01-21 03:55
A pointer in the 2.7 subprocess docs to subprocess32 does seem like a good idea, its what i tell everyone to do anyways. :)

If you've got a patch for this in 2.7 feel free to add it here and I can take a look.

leaving this open as a reminder to me to update the docs at the very least.
msg216305 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2014-04-15 15:10
I added a pointer to subprocess32 in the 2.7 subprocess docs in dd52365c8721.
Date User Action Args
2022-04-11 14:57:57adminsetgithub: 64517
2016-06-28 14:12:07r.david.murraylinkissue27406 superseder
2014-04-15 15:10:31gregory.p.smithsetstatus: open -> closed
resolution: wont fix
messages: + msg216305

stage: resolved
2014-02-10 06:14:01martin.pantersetnosy: + martin.panter
2014-02-03 01:32:35vstinnersettitle: subprocess.Popen can hang in threaded applications -> subprocess.Popen can hang in threaded applications in Python 2
2014-01-21 03:55:13gregory.p.smithsetstatus: closed -> open
assignee: gregory.p.smith
messages: + msg208606
2014-01-20 22:26:31Andrew.Lutomirskisetstatus: open -> closed

messages: + msg208585
2014-01-20 22:19:39Andrew.Lutomirskisetstatus: closed -> open
resolution: wont fix -> (no value)
messages: + msg208584
2014-01-20 22:07:40gregory.p.smithsetstatus: open -> closed

nosy: + gregory.p.smith
messages: + msg208582

resolution: wont fix
2014-01-20 20:35:09r.david.murraysetnosy: + vstinner, r.david.murray, gps
messages: + msg208578
2014-01-20 20:30:19Andrew.Lutomirskicreate