Title: read stuck with multithreading and simultaneous subprocess.Popen
Type: behavior Stage:
Components: Library (Lib), Windows Versions: Python 3.1, Python 3.2, Python 2.7
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: SAPikachu, gjb1002, sbt, vstinner
Priority: normal Keywords:

Created on 2011-08-12 03:20 by SAPikachu, last changed 2014-06-09 15:28 by gjb1002.

File name Uploaded Description Edit SAPikachu, 2011-08-12 03:20 test case
Messages (5)
msg141939 - (view) Author: Joe Hu (SAPikachu) Date: 2011-08-12 03:20
When multiple threads create child processes simultaneously and redirect their stdout using subprocess.Popen, at least one thread will stuck on reading the stdout after its child process exited, until all other processes are also exited.

The test case reproduces the problem. It's always reproducible on my system (Python 3.1 on Windows 7 x64 / Python 3.2 on Windows 7 x86).

Here is my suspicion: 
When Popen is called by two threads simultaneously, the latter child processes may be started before pipe handles for the former process are closed, causing the handles be incorrectly inherited by the latter process. So these handles can only be closed after all the two processes exit, and only after that,* can detect EOF and return.
msg220095 - (view) Author: Geoffrey Bache (gjb1002) Date: 2014-06-09 14:25
Just ran into this on Python 2.6 also.
msg220096 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-06-09 14:36
The PEP 446 partially fixes this issue. The issue #19764 should fix it completly. Since you are using Python 2, you should not wait until the issue is fixed, but work around it.

To workaround the issue: use you own lock around the creation of processes. Example:
lock = threading.Lock()

def run_command(...):
   with lock:
      proc = subprocess.Popen(...)
   return proc.communicate()

The problem is that a thread B may inherit the handle of a pipe from handle A because the pip is marked as inheritable, and the subprocess module must use CreateProcess() with bInheritHandles parameter set to True to be able to redirect stdout.

Said differently: the subprocess is not thread-safe, you have to use your own lock to workaround race conditions.
msg220097 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-06-09 14:37
Oh by the way, this issue was fixed on UNIX in Python 3.2: all file descriptors are now closed by default (close_fds=True by default on UNIX).
msg220099 - (view) Author: Geoffrey Bache (gjb1002) Date: 2014-06-09 15:28
Thanks Victor, yes I already created my own lock which fixed the issue for me. 

Maybe it would be worth adding a note to the documentation about this in the meantime (especially for Python 2)?
Date User Action Args
2014-06-09 15:28:14gjb1002setmessages: + msg220099
2014-06-09 14:37:37vstinnersetmessages: + msg220097
2014-06-09 14:36:08vstinnersetnosy: + vstinner, sbt
messages: + msg220096
2014-06-09 14:25:43gjb1002setmessages: + msg220095
versions: + Python 2.7
2014-06-09 14:23:06gjb1002setnosy: + gjb1002
2011-08-12 03:20:23SAPikachucreate