This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: read stuck with multithreading and simultaneous subprocess.Popen
Type: behavior Stage: resolved
Components: Library (Lib), Windows Versions: Python 3.1, Python 3.2, Python 2.7
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: SAPikachu, gjb1002, iritkatriel, sbt, vstinner
Priority: normal Keywords:

Created on 2011-08-12 03:20 by SAPikachu, last changed 2022-04-11 14:57 by admin. This issue is now closed.

File name Uploaded Description Edit SAPikachu, 2011-08-12 03:20 test case
Messages (7)
msg141939 - (view) Author: Joe Hu (SAPikachu) Date: 2011-08-12 03:20
When multiple threads create child processes simultaneously and redirect their stdout using subprocess.Popen, at least one thread will stuck on reading the stdout after its child process exited, until all other processes are also exited.

The test case reproduces the problem. It's always reproducible on my system (Python 3.1 on Windows 7 x64 / Python 3.2 on Windows 7 x86).

Here is my suspicion: 
When Popen is called by two threads simultaneously, the latter child processes may be started before pipe handles for the former process are closed, causing the handles be incorrectly inherited by the latter process. So these handles can only be closed after all the two processes exit, and only after that,* can detect EOF and return.
msg220095 - (view) Author: Geoffrey Bache (gjb1002) Date: 2014-06-09 14:25
Just ran into this on Python 2.6 also.
msg220096 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-06-09 14:36
The PEP 446 partially fixes this issue. The issue #19764 should fix it completly. Since you are using Python 2, you should not wait until the issue is fixed, but work around it.

To workaround the issue: use you own lock around the creation of processes. Example:
lock = threading.Lock()

def run_command(...):
   with lock:
      proc = subprocess.Popen(...)
   return proc.communicate()

The problem is that a thread B may inherit the handle of a pipe from handle A because the pip is marked as inheritable, and the subprocess module must use CreateProcess() with bInheritHandles parameter set to True to be able to redirect stdout.

Said differently: the subprocess is not thread-safe, you have to use your own lock to workaround race conditions.
msg220097 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-06-09 14:37
Oh by the way, this issue was fixed on UNIX in Python 3.2: all file descriptors are now closed by default (close_fds=True by default on UNIX).
msg220099 - (view) Author: Geoffrey Bache (gjb1002) Date: 2014-06-09 15:28
Thanks Victor, yes I already created my own lock which fixed the issue for me. 

Maybe it would be worth adding a note to the documentation about this in the meantime (especially for Python 2)?
msg380459 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2020-11-06 17:30
I will close this as out of date unless someone objects.
msg380460 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-11-06 17:33
> Oh by the way, this issue was fixed on UNIX in Python 3.2: all file descriptors are now closed by default (close_fds=True by default on UNIX).

Issue fixed in Python 3.4 (sorry, 3.4, not 3.2):
Date User Action Args
2022-04-11 14:57:20adminsetgithub: 56948
2020-11-06 17:33:59vstinnersetstatus: pending -> closed
resolution: fixed
messages: + msg380460

stage: resolved
2020-11-06 17:30:22iritkatrielsetstatus: open -> pending
nosy: + iritkatriel
messages: + msg380459

2014-06-09 15:28:14gjb1002setmessages: + msg220099
2014-06-09 14:37:37vstinnersetmessages: + msg220097
2014-06-09 14:36:08vstinnersetnosy: + vstinner, sbt
messages: + msg220096
2014-06-09 14:25:43gjb1002setmessages: + msg220095
versions: + Python 2.7
2014-06-09 14:23:06gjb1002setnosy: + gjb1002
2011-08-12 03:20:23SAPikachucreate