This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: subprocess.Popen() hangs in multi-threaded code
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 2.7
process
Status: closed Resolution: duplicate
Dependencies: Superseder: subprocess.Popen can hang in threaded applications in Python 2
View: 20318
Assigned To: Nosy List: kristjan.jonsson, r.david.murray
Priority: normal Keywords:

Created on 2016-06-28 14:09 by kristjan.jonsson, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Messages (2)
msg269432 - (view) Author: Kristján Valur Jónsson (kristjan.jonsson) * (Python committer) Date: 2016-06-28 14:09
On a quad-core raspberrypi, i have experienced that subprocess.Popen() sometimes does not return immediatelly, but only much later, when an unrelated process has exited.
Debugging the issue, I find the parent process hanging in 
                # Wait for exec to fail or succeed; possibly raising exception
                # Exception limited to 1M
                data = _eintr_retry_call(os.read, errpipe_read, 1048576)

This behaviour is consistent with the problem described in pipe_cloexec():


def pipe_cloexec(self):
            """Create a pipe with FDs set CLOEXEC."""
            # Pipes' FDs are set CLOEXEC by default because we don't want them
            # to be inherited by other subprocesses: the CLOEXEC flag is removed
            # from the child's FDs by _dup2(), between fork() and exec().
            # This is not atomic: we would need the pipe2() syscall for that.
            r, w = os.pipe()
            self._set_cloexec_flag(r)
            self._set_cloexec_flag(w)
            return r, w

In short:  It appears that occasionally the pipe FD is leaked to a different subprocess (started on a separate thread) before the cloexec flags can be set.  This causes the parent process to wait until that other instance of the file descriptor is closed, i.e. when that other unrelated process exits.

I currently have a workaround which involves using a threading.Lock() around the call.

This is not very nice, however.  Also, there is #Issue12196 which could be backported to 2.7 to address this problem.
msg269433 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2016-06-28 14:12
This is a duplicate of issue 20318.  The advice is to use subprocess32.
History
Date User Action Args
2022-04-11 14:58:33adminsetgithub: 71593
2016-06-28 14:12:07r.david.murraysetstatus: open -> closed

superseder: subprocess.Popen can hang in threaded applications in Python 2

nosy: + r.david.murray
messages: + msg269433
resolution: duplicate
stage: resolved
2016-06-28 14:09:06kristjan.jonssoncreate