New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
"subprocess" can raise OSError (EPIPE) when communicating with short-lived processes #55172
Comments
If we start a short-lived process which finishes before we begin communicating with it (e.g. by crashing), we can receive a SIGPIPE due to the receiving process no longer existing. This becomes an EPIPE, which becomes an: Arguably this is a bug; if the subprocess could crash, the user currently has to check for it by both monitoring the returncode _and_ catching OSError (then examining for this specific errno), which seems ugly to me. I'm attaching a patch for subprocess which handles this case, masking the OSError within the Popen implementation, so that you have to test for it within the returncode, in one place, rather than wrap every call with a try/except. It could be argued that this is incorrect, as it masks under-reads of stdin by the subprocess. However I believe a sanely-written subprocess ought to indicate success/failure back with its return code. This was originally filed downstream within the Red Hat bugzilla instance as: The handler part of the patch is based on a patch attached there by Federico Simoncelli; I don't know if he has signed a PSF contributor agreement, however the size of the patch is sufficiently small that I suspect it's not copyrightable, and I've somewhat rewritten it since; I wrote the unit test. |
It seems quite orthogonal. The subprocess could have terminated successfully while not all of the stdin bytes were consumed; EPIPE informs you of the latter. |
I agree they are orthogonal. The problem is that Popen.communicate doesn't support a way to ignore the exception and keep reading from stdout and sterr until the process dies. |
You are right, this is suboptimal. It would also be a behaviour change from currently, but it seems beneficial. |
I'd argue that this is not a feature request but a bug. I did some testing of this issue and the problem is that EPIPE is only generated sometimes depending on the time the process takes to finish, the size of the data sent, the underlying mechanism used (select vs poll) and the whether anything happens between the starting of the process and the communicate() call. Here are some results (on my PC, I think some of these will vary on others): With select: Only stdin (neither select or poll): (all of my tests appear to fail on Windows, they also generate EINVAL sometimes) I think it's best to remove all this inconsistency and fix it so that EPIPE is never generated and then backport it to 2.7, 3.1, 3.2. Attached is a patch which fixes it for poll, select, windows and adds two tests. |
The patch looks good to me. |
New changeset c10d55c51d81 by Ross Lagerwall in branch '2.7': New changeset 158495d49f58 by Ross Lagerwall in branch '3.1': |
Committed, thanks. |
self.stdout.close() also can fail with EPIPE or EINVAL if the stream is buffered (text streams are buffered). I.e. communicate() in text mode can loss the data. See also bpo-21619. |
FTR bpo-26372 has been opened about Serhiy’s bug with stdin.close() raising EPIPE. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: