classification
Title: "subprocess" can raise OSError (EPIPE) when communicating with short-lived processes
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: rosslagerwall Nosy List: Yaniv.Aknin, amaury.forgeotdarc, dmalcolm, dwalczak, giampaolo.rodola, gregory.p.smith, martin.panter, mcrute, pitrou, python-dev, rosslagerwall, serhiy.storchaka, simon3z
Priority: normal Keywords: patch

Created on 2011-01-20 20:23 by dmalcolm, last changed 2016-02-18 00:14 by martin.panter. This issue is now closed.

Files
File name Uploaded Description Edit
py3k-handle-EPIPE-for-short-lived-subprocesses-2011-01-20-001.patch dmalcolm, 2011-01-20 20:23 review
10963.patch rosslagerwall, 2011-03-31 12:35 review
Messages (11)
msg126643 - (view) Author: Dave Malcolm (dmalcolm) (Python committer) Date: 2011-01-20 20:23
If we start a short-lived process which finishes before we begin communicating with it (e.g. by crashing), we can receive a SIGPIPE due to the receiving process no longer existing.  This becomes an EPIPE, which becomes an:
  OSError: [Errno 32] Broken pipe

Arguably this is a bug; if the subprocess could crash, the user currently has to check for it by both monitoring the returncode _and_ catching OSError (then examining for this specific errno), which seems ugly to me.

I'm attaching a patch for subprocess which handles this case, masking the OSError within the Popen implementation, so that you have to test for it within the returncode, in one place, rather than wrap every call with a try/except.

It could be argued that this is incorrect, as it masks under-reads of stdin by the subprocess.  However I believe a sanely-written subprocess ought to indicate success/failure back with its return code.

This was originally filed downstream within the Red Hat bugzilla instance as:
  https://bugzilla.redhat.com/show_bug.cgi?id=667431

The handler part of the patch is based on a patch attached there by Federico Simoncelli; I don't know if he has signed a PSF contributor agreement, however the size of the patch is sufficiently small that I suspect it's not copyrightable, and I've somewhat rewritten it since; I wrote the unit test.
msg126644 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-01-20 20:26
> It could be argued that this is incorrect, as it masks under-reads of
> stdin by the subprocess.  However I believe a sanely-written subprocess 
> ought to indicate success/failure back with its return code.

It seems quite orthogonal. The subprocess could have terminated successfully while not all of the stdin bytes were consumed; EPIPE informs you of the latter.
msg126934 - (view) Author: Federico Simoncelli (simon3z) Date: 2011-01-24 17:08
I agree they are orthogonal. The problem is that Popen.communicate doesn't support a way to ignore the exception and keep reading from stdout and sterr until the process dies.
When the child closes the stdin its (error/debug/info) messages on stout/sterr will be lost.
I am not just concerned about checking both the return-code and this exception, I am also worried about losing important information.
msg126935 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-01-24 17:11
You are right, this is suboptimal. It would also be a behaviour change from currently, but it seems beneficial.
msg132662 - (view) Author: Ross Lagerwall (rosslagerwall) (Python committer) Date: 2011-03-31 12:35
I'd argue that this is not a feature request but a bug.

I did some testing of this issue and the problem is that EPIPE is only generated sometimes depending on the time the process takes to finish, the size of the data sent, the underlying mechanism used (select vs poll) and the whether anything happens between the starting of the process and the communicate() call.

Here are some results (on my PC, I think some of these will vary on others):
With poll:
 [sys.executable, 'c', 'pass']- no error
 ['dd', 'option=bad'] - varies between EPIPE and no error
 ['dd', 'option=bad'], sleep(1) - EPIPE

With select:
 [sys.executable, 'c', 'pass']- EPIPE
 ['dd', 'option=bad'] - EPIPE
 ['dd', 'option=bad'], sleep(1) - EPIPE

Only stdin (neither select or poll):
 [sys.executable, 'c', 'pass']- no error (error in 2.7)
 ['dd', 'option=bad'] - no error (error in 2.7)
 ['dd', 'option=bad'], sleep(1) - EPIPE

(all of my tests appear to fail on Windows, they also generate EINVAL sometimes)

I think it's best to remove all this inconsistency and fix it so that EPIPE is never generated and then backport it to 2.7, 3.1, 3.2.

Attached is a patch which fixes it for poll, select, windows and adds two tests.
msg132730 - (view) Author: Ross Lagerwall (rosslagerwall) (Python committer) Date: 2011-04-01 06:34
Marked #6457 as a duplicate. See #6457 for more discussion.
msg132806 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-04-02 17:50
> I think it's best to remove all this inconsistency and fix it so that
> EPIPE is never generated and then backport it to 2.7, 3.1, 3.2.
> 
> Attached is a patch which fixes it for poll, select, windows and adds
> two tests.

The patch looks good to me.
msg133031 - (view) Author: Roundup Robot (python-dev) Date: 2011-04-05 14:10
New changeset c10d55c51d81 by Ross Lagerwall in branch '2.7':
Issue #10963: Ensure that subprocess.communicate() never raises EPIPE.
http://hg.python.org/cpython/rev/c10d55c51d81

New changeset 158495d49f58 by Ross Lagerwall in branch '3.1':
Issue #10963: Ensure that subprocess.communicate() never raises EPIPE.
http://hg.python.org/cpython/rev/158495d49f58
msg133033 - (view) Author: Ross Lagerwall (rosslagerwall) (Python committer) Date: 2011-04-05 14:14
Committed, thanks.
msg236951 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-03-01 09:31
self.stdout.close() also can fail with EPIPE or EINVAL if the stream is buffered (text streams are buffered). I.e. communicate() in text mode can loss the data. See also issue21619.
msg260416 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-02-18 00:14
FTR Issue 26372 has been opened about Serhiy’s bug with stdin.close() raising EPIPE.
History
Date User Action Args
2016-02-18 00:14:03martin.pantersetnosy: + martin.panter
messages: + msg260416
2015-03-01 09:31:25serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg236951
2011-04-05 14:14:29rosslagerwallsetstatus: open -> closed
type: enhancement -> behavior
messages: + msg133033

assignee: rosslagerwall
resolution: fixed
stage: patch review -> resolved
2011-04-05 14:10:14python-devsetnosy: + python-dev
messages: + msg133031
2011-04-02 17:50:03pitrousetmessages: + msg132806
2011-04-01 06:34:01rosslagerwallsetnosy: + amaury.forgeotdarc, dwalczak, mcrute, Yaniv.Aknin
messages: + msg132730
2011-03-31 12:35:19rosslagerwallsetfiles: + 10963.patch
nosy: + rosslagerwall
messages: + msg132662

2011-01-24 18:30:53giampaolo.rodolasetnosy: + giampaolo.rodola
2011-01-24 17:11:30pitrousetnosy: + gregory.p.smith
type: enhancement
messages: + msg126935
2011-01-24 17:08:01simon3zsetnosy: + simon3z
messages: + msg126934
2011-01-20 20:26:56pitrousetnosy: + pitrou
messages: + msg126644
2011-01-20 20:23:29dmalcolmcreate