classification
Title: Cleaning up a subprocess with a broken pipe
Type: resource usage Stage: resolved
Components: Library (Lib) Versions: Python 3.5, Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: Arfrever, akira, dmalcolm, martin.panter, pitrou, python-dev, rosslagerwall, serhiy.storchaka, vstinner
Priority: normal Keywords: patch

Created on 2014-05-31 12:57 by martin.panter, last changed 2015-03-08 08:42 by serhiy.storchaka. This issue is now closed.

Files
File name Uploaded Description Edit
pipe-cleanup.patch martin.panter, 2014-12-16 10:47 review
overflow-pipe-test.patch martin.panter, 2015-03-07 01:41 review
overflow-pipe-test-2.patch serhiy.storchaka, 2015-03-08 07:00 review
Messages (26)
msg219450 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2014-05-31 12:57
The documentation for the “subprocess” module says that a “with” statement will “wait for” the process, implying that it does not leave a zombie. However this is not the case if there is buffered input data:

$ python3 -Wall -bt -q
>>> import subprocess
>>> with subprocess.Popen(("true",), stdin=subprocess.PIPE, bufsize=-1) as p:
...     from time import sleep; sleep(1)  # Wait for pipe to be broken
...     p.stdin.write(b"buffered data")
... 
13
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
  File "/usr/lib/python3.4/subprocess.py", line 899, in __exit__
    self.stdin.close()
BrokenPipeError: [Errno 32] Broken pipe
>>> # (Hit Ctrl-Z here)
[1]+  Stopped                 python3 -Wall -bt -q
[Exit 148]
$ ps
  PID TTY          TIME CMD
15867 pts/5    00:00:00 python3
15869 pts/5    00:00:00 true <defunct>
15873 pts/5    00:00:00 ps
32227 pts/5    00:00:10 bash

Similarly, calling Popen.communicate() does not clean the process up either if there is buffered input data and the process has already exited. The documentation does not spell out how a broken pipe is handled in communicate(), but after reading Issue 10963 I see that in many other cases it is meant to be ignored.

The best way to clean up a subprocess that I have come up with to close the pipe(s) and call wait() in two separate steps, such as:

try:
    proc.stdin.close()
except BrokenPipeError:
    pass
proc.wait()
msg232732 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2014-12-16 10:47
Here is a patch to fix this by calling wait() even if stdin.close() fails, including a test case. With my patch, the subprocess context manager __exit__() will still raise a BrokenPipeError, but no zombie will be left.
msg236873 - (view) Author: Roundup Robot (python-dev) Date: 2015-02-28 10:46
New changeset b5e9ddbdd4a7 by Serhiy Storchaka in branch '3.4':
Issue #21619: Popen objects no longer leave a zombie after exit in the with
https://hg.python.org/cpython/rev/b5e9ddbdd4a7

New changeset cdac249808a8 by Serhiy Storchaka in branch 'default':
Issue #21619: Popen objects no longer leave a zombie after exit in the with
https://hg.python.org/cpython/rev/cdac249808a8
msg236874 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-02-28 10:48
Thank you for your contribution Martin.
msg236875 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-02-28 10:50
Why not ignoring BrokenPipeError like communicate()?
msg236877 - (view) Author: Roundup Robot (python-dev) Date: 2015-02-28 11:28
New changeset 1b4d916329e7 by Serhiy Storchaka in branch '3.4':
Fixed a test for issue #21619 on Windows.
https://hg.python.org/cpython/rev/1b4d916329e7

New changeset eae459e35cb9 by Serhiy Storchaka in branch 'default':
Fixed a test for issue #21619 on Windows.
https://hg.python.org/cpython/rev/eae459e35cb9
msg236949 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-03-01 08:54
The test still sporadically fails on Windows:

http://buildbot.python.org/all/builders/x86%20Windows7%203.x/builds/9323/steps/test/logs/stdio
======================================================================
FAIL: test_broken_pipe_cleanup (test.test_subprocess.ContextManagerTests)
Broken pipe error should not prevent wait() (Issue 21619)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\test_subprocess.py", line 2515, in test_broken_pipe_cleanup
    self.assertRaises(OSError, proc.__exit__, None, None, None)
AssertionError: OSError not raised by __exit__

----------------------------------------------------------------------
msg236950 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-03-01 09:10
> Why not ignoring BrokenPipeError like communicate()?

Why communicate() ignores BrokenPipeError?
msg236952 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-03-01 09:32
It was added in issue10963. I don't know if this way is applicable to this issue.
msg236961 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-03-01 13:30
Le dimanche 1 mars 2015, Serhiy Storchaka <report@bugs.python.org> a écrit :
>
> Why communicate() ignores BrokenPipeError?
>

It's more convinient. There is nothing useful you can do on pipe error in
communicate().

For __exit__, what do you want to on broken pipe error? I did't check yet:
the process always exit after __exit__?
msg236964 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-03-01 13:48
When you write to the file you don't want the error was silently ignored.

    with open(filename, 'w') as f:
        f.write(content)

And also I don't want the error was silently ignored when write to the subprocess.

    with subprocess.Popen(cmd, universal_newlines=True, stdin=subprocess.PIPE) as p:
        p.stdin.write(content)
msg237005 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-03-02 02:19
It seems two different issues have popped up:

## 1. Windows behaviour ##

Windows apparently doesn’t handle broken pipes consistently, sometimes raising an EINVAL error, and sometimes not raising any error. I don’t know if this is a problem with Python, or a quirk with Windows. (Just tried with Wine, and it always raises ENOSPC instead, but I guess that’s a Wine-specific quirk.)

Perhaps we can change the offending test case to the following, at least until someone can investigate and explain what is going on:

...
if mswindows:
    try:
        proc.__exit__(None, None, None)
    except OSError:  # Sometimes raises EINVAL, sometimes no error
        pass
else:
    self.assertRaises(BrokenPipeError, proc.__exit__, None, None, None)
...

## 2. Suppressing BrokenPipeError ##

I don’t have any strong opinions. I think in the past when writing to a process’s input raises BrokenPipeError, I have tended to skip the writing and go on to analyse the process’s output and/or exit status (the same as if writing succeeded).

Some arguments for raising BrokenPipeError (current behaviour):

* No change in behaviour
* Slightly simpler implementation
* Consistent with the way stdin.close() works
* Potentially more flexible if the user cares about handling a broken pipe specially.
* Exiting the context manager does not return any data, so no return value is lost by raising an exception. In contrast, one of the reasons communicate() was changed to sometimes suppress the exception is so that the captured output is returned and not lost.

Arguments for suppressing BrokenPipeError:

* Consistent with the way communicate() is meant to work, according to Issue 10963.
* Probably more useful in normal use cases
* User can always explicitly call stdin.close() if they really want to handle the BrokenPipeError
* Avoid possible confusion and hiding a more important exception, if one was already raised inside the context manager, for example if a subprocess crash was detected, or if stdin.write() already raised its own BrokenPipeError.

Replying to Victor: “the process always exit after __exit__?”:

__exit__ is meant to call wait(), so it only returns when the subprocess exits. It could potentially still be running despite closing its input pipe. This issue was originally about making sure __exit__ called wait() in all cases.
msg237098 - (view) Author: Akira Li (akira) * Date: 2015-03-03 00:34
On Windows behavior
http://stackoverflow.com/questions/23688492/oserror-errno-22-invalid-argument-in-subprocess
msg237105 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-03-03 04:47
Thanks for that link; the answer by Eryksun <https://stackoverflow.com/questions/23688492/oserror-errno-22-invalid-argument-in-subprocess#28020096> is particularly enlightening. Apparently EINVAL actually represents an underlying broken pipe condition in Windows. Anyway, as far as the test is concerned, it doesn’t matter what particular exception is raised.

This still doesn’t explain Serhiy’s observation that sometimes no exception is raised at all, <https://bugs.python.org/issue21619#msg236949>. Does Windows sometimes not report broken pipe errors, or is my signalling logic not working and there is a race condition? The whole point of the test is to trigger an exception when stdin is closed, so it would be nice to be able to reliably do that on Windows.
msg237109 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-03-03 07:12
A few months ago, I modified Popen.communicate() to handle EINVAL on
Windows.
msg237116 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-03-03 09:53
I opened the issue #23570: Change "with subprocess.Popen():" (context manager) to ignore broken pipe error.

> FAIL: test_broken_pipe_cleanup (test.test_subprocess.ContextManagerTests)

Serhiy: see existing test_communicate_epipe() and test_communicate_epipe_only_stdin() tests which are now reliable and portable.

You should write more data (2**20 bytes) and set the buffer size to a value larger than the input data, to buffer all data, so the write occurs at stdin.close() in Popen.__exit__().

By the way, instead of an hardcoded value (2**20), support.PIPE_MAX_SIZE may be more appropriate.
msg237119 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-03-03 11:08
Aha! So perhaps Windows can accept a small amount of data into its pipe buffer even if we know the pipe has been broken. That kind of makes sense. Test case could be modified to:

proc = subprocess.Popen([...], bufsize=support.PIPE_MAX_SIZE, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
proc.stdout.read()  # Make sure subprocess has closed its input
proc.stdin.write(bytes(support.PIPE_MAX_SIZE))
self.assertIsNone(proc.returncode)
# Expect EPIPE under POSIX and EINVAL under Windows
self.assertRaises(OSError, proc.__exit__, None, None, None)
msg237121 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-03-03 11:27
Could you provide a patch Martin?
msg237124 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-03-03 12:01
I would be safer to use a bufsize a little bit larger :-)

proc = subprocess.Popen([...], bufsize=support.PIPE_MAX_SIZE * 2,
stdin=subprocess.PIPE, stdout=subprocess.PIPE)
...
proc.stdin.write(b'x' * support.PIPE_MAX_SIZE)
msg237226 - (view) Author: Roundup Robot (python-dev) Date: 2015-03-05 01:42
New changeset 77a978716517 by Victor Stinner in branch '3.4':
Issue #21619: Try to fix test_broken_pipe_cleanup()
https://hg.python.org/cpython/rev/77a978716517
msg237249 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-03-05 08:36
Looks as tests are fixed. Thank you Victor.
msg237409 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-03-07 01:41
Thanks for getting the test working. Just to tidy things up here I would like to get rid of my stdout signalling in the test, which is no longer needed and could be misleading. See overflow-pipe-test.patch.
msg237484 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-03-08 00:32
overflow-pipe-test.patch looks good to me.
msg237514 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-03-08 06:59
I think we should add __enter__ for consistency.
msg237515 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-03-08 07:03
Sure, new version is fine by me
msg237516 - (view) Author: Roundup Robot (python-dev) Date: 2015-03-08 07:18
New changeset 4ea40dc3d26d by Serhiy Storchaka in branch '3.4':
Issue #21619: Cleaned up test_broken_pipe_cleanup.
https://hg.python.org/cpython/rev/4ea40dc3d26d

New changeset 41ce95a5b2d8 by Serhiy Storchaka in branch 'default':
Issue #21619: Cleaned up test_broken_pipe_cleanup.
https://hg.python.org/cpython/rev/41ce95a5b2d8
History
Date User Action Args
2015-03-08 08:42:12serhiy.storchakasetstatus: open -> closed
2015-03-08 07:18:39python-devsetmessages: + msg237516
2015-03-08 07:03:55martin.pantersetmessages: + msg237515
2015-03-08 07:00:36serhiy.storchakasetfiles: + overflow-pipe-test-2.patch
2015-03-08 06:59:00serhiy.storchakasetmessages: + msg237514
2015-03-08 00:32:34vstinnersetmessages: + msg237484
2015-03-07 01:41:26martin.pantersetstatus: closed -> open
files: + overflow-pipe-test.patch
messages: + msg237409
2015-03-05 08:36:52serhiy.storchakasetstatus: open -> closed
resolution: fixed
messages: + msg237249

stage: needs patch -> resolved
2015-03-05 01:42:13python-devsetmessages: + msg237226
2015-03-04 10:30:49serhiy.storchakasetstage: needs patch
2015-03-03 12:01:23vstinnersetmessages: + msg237124
2015-03-03 11:27:27serhiy.storchakasetmessages: + msg237121
2015-03-03 11:08:57martin.pantersetmessages: + msg237119
2015-03-03 09:53:50vstinnersetmessages: + msg237116
2015-03-03 07:12:21vstinnersetmessages: + msg237109
2015-03-03 04:47:30martin.pantersetmessages: + msg237105
2015-03-03 00:34:23akirasetnosy: + akira
messages: + msg237098
2015-03-02 02:19:20martin.pantersetmessages: + msg237005
2015-03-01 13:48:43serhiy.storchakasetmessages: + msg236964
2015-03-01 13:30:10vstinnersetmessages: + msg236961
2015-03-01 11:49:40Arfreversetnosy: + Arfrever
2015-03-01 09:32:39serhiy.storchakasetnosy: + pitrou, dmalcolm, rosslagerwall
messages: + msg236952

resolution: fixed -> (no value)
stage: resolved -> (no value)
2015-03-01 09:10:30serhiy.storchakasetmessages: + msg236950
2015-03-01 08:54:17serhiy.storchakasetstatus: closed -> open

messages: + msg236949
2015-02-28 11:28:56python-devsetmessages: + msg236877
2015-02-28 10:50:07vstinnersetmessages: + msg236875
2015-02-28 10:49:36serhiy.storchakasetstatus: open -> closed
2015-02-28 10:48:01serhiy.storchakasetversions: + Python 3.5
resolution: fixed
messages: + msg236874

type: resource usage
stage: patch review -> resolved
2015-02-28 10:46:47python-devsetnosy: + python-dev
messages: + msg236873
2015-02-28 10:23:44serhiy.storchakasetassignee: serhiy.storchaka

nosy: + serhiy.storchaka
stage: patch review
2014-12-17 23:58:38vstinnersetnosy: + vstinner
2014-12-16 10:47:26martin.pantersetfiles: + pipe-cleanup.patch
keywords: + patch
messages: + msg232732
2014-05-31 12:57:05martin.pantercreate