classification
Title: Change "with subprocess.Popen():" (context manager) to ignore broken pipe error
Type: Stage:
Components: Versions: Python 3.5
process
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: Nosy List: akira, martin.panter, neologix, serhiy.storchaka, vstinner
Priority: normal Keywords: patch

Created on 2015-03-03 09:49 by vstinner, last changed 2016-02-17 17:21 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
popen_exit.patch vstinner, 2015-03-03 09:49 review
popen_exit-2.patch vstinner, 2015-03-03 12:08 review
Messages (13)
msg237115 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-03-03 09:49
The Popen.communicate() method ignores broken pipe error when writing to stdin. I propose to modify Popen.__exit__() to do the same in Python 3.5.

Attached patch implements this suggestion and document it. I added this paragraph to Popen doc:

"The context manager ignores broken pipe errors when closing the process’s stdin: call explicitly proc.stdin.flush() and proc.stdin.close() to get broken pipe errors."

So it's still possible to get broken pipe errors if you need them.

Do you know applications or libraries which rely on broken pipe errors and do something different than just ignoring them?

I prefer to leave Python 3.4 unchanged to avoid subtle behaviour changes in minor Python releases.

See also:

- issue #21619 which modified Popen.__exit__() to call wait() even if stdin.close() raised an exception
- issue #19612 which modified communicate() to handle EINVAL on stdin.write() on Windows
msg237117 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-03-03 10:40
I left some minor comments on the documentation.

As a side effect of your rearranging of _stdin_write(), I think it would fix the bug with communicate() leaking a BrokenPipeError and leaving a zombie when there is less than a buffer’s worth of data to send.
msg237120 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-03-03 11:26
Do you want to modify IOBase.__exit__ to ignore I/O errors in close()? I think such changes should be discussed in Python-Dev.
msg237122 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-03-03 11:41
> Do you want to modify IOBase.__exit__ to ignore I/O errors in close()?

Nope. On files, you want to want to know if your data has been fully written. For a subprocess, it's different. You only expect best effort.

The BrokenPipeError exception is raised by Python when an OS function fails with EPIPE. This exception has the same purpose than the SIGPIPE signal: warn the application that it's now useless to send more data, the consumer will ignore it. By the way, since Python checks the result of *all* OS functions, SIGPIPE is simply ignored. SIGPIPE and EPIPE have the same purpose.

In the subprocess module, if we get BrokenPipeError, it means that the child process stopped reading from stdin (closed it or the process already exited).

Popen.communicate() must close stdin, so why would we pass BrokenPipeError to the caller? It's useless, we just stop writing and close the pipe.

Since Popen.__exit__() also closes stdin, I use the same rationale: it useless to pass BrokenPipeError to the caller. The caller expects that the process exited.

Anyway, we closed all pipes, it's not more possible to communicate with the child process. The following example displays "is proc stdin closed? True":
---
import subprocess, sys

args = [sys.executable, '-c', 'pass']
proc = subprocess.Popen(args, stdin=subprocess.PIPE)
try:
    with proc:
        proc.stdin.write(b'x' * (2**20))
except BrokenPipeError:
    pass
print("is proc stdin closed?", proc.stdin.closed)
---

See also: "Why does SIGPIPE exist?"
https://stackoverflow.com/questions/8369506/why-does-sigpipe-exist
msg237125 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-03-03 12:08
New patch to fix the bug seen by Serhiy.

> Anyway, we closed all pipes

Oh, I forgot to explain that TextIOWrapper.close() closes the buffered file even if flush() raised an exception. BufferedWriter.close() does the same.

So stdin.close() always closes the text (binary or text, buffered or not). It's not more possible to use stdin after stdin.close(), even if stdin.close() raised an exception.
msg237129 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-03-03 12:58
I don't see a difference between buffered file and Popen object. Both are useless 
after closing, both can raise an exception when flush a buffer on closing. Why 
suppress an exception in one case but not in other? I think this question 
needs wider discussion in Python-Dev.
msg237130 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-03-03 13:06
> New patch to fix the bug seen by Serhiy.

I thought about different solution:

    try:
        if input:
            self.stdin.write(input)
    finally:
        self.stdin.close()

But your approach looks working too,
msg237131 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-03-03 13:13
> I thought about different solution:

Your solution is different: I would prefer to also ignore broken pipe errors on close(). I'm not sure that close() can raise a BrokenPipeError in practice.
msg237133 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-03-03 13:17
> Your solution is different: I would prefer to also ignore broken pipe errors
> on close(). I'm not sure that close() can raise a BrokenPipeError in
> practice.

Of course all this code should be inside try/except block that ignores a 
BrokenPipeError.
msg237498 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-03-08 02:11
After understanding the Windows test failure in Issue 21619, I am starting to believe that code relying on a BrokenPipeError or EPIPE is flawed. It is an inherent unavoidable race condition with the receiving end of the pipe, as long as another thread or process is involved, which is always the case for the subprocess module. Another way of looking at it is that there is no guarantee that your data will have been (or will be) received, even if stdin.close() succeeds and does not raise EPIPE or similar. This is because piped data is buffered by the OS.

So the proposed change wouldn’t be a significant disadvantage, except for code that is already flawed. It is analogous to the argument used for ignoring EINTR, because depending on it for handling signals is inherently racy.
msg240186 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-04-06 21:31
No consensus was found, I close the issue.
msg260403 - (view) Author: Akira Li (akira) * Date: 2016-02-17 17:07
Should this issue be reopened in light of
http://bugs.python.org/issue26372 (Popen.communicate not ignoring
BrokenPipeError)?

If .close() shouldn't raise BrokenPipeError in .communicate() (and it
shouldn't) then it seems logical that .close() shouldn't raise
BrokenPipeError in .__exit__() too (and in other subprocess.Popen
methods that do not return until the child process is dead such as
subprocess.run())
msg260404 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-02-17 17:21
"Should this issue be reopened in light of http://bugs.python.org/issue26372 (Popen.communicate not ignoring BrokenPipeError)?"

I don't like reopen old issues. IMHO the two issues are different enough to justify two entries in the bug tracker.
History
Date User Action Args
2016-02-17 17:21:23vstinnersetmessages: + msg260404
2016-02-17 17:07:50akirasetnosy: + akira
messages: + msg260403
2015-04-06 21:31:28vstinnersetstatus: open -> closed
resolution: rejected
messages: + msg240186
2015-03-08 02:11:26martin.pantersetmessages: + msg237498
2015-03-03 13:17:49serhiy.storchakasetmessages: + msg237133
2015-03-03 13:13:56vstinnersetmessages: + msg237131
2015-03-03 13:06:42serhiy.storchakasetmessages: + msg237130
2015-03-03 12:58:31serhiy.storchakasetmessages: + msg237129
2015-03-03 12:08:43vstinnersetfiles: + popen_exit-2.patch

messages: + msg237125
2015-03-03 11:41:44vstinnersetnosy: + neologix
messages: + msg237122
2015-03-03 11:26:57serhiy.storchakasetmessages: + msg237120
2015-03-03 10:40:34martin.pantersetmessages: + msg237117
2015-03-03 09:49:16vstinnercreate