classification
Title: subprocess.Popen(stderr=STDOUT) fails to redirect subprocess stderr to stdout
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.6, Python 3.5, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: akira, gregory.p.smith, josh.r, martin.panter, python-dev, r.david.murray, vstinner
Priority: normal Keywords: patch

Created on 2014-08-25 21:46 by akira, last changed 2016-05-13 10:00 by martin.panter. This issue is now closed.

Files
File name Uploaded Description Edit
subprocess-stderr_redirect_with_no_stdout_redirect.diff akira, 2014-08-25 21:46 review
subprocess-stderr_redirect_with_no_stdout_redirect-2.diff akira, 2016-05-08 12:46 print -> .write review
Messages (7)
msg225898 - (view) Author: Akira Li (akira) * Date: 2014-08-25 21:46
The following command should not produce any output but it does:

  $ ./python >/dev/null -c 'import subprocess as S, sys; S.call([sys.executable, "-c", "import sys; print(42, file=sys.stderr)"], stderr=S.STDOUT)'

Its stdout is redirected to /dev/null. It starts a subprocess with its
stderr redirected to stdout. See "Redirect subprocess stderr to
stdout" [1] on Stackoverflow.

[1] http://stackoverflow.com/questions/11495783/redirect-subprocess-stderr-to-stdout

I've uploaded a patch that fixes the issue on POSIX.

Please, run the provided test (in the patch), to see whether the code
should be fixed on Windows too (it might work as is there).

No documentation changes are required.

Please, review.
msg225907 - (view) Author: Josh Rosenberg (josh.r) * (Python triager) Date: 2014-08-25 23:48
Okay, dumb question: Is there a reason the Windows code explicitly initializes c2pwrite in the "stdout not passed" case, while the Linux code leaves it as -1? Windows doesn't look like it would have the problem (because c2pwrite is always set to a non-default value), and it seems like the fix for Linux could just mimic the Windows approach; the code that sets errwrite wouldn't change, but instead of a "pass", when stdout is None, we'd explicitly set it to os.STDOUT_FILENO, and the stderr=subprocess.STDOUT (stdout unset) case would work automatically, and the code would be more similar.

Haven't explored the negative consequences of that change, if any.
msg226020 - (view) Author: Akira Li (akira) * Date: 2014-08-28 12:10
Josh, on Windows, if at least one standard stream is replaced; all three
hStdInput, hStdOutput, hStdError handles are provided
(all-or-nothing). On POSIX, standard streams stdin (0), stdout (1),
stderr (2) are always inherited from the parent. Each stream can be
manipulated independently. c2pwrite=-1 is different from providing
c2pwrite=1 (STDOUT_FILENO) explicitly e.g., set_inheritable() call is
made after the fork() in the latter case.

My patch leads to dup2(fileno(stdout), STDERR_FILENO) when stdout is
None and stderr=STDOUT on POSIX i.e., it redirects stderr to the
inherited stdout (like 2>&1 in the shell). It has no effect otherwise.

sys.__stdout__ is used so that the call fails sooner without fork() if
fileno(stdout) is not a valid file descriptor when python initializes
(daemon, GUI). __stdout__-based solution doesn't support a case when
fileno(stdout) is changed later in the program e.g., using freopen() on
some systems.
msg265136 - (view) Author: Akira Li (akira) * Date: 2016-05-08 12:46
Updated the patch to address vadmium's review comments.
msg265139 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-05-08 13:27
I think this patch is pretty good. I will try to commit it in the next few days.
msg265451 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-05-13 07:55
New changeset 642933771fa5 by Martin Panter in branch '3.5':
Issue #22274: Redirect stderr=STDOUT when stdout not redirected, by Akira Li
https://hg.python.org/cpython/rev/642933771fa5

New changeset 5979e7aadd59 by Martin Panter in branch 'default':
Issue #22274: Merge stderr=STDOUT fix from 3.5
https://hg.python.org/cpython/rev/5979e7aadd59

New changeset 5f46ecaf8c6e by Martin Panter in branch '2.7':
Issue #22274: Redirect stderr=STDOUT when stdout not redirected, by Akira Li
https://hg.python.org/cpython/rev/5f46ecaf8c6e
msg265460 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-05-13 10:00
Thanks for the patch Akira.
History
Date User Action Args
2017-05-26 18:33:14eryksunlinkissue30488 superseder
2016-05-13 10:00:07martin.pantersetstatus: open -> closed
resolution: fixed
messages: + msg265460

stage: patch review -> resolved
2016-05-13 07:55:51python-devsetnosy: + python-dev
messages: + msg265451
2016-05-08 13:27:17martin.pantersetnosy: + martin.panter

messages: + msg265139
versions: + Python 2.7, Python 3.5, Python 3.6
2016-05-08 12:46:34akirasetfiles: + subprocess-stderr_redirect_with_no_stdout_redirect-2.diff

messages: + msg265136
versions: - Python 3.4
2014-08-28 12:10:11akirasetmessages: + msg226020
2014-08-26 00:34:42vstinnersetnosy: + vstinner
2014-08-25 23:48:35josh.rsetnosy: + josh.r
messages: + msg225907
2014-08-25 22:13:44r.david.murraysetnosy: + r.david.murray
2014-08-25 22:00:27ned.deilysetnosy: + gregory.p.smith

stage: patch review
2014-08-25 21:46:01akiracreate