Title: Documentation lacks clear warning of subprocess issue with pythonw
Type: behavior Stage:
Components: Documentation, Windows Versions: Python 3.7, Python 3.6, Python 2.7
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: Steve Barnes, docs@python, eryksun, paul.moore, r.david.murray, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2017-03-16 16:58 by Steve Barnes, last changed 2017-03-17 04:00 by eryksun.

Messages (3)
msg289722 - (view) Author: Steve Barnes (Steve Barnes) * Date: 2017-03-16 16:58
When running under pythonw, or pyinstaller with the -w flag, modules that use subprocess calls such as popen, run, etc. will crash if the default `stdout=None, stderr=None` behaviour is used rather than PIPE. This is an obscure problem which is very hard to debug yet there is no warning in the documentation on this.

I would like to suggest adding a :warning:`stdout=None, stderr=None` must not be used in any of the calls in this module when running under pythonw due to the lack of sys.stdout & sys.stderr in that case. Please use `stdout=PIPE, stderr=PIPE` instead.

A patch against the default branch would be:
diff -r 4243df51fe43 Doc/library/subprocess.rst
--- a/Doc/library/subprocess.rst	Fri Feb 10 14:19:36 2017 +0100
+++ b/Doc/library/subprocess.rst	Thu Mar 16 16:56:24 2017 +0000
@@ -33,6 +33,13 @@
 function for all use cases it can handle. For more advanced use cases, the
 underlying :class:`Popen` interface can be used directly.
+.. warning:: Do not use default parameters on Windows with pythonw.
+	As pythonw deletes `sys.stdout` & `sys.stderr` the use of the default 
+	parameters, `stdout=None, stderr=None,`, which defaults to being
+	`stdout=sys.stdout, stderr=sys.stderr,` may cause unexpected crashes
+	it is recommended to use `stdout=PIPE, stderr=PIPE,` instead.
 The :func:`run` function was added in Python 3.5; if you need to retain
 compatibility with older versions, see the :ref:`call-function-trio` section.
msg289729 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2017-03-16 19:03
A warning is not appropriate (we reserve those for things that are security related, pretty much). A sentence might be, though.  For example, we could change the initial discussion of the run function to say:

   This does not capture stdout or stderr by default. To do so, pass PIPE for the stdout and/or stderr arguments.  You *must* do this if you run your python program with pythonw on windows, since pythonw closes the standard streams.

Here I'm adding the windows sentence to the existing "This does not capture" sentence.
msg289744 - (view) Author: Eryk Sun (eryksun) * Date: 2017-03-17 04:00
An exception (not a crash) is possible if a standard handle is invalid and a Popen call tries to replace one or two of the other standard handles (e.g. stdin is invalid and you pass the argument stdout=PIPE). Note that subprocess uses the standard handles directly, so this problem is unrelated to Python's sys.std* streams. IMO, the presence of an invalid handle in the standard handles should not cause Popen to fail. This is an old problem, and it should have been addressed a long time ago by substituting a handle for \\.\NUL.

In Windows 8+ this isn't particular to running pythonw.exe. In older versions such as Windows 7, if you start pythonw.exe from a console application, then the parent's standard handle values are copied to the pythonw.exe child. But these handles are invalid because pythonw.exe doesn't automatically attach to its parent's console. (You have to manually call AttachConsole(-1) or AllocConsole(), but that's complicated because you may need to reset the CRT's standard file descriptors and FILE streams and Pythons sys.std* streams.)

Note that pythonw.exe doesn't delete or close the standard streams. They're normally set to None because, in the default case, the Windows standard handles aren't valid in pythonw.exe, and thus the CRT maps its standard file descriptors (0, 1, and 2) to INVALID_HANDLE_VALUE (i.e. -1 cast as a pointer). For example:

    C:\Temp>pyw -2 -c "import msvcrt; open('test.txt', 'w').write(str(msvcrt.get_osfhandle(1)))"

    C:\Temp>type test.txt

You can override this by starting pythonw.exe with explicit standard handles. For example, the following redirects stderr (2) to stdout (1) and redirects stdout to a pipe:

    C:\>set "script=import sys; print(sys.executable); print(sys.stdout); print(sys.stderr)"

    C:\>2>&1 pyw -2 -c "%script%" | more
    C:\Program Files\Python27\pythonw.exe
    <open file '<stdout>', mode 'w' at 0x00000000021FB0C0>
    <open file '<stderr>', mode 'w' at 0x00000000021FB150>

    C:\>2>&1 pyw -3 -c "%script%" | more
    C:\Program Files\Python36\pythonw.exe
    <_io.TextIOWrapper name='<stdout>' mode='w' encoding='cp1252'>
    <_io.TextIOWrapper name='<stderr>' mode='w' encoding='cp1252'>
Date User Action Args
2017-03-17 04:00:36eryksunsetnosy: + eryksun, paul.moore, tim.golden, zach.ware, steve.dower
messages: + msg289744
components: + Windows
2017-03-16 19:03:36r.david.murraysetnosy: + r.david.murray
messages: + msg289729
2017-03-16 16:58:21Steve Barnescreate