classification
Title: Documentation lacks clear warning of subprocess issue with pythonw
Type: behavior Stage: resolved
Components: Documentation, Windows Versions: Python 3.10, Python 3.9, Python 3.8
process
Status: closed Resolution: duplicate
Dependencies: Superseder: subprocess failing in GUI applications on Windows
View: 3905
Assigned To: docs@python Nosy List: Steve Barnes, docs@python, eryksun, paul.moore, r.david.murray, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2017-03-16 16:58 by Steve Barnes, last changed 2021-03-04 16:02 by eryksun. This issue is now closed.

Messages (5)
msg289722 - (view) Author: Steve Barnes (Steve Barnes) * Date: 2017-03-16 16:58
When running under pythonw, or pyinstaller with the -w flag, modules that use subprocess calls such as popen, run, etc. will crash if the default `stdout=None, stderr=None` behaviour is used rather than PIPE. This is an obscure problem which is very hard to debug yet there is no warning in the documentation on this.

I would like to suggest adding a :warning:`stdout=None, stderr=None` must not be used in any of the calls in this module when running under pythonw due to the lack of sys.stdout & sys.stderr in that case. Please use `stdout=PIPE, stderr=PIPE` instead.

A patch against the default branch would be:
diff -r 4243df51fe43 Doc/library/subprocess.rst
--- a/Doc/library/subprocess.rst	Fri Feb 10 14:19:36 2017 +0100
+++ b/Doc/library/subprocess.rst	Thu Mar 16 16:56:24 2017 +0000
@@ -33,6 +33,13 @@
 function for all use cases it can handle. For more advanced use cases, the
 underlying :class:`Popen` interface can be used directly.
 
+.. warning:: Do not use default parameters on Windows with pythonw.
+    
+	As pythonw deletes `sys.stdout` & `sys.stderr` the use of the default 
+	parameters, `stdout=None, stderr=None,`, which defaults to being
+	`stdout=sys.stdout, stderr=sys.stderr,` may cause unexpected crashes
+	it is recommended to use `stdout=PIPE, stderr=PIPE,` instead.
+
 The :func:`run` function was added in Python 3.5; if you need to retain
 compatibility with older versions, see the :ref:`call-function-trio` section.
msg289729 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2017-03-16 19:03
A warning is not appropriate (we reserve those for things that are security related, pretty much). A sentence might be, though.  For example, we could change the initial discussion of the run function to say:

   This does not capture stdout or stderr by default. To do so, pass PIPE for the stdout and/or stderr arguments.  You *must* do this if you run your python program with pythonw on windows, since pythonw closes the standard streams.

Here I'm adding the windows sentence to the existing "This does not capture" sentence.
msg289744 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2017-03-17 04:00
An exception (not a crash) is possible if a standard handle is invalid and a Popen call tries to replace one or two of the other standard handles (e.g. stdin is invalid and you pass the argument stdout=PIPE). Note that subprocess uses the standard handles directly, so this problem is unrelated to Python's sys.std* streams. IMO, the presence of an invalid handle in the standard handles should not cause Popen to fail. This is an old problem, and it should have been addressed a long time ago by substituting a handle for \\.\NUL.

In Windows 8+ this isn't particular to running pythonw.exe. In older versions such as Windows 7, if you start pythonw.exe from a console application, then the parent's standard handle values are copied to the pythonw.exe child. But these handles are invalid because pythonw.exe doesn't automatically attach to its parent's console. (You have to manually call AttachConsole(-1) or AllocConsole(), but that's complicated because you may need to reset the CRT's standard file descriptors and FILE streams and Pythons sys.std* streams.)

Note that pythonw.exe doesn't delete or close the standard streams. They're normally set to None because, in the default case, the Windows standard handles aren't valid in pythonw.exe, and thus the CRT maps its standard file descriptors (0, 1, and 2) to INVALID_HANDLE_VALUE (i.e. -1 cast as a pointer). For example:

    C:\Temp>pyw -2 -c "import msvcrt; open('test.txt', 'w').write(str(msvcrt.get_osfhandle(1)))"

    C:\Temp>type test.txt
    18446744073709551614

You can override this by starting pythonw.exe with explicit standard handles. For example, the following redirects stderr (2) to stdout (1) and redirects stdout to a pipe:

    C:\>set "script=import sys; print(sys.executable); print(sys.stdout); print(sys.stderr)"

    C:\>2>&1 pyw -2 -c "%script%" | more
    C:\Program Files\Python27\pythonw.exe
    <open file '<stdout>', mode 'w' at 0x00000000021FB0C0>
    <open file '<stderr>', mode 'w' at 0x00000000021FB150>

    C:\>2>&1 pyw -3 -c "%script%" | more
    C:\Program Files\Python36\pythonw.exe
    <_io.TextIOWrapper name='<stdout>' mode='w' encoding='cp1252'>
    <_io.TextIOWrapper name='<stderr>' mode='w' encoding='cp1252'>
msg387748 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-02-26 21:17
This problem can be addressed most easily in _get_handles() by combining the case of an invalid standard handle with that of a NULL standard handle, for which a file handle is validated via GetFileType(). However, that leaves a small window for the handle to become invalid before duplicating it. Instead, _get_handles() could deal with a failed _make_inheritable() call by substituting a pipe handle.
msg388100 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-03-04 16:02
I forgot about the much older issue for this problem. I'm closing this issue, because the broken behavior should be fixed, not documented.
History
Date User Action Args
2021-03-04 16:02:03eryksunsetstatus: open -> closed
superseder: subprocess failing in GUI applications on Windows
messages: + msg388100

resolution: duplicate
stage: resolved
2021-02-26 21:17:25eryksunsetmessages: + msg387748
versions: + Python 3.8, Python 3.9, Python 3.10, - Python 2.7, Python 3.6, Python 3.7
2017-03-17 04:00:36eryksunsetnosy: + eryksun, paul.moore, tim.golden, zach.ware, steve.dower
messages: + msg289744
components: + Windows
2017-03-16 19:03:36r.david.murraysetnosy: + r.david.murray
messages: + msg289729
2017-03-16 16:58:21Steve Barnescreate