classification
Title: pythonw.exe fails with redirected stderr
Type: Stage:
Components: Windows Versions: Python 2.7
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: amaury.forgeotdarc, techtonik
Priority: normal Keywords:

Created on 2013-06-25 09:09 by techtonik, last changed 2013-06-26 07:46 by amaury.forgeotdarc. This issue is now closed.

Messages (8)
msg191839 - (view) Author: anatoly techtonik (techtonik) Date: 2013-06-25 09:09
---cut test.py---
print("-1-")

open("-2-", "w").write("-3-")
---cut test.py---


> C:\Python27\pythonw.exe test.py > -4-
> type -4-
-1-

> C:\Python27\pythonw.exe test.py 2> -4-
> type -4-
close failed in file object destructor:
sys.excepthook is missing
lost sys.stderr

> C:\Python27\python.exe test.py 2> -4-
-1-
>type -4-
>


This may also affect subprocess calls under pythonw.exe  I am running Python 2.7.3
msg191840 - (view) Author: anatoly techtonik (techtonik) Date: 2013-06-25 09:26
This subprocess.communicate() call fails with pythonw.exe

--cut testhg.py--
import subprocess

hg = "hg"
output, _err = subprocess.Popen([hg, 'id', '-nib'],
                   stdout=subprocess.PIPE).communicate()
open("-hg-", "w").write(output)
--cut testhg.py--


When testhg.py is run with python.exe from cmd.exe session, the -hg- file is created ok. When pythonw.exe is used, nothing happens. When redirecting stderr from pythonw.exe with:

C:\Python27\pythonw.exe testhg.py 2>1

1 contains the following stacktrace:
Traceback (most recent call last):
  File "testhg.py", line 5, in <module>
    stdout=subprocess.PIPE).communicate()
  File "C:\Python27\lib\subprocess.py", line 672, in __init__
    errread, errwrite) = self._get_handles(stdin, stdout, stderr)
  File "C:\Python27\lib\subprocess.py", line 787, in _get_handles
    p2cread = self._make_inheritable(p2cread)
  File "C:\Python27\lib\subprocess.py", line 826, in _make_inheritable
    _subprocess.DUPLICATE_SAME_ACCESS)
WindowsError: [Error 6] The handle is invalid
msg191841 - (view) Author: anatoly techtonik (techtonik) Date: 2013-06-25 09:28
This was meant to be a separate issue. :/
msg191842 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2013-06-25 10:26
Yes, in pythonw.exe the C stderr is not really usable, and this leads to unpredictable results.
It's difficult to fix in python2 though; python3 has fixed this issue, but in a way that cannot be backported.

Some workarounds:
- don't use pythonw.exe in a console, it's meant to be a *windows* application, without standard streams. I'm actually surprised of the behavior of your first example.

- use python3, where pythonw will set sys.stderr to None, and prints will be silently discarded.
msg191863 - (view) Author: anatoly techtonik (techtonik) Date: 2013-06-25 15:33
It is not about modifying Python in general, it is about patching pythonw.exe or subprocess or documenting how to make subprocess calls compatible with pythonw.exe


"don't use pythonw.exe in a console"

And how to debug the issue? Maybe the only solution in not using pythonw.exe at all if you want *windows* app solution, and do custom management of hidden windows.

http://code.google.com/p/spyderlib/source/browse/CHANGELOG
http://code.google.com/p/spyderlib/source/browse/spyderlib/utils/windows.py
msg191866 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2013-06-25 16:19
The proper solution if you use pythonw.exe is to define sys.stdout/sys.stderr yourself in pythonw.exe::
    sys.stdout = open('c:/temp/output.txt', 'w')

IOW, do the redirect from inside the program, don't rely on the shell which (by design?) does not work correctly with "/subsytem:windows" binaries.
Again, this is a workaround for 2.7, because no backward-compatible fix could be found.
msg191897 - (view) Author: anatoly techtonik (techtonik) Date: 2013-06-26 07:25
I am not using pythonw.exe, it is the option users prefer to run the program. 

pythonw.exe is a binary, how do you propose to patch that? Or is it translated to .exe with RPython?


Can you be more specific  what shell  does not work correctly, what exactly does not work correctly, and what is the backward-incompatible behaviour that you want to avoid?

pythonw.exe is meant to suppresses the terminal window on startup (console window to be exact), but not to kill vital streams for an application. I posted links Spyder IDE source to show how it should be done.
msg191898 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2013-06-26 07:46
RPython... let's be serious. The code of pythonw.exe is very simple, see PC/WinMain.c.

No, pythonw.exe is not "meant to suppresses the terminal window on startup". This is only a consequence of being a "windows application". There is a lot of documentation about this, for example:
http://comsci.liu.edu/~murali/win32gui/Win32Apps.htm

- python.exe is a regular console application, with a main() function.
- pythonw.exe is a gui application, with a WinMain() function; as a consequence, stdout/stderr won't work properly.

Again, it's a won't fix for 2.7, unless someone comes with a patch. But this has already been tried, and be careful, different versions of the msvcrt differ in behavior here.
History
Date User Action Args
2013-06-26 07:46:33amaury.forgeotdarcsetstatus: open -> closed
resolution: wont fix
messages: + msg191898
2013-06-26 07:25:59techtoniksetstatus: closed -> open
resolution: wont fix -> (no value)
messages: + msg191897
2013-06-25 16:19:36amaury.forgeotdarcsetstatus: open -> closed
resolution: wont fix
messages: + msg191866
2013-06-25 15:33:05techtoniksetstatus: closed -> open
resolution: wont fix -> (no value)
messages: + msg191863
2013-06-25 10:26:15amaury.forgeotdarcsetstatus: open -> closed

nosy: + amaury.forgeotdarc
messages: + msg191842

resolution: wont fix
2013-06-25 09:28:20techtoniksetmessages: + msg191841
2013-06-25 09:26:22techtoniksetmessages: + msg191840
2013-06-25 09:10:27techtoniksetcomponents: + Windows
2013-06-25 09:09:58techtoniksettitle: pythonw.exe fails with redirected stdett -> pythonw.exe fails with redirected stderr
2013-06-25 09:09:46techtonikcreate