classification
Title: py3k: pythonw.exe fails because std streams a missing
Type: crash Stage:
Components: Windows Versions: Python 3.0
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: amaury.forgeotdarc, christian.heimes, gvanrossum
Priority: high Keywords:

Created on 2007-11-10 00:00 by christian.heimes, last changed 2008-01-06 22:29 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
py3k_fileio_closed.patch christian.heimes, 2007-11-11 00:10
py3k_fileio_fixes.patch christian.heimes, 2007-11-11 02:55
Messages (26)
msg57342 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2007-11-10 00:00
pythonw.exe fails to run with a runtime error. python.exe works as
expected. While the bug itself isn't serious it should either be fixed
or pythonw.exe be omitted from the next alpha release.
msg57343 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2007-11-10 00:13
Update:

pythonw fails because the standard streams can't be initialized.
fileno(stdin), fileno(stdout) and fileno(stderr) are returning -2.
msg57344 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2007-11-10 01:20
Python 2.6 (svn trunk) doesn't check the file handlers when it creates
sys.stdin, stdout and stderr. write() operations to stdout and stderr
don't fail although the data is written into nowhere land. stdin.read()
fails with IOError: [Errno 9] Bad file descriptor.

How should I address the problem in py3k?
msg57349 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2007-11-10 14:44
Congratulations Amaury and welcome on board! :)

I like to get your opinion on the problem. So far I've figured out that
Windows doesn't create the std streams for Windows WinMain() programs.
As first step towards the solution I like to separate the close check
from the fd. Currently Module/_fileio.c:file_close() sets the fd to a
negative value and open fails immediately with a negative fd. I want to
add a closed flag to the struct and move the fd < 0 check to the read
and write operations.

That's going to fix stdin and stdout for non console based programs on
Windows. For stderr I've to think about a better solution to avoid an
infinite loop (stderr.write() -> exception -> stderr.write() ...). Maybe
I could add some more methods to the dumb writer to make it more file
like and use it?
msg57363 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2007-11-11 00:10
PATCH:
* remove the analogy fd < 0 -> file is closed from _fileio.FileIO
* added new flag closed to _fileio.FileIO
* renamed closefd to close_fd to distinguish it from closed
* make it impossible to instantiate another stdprinter
* added repr and fileno methods to stdprinter

Guido:
Are you fine with the changes? The patch doesn't fix the problem (yet)
but it's the first step towards a solution.
msg57365 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2007-11-11 02:55
The new patch fixes the startup problem with pythonw.exe on Windows. I
still wonder if print(sometest) is raising an exception when stdout is
not available.
msg57373 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2007-11-11 17:37
Hmm... In internal_close() there's still a test for self->fd >= 0. I'm
not sure if this is an oversight or intentional.

Also, I don't understand under what circumstances fds < 0 can occur. I
presume this is only on Windows. Can you point me to docs for this
fact?
msg57374 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2007-11-11 17:48
Guido van Rossum wrote:
> Hmm... In internal_close() there's still a test for self->fd >= 0. I'm
> not sure if this is an oversight or intentional.

I'll check it later.

The patch still contains some debugging code that redirects stdout and
stderr to a file when PY_STDERR_FILE is defined.

> Also, I don't understand under what circumstances fds < 0 can occur. I
> presume this is only on Windows. Can you point me to docs for this
> fact?

It happens when a script is run with pythonw.exe (pyw extension).
PythonW.exe isn't a console application but a GUI app which doesn't
create a console window. However GUI apps don't have valid standard
streams because stdin, stdout and stderr aren't connected.

Here are some links that shed some light on the problem:

http://mail.python.org/pipermail/python-dev/2001-January/011423.html
http://www.halcyon.com/~ast/dload/guicon.htm
http://msdn2.microsoft.com/en-us/library/3x292kth(VS.80).aspx

The patch creates another problem:
http://bugs.python.org/issue1422

Christian
msg57397 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2007-11-12 13:09
I made some checks with the vc2005 (msvcr80) runtime library:
- fd==-2 corresponds to the _NO_CONSOLE_FILENO constant.
A comment in the CRT code says:
    /*
     * For stdin, stdout & stderr, we use _NO_CONSOLE_FILENO (a value
     * different from _INVALID_HANDLE_VALUE to distinguish between
     * a failure in opening a file & a program run without a console.
     */

- in this case, stderr is a buffered FILE*, but the flush() method
always fails. This makes not difference for python2.5, because it never
looks at the return value of fprintf (which is not very consistent, BTW).

Since pythonw (2.5) silently discards any output to stderr, we could
achieve the same by opening the nul file...

--- c:/temp/t	2007-11-12 13:54:34.105463200 +0100
+++ c:/afa/python/py3k/Modules/_fileio.c	2007-11-12 13:52:42.576675100 +0100
@@ -149,6 +149,15 @@
 
 	if (PyArg_ParseTupleAndKeywords(args, kwds, "i|si:fileio",
 					kwlist, &fd, &mode, &closefd)) {
+#ifdef MS_WINDOWS
+		/* Windows sets the descriptor to _NO_CONSOLE_FILENO when */
+		/* the program is run without a console */
+		if (fd == -2)
+		{
+			fd = open("nul", "r+");
+		}
+		else
+#endif
 		if (fd < 0) {
 			PyErr_SetString(PyExc_ValueError,
 					"Negative filedescriptor");
msg57398 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2007-11-12 14:46
The patch is a good idea. However it doesn't work for VS 2003 and MSVCR71:

import sys
f = open("stderr.txt", "w")
f.write("stdin: %i\n" % sys.stdin.fileno())
f.write("stdout: %i\n" % sys.stdout.fileno())
f.write("stderr: %i\n" % sys.stderr.fileno())

> pythonw.exe pywtest.py
> type stderr.txt
stdin: -1
stdout: -1
stderr: -1

/me sends another hate letter to Redmond.
msg57399 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2007-11-12 15:28
Doh, you're right:
> c:\python24\pythonw -c "import sys;print sys.stderr.fileno()"|more
-1
> c:\python24-vc8\pythonw -c "import sys;print sys.stderr.fileno()"|more
-2

/me needs to get the code of msvcrt7.

We could simply check for (fd<0) instead, but it's better to reserve
this special processing to stdin/stdout/stderr. maybe somewhere in
pythonrun.c. I'll try a patch later tonight.
msg57415 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2007-11-12 17:51
IMO you don't need the 'closed' flag and you should continue to test for
fd < 0.  Whether it's -1 or -2, you still can't write to it...
msg57417 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2007-11-12 18:14
W/o the closed flag it's impossible to distinguish between closed fd and
invalid fd.
msg57425 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2007-11-12 20:11
I don't understand. When does the difference matter?

On Nov 12, 2007 10:14 AM, Christian Heimes <report@bugs.python.org> wrote:
>
> Christian Heimes added the comment:
>
> W/o the closed flag it's impossible to distinguish between closed fd and
> invalid fd.
msg57426 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2007-11-12 20:57
Guido van Rossum wrote:
> Guido van Rossum added the comment:
> 
> I don't understand. When does the difference matter?

When the check for fd < 0 is removed from fileio_init() there is no way
to tell if fd < 0 means fd closed or invalid fd.

In order to fix the problem with GUI apps on Windows the check for fd <
0 has to be removed (Python 2.x way). Or we use Amaury's way and open
NUL for reading and writing as a substitute for the standard streams.

Christian
msg57427 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2007-11-12 21:09
> > I don't understand. When does the difference matter?
>
> When the check for fd < 0 is removed from fileio_init() there is no way
> to tell if fd < 0 means fd closed or invalid fd.

I still don't understand. Why do you need to treat a closed fd
different from an invalid fd. In both cases you can't use it and you
shouldn't close it.

> In order to fix the problem with GUI apps on Windows the check for fd <
> 0 has to be removed (Python 2.x way). Or we use Amaury's way and open
> NUL for reading and writing as a substitute for the standard streams.

I'd suggest that, on Windows, sys.std{in,out.err} should be set to
None instead of a file object when their file descriptor is invalid.
That way print and write requests will fail immediately instead of
nondeterministically. With an invalid file that only blows upwhen
trying to flush you may be able to write a small traceback but it will
still blow up if the traceback is big.

Is there a downside to print/write blowing up right away in apps using pythonw?
msg57428 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2007-11-12 21:32
Guido van Rossum wrote:
> I still don't understand. Why do you need to treat a closed fd
> different from an invalid fd. In both cases you can't use it and you
> shouldn't close it.

I don't want to treat the fd differently. I want to return a sensible
error message that is different from "file closed" when the fd is invalid.

> I'd suggest that, on Windows, sys.std{in,out.err} should be set to
> None instead of a file object when their file descriptor is invalid.
> That way print and write requests will fail immediately instead of
> nondeterministically. With an invalid file that only blows upwhen
> trying to flush you may be able to write a small traceback but it will
> still blow up if the traceback is big.

But wouldn't that cause a fatal error when sys.stderr is missing and
Python can't write the traceback to a file like object?

*testing*

No, it works:

object  : Exception('msg',)
type    : Exception
refcount: 4
address : 0x839d274
lost sys.stderr

I've an alternative solution based on Amaurgy's idea. Python could set
replacements for stdin, stdout and stderr before it sets up the
preliminary stderr:

#ifdef MS_WINDOWS
	/* The standard streams of Windows GUI apps aren't connected. */
	if (fileno(stdin) < 0) {
		if (freopen("NUL", "rb", stdin) == NULL)
			Py_FatalError("Py_Initialize: failed to replace stdin");
	}
	if (fileno(stdout) < 0) {
		if (freopen("NUL", "wb", stdout) == NULL)
			Py_FatalError("Py_Initialize: failed to replace stdout");
	}
	if (fileno(stderr) < 0) {
		if (freopen("NUL", "wb", stderr) == NULL)
			Py_FatalError("Py_Initialize: failed to replace stderr");
	}
#endif
msg57429 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2007-11-12 21:34
The mail gateway swallows examples:

>>> import sys
>>> sys.stderr = None
>>> raise Exception("msg")
>>> del sys.stderr
>>> raise Exception("msg")
object  : Exception('msg',)
type    : Exception
refcount: 4
address : 0x839d274
lost sys.stderr
msg57431 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2007-11-12 22:10
> I don't want to treat the fd differently. I want to return a sensible
> error message that is different from "file closed" when the fd is invalid.

Doesn't sound too important.  Anyway, from which call do you want to
issue different messages?

I'd rather see sys.stdout == None than opening NUL.
msg57437 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2007-11-13 02:26
Fixed in r58959

I followed your wish and set sys.stdin, stdout and stderr to None
together with __std?__. I also kept the check for fd < 0 in
fileio_init(). A negative fd still raises the correct error with errno 9
(bad file descriptor).
msg57453 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2007-11-13 14:37
I consider the bug fixed and closed. Objections?
msg57454 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2007-11-13 15:06
(Sorry, I cannot test just now)
What happens if pythonw.exe calls the print() function?
Please tell me that it does not throw an exception.
msg57455 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2007-11-13 15:26
Since r58962 it doesn't ;)
msg57457 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2007-11-13 15:57
Is it the only possibility?

On Windows, it is quite common to print to stdout for debugging
purposes, then deploy the application with pythonw.exe which suppresses
the console. Without any change to the code. Most pygame programs I know
work this way, and C programs have the same behaviour.

Prints should work, even if they discard their output.
msg57459 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2007-11-13 16:35
As far as I can see print() works if sys.stdout is either None (discard
output ASAP) or a file like object. Even print(file=syst.stderr) works.

sys.stdout.write() is going to fail when sys.stdout is None but that's
not my concern. It's another well documented difference between Python
2.x and 3.x. If people still need a valid but no op stdout they can set
up their own:

class NullWriter:
    def write(self, data):
        pass

if sys.stdout is None:
    sys.stdout = NullWriter()

Done ;)
msg57460 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2007-11-13 17:07
> As far as I can see print() works if sys.stdout is either None
> (discard output ASAP) or a file like object. Even
> print(file=syst.stderr) works.

Ah, I prefer this.

> sys.stdout.write() is going to fail when sys.stdout is None 
> but that's not my concern. 

It's not very important indeed.

I'm happy with the current behaviour. Let's close this issue.
History
Date User Action Args
2008-01-06 22:29:45adminsetkeywords: - py3k
versions: Python 3.0
2007-11-13 17:07:29amaury.forgeotdarcsetmessages: + msg57460
2007-11-13 16:35:22christian.heimessetmessages: + msg57459
2007-11-13 15:57:58amaury.forgeotdarcsetmessages: + msg57457
2007-11-13 15:26:16christian.heimessetmessages: + msg57455
2007-11-13 15:06:50amaury.forgeotdarcsetmessages: + msg57454
2007-11-13 14:37:01christian.heimessetstatus: open -> closed
messages: + msg57453
2007-11-13 02:26:50christian.heimessetresolution: fixed
messages: + msg57437
2007-11-12 22:10:36gvanrossumsetmessages: + msg57431
2007-11-12 21:34:19christian.heimessetmessages: + msg57429
2007-11-12 21:32:20christian.heimessetmessages: + msg57428
2007-11-12 21:09:46gvanrossumsetmessages: + msg57427
2007-11-12 20:57:22christian.heimessetmessages: + msg57426
2007-11-12 20:11:54gvanrossumsetmessages: + msg57425
2007-11-12 18:14:32christian.heimessetmessages: + msg57417
2007-11-12 17:51:15gvanrossumsetmessages: + msg57415
2007-11-12 15:28:10amaury.forgeotdarcsetmessages: + msg57399
2007-11-12 14:46:26christian.heimessetmessages: + msg57398
2007-11-12 13:09:46amaury.forgeotdarcsetmessages: + msg57397
2007-11-11 17:48:43christian.heimessetmessages: + msg57374
2007-11-11 17:37:58gvanrossumsetmessages: + msg57373
2007-11-11 02:55:35christian.heimessetfiles: + py3k_fileio_fixes.patch
messages: + msg57365
2007-11-11 00:10:46christian.heimessetfiles: + py3k_fileio_closed.patch
nosy: + gvanrossum
messages: + msg57363
2007-11-10 14:44:46christian.heimessetnosy: + amaury.forgeotdarc
messages: + msg57349
title: py3k: pythonw.exe fails to run -> py3k: pythonw.exe fails because std streams a missing
2007-11-10 01:20:32christian.heimessetmessages: + msg57344
2007-11-10 00:13:27christian.heimessetmessages: + msg57343
2007-11-10 00:00:27christian.heimescreate