pythonw.exe fails to run with a runtime error. python.exe works as
expected. While the bug itself isn't serious it should either be fixed
or pythonw.exe be omitted from the next alpha release.

Update:

pythonw fails because the standard streams can't be initialized.
fileno(stdin), fileno(stdout) and fileno(stderr) are returning -2.

Python 2.6 (svn trunk) doesn't check the file handlers when it creates
sys.stdin, stdout and stderr. write() operations to stdout and stderr
don't fail although the data is written into nowhere land. stdin.read()
fails with IOError: [Errno 9] Bad file descriptor.

How should I address the problem in py3k?

Congratulations Amaury and welcome on board! :)

I like to get your opinion on the problem. So far I've figured out that
Windows doesn't create the std streams for Windows WinMain() programs.
As first step towards the solution I like to separate the close check
from the fd. Currently Module/_fileio.c:file_close() sets the fd to a
negative value and open fails immediately with a negative fd. I want to
add a closed flag to the struct and move the fd < 0 check to the read
and write operations.

That's going to fix stdin and stdout for non console based programs on
Windows. For stderr I've to think about a better solution to avoid an
infinite loop (stderr.write() -> exception -> stderr.write() ...). Maybe
I could add some more methods to the dumb writer to make it more file
like and use it?

PATCH:
* remove the analogy fd < 0 -> file is closed from _fileio.FileIO
* added new flag closed to _fileio.FileIO
* renamed closefd to close_fd to distinguish it from closed
* make it impossible to instantiate another stdprinter
* added repr and fileno methods to stdprinter

Guido:
Are you fine with the changes? The patch doesn't fix the problem (yet)
but it's the first step towards a solution.

The new patch fixes the startup problem with pythonw.exe on Windows. I
still wonder if print(sometest) is raising an exception when stdout is
not available.

Hmm... In internal_close() there's still a test for self->fd >= 0. I'm
not sure if this is an oversight or intentional.

Also, I don't understand under what circumstances fds < 0 can occur. I
presume this is only on Windows. Can you point me to docs for this
fact?

Guido van Rossum wrote:
> Hmm... In internal_close() there's still a test for self->fd >= 0. I'm
> not sure if this is an oversight or intentional.

I'll check it later.

The patch still contains some debugging code that redirects stdout and
stderr to a file when PY_STDERR_FILE is defined.

> Also, I don't understand under what circumstances fds < 0 can occur. I
> presume this is only on Windows. Can you point me to docs for this
> fact?

It happens when a script is run with pythonw.exe (pyw extension).
PythonW.exe isn't a console application but a GUI app which doesn't
create a console window. However GUI apps don't have valid standard
streams because stdin, stdout and stderr aren't connected.

Here are some links that shed some light on the problem:

http://mail.python.org/pipermail/python-dev/2001-January/011423.html
http://www.halcyon.com/~ast/dload/guicon.htm
http://msdn2.microsoft.com/en-us/library/3x292kth(VS.80).aspx

The patch creates another problem:
http://bugs.python.org/issue1422

Christian

I made some checks with the vc2005 (msvcr80) runtime library:
- fd==-2 corresponds to the _NO_CONSOLE_FILENO constant.
A comment in the CRT code says:
    /*
     * For stdin, stdout & stderr, we use _NO_CONSOLE_FILENO (a value
     * different from _INVALID_HANDLE_VALUE to distinguish between
     * a failure in opening a file & a program run without a console.
     */

- in this case, stderr is a buffered FILE*, but the flush() method
always fails. This makes not difference for python2.5, because it never
looks at the return value of fprintf (which is not very consistent, BTW).

Since pythonw (2.5) silently discards any output to stderr, we could
achieve the same by opening the nul file...

--- c:/temp/t	2007-11-12 13:54:34.105463200 +0100
+++ c:/afa/python/py3k/Modules/_fileio.c	2007-11-12 13:52:42.576675100 +0100
@@ -149,6 +149,15 @@
 
 	if (PyArg_ParseTupleAndKeywords(args, kwds, "i|si:fileio",
 					kwlist, &fd, &mode, &closefd)) {
+#ifdef MS_WINDOWS
+		/* Windows sets the descriptor to _NO_CONSOLE_FILENO when */
+		/* the program is run without a console */
+		if (fd == -2)
+		{
+			fd = open("nul", "r+");
+		}
+		else
+#endif
 		if (fd < 0) {
 			PyErr_SetString(PyExc_ValueError,
 					"Negative filedescriptor");

The patch is a good idea. However it doesn't work for VS 2003 and MSVCR71:

import sys
f = open("stderr.txt", "w")
f.write("stdin: %i\n" % sys.stdin.fileno())
f.write("stdout: %i\n" % sys.stdout.fileno())
f.write("stderr: %i\n" % sys.stderr.fileno())

> pythonw.exe pywtest.py
> type stderr.txt
stdin: -1
stdout: -1
stderr: -1

/me sends another hate letter to Redmond.

Doh, you're right:
> c:\python24\pythonw -c "import sys;print sys.stderr.fileno()"|more
-1
> c:\python24-vc8\pythonw -c "import sys;print sys.stderr.fileno()"|more
-2

/me needs to get the code of msvcrt7.

We could simply check for (fd<0) instead, but it's better to reserve
this special processing to stdin/stdout/stderr. maybe somewhere in
pythonrun.c. I'll try a patch later tonight.

IMO you don't need the 'closed' flag and you should continue to test for
fd < 0.  Whether it's -1 or -2, you still can't write to it...

W/o the closed flag it's impossible to distinguish between closed fd and
invalid fd.

I don't understand. When does the difference matter?

On Nov 12, 2007 10:14 AM, Christian Heimes <report@bugs.python.org> wrote:
>
> Christian Heimes added the comment:
>
> W/o the closed flag it's impossible to distinguish between closed fd and
> invalid fd.

Guido van Rossum wrote:
> Guido van Rossum added the comment:
> 
> I don't understand. When does the difference matter?

When the check for fd < 0 is removed from fileio_init() there is no way
to tell if fd < 0 means fd closed or invalid fd.

In order to fix the problem with GUI apps on Windows the check for fd <
0 has to be removed (Python 2.x way). Or we use Amaury's way and open
NUL for reading and writing as a substitute for the standard streams.

Christian

> > I don't understand. When does the difference matter?
>
> When the check for fd < 0 is removed from fileio_init() there is no way
> to tell if fd < 0 means fd closed or invalid fd.

I still don't understand. Why do you need to treat a closed fd
different from an invalid fd. In both cases you can't use it and you
shouldn't close it.

> In order to fix the problem with GUI apps on Windows the check for fd <
> 0 has to be removed (Python 2.x way). Or we use Amaury's way and open
> NUL for reading and writing as a substitute for the standard streams.

I'd suggest that, on Windows, sys.std{in,out.err} should be set to
None instead of a file object when their file descriptor is invalid.
That way print and write requests will fail immediately instead of
nondeterministically. With an invalid file that only blows upwhen
trying to flush you may be able to write a small traceback but it will
still blow up if the traceback is big.

Is there a downside to print/write blowing up right away in apps using pythonw?

Guido van Rossum wrote:
> I still don't understand. Why do you need to treat a closed fd
> different from an invalid fd. In both cases you can't use it and you
> shouldn't close it.

I don't want to treat the fd differently. I want to return a sensible
error message that is different from "file closed" when the fd is invalid.

> I'd suggest that, on Windows, sys.std{in,out.err} should be set to
> None instead of a file object when their file descriptor is invalid.
> That way print and write requests will fail immediately instead of
> nondeterministically. With an invalid file that only blows upwhen
> trying to flush you may be able to write a small traceback but it will
> still blow up if the traceback is big.

But wouldn't that cause a fatal error when sys.stderr is missing and
Python can't write the traceback to a file like object?

*testing*

No, it works:

object  : Exception('msg',)
type    : Exception
refcount: 4
address : 0x839d274
lost sys.stderr

I've an alternative solution based on Amaurgy's idea. Python could set
replacements for stdin, stdout and stderr before it sets up the
preliminary stderr:

#ifdef MS_WINDOWS
	/* The standard streams of Windows GUI apps aren't connected. */
	if (fileno(stdin) < 0) {
		if (freopen("NUL", "rb", stdin) == NULL)
			Py_FatalError("Py_Initialize: failed to replace stdin");
	}
	if (fileno(stdout) < 0) {
		if (freopen("NUL", "wb", stdout) == NULL)
			Py_FatalError("Py_Initialize: failed to replace stdout");
	}
	if (fileno(stderr) < 0) {
		if (freopen("NUL", "wb", stderr) == NULL)
			Py_FatalError("Py_Initialize: failed to replace stderr");
	}
#endif

The mail gateway swallows examples:

>>> import sys
>>> sys.stderr = None
>>> raise Exception("msg")
>>> del sys.stderr
>>> raise Exception("msg")
object  : Exception('msg',)
type    : Exception
refcount: 4
address : 0x839d274
lost sys.stderr

> I don't want to treat the fd differently. I want to return a sensible
> error message that is different from "file closed" when the fd is invalid.

Doesn't sound too important.  Anyway, from which call do you want to
issue different messages?

I'd rather see sys.stdout == None than opening NUL.

Fixed in r58959

I followed your wish and set sys.stdin, stdout and stderr to None
together with __std?__. I also kept the check for fd < 0 in
fileio_init(). A negative fd still raises the correct error with errno 9
(bad file descriptor).

I consider the bug fixed and closed. Objections?

(Sorry, I cannot test just now)
What happens if pythonw.exe calls the print() function?
Please tell me that it does not throw an exception.

Since r58962 it doesn't ;)

Is it the only possibility?

On Windows, it is quite common to print to stdout for debugging
purposes, then deploy the application with pythonw.exe which suppresses
the console. Without any change to the code. Most pygame programs I know
work this way, and C programs have the same behaviour.

Prints should work, even if they discard their output.

As far as I can see print() works if sys.stdout is either None (discard
output ASAP) or a file like object. Even print(file=syst.stderr) works.

sys.stdout.write() is going to fail when sys.stdout is None but that's
not my concern. It's another well documented difference between Python
2.x and 3.x. If people still need a valid but no op stdout they can set
up their own:

class NullWriter:
    def write(self, data):
        pass

if sys.stdout is None:
    sys.stdout = NullWriter()

Done ;)

> As far as I can see print() works if sys.stdout is either None
> (discard output ASAP) or a file like object. Even
> print(file=syst.stderr) works.

Ah, I prefer this.

> sys.stdout.write() is going to fail when sys.stdout is None 
> but that's not my concern. 

It's not very important indeed.

I'm happy with the current behaviour. Let's close this issue.