bash$ python3.1 -c 'pass' 2>&-
Aborted (core dumped)

(I verified, the core dump belongs to python.)

If you remove the redirection thingy at the end, it works.

Not sure why I ever wrote that code, but it has been working since
forever up to python3.0.

The problem is the check_fd in _fileio.c checks fstat of 2, which
returns EBADFD. I'm not sure what about this redirection makes it a bad
file descriptor, though..

After some searching around with Google, "2>&-" means "close file
descriptor 2" (i.e., standard error).

Please note that normally an error message is output, but of course it
doesn't display since stderr is invalid :-)

It's clearer if you close stdout instead:

$ ./python -c 'pass' >&-
Fatal Python error: Py_Initialize: can't initialize sys standard streams
OSError: [Errno 9] Bad file descriptor
Abandon

If we want to allow for closed {stdin, stdout, stderr}, I'm not sure
what the semantics should be. Should sys.std{in, out, err} be None? Or a
file object which always throws an error?

Under Python 2.x, you don't get a crash but the behaviour is quite
unhelpful anyway:

$ python -c 'print 1' >&-
close failed in file object destructor:
Error in sys.excepthook:

Original exception was:

Is it even possible to portably test the validity of a file descriptor
without trying to write/read it?

When I first saw this bug, my gut feeling was "well, don't do that
then!"  However, I then recalled that Windows GUI applications have no
stdin, stdout, or stderr.  

Python 2 will raise IOError: Bad File Descriptor when the user tries to
write to stdout or stderr (more accurately, it raises the exception when
trying to flush data to the file descriptor).  

I just tested pythonw.exe.  If I set sys.stderr by hand to a file, then
write to sys.stdout, 2.6 will correctly write the exception to the file.
 3.1 exits silently.

> 3.1 exits silently.
Did you use "print"? pythonw.exe 3.1 sets sys.stdout to None.
if you use sys.stdout.write, you get an exception. But print() silently
does nothing if the file is None.

For what it's worth, the code in question is used here (using "import
distutils" instead of "pass"):

http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/config/python.m4?rev=1.15;content-type=text%2Fx-cvsweb-markup

This is obviously a completely gratuitous variant on 2>/dev/null, but it
has apparently been working since forever.  I'll probably go make the
change anyway.

Nevertheless, I think Python shouldn't core dump.  It may choose to exit
doing nothing (useful) if it doesn't want to deal with this case.

Check this for possible alternative behaviors:

$ ls 1>&-
ls: write error: Bad file descriptor
($? = 2)
$ ls 1>&- 2>&-
($? = 2, no output)

sorry, title restored!

> If we want to allow for closed {stdin, stdout, stderr}, I'm not sure
> what the semantics should be. Should sys.std{in, out, err} be None? Or a
> file object which always throws an error?

I would say it should be a *pseudo*-file object which always throws a *descriptive* error.  Note that setting sys.stdout to None makes print() do nothing rather than report an error:

>>> sys.stdout = None
>>> print('abc')

See also issue6501.

On the second thought, as long as python used fd 2 as the "message stream of last resort", we should probably not allow it to run with fd 2 closed. The problem is that in this case fd 2 may become associated with a very important file contents of which you don't want to see replaced with a python error message.

That's an interesting point.

Do you know of places where we use fd 2 instead of sys.stderr?

> That's an interesting point.
> 
> Do you know of places where we use fd 2 instead of sys.stderr?

We normally don't. One reason is that buffering inside sys.stderr can
make ordering of output incorrect. There are some places in C code where
we do "fprintf(stderr, ...)" but that's for specialized debugging
(disabled in normal builds) or fatal error messages.

On Thu, Feb 3, 2011 at 2:44 PM, Antoine Pitrou <report@bugs.python.org> wrote:
..
>> Do you know of places where we use fd 2 instead of sys.stderr?
>
> We normally don't. One reason is that buffering inside sys.stderr can
> make ordering of output incorrect. There are some places in C code where
> we do "fprintf(stderr, ...)" but that's for specialized debugging
> (disabled in normal builds) or fatal error messages.

This is the case that I had in mind.  What does non-debug build do on
a fatal error?  Also, can we be sure that Python does not call C
library functions that write to stderr behind the scenes?  If vanilla
Python is safe to run with closed fd 2, that may not be the case for
3rd party extensions.    What is the use case for "python >&-"?    Is
it important enough to justify the risk of accidental data loss?

> On Thu, Feb 3, 2011 at 2:44 PM, Antoine Pitrou <report@bugs.python.org> wrote:
> ..
>> Do you know of places where we use fd 2 instead of sys.stderr?
>
> We normally don't.

Hmm, grep "fprintf(stderr," returned 122 hits in the py3k branch.
Are you sure these are all debug-build only?

> > We normally don't. One reason is that buffering inside sys.stderr can
> > make ordering of output incorrect. There are some places in C code where
> > we do "fprintf(stderr, ...)" but that's for specialized debugging
> > (disabled in normal builds) or fatal error messages.
> 
> This is the case that I had in mind.  What does non-debug build do on
> a fatal error?

It uses fprintf(stderr, ...). That's the only thing it can do (there's
no way sys.stderr is guaranteed to be usable at that point). If C stderr
is invalid, then too bad.

> Also, can we be sure that Python does not call C
> library functions that write to stderr behind the scenes?

I think you can guess the answer :)

> What is the use case for "python >&-"?    Is
> it important enough to justify the risk of accidental data loss?

I don't think so. One more important use case is when running a Unix
daemon, which has (AFAIK) to close all std handles. I don't know how
that interacts with using C stderr, especially if the handle closing is
done in Python (and therefore only calls C close() and not fclose()!).

Perhaps we should provide a sys function to fclose() C std{in,out,err}.

Le jeudi 03 février 2011 à 19:59 +0000, Alexander Belopolsky a écrit :
> Alexander Belopolsky <belopolsky@users.sourceforge.net> added the comment:
> 
> > On Thu, Feb 3, 2011 at 2:44 PM, Antoine Pitrou <report@bugs.python.org> wrote:
> > ..
> >> Do you know of places where we use fd 2 instead of sys.stderr?
> >
> > We normally don't.
> 
> Hmm, grep "fprintf(stderr," returned 122 hits in the py3k branch.
> Are you sure these are all debug-build only?

"grep -C2" seems to say most of them are. I haven't done a survey.

On Thu, Feb 3, 2011 at 11:56 AM, Alexander Belopolsky
<report@bugs.python.org> wrote:
> 3rd party extensions.    What is the use case for "python >&-"?    Is
> it important enough to justify the risk of accidental data loss?

I don't think closing stderr via the command line is an important use
case, but pythonw.exe and Unix daemon processes are important use
cases.

> I don't think so. One more important use case is when running a Unix
> daemon, which has (AFAIK) to close all std handles.

I just took a look at http://pypi.python.org/pypi/python-daemon/, and it
uses dup2() to redirect standard streams, which is far nicer.

On Thu, Feb 3, 2011 at 12:18 PM, Antoine Pitrou <report@bugs.python.org> wrote:
> I just took a look at http://pypi.python.org/pypi/python-daemon/, and it
> uses dup2() to redirect standard streams, which is far nicer.

I'm more worried about the case where a daemon launches python.

At startup, could we check that 2 and 3 are valid file descriptors,
and, if not, open /dev/null?  That way, they cannot later be
inadvertently assigned to some other file?

Attached patch allows Python to run even if no standard stream is available. I use dup() to detect whether a fd is valid.

(No Rietveld link):

+is_valid_fd(int fd)
[...]
+    dummy_fd = dup(fd);
+    if (dummy_fd < 0)
+        return 0;
+    close(dummy_fd);

Why not use fstat() instead (does Windows have fstat()? And dup()?).

+    @unittest.skipIf(os.name == 'nt', "test needs POSIX semantics")
+    def test_no_stdin(self):

It would maybe be more direct with skipUnless(os.name == 'posix').

Finally, it's not that important, but it could maybe be possible to factorize the code, i.e. make a helper function that takes a list of streams and defines the preexec() function and code to test those streams, and then just call:

def test_no_stdin(self):
    out, err = self._test_with_closed_streams(['stdin'])
    [...]

def test_no_streams(self):
    out, err = self._test_with_closed_streams(['stdin', 'stdout', 'stderr'])
    [...]

> +is_valid_fd(int fd)
> [...]
> +    dummy_fd = dup(fd);
> +    if (dummy_fd < 0)
> +        return 0;
> +    close(dummy_fd);
> 
> Why not use fstat() instead (does Windows have fstat()? And dup()?).

Windows has dup(), but no fstat().

> +    @unittest.skipIf(os.name == 'nt', "test needs POSIX semantics")
> +    def test_no_stdin(self):
> 
> It would maybe be more direct with skipUnless(os.name == 'posix').

Hmm, indeed.

> Finally, it's not that important, but it could maybe be possible to
> factorize the code, i.e. make a helper function that takes a list of
> streams and defines the preexec() function and code to test those
> streams, and then just call:

Ah, indeed perhaps.

Updated patch.

> Updated patch.
>

LGTM.

New changeset f15943505db0 by Antoine Pitrou in branch '3.2':
Issue #7111: Python can now be run without a stdin, stdout or stderr stream.
http://hg.python.org/cpython/rev/f15943505db0

New changeset c86fb10eaf68 by Antoine Pitrou in branch 'default':
Issue #7111: Python can now be run without a stdin, stdout or stderr stream.
http://hg.python.org/cpython/rev/c86fb10eaf68

Thanks, committed.