classification
Title: abort when stderr is closed
Type: crash Stage: committed/rejected
Components: Interpreter Core Versions: Python 3.3, Python 3.2
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, amaury.forgeotdarc, belopolsky, benjamin.peterson, doko, exarkun, loewis, naufraghi, neologix, petere, pitrou, python-dev, stutzbach
Priority: normal Keywords: patch

Created on 2009-10-12 08:23 by petere, last changed 2011-11-28 18:22 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
nostdio.patch pitrou, 2011-11-26 22:25
nostdio2.patch pitrou, 2011-11-27 22:25
Messages (26)
msg93891 - (view) Author: Peter Eisentraut (petere) * Date: 2009-10-12 08:23
bash$ python3.1 -c 'pass' 2>&-
Aborted (core dumped)

(I verified, the core dump belongs to python.)

If you remove the redirection thingy at the end, it works.

Not sure why I ever wrote that code, but it has been working since
forever up to python3.0.
msg93917 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2009-10-13 03:27
The problem is the check_fd in _fileio.c checks fstat of 2, which
returns EBADFD. I'm not sure what about this redirection makes it a bad
file descriptor, though..
msg93919 - (view) Author: Daniel Stutzbach (stutzbach) (Python committer) Date: 2009-10-13 12:31
After some searching around with Google, "2>&-" means "close file
descriptor 2" (i.e., standard error).
msg93946 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-10-13 23:13
Please note that normally an error message is output, but of course it
doesn't display since stderr is invalid :-)

It's clearer if you close stdout instead:

$ ./python -c 'pass' >&-
Fatal Python error: Py_Initialize: can't initialize sys standard streams
OSError: [Errno 9] Bad file descriptor
Abandon

If we want to allow for closed {stdin, stdout, stderr}, I'm not sure
what the semantics should be. Should sys.std{in, out, err} be None? Or a
file object which always throws an error?

Under Python 2.x, you don't get a crash but the behaviour is quite
unhelpful anyway:

$ python -c 'print 1' >&-
close failed in file object destructor:
Error in sys.excepthook:

Original exception was:
msg93950 - (view) Author: Daniel Stutzbach (stutzbach) (Python committer) Date: 2009-10-14 00:11
Is it even possible to portably test the validity of a file descriptor
without trying to write/read it?

When I first saw this bug, my gut feeling was "well, don't do that
then!"  However, I then recalled that Windows GUI applications have no
stdin, stdout, or stderr.  

Python 2 will raise IOError: Bad File Descriptor when the user tries to
write to stdout or stderr (more accurately, it raises the exception when
trying to flush data to the file descriptor).  

I just tested pythonw.exe.  If I set sys.stderr by hand to a file, then
write to sys.stdout, 2.6 will correctly write the exception to the file.
 3.1 exits silently.
msg93969 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2009-10-14 08:56
> 3.1 exits silently.
Did you use "print"? pythonw.exe 3.1 sets sys.stdout to None.
if you use sys.stdout.write, you get an exception. But print() silently
does nothing if the file is None.
msg93971 - (view) Author: Peter Eisentraut (petere) * Date: 2009-10-14 09:55
For what it's worth, the code in question is used here (using "import
distutils" instead of "pass"):

http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/config/python.m4?rev=1.15;content-type=text%2Fx-cvsweb-markup

This is obviously a completely gratuitous variant on 2>/dev/null, but it
has apparently been working since forever.  I'll probably go make the
change anyway.

Nevertheless, I think Python shouldn't core dump.  It may choose to exit
doing nothing (useful) if it doesn't want to deal with this case.

Check this for possible alternative behaviors:

$ ls 1>&-
ls: write error: Bad file descriptor
($? = 2)
$ ls 1>&- 2>&-
($? = 2, no output)
msg94425 - (view) Author: Matteo Bertini (naufraghi) Date: 2009-10-24 15:30
sorry, title restored!
msg127773 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2011-02-02 22:25
> If we want to allow for closed {stdin, stdout, stderr}, I'm not sure
> what the semantics should be. Should sys.std{in, out, err} be None? Or a
> file object which always throws an error?

I would say it should be a *pseudo*-file object which always throws a *descriptive* error.  Note that setting sys.stdout to None makes print() do nothing rather than report an error:

>>> sys.stdout = None
>>> print('abc')

See also issue6501.
msg127775 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2011-02-02 22:36
On the second thought, as long as python used fd 2 as the "message stream of last resort", we should probably not allow it to run with fd 2 closed. The problem is that in this case fd 2 may become associated with a very important file contents of which you don't want to see replaced with a python error message.
msg127803 - (view) Author: Daniel Stutzbach (stutzbach) (Python committer) Date: 2011-02-03 18:33
That's an interesting point.

Do you know of places where we use fd 2 instead of sys.stderr?
msg127809 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-02-03 19:44
> That's an interesting point.
> 
> Do you know of places where we use fd 2 instead of sys.stderr?

We normally don't. One reason is that buffering inside sys.stderr can
make ordering of output incorrect. There are some places in C code where
we do "fprintf(stderr, ...)" but that's for specialized debugging
(disabled in normal builds) or fatal error messages.
msg127811 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2011-02-03 19:56
On Thu, Feb 3, 2011 at 2:44 PM, Antoine Pitrou <report@bugs.python.org> wrote:
..
>> Do you know of places where we use fd 2 instead of sys.stderr?
>
> We normally don't. One reason is that buffering inside sys.stderr can
> make ordering of output incorrect. There are some places in C code where
> we do "fprintf(stderr, ...)" but that's for specialized debugging
> (disabled in normal builds) or fatal error messages.

This is the case that I had in mind.  What does non-debug build do on
a fatal error?  Also, can we be sure that Python does not call C
library functions that write to stderr behind the scenes?  If vanilla
Python is safe to run with closed fd 2, that may not be the case for
3rd party extensions.    What is the use case for "python >&-"?    Is
it important enough to justify the risk of accidental data loss?
msg127812 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2011-02-03 19:59
> On Thu, Feb 3, 2011 at 2:44 PM, Antoine Pitrou <report@bugs.python.org> wrote:
> ..
>> Do you know of places where we use fd 2 instead of sys.stderr?
>
> We normally don't.

Hmm, grep "fprintf(stderr," returned 122 hits in the py3k branch.
Are you sure these are all debug-build only?
msg127813 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-02-03 20:06
> > We normally don't. One reason is that buffering inside sys.stderr can
> > make ordering of output incorrect. There are some places in C code where
> > we do "fprintf(stderr, ...)" but that's for specialized debugging
> > (disabled in normal builds) or fatal error messages.
> 
> This is the case that I had in mind.  What does non-debug build do on
> a fatal error?

It uses fprintf(stderr, ...). That's the only thing it can do (there's
no way sys.stderr is guaranteed to be usable at that point). If C stderr
is invalid, then too bad.

> Also, can we be sure that Python does not call C
> library functions that write to stderr behind the scenes?

I think you can guess the answer :)

> What is the use case for "python >&-"?    Is
> it important enough to justify the risk of accidental data loss?

I don't think so. One more important use case is when running a Unix
daemon, which has (AFAIK) to close all std handles. I don't know how
that interacts with using C stderr, especially if the handle closing is
done in Python (and therefore only calls C close() and not fclose()!).

Perhaps we should provide a sys function to fclose() C std{in,out,err}.
msg127814 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-02-03 20:07
Le jeudi 03 février 2011 à 19:59 +0000, Alexander Belopolsky a écrit :
> Alexander Belopolsky <belopolsky@users.sourceforge.net> added the comment:
> 
> > On Thu, Feb 3, 2011 at 2:44 PM, Antoine Pitrou <report@bugs.python.org> wrote:
> > ..
> >> Do you know of places where we use fd 2 instead of sys.stderr?
> >
> > We normally don't.
> 
> Hmm, grep "fprintf(stderr," returned 122 hits in the py3k branch.
> Are you sure these are all debug-build only?

"grep -C2" seems to say most of them are. I haven't done a survey.
msg127815 - (view) Author: Daniel Stutzbach (stutzbach) (Python committer) Date: 2011-02-03 20:12
On Thu, Feb 3, 2011 at 11:56 AM, Alexander Belopolsky
<report@bugs.python.org> wrote:
> 3rd party extensions.    What is the use case for "python >&-"?    Is
> it important enough to justify the risk of accidental data loss?

I don't think closing stderr via the command line is an important use
case, but pythonw.exe and Unix daemon processes are important use
cases.
msg127817 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-02-03 20:18
> I don't think so. One more important use case is when running a Unix
> daemon, which has (AFAIK) to close all std handles.

I just took a look at http://pypi.python.org/pypi/python-daemon/, and it
uses dup2() to redirect standard streams, which is far nicer.
msg127821 - (view) Author: Daniel Stutzbach (stutzbach) (Python committer) Date: 2011-02-03 20:49
On Thu, Feb 3, 2011 at 12:18 PM, Antoine Pitrou <report@bugs.python.org> wrote:
> I just took a look at http://pypi.python.org/pypi/python-daemon/, and it
> uses dup2() to redirect standard streams, which is far nicer.

I'm more worried about the case where a daemon launches python.

At startup, could we check that 2 and 3 are valid file descriptors,
and, if not, open /dev/null?  That way, they cannot later be
inadvertently assigned to some other file?
msg148425 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-11-26 22:25
Attached patch allows Python to run even if no standard stream is available. I use dup() to detect whether a fd is valid.
msg148449 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2011-11-27 10:57
(No Rietveld link):

+is_valid_fd(int fd)
[...]
+    dummy_fd = dup(fd);
+    if (dummy_fd < 0)
+        return 0;
+    close(dummy_fd);

Why not use fstat() instead (does Windows have fstat()? And dup()?).

+    @unittest.skipIf(os.name == 'nt', "test needs POSIX semantics")
+    def test_no_stdin(self):

It would maybe be more direct with skipUnless(os.name == 'posix').

Finally, it's not that important, but it could maybe be possible to factorize the code, i.e. make a helper function that takes a list of streams and defines the preexec() function and code to test those streams, and then just call:

def test_no_stdin(self):
    out, err = self._test_with_closed_streams(['stdin'])
    [...]

def test_no_streams(self):
    out, err = self._test_with_closed_streams(['stdin', 'stdout', 'stderr'])
    [...]
msg148460 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-11-27 22:14
> +is_valid_fd(int fd)
> [...]
> +    dummy_fd = dup(fd);
> +    if (dummy_fd < 0)
> +        return 0;
> +    close(dummy_fd);
> 
> Why not use fstat() instead (does Windows have fstat()? And dup()?).

Windows has dup(), but no fstat().

> +    @unittest.skipIf(os.name == 'nt', "test needs POSIX semantics")
> +    def test_no_stdin(self):
> 
> It would maybe be more direct with skipUnless(os.name == 'posix').

Hmm, indeed.

> Finally, it's not that important, but it could maybe be possible to
> factorize the code, i.e. make a helper function that takes a list of
> streams and defines the preexec() function and code to test those
> streams, and then just call:

Ah, indeed perhaps.
msg148461 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-11-27 22:25
Updated patch.
msg148505 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2011-11-28 18:03
> Updated patch.
>

LGTM.
msg148507 - (view) Author: Roundup Robot (python-dev) Date: 2011-11-28 18:16
New changeset f15943505db0 by Antoine Pitrou in branch '3.2':
Issue #7111: Python can now be run without a stdin, stdout or stderr stream.
http://hg.python.org/cpython/rev/f15943505db0

New changeset c86fb10eaf68 by Antoine Pitrou in branch 'default':
Issue #7111: Python can now be run without a stdin, stdout or stderr stream.
http://hg.python.org/cpython/rev/c86fb10eaf68
msg148508 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-11-28 18:22
Thanks, committed.
History
Date User Action Args
2011-11-28 18:22:26pitrousetstatus: open -> closed
resolution: fixed
messages: + msg148508

stage: patch review -> committed/rejected
2011-11-28 18:16:23python-devsetnosy: + python-dev
messages: + msg148507
2011-11-28 18:03:07neologixsetmessages: + msg148505
2011-11-27 22:25:40pitrousetfiles: + nostdio2.patch

messages: + msg148461
2011-11-27 22:14:17pitrousetmessages: + msg148460
2011-11-27 10:57:33neologixsetmessages: + msg148449
2011-11-27 00:39:38Arfreversetnosy: + Arfrever
2011-11-26 22:25:53pitrousetfiles: + nostdio.patch

versions: + Python 3.3, - Python 3.1
keywords: + patch
nosy: + neologix

messages: + msg148425
stage: patch review
2011-02-03 20:49:11stutzbachsetnosy: loewis, doko, exarkun, amaury.forgeotdarc, belopolsky, pitrou, benjamin.peterson, stutzbach, naufraghi, petere
messages: + msg127821
2011-02-03 20:18:48pitrousetnosy: loewis, doko, exarkun, amaury.forgeotdarc, belopolsky, pitrou, benjamin.peterson, stutzbach, naufraghi, petere
messages: + msg127817
2011-02-03 20:12:57stutzbachsetnosy: loewis, doko, exarkun, amaury.forgeotdarc, belopolsky, pitrou, benjamin.peterson, stutzbach, naufraghi, petere
messages: + msg127815
2011-02-03 20:07:11pitrousetnosy: loewis, doko, exarkun, amaury.forgeotdarc, belopolsky, pitrou, benjamin.peterson, stutzbach, naufraghi, petere
messages: + msg127814
2011-02-03 20:06:46pitrousetnosy: loewis, doko, exarkun, amaury.forgeotdarc, belopolsky, pitrou, benjamin.peterson, stutzbach, naufraghi, petere
messages: + msg127813
2011-02-03 20:05:09pitrousetnosy: + exarkun, loewis
2011-02-03 19:59:24belopolskysetnosy: doko, amaury.forgeotdarc, belopolsky, pitrou, benjamin.peterson, stutzbach, naufraghi, petere
messages: + msg127812
2011-02-03 19:56:37belopolskysetnosy: doko, amaury.forgeotdarc, belopolsky, pitrou, benjamin.peterson, stutzbach, naufraghi, petere
messages: + msg127811
2011-02-03 19:44:47pitrousetnosy: doko, amaury.forgeotdarc, belopolsky, pitrou, benjamin.peterson, stutzbach, naufraghi, petere
messages: + msg127809
2011-02-03 18:33:02stutzbachsetnosy: doko, amaury.forgeotdarc, belopolsky, pitrou, benjamin.peterson, stutzbach, naufraghi, petere
messages: + msg127803
2011-02-02 22:36:38belopolskysetnosy: doko, amaury.forgeotdarc, belopolsky, pitrou, benjamin.peterson, stutzbach, naufraghi, petere
messages: + msg127775
title: abort when stderr is moved -> abort when stderr is closed
2011-02-02 22:25:30belopolskysetnosy: + belopolsky
messages: + msg127773
2009-10-24 20:53:58benjamin.petersonsettitle: core dump when stderr is moved -> abort when stderr is moved
2009-10-24 15:30:13naufraghisetmessages: + msg94425
title: stdout closed -> core dump when stderr is moved
2009-10-24 15:28:47naufraghisettitle: core dump when stderr is moved -> stdout closed
2009-10-24 15:28:18naufraghisetnosy: + naufraghi
2009-10-14 09:55:12peteresetmessages: + msg93971
2009-10-14 08:56:37amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg93969
2009-10-14 00:11:12stutzbachsetmessages: + msg93950
2009-10-13 23:13:20pitrousetpriority: normal
versions: + Python 3.2
nosy: + pitrou

messages: + msg93946
2009-10-13 12:31:54stutzbachsetnosy: + stutzbach
messages: + msg93919
2009-10-13 03:27:49benjamin.petersonsetnosy: + benjamin.peterson
messages: + msg93917
2009-10-12 09:56:37dokosetnosy: + doko
2009-10-12 08:23:19peterecreate