The following script fails:

from time import sleep

try:
    while True:
        sleep(0.1)
        print "blah"
except KeyboardInterrupt:
    pass


when being launched with redirections like this:
$ python test.py | tee /dev/null

and the tee process gots killed *before* the python process (by e.g. hitting ^C in most shells). The output is:

close failed in file object destructor:
sys.excepthook is missing
lost sys.stderr

When the python process is SIGINTed prior to the tee process, everything goes as expected: buffered output to stdout gots flushed and seen on the console, the script finishes successfully, with zero exit status, tee finishes too. When the tee receives SIGINT first, and then python, script loses the output and prints the mentioned message. strace shows that python is receiving (but probably not handling properly) "Broken pipe" error in response to write(2):
[pid 17874] write(1, "blah\nblah\nblah\nblah\nblah\nblah\nbl"..., 680) = -1 EPIPE (Broken pipe)

Have you tried this with 3.2 (or 3.1)?
What behavior do you expect when pipe goes first?

The behavior under question here is how the interpreter handles a broken pipe during shutdown.  The following code behaves as expected when either (a) the process receives a SIGINT or (b) the output pipe is closed

import sys
try:
    while True:
        print 'y'
except KeyboardInterrupt:
    sys.exit(130)
except (IOError, OSError):
    sys.exit(141)

However, in the case of a SIGINT sent to the entire process group e.g. from a shell prompt, the interpreter may receive the SIGINT before detecting the broken pipe.  In this case, the interpreter prints an error somewhere during the SIGINT handler or subsequent shutdown.

In Python 2.4.6 and 2.5.2 the error is
    close failed: [Errno 32] Broken pipe

In Python 2.7.2 the error is
    close failed in file object destructor:
    sys.excepthook is missing
    lost sys.stderr

In Python 3.2 the error is
    Exception IOError: IOError(32, 'Broken pipe') in <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'> ignored

I have not succeeded in triggering the error in Python 3.1.2, despite ensuring the signal arrived before the next write(2) call after the output pipe was closed.  I have attached some straces, but this difference is probably spurious unless reproduced with one of the later bugfix 3.1 releases.

In all versions, the program properly terminates with code 130, or 141 if the write(2) error occurs first.

I would say the bug here if any is inconsistent behavior, though that isn't surprising between major versions.  The 2.7 error message is the only misleading one - stderr was never "lost", and we don't really care about sys.excepthook here.

It seems this can be triggered easily with echo, since that appears to reliably close stdin on startup (Discovered via http://stackoverflow.com/questions/16314321).

Compare (using a recent trunk build, although I see the same behaviour with the system 3.3 installation):

$ ../py3k/python -c 'print("Hello world!")' | echo

Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
BrokenPipeError: [Errno 32] Broken pipe

$ ../py3k/python -c 'print("Hello world!")' | xargs echo
Hello world!

$ ../py3k/python -c 'print("Hello world!")' | cat
Hello world!

Given that flush/close related exceptions are automatically ignored in destructors, something else must be going on here.

It turns out that the *explicit* flush calls during interpreter finalization use PyErr_WriteUnraisable on failure to flush stdout instead of silently ignoring the exceptions (http://hg.python.org/cpython/file/ed011b0d7daf/Python/pythonrun.c#l457).

This is reasonable, since a broken pipeline *is* an error that the user may need to do something about (e.g. as with echo, they next pipeline stage may not being invoked correctly, thus breaking the pipe)

However, in the specific case of a broken pipe, I believe it is desirable for us to write out a cleaner message than the generic PyErr_WriteUnraisable output.

I'm not sure I understand Nick's suggestion. As far as I can tell, the issue is to detect that write() (really fwrite() in 2.x) failed and output the error.

Ah, actually, the real issue is that sys.stderr is gone by the time we try to print the exception (which explains the "lost sys.stderr" message).

Here is a patch, lacking tests.
How important it is to fix this in 2.7 I'm not sure. People are certainly used to the quirk now, and it's generally harmless.

My specific suggestion was aimed at 3.4 - I think reporting the failure to
flush stdout on stderr is the right thing to do (since data may have been
lost and people can suppress it by closing stdout early, including in an
atexit handler), but I also think it's worth improving the error message in
the specific case of a broken pipe error.

However, I also agree the error output is more wrong in 2.7, so improving
that would be good, too.

> My specific suggestion was aimed at 3.4 - I think reporting the failure to
> flush stdout on stderr is the right thing to do (since data may have been
> lost and people can suppress it by closing stdout early, including in an
> atexit handler), but I also think it's worth improving the error message in
> the specific case of a broken pipe error.

How would you improve it?

Suppress the traceback, avoid printing the exception repr and instead
display something more like "Broken pipe: receiving end of stdout closed
prior to interpreter shutdown"

my users are reporting one of these issues, and while I can easily catch IOError on Python 2, I can't catch anything on Python 3, and pointing SIGPIPE to SIG_DFL isn't an option because this is for Alembic migrations and the command needs to complete its work.

What is/will be the recommended solution for people who want their programs to finish totally but not complain when their pipe is cut off in Python 3?

See also issue24864 which is not *quite* a dupe, I also found that it exits 0, unreasonably so.

The reporting thing is interesting, but the main thing I care about is that we can catch it and do something reasonable with it... and that if not caught it signals an error :)

@zzeeek

For Python 3 pipeline tools you need something like this:

try:
   <all your stuff>
finally:
   try:
       sys.stdout.flush()
   finally:
       try:
           sys.stdout.close()
       finally:  
           try:
               sys.stderr.flush()
           finally:
               sys.stderr.close()

This will:
 - capture any exceptions *you've* raised as the context for the errors raised in this handler
 - expose any exceptions generated during this thing itself
 - prevent the interpreter dying during shutdown in flush_std_files by closing the files (you can't easily wipe out the pending writes that have failed)

You could alternatively reopen stdout and stderr on the same fds, giving you new buffers with no pending writes. Anyhow - the key thing is:

- ensure that at shutdown, there are no pending writes for the interpreter lifecycle to flush, and you won't hit this bug.

Oh, just saw your comment Martin; yes, this does look like a dupe.

Bah, wrong issue. Sorry.

Issue 33550 was opened about Mike’s case of ignoring broken pipe conditions.

BTW a side effect of closing sys.stderr is that error messages reported by interpreter shutdown will be missing (even if there was no broken pipe). For example, exception messages reported by sys.excepthook.