classification
Title: stdout stderr redirection mess
Type: Stage: resolved
Components: Versions: Python 3.2, Python 3.3, Python 3.4, Python 2.7
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: lukasz.langa, neologix, pitrou, schmir, serhiy.storchaka, techtonik, terry.reedy
Priority: normal Keywords:

Created on 2012-12-15 00:56 by techtonik, last changed 2012-12-30 23:59 by schmir. This issue is now closed.

Files
File name Uploaded Description Edit
test_unicode_fname.py techtonik, 2012-12-15 00:56
Messages (14)
msg177525 - (view) Author: anatoly techtonik (techtonik) Date: 2012-12-15 00:56
Tested on Windows. When Python output is redirected with:

> py test_unicode_fname.py > test.log 2>&1

In Python 2.7 the traceback is at the end of file, in Python 3.3 it is at the beginning. In should be in the middle, and it appears on the screen correctly.

From issue16656.
msg177912 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2012-12-22 00:55
Is that really the simplest example that will exhibit the cited behavior? 

Why do you think this is a Python issue rather than a Windows issue?
What happens, for instance, with a C program than alternately outputs to stdout and stderr?
msg177922 - (view) Author: anatoly techtonik (techtonik) Date: 2012-12-22 04:37
I don't know. I don't write C.
msg177928 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2012-12-22 09:12
It's not an issue at all, it's simply buffering in effect.
Depending on the version, stdout/stderr is either unbuffured or fully buffered, see http://bugs.python.org/issue13597

Simply pass '-u' and streams will be unbuffered, and appear in the order in chich the print() calls are made.
msg177929 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2012-12-22 09:16
It looks as for example stdout is line-buffered when output to terminal and block-buffered when redirected to file or pipe and stderr is line-buffered in any case in Python 3.3 (this a common behavior for all C programs). Or something like.

Python has an option -u to control this.
msg177942 - (view) Author: anatoly techtonik (techtonik) Date: 2012-12-22 13:49
I am not convinced that "won't fix" is the solution. If you're redirecting both output streams to the same destination, you expect that the output will appear in the final file exactly as it appears on the screen.

tag:wart
msg177946 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2012-12-22 17:44
> I am not convinced that "won't fix" is the solution. If you're redirecting both output streams to the same destination, you expect that the output will appear in the final file exactly as it appears on the screen.

There's nothing to fix.
The order in which the output appears in the final file depends on the
buffering.
msg177947 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2012-12-22 17:55
To be more specific, for performance reasons, you definitely don't
want stdout to be line buffered when redirected to a file, but fully
buffered.
It makes sense to have stderr unbuffered or line buffered either way
to make sure that errors get printed - right away.
See http://www.delorie.com/gnu/docs/glibc/libc_226.html for an explanation.
msg177952 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-12-22 20:12
And arguably you shouldn't redirect stdout and stderr to the same place, except for temporary debugging purposes.
msg177954 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2012-12-22 20:37
Anatoly, we are not going to change the default buffering to the most inefficient possible to meet your expectation -- especially when there is a simple option to give you what you want. Unix/C and hence windows has two output streams because people often want the two streams to go to different places. Anyone who knows and uses the magic incantation '2>&1' to mix them together had better know (or learn) what this really means and what the options are. This issue has nothing in particular to do with Python.
msg178005 - (view) Author: anatoly techtonik (techtonik) Date: 2012-12-23 18:49
Can Python detect when output file descriptor for both stderr and stdout streams is the same and use the single buffer itself?
msg178009 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2012-12-23 19:15
No.

Please, don't reopen without a patch. Better yet: move the discussion to python-ideas. Better yet: simply accept that you should either use -u or stop treating stdout/stderr as synchronized.

This issue has been closed twice already by distinct contributors. To keep reopening it is passive–aggressive behaviour, so is the wording of the title you chose for the report.
msg178034 - (view) Author: anatoly techtonik (techtonik) Date: 2012-12-24 07:37
I need to track issues the same way as you. You act as if the issue is resolved while it is (or rather wasn't) until the last comment that Python is unable to detect if stdout and stderr stream share the same output descriptor.

And your separation of the world to "distinct contributors" and everybody else is the source of this conflict - not somebody doing some passive actions.
msg178628 - (view) Author: Ralf Schmitt (schmir) Date: 2012-12-30 23:59
flushing stdout may help before writing to stderr (or at least when writing a traceback). see http://utcc.utoronto.ca/~cks/space/blog/programming/HowToWriteToStderr for a discussion.

I don't think it's unreasonable to implement that in python core.
History
Date User Action Args
2012-12-30 23:59:35schmirsetnosy: + schmir
messages: + msg178628
2012-12-24 07:37:38techtoniksetmessages: + msg178034
2012-12-23 19:15:37lukasz.langasetstatus: open -> closed
nosy: + lukasz.langa
messages: + msg178009

2012-12-23 18:49:15techtoniksetstatus: closed -> open

messages: + msg178005
2012-12-22 20:37:34terry.reedysetmessages: + msg177954
2012-12-22 20:12:05pitrousetstatus: open -> closed
nosy: + pitrou
messages: + msg177952

2012-12-22 17:55:44neologixsetmessages: + msg177947
2012-12-22 17:44:45neologixsetstatus: pending -> open

messages: + msg177946
2012-12-22 13:49:43techtoniksetstatus: closed -> pending

messages: + msg177942
2012-12-22 09:16:07serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg177929
2012-12-22 09:12:29neologixsetstatus: open -> closed

nosy: + neologix
messages: + msg177928

resolution: not a bug
stage: resolved
2012-12-22 04:37:39techtoniksetmessages: + msg177922
2012-12-22 00:55:57terry.reedysetnosy: + terry.reedy

messages: + msg177912
versions: - Python 2.6, Python 3.1
2012-12-15 00:56:53techtonikcreate