classification
Title: Document that print may need explicit flushing
Type: enhancement Stage: resolved
Components: Documentation Versions: Python 3.3, Python 3.2
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: docs@python Nosy List: amaury.forgeotdarc, cameron, docs@python, eric.araujo, georg.brandl, haypo, python-dev, techtonik, terry.reedy
Priority: normal Keywords:

Created on 2011-03-22 10:37 by techtonik, last changed 2012-01-14 05:07 by python-dev. This issue is now closed.

Files
File name Uploaded Description Edit
printtest3.py techtonik, 2011-03-22 10:37
printtest2.py techtonik, 2011-03-22 10:37
print-flush.patch amaury.forgeotdarc, 2011-03-22 23:27 review
Messages (27)
msg131736 - (view) Author: anatoly techtonik (techtonik) Date: 2011-03-22 10:37
print() function for some reason buffers output on Windows if end!='\n'. 

In attached examples "Processing.. " string is shown in Python 3.2 only after the actual processing is finished, while in Python 2.6 it is shown before.
msg131737 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-03-22 11:16
printtest2.py displays directly "Processing.. " on Windows, but not on Linux. It looks like stdout is not buffered on Windows, which looks like a bug to bug :-) I think that it is safer to always call sys.stdout.flush() to ensure that your message is directly displayed. With Python 2, you can use -u flag (unbuffered output) to avoid the explicit flush, but this is very inefficient (slow).

Python 3 uses line buffers, even with python3 -u, for better performances. If you want to see directly "Processing.. ", as Python 2, call sys.stdout.flush().

It is not a regression, it is a choice to be efficient.
msg131738 - (view) Author: anatoly techtonik (techtonik) Date: 2011-03-22 11:33
From my perspective it is a regression on Windows and a bug in Linux version of Python 2.x, which unfortunately can not be fixed, because of 2.x release process.

If the fact that print statement doesn't output anything when called is not a bug - then should be documented long ago.
msg131739 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-03-22 11:47
> From my perspective it is a regression on Windows and a bug in Linux
> version of Python 2.x, which unfortunately can not be fixed,
> because of 2.x release process.

Line buffering is used by default on most operating systems (ok, maybe not Windows, which looks strange to me) and is not specific to Python.

Yes, Python has subtle differences on different operating systems, but at least it looks like Python 3 has the same behaviour on Linux and Windows ;-)

> If the fact that print statement doesn't output anything when called
> is not a bug - then should be documented long ago.

Can you please write a patch for the doc? Reopen the issue if you have a patch.
msg131741 - (view) Author: anatoly techtonik (techtonik) Date: 2011-03-22 11:49
How about making print() user-friendly with flushing after every call, and if you want explicitly want speed - use buffered sys.stdout.write/flush()?
msg131742 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-03-22 11:57
> How about making print() user-friendly with flushing after every call,
> and if you want explicitly want speed - use buffered
> sys.stdout.write/flush()?

This is exactly the -u option of Python 2: use it if you would like a completly unbuffered sys.stdout in a portable way.

In Python 3, it is only useful to flush at each line, even if the output is not a console, but a pipe (e.g. output redirected to a file). But nobody asked yet to have a fully unbuffered sys.stdout in Python 3.
msg131743 - (view) Author: anatoly techtonik (techtonik) Date: 2011-03-22 12:07
You must realize that the most common use case for print(..., end!='\n') is when you want to notify user about intermediate progress of a very long operation.

Making documentation for simple print() statement overloaded with low level buffering details makes language seem overly complicated for new users.
msg131755 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2011-03-22 17:28
We are talking about different things here:

- When python is run from a console, sys.stdout is line buffered. sys.stdout.write() flushes if there is a carriage return. No need to change anything here.

- print() could call file.flush() if file.isatty(), *after* the multiple calls to file.write().
msg131800 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-03-22 22:44
Python 3.2, WinXP, IDLE edit window, F5 Run:
'Processing ...' appears immediately, 'Done' 3 sec later.
The only difference between printtest2/3 is where 'Done' appears.

Behavior is same when pasting into interactive interpreter -- has to be since the first two prints are executed before the sleep. I presume interpreter flushes before or after printing next prompt. IDLE must do same even when running from editor even when not printing prompts.

Anatoly, I gather you ran files by some other method.

I disagree with closing this yet. Print was designed to be a simplified wrapper around sys.stdout.write (and now, any file.write). Requiring that one import sys to use print goes against that.

I have always experienced and expected Python's print to screen to be immediately visible. I thought that was pretty standard in other languages with a print-to-screen separate from general file-write. When printing to screen, efficiency is, I believe, a non-issue. Writing to files or the net is a whole different ballgame.

Amaury's idea of checking isatty seems good. Otherwise, print should gain an optional, no-default flush=True/False parameter (no default because behavior currently varies.) Until then, non-flushing *should* be documented. The current doc for print does not seem to address the issue either way.

Amaury, I though 'run from console' more or less meant 'isatty', so I am not sure I understand your comment.
msg131808 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2011-03-22 23:27
The attached patch calls "if file.isatty(): file.flush()" at the end of the print function:
- only when an "end" argument was specified
- errors in file.isatty() are ignored (and then no flush occurs)
msg131811 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2011-03-22 23:29
> Python 3.2, WinXP, IDLE edit window, F5 Run:
> 'Processing ...' appears immediately, 'Done' 3 sec later.

Terry, IDLE is completely different, its sys.stdout completely bypasses the new io stack, and there is no buffering...
msg131859 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-03-23 09:25
amaury> When python is run from a console, sys.stdout is line buffered.
amaury> sys.stdout.write() flushes if there is a carriage return.
amaury> No need to change anything here.

Anatoly would like a flush after all calls to print().

> print() could call file.flush() if file.isatty(), *after* the multiple
> calls to file.write().

I vote +0 to change print(), call sys.stdout.flush(), if:

 - file option is not used (and so, sys.stdout is used)
 - sys.stdout is a TTY
 - end option is used (fast heuristic to check if print will write a newline or not, a better one whould be to check if end contains a newline character or not, but we had to check for \n and/or \r, for a little gain)

But I don't want to change print() for print(text, file=file), because it would make Python slower and print(... file=file) is not used to an interactive prompt or to display informations to the user.

> Behavior is same when pasting into interactive interpreter ...
> I presume interpreter flushes before or after printing next prompt.

Did you wrote all commands on the same line? Python does change stdout buffer in interactive mode:
------------
    if (Py_UnbufferedStdioFlag) {
#ifdef HAVE_SETVBUF
        setvbuf(stdin,  (char *)NULL, _IONBF, BUFSIZ);
        setvbuf(stdout, (char *)NULL, _IONBF, BUFSIZ);
        setvbuf(stderr, (char *)NULL, _IONBF, BUFSIZ);
#else /* !HAVE_SETVBUF */
        setbuf(stdin,  (char *)NULL);
        setbuf(stdout, (char *)NULL);
        setbuf(stderr, (char *)NULL);
#endif /* !HAVE_SETVBUF */
    }
    else if (Py_InteractiveFlag) {
#ifdef MS_WINDOWS
        /* Doesn't have to have line-buffered -- use unbuffered */
        /* Any set[v]buf(stdin, ...) screws up Tkinter :-( */
        setvbuf(stdout, (char *)NULL, _IONBF, BUFSIZ);
#else /* !MS_WINDOWS */
#ifdef HAVE_SETVBUF
        setvbuf(stdin,  (char *)NULL, _IOLBF, BUFSIZ);
        setvbuf(stdout, (char *)NULL, _IOLBF, BUFSIZ);
#endif /* HAVE_SETVBUF */
#endif /* !MS_WINDOWS */
        /* Leave stderr alone - it should be unbuffered anyway. */
    }
#ifdef __VMS
    else {
        setvbuf (stdout, (char *)NULL, _IOLBF, BUFSIZ);
    }
#endif /* __VMS */
------------
(it doesn't check if stdout is a TTY or not, but I don't think that it is very useful to use the interactive mode outside a TTY)

> I have always experienced and expected Python's print to screen
> to be immediately visible. I thought that was pretty standard
> in other languages with a print-to-screen separate from
> general file-write.

Did you try Perl, Ruby, bash and other languages? I know that at least the C language requires an explicit call to fflush(stdout). I always used that.

> Terry, IDLE is completely different, its sys.stdout completely
> bypasses the new io stack, and there is no buffering...

As I wrote: "unbuffered mode" is not implemented for TextIOWrapper. So even with python3 -u, sys.stdout.write("abc") doesn't flush immediatly into the underlying FileIO.
msg131909 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-03-23 19:25
I completely agree that file/socket output should be left alone. Flushing char by char to either is a bit insane. The two interactive to screen use cases I can think of are text progress meters, mentioned by Anatoly, such as :
  Working .... (1 dot printed at intervals)
and timed text like

import time
for c in 'Similated 10 cps teletype output':
    print(c,end='')
    time.sleep(.1)
print()

which works fine from IDLE and whose non-functioning when started otherwise would puzzle any beginner and many beyond.
msg150924 - (view) Author: anatoly techtonik (techtonik) Date: 2012-01-09 08:02
I've tried to switch to Python 3 once more and stumbled upon this problem once more.

Seems like this regression got stale. Last Victor's proposal seems reasonable for me. Should we open a new, more clear bug report and close this one?
msg150930 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2012-01-09 11:03
No, I don't think so. Another issue will not magically create more time for anyone.
msg150932 - (view) Author: anatoly techtonik (techtonik) Date: 2012-01-09 11:35
On Mon, Jan 9, 2012 at 2:03 PM, Terry J. Reedy <report@bugs.python.org>wrote:

>
> Terry J. Reedy <tjreedy@udel.edu> added the comment:
>
> No, I don't think so. Another issue will not magically create more time
> for anyone.
>

But anyone will waste less time to get to the outcome of discussion.
-- 
anatoly t.
msg150955 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-01-09 16:59
> You must realize that the most common use case for print(..., end!='\n') is when you want
> to notify user about intermediate progress of a very long operation.

References needed.
msg150961 - (view) Author: anatoly techtonik (techtonik) Date: 2012-01-09 18:41
> Making documentation for simple print() statement overloaded with low level buffering details makes language seem overly complicated for new users.

Why don't anybody require references for that?
msg150971 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2012-01-09 19:59
The current doc says

"print([object, ...], *, sep=' ', end='\n', file=sys.stdout) 
Print object(s) to the stream file, separated by sep and followed by end. sep, end and file, if present, must be given as keyword arguments.

All non-keyword arguments are converted to strings like str() does and written to the stream, separated by sep and followed by end. Both sep and end must be strings; they can also be None, which means to use the default values. If no object is given, print() will just write end.

The file argument must be an object with a write(string) method; if it is not present or None, sys.stdout will be used."

(The bit about None, said twice, could be factored out and said once after the second sentence.)

This is exactly what print does and Guido today (Python ideas) said that is what it should do and that "Apps that need flushing should call flush()." So a code change is rejected.

The issue title was incorrect. The print function does not do any buffering. The file object it writes to may. Even sys.stdout may or may not. 

We could add at the end a sentence or two something like

"Output buffering is determined by *file*. Call file.flush() to ensure, for instance, immediate appearance on a screen."
msg151080 - (view) Author: Roundup Robot (python-dev) Date: 2012-01-11 19:15
New changeset bc043cef94f2 by Terry Jan Reedy in branch '3.2':
Closes #11633 Clarify print buffering.
http://hg.python.org/cpython/rev/bc043cef94f2

New changeset fb0d61fd1753 by Terry Jan Reedy in branch 'default':
Merge with 3.2
http://hg.python.org/cpython/rev/fb0d61fd1753
msg151081 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2012-01-11 19:16
#13761 proposes to add flush=False param with option for True.
msg151140 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-01-12 17:10
I agree with the python-ideas message that ``sys.stdout.flush()`` is surprising / possibly misleading and should be ``file.flush()``.  If the other bug report about adding a flush argument is rejected, please consider this.  Thanks :)
msg151149 - (view) Author: Roundup Robot (python-dev) Date: 2012-01-12 19:52
New changeset 4a767054551b by Terry Jan Reedy in branch '3.2':
#11633 At least 2 people prefer earlier revision.
http://hg.python.org/cpython/rev/4a767054551b

New changeset 22688f5f9d0f by Terry Jan Reedy in branch 'default':
Merge #11633 At least 2 people prefer earlier revision.
http://hg.python.org/cpython/rev/22688f5f9d0f
msg151212 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-01-14 03:18
Thank you sir.  Should the doc edit be backported to the 2.7 docs, with a mention that it’s only on unix?
msg151213 - (view) Author: Cameron Simpson (cameron) Date: 2012-01-14 03:34
Putting the wording into 2.7 might be nice, but I thought it was in bugfix only mode.

Regarding UNIX only, I'd avoid it; any file may be buffered in almost any way on any platform. Saying an explicit flush call may be necessary for immediate output is _not_ UNIX only and would be very misleading. Remembering that ~UNIX != Windows.

Telling users to explicitly call flush to ensure immediate output where that is necessary ensures portable coding (or ought to, user pigheadedness discounted:-)
msg151223 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-01-14 04:36
Bug fixes include doc improvements, so 2.7 is fair game.

Thanks for your suggestion to not mention specific platforms.  Let’s just backport the 3.2 text.
msg151227 - (view) Author: Roundup Robot (python-dev) Date: 2012-01-14 05:07
New changeset 8935a33773b9 by Terry Jan Reedy in branch '2.7':
#11633 about buffering of print
http://hg.python.org/cpython/rev/8935a33773b9
History
Date User Action Args
2012-01-14 05:07:41python-devsetmessages: + msg151227
2012-01-14 04:36:00eric.araujosetmessages: + msg151223
2012-01-14 03:34:59cameronsetmessages: + msg151213
2012-01-14 03:18:41eric.araujosetmessages: + msg151212
2012-01-12 19:52:07python-devsetmessages: + msg151149
2012-01-12 17:10:55eric.araujosetmessages: + msg151140
2012-01-12 01:26:40cameronsetnosy: + cameron
2012-01-11 19:16:36terry.reedysetmessages: + msg151081
2012-01-11 19:15:16python-devsetstatus: open -> closed

nosy: + python-dev
messages: + msg151080

resolution: fixed
stage: needs patch -> resolved
2012-01-09 19:59:39terry.reedysetassignee: docs@python
components: + Documentation, - Interpreter Core
title: Document that print buffers output when end='' -> Document that print may need explicit flushing
nosy: + docs@python

messages: + msg150971
stage: needs patch
2012-01-09 18:41:28techtoniksetmessages: + msg150961
2012-01-09 16:59:09eric.araujosettitle: regression: print buffers output when end='' -> Document that print buffers output when end=''
nosy: + eric.araujo

messages: + msg150955

keywords: - patch, 3.2regression
2012-01-09 11:36:00techtoniksetmessages: + msg150932
2012-01-09 11:03:03terry.reedysetmessages: + msg150930
2012-01-09 08:02:40techtoniksetmessages: + msg150924
2011-03-23 19:25:00terry.reedysetmessages: + msg131909
2011-03-23 09:25:10hayposetnosy: georg.brandl, terry.reedy, amaury.forgeotdarc, haypo, techtonik
messages: + msg131859
2011-03-22 23:41:34eric.araujosetkeywords: + 3.2regression
assignee: docs@python -> (no value)
components: + Interpreter Core, - Documentation, Library (Lib)
nosy: + georg.brandl, - docs@python
2011-03-22 23:29:17amaury.forgeotdarcsetnosy: terry.reedy, amaury.forgeotdarc, haypo, techtonik, docs@python
messages: + msg131811
2011-03-22 23:27:16amaury.forgeotdarcsetfiles: + print-flush.patch

messages: + msg131808
keywords: + patch
nosy: terry.reedy, amaury.forgeotdarc, haypo, techtonik, docs@python
2011-03-22 22:44:12terry.reedysetstatus: closed -> open

type: enhancement
assignee: docs@python
components: + Documentation, Library (Lib)
versions: + Python 3.3
nosy: + docs@python, terry.reedy

messages: + msg131800
resolution: not a bug -> (no value)
2011-03-22 17:28:31amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg131755
2011-03-22 12:07:09techtoniksetmessages: + msg131743
2011-03-22 11:57:45hayposetmessages: + msg131742
2011-03-22 11:49:33techtoniksetmessages: + msg131741
2011-03-22 11:47:04hayposetmessages: + msg131739
2011-03-22 11:33:05techtoniksetmessages: + msg131738
2011-03-22 11:16:25hayposetstatus: open -> closed

nosy: + haypo
messages: + msg131737

resolution: not a bug
2011-03-22 10:37:47techtoniksetfiles: + printtest2.py
2011-03-22 10:37:22techtonikcreate