classification
Title: An article on Python 3 stdout and stderr output buffering
Type: Stage: resolved
Components: Documentation Versions:
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: docs@python Nosy List: Anatoli Babenia, docs@python, gvanrossum, tedder
Priority: normal Keywords:

Created on 2020-07-31 10:10 by Anatoli Babenia, last changed 2020-08-03 04:49 by Anatoli Babenia. This issue is now closed.

Messages (8)
msg374633 - (view) Author: Anatoli Babenia (Anatoli Babenia) Date: 2020-07-31 10:10
It is hard to find info why Python 3 buffers stdout/stderr. The buffering causes problems when debugging Python apps in Docker and Kubernetes, and it is unclear if it is Python 3 who starts to buffer stdout if no tty is attached, it is Docker, or it is Kubernetes.

The only bit of info that could be searched is the description of -u option https://docs.python.org/3.8/using/cmdline.html?#cmdoption-u which is not linked to any article.

The `-u` description also says.

> Changed in version 3.7: The text layer of the stdout and stderr streams now is unbuffered.

However, I don't understand what is the text layers of stdout. And there is no description of behaviour when the output is not attached, and when the output is redirected.
msg374706 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-08-03 00:53
Maybe you’ll get some explanation or pointers on python-list. Or were you volunteering to write something?
msg374714 - (view) Author: Anatoli Babenia (Anatoli Babenia) Date: 2020-08-03 04:21
Not sure I can volunteer - need to find a sustenance ASAP. But given that nothing moved in this direction over the past 6 months, another week spent not writing this stuff won't change anything.

In any case it would start with the following problem - "Logs don't show up if running in Kubernetes" - https://github.com/bottlepy/bottle/issues/1130

I've encountered it a 3 days session on adding CI to my favourite cheat sheet server https://github.com/chubin/cheat.sh/pull/224/commits/10bfaab38cc649d765637bf4af4791266d21a3f7 and I am still not sure I am right. That's why I want to read something about it.
msg374716 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-08-03 04:25
I don't think that waiting another six months is going to make this happen either. Maybe you can petition realpython.com to publish something about this.

Honestly it sounds like this ties too many different topics together to easily fit in the library docs: in order to make a large number of different use cases easy, how I/O works under the hood in Python is truly complex.

So I'm going to close this rather than have it add to our issue count.
msg374719 - (view) Author: Anatoli Babenia (Anatoli Babenia) Date: 2020-08-03 04:29
To avoid too much outside reference, the commit message I am unsure of is this.

> Run Python 3 in unbuffered mode to see lines in `docker logs` as soon as they appear. When Python 3 is not attached to a terminal, it turns on the buffering, like when `docker` runs in a background.

I am not sure.

1. How Python detects it is attached to terminal or not.
2. That it is Python that turns on stdout buffering.
3. What happens to stdout with child processes when they run with docker in different modes - foreground, background (-d), interactive (-i), with tty (-t) and without tty.
msg374720 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-08-03 04:33
I don't know anything about Docker, so I can't help you there. Please don't reopen the issue again.
msg374721 - (view) Author: Anatoli Babenia (Anatoli Babenia) Date: 2020-08-03 04:48
I didn't reopen the issue explicitly. Just pressed the submit button again when the notification about edited bug appeared.
msg374722 - (view) Author: Anatoli Babenia (Anatoli Babenia) Date: 2020-08-03 04:49
The point is that without a clear description of buffering in Python 3, it is impossible to understand what goes when Python is wrapped by Docker, supervisord, redirected to file etc.
History
Date User Action Args
2020-08-03 04:49:46Anatoli Babeniasetmessages: + msg374722
2020-08-03 04:48:13Anatoli Babeniasetmessages: + msg374721
2020-08-03 04:33:06gvanrossumsetstatus: open -> closed
resolution: wont fix
messages: + msg374720
2020-08-03 04:29:35Anatoli Babeniasetstatus: closed -> open
resolution: wont fix -> (no value)
messages: + msg374719
2020-08-03 04:25:06gvanrossumsetstatus: open -> closed
resolution: wont fix
messages: + msg374716

stage: resolved
2020-08-03 04:21:48Anatoli Babeniasetmessages: + msg374714
2020-08-03 00:53:52gvanrossumsetnosy: + gvanrossum
messages: + msg374706
2020-08-02 03:30:08teddersetnosy: + tedder
2020-07-31 10:10:32Anatoli Babeniacreate