First a minor correction:
> The new requirement would be that a correct app also needs to flush between a sequence of buffer.writes (that end in an incomplete line, or always if PYTHONUNBUFFERED or python -u is used), and a sequence of writes.

That should be "and only if PYTHONUNBUFFERED or python -u is not used".

I also said:
> If an app sets the .buffer attribute of sys.std{out,err}, it would fall back to using that buffer in the same way as when the fd is redirected.

but the .buffer attribute is readonly, so this case can't occur.

Glenn Linderman wrote:
> Would it suffice if the new scheme internally flushed after every buffer.write?  It wouldn't be needed after write, because the correct application would already do one there?

Yes, that would be sufficient.

> Am I off-base in supposing that the performance of buffer.write is expected to include a flush (because it isn't expected to be buffered)?

It is expected to be line-buffered. So an app might expect that printing characters one-at-a-time will have reasonable performance.

In any case, given that the buffer of the initial std{out,err} will always be a BufferedWriter object (since .buffer is readonly), it would be possible for the TextIOWriter to test a dirty flag in the BufferedWriter, in order to check efficiently whether the buffer needs flushing on each write. I've looked at the implementation complexity cost of this, and it doesn't seem too bad.

A similar issue arises for stdin: to maintain strict compatibility, every read from a TextIOWrapper attached to an input console would have to drain the buffer of its buffer object, in case the app has read from it. This is a bit tricky because the bytes drained from the buffer have to be converted to Unicode, so what happens if they end part-way through a multibyte character? Ugh, I'll have to think about that one.

Victor STINNER wrote:
> Some developers already think that adding sys.stdout.flush() after
print("Processing.. ", end='') is too hard (#11633).

IIUC, that bug is about the behaviour of 'print', and didn't suggest to change the fact that sys.stdout is line-buffered.


By the way, are these changes going to be in a major release? If I understand correctly, the layout of structs (for standard library types not prefixed with '_', such as 'buffered' in bufferedio.c or 'textio' in textio.c) can change with major releases but not with minor releases, correct?