Title: Running Python in unbuffered mode may not write all contents to the console
Type: behavior Stage:
Components: IO Versions: Python 3.8
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: eryksun, fabioz
Priority: normal Keywords:

Created on 2020-10-15 13:22 by fabioz, last changed 2020-10-15 19:00 by eryksun.

Messages (2)
msg378684 - (view) Author: Fabio Zadrozny (fabioz) Date: 2020-10-15 13:22
When running Python in unbuffered mode it may fail to write all the contents to the actual console (on Windows).

The code below can reproduce the issue: 

import sys
s = ''
for i in range(1,301):
    s += f"{str(i*100).zfill(10)}{'x' * 89}\n"


When calling it with `python -u` it'll write only up to line 15000 and when calling it with `python` it'll write up to line 30000.

This fails because in `_textiowrapper_writeflush` it doesn't verify if all the contents have been indeed written and thus fails in a partial write. In buffered mode it works because `_io_BufferedWriter_write_impl` does the job properly.

I'm a bit uncertain on why doesn't `_io__WindowsConsoleIO_write_impl` itself do the loop to write everything instead of leaving it up to callers to do that work (apparently due to issue11395 it says that it only writes partially, but maybe the fix could've been to loop inside of `_io__WindowsConsoleIO_write_impl` to write everything instead of expecting callers to handle partial writes...
msg378693 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2020-10-15 19:00
Text mode without a buffer isn't reliable. That said, Python 3.9 no longer supports Windows 7, so it can remove the 32 KiB limit on console I/O files.

The size limit in Windows 7 and earlier is due to the LPC-based pseudo-files that it uses for I/O. Under the hood, console pseudo-files use a 64 KiB heap that's shared between the console host process and client processes. 

In Windows 8+, console files are real files that use the ConDrv device instead of LPC, in which case there is practically no limit on the size of buffers that can be read and written.
Date User Action Args
2020-10-15 19:00:29eryksunsetnosy: + eryksun
messages: + msg378693
2020-10-15 13:22:04fabiozcreate