classification
Title: Running Python in unbuffered mode may not write all contents to the console
Type: behavior Stage: patch review
Components: IO Versions: Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: eryksun, fabioz
Priority: normal Keywords: patch

Created on 2020-10-15 13:22 by fabioz, last changed 2021-06-11 18:26 by fabioz.

Pull Requests
URL Status Linked Edit
PR 26678 open fabioz, 2021-06-11 18:17
Messages (3)
msg378684 - (view) Author: Fabio Zadrozny (fabioz) * Date: 2020-10-15 13:22
When running Python in unbuffered mode it may fail to write all the contents to the actual console (on Windows).

The code below can reproduce the issue: 

```
import sys
s = ''
for i in range(1,301):
    s += f"{str(i*100).zfill(10)}{'x' * 89}\n"

sys.stdout.write(s)
```

When calling it with `python -u code.py` it'll write only up to line 15000 and when calling it with `python code.py` it'll write up to line 30000.

This fails because in `_textiowrapper_writeflush` it doesn't verify if all the contents have been indeed written and thus fails in a partial write. In buffered mode it works because `_io_BufferedWriter_write_impl` does the job properly.

I'm a bit uncertain on why doesn't `_io__WindowsConsoleIO_write_impl` itself do the loop to write everything instead of leaving it up to callers to do that work (apparently due to issue11395 it says that it only writes partially, but maybe the fix could've been to loop inside of `_io__WindowsConsoleIO_write_impl` to write everything instead of expecting callers to handle partial writes...
msg378693 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2020-10-15 19:00
Text mode without a buffer isn't reliable. That said, Python 3.9 no longer supports Windows 7, so it can remove the 32 KiB limit on console I/O files.

The size limit in Windows 7 and earlier is due to the LPC-based pseudo-files that it uses for I/O. Under the hood, console pseudo-files use a 64 KiB heap that's shared between the console host process and client processes. 

In Windows 8+, console files are real files that use the ConDrv device instead of LPC, in which case there is practically no limit on the size of buffers that can be read and written.
msg395654 - (view) Author: Fabio Zadrozny (fabioz) * Date: 2021-06-11 18:26
Seems fair. I just did a pull request to remove those limits. 

Please let me know if you think something else is needed there.
History
Date User Action Args
2021-06-11 18:26:14fabiozsetmessages: + msg395654
2021-06-11 18:17:40fabiozsetkeywords: + patch
stage: patch review
pull_requests: + pull_request25264
2020-10-15 19:00:29eryksunsetnosy: + eryksun
messages: + msg378693
2020-10-15 13:22:04fabiozcreate