classification
Title: Support unbuffered TextIOWrapper
Type: Stage:
Components: IO Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: haypo, serhiy.storchaka
Priority: normal Keywords:

Created on 2017-10-12 15:09 by haypo, last changed 2017-10-12 16:19 by serhiy.storchaka.

Messages (7)
msg304250 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-10-12 15:09
It seems like that some methods of the io.TextIOWrapper class requires that its buffer object has the read1() method, whereas the constructor checks if the buffer has a read1() method and the TextIOWrapper _read_chunk() method is able to call buffer.read() if buffer doesn't have read1().

This issue may help to get fully unbuffered sys.stdin, at least when replaced manually:

stdin = sys.stdin
sys.stdin = open(0, "r", buffering=0, encoding=stdin.encoding, errors=stdin.errors, newline=stdin.newline)
msg304252 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-10-12 15:12
I created this issue because of this comment in create_stdio() of Python/pylifecycle.c:
---
    /* stdin is always opened in buffered mode, first because it shouldn't
       make a difference in common use cases, second because TextIOWrapper
       depends on the presence of a read1() method which only exists on
       buffered streams.
    */
    if (Py_UnbufferedStdioFlag && write_mode)
        buffering = 0;
    else
        buffering = -1;
---
msg304253 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-10-12 15:25
Oh, currently it's not possible to create an unbuffered read-only TextIOWrapper object:

haypo@selma$ python3
Python 3.6.2 (default, Oct  2 2017, 16:51:32) 
>>> open("/etc/issue", "r", 0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: can't have unbuffered text I/O
msg304258 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-10-12 15:30
The falling back to read() was implemented in issue12591. What methods still require read1()?
msg304260 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-10-12 15:46
> The falling back to read() was implemented in issue12591. What methods still require read1()?

I read the Python implementation of TextIOWrapper in _pyio:

* seek() calls buffer.read() "if chars_to_skip:"... not sure that it can happen if buffer is a FileIO?
* read(None)  and read(-1) to read "everything" (until EOF): buffer.read()

It seems like read(None) also calls buffer.read() in the C implementation, _io_TextIOWrapper_read_impl():

        /* Read everything */
        PyObject *bytes = _PyObject_CallMethodId(self->buffer, &PyId_read, NULL);
        PyObject *decoded;
        if (bytes == NULL)
            goto fail;

and in _io_TextIOWrapper_seek_impl():

        /* Just like _read_chunk, feed the decoder and save a snapshot. */
        PyObject *input_chunk = _PyObject_CallMethodId(
            self->buffer, &PyId_read, "i", cookie.bytes_to_feed);
msg304262 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-10-12 15:48
Context: I created this issue as a follow-up of the discussion on the -u command line option, bpo-28647, which still leaves sys.stdin *buffered* in the current master branch.
msg304266 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-10-12 16:19
What methods still require read1()?
History
Date User Action Args
2017-10-12 16:19:18serhiy.storchakasetmessages: + msg304266
2017-10-12 15:48:40hayposetmessages: + msg304262
2017-10-12 15:46:21hayposetmessages: + msg304260
2017-10-12 15:30:50serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg304258
2017-10-12 15:25:04hayposetmessages: + msg304253
2017-10-12 15:12:26hayposetmessages: + msg304252
2017-10-12 15:09:32haypocreate