This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: ValueError: can't have unbuffered text I/O for io.open(1, 'wt', 0)
Type: behavior Stage:
Components: IO Versions: Python 3.10, Python 3.9, Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: amaury.forgeotdarc, benjamin.peterson, hynek, martin.panter, pitrou, rbcollins, serhiy.storchaka, stutzbach
Priority: normal Keywords:

Created on 2013-03-12 14:42 by rbcollins, last changed 2022-04-11 14:57 by admin.

Messages (7)
msg184025 - (view) Author: Robert Collins (rbcollins) * (Python committer) Date: 2013-03-12 14:42
The io library rejects unbuffered text I/O, but this is not documented - and in fact can be manually worked around:
    binstdout = io.open(sys.stdout.fileno(), 'wt', 0)
    sys.stdout = io.TextIOWrapper(binstdout, encoding=sys.stdout.encoding)
will get a sys.stdout that is unbuffered.

Note that writing to a pipe doesn't really need to care about buffering anyway, if the user writes 300 characters, the codec will output a single block and the IO made will be one write:

This test script:
import sys
import io
stream = io.TextIOWrapper(io.open(sys.stdout.fileno(), 'wb', 0), encoding='utf8')
for r in range(10):
  stream.write(u'\u1234'*500)

When run under strace -c does exactly 10 writes: so the performance is predictable. IMO it doesn't make sense to prohibit unbuffered text write I/O. readers may be another matter, but that doesn't suffer the same latency issues.
msg184040 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2013-03-12 18:58
The proposed workaround seems to work ("wb" instead of "wt"!), with the following restrictions:

- it's not really unbuffered: the encoder has its own buffers (OK, in the stdlib only 'idna' encoding will retain data)

- it won't work for reading: TextIOWrapper calls the read1() method, which is only defined by BufferedIO objects.

IMO this explains why it's not a supported combination in io.open().
msg184041 - (view) Author: Robert Collins (rbcollins) * (Python committer) Date: 2013-03-12 19:05
Huh, I didn't realise idna would retain data! But that will still be within the TextIOWrapper itself, right?

And a stream opened 'wt' cannot be read from anyway, so the read1 limitation is irrelevant.
msg184045 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2013-03-12 19:30
> But that will still be within the TextIOWrapper itself, right?

Yes. And I just noticed that the _io module (the C version) will also buffer encoded bytes, up to f._CHUNK_SIZE.

On the other hand, TextIOWrapper is broken for buffering codecs, encode() is never called with final=True

    >>> import io
    >>> buffer = io.BytesIO()      # <-- not really buffered, right?
    >>> output = io.TextIOWrapper(buffer, encoding='idna')
    >>> output.write("www.somesite.com")
    16
    >>> print(buffer.getvalue())
    b''                            # <-- ok, _CHUNK_SIZE buffering
    >>> output.flush()
    >>> print(buffer.getvalue())
    b'www.somesite.'               # <-- the last word is missing!
    >>> output.close()
    >>> print(buffer.getvalue())
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ValueError: I/O operation on closed file.


And it's even worse with python 2.7::

    >>> import io as io
    >>> buffer = io.BytesIO()
    >>> output = io.TextIOWrapper(buffer, encoding='idna')
    >>> output.write("www.somesite.com")
    Traceback (most recent call last):
      File "<stdin>", line 3, in <module>
    TypeError: must be unicode, not str
msg184072 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-03-13 08:16
> - it won't work for reading: TextIOWrapper calls the read1() method, which is only defined by BufferedIO objects.

Since 3.3 TextIOWrapper works with raw IO objects (issue12591).

> Yes. And I just noticed that the _io module (the C version) will also buffer encoded bytes, up to f._CHUNK_SIZE.

Use write_through=True to disable this.
msg184073 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-03-13 09:23
> > - it won't work for reading: TextIOWrapper calls the read1()
> > method, which is only defined by BufferedIO objects.
> 
> Since 3.3 TextIOWrapper works with raw IO objects (issue12591).

It won't be technically unbuffered, though.
msg332739 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2018-12-30 00:00
It is documented that buffering=0 is not supported in text mode. Look a handful of paragraphs down from <https://docs.python.org/release/3.7.2/library/functions.html#open>:

“Pass 0 to switch buffering off (only allowed in binary mode)”

Amaury’s problem with the IDNA buffering encoder has now been reported separately in Issue 35611.
History
Date User Action Args
2022-04-11 14:57:42adminsetgithub: 61606
2020-11-06 22:51:34iritkatrielsetversions: + Python 3.8, Python 3.9, Python 3.10, - Python 2.7, Python 3.2, Python 3.3, Python 3.4
2018-12-30 00:00:51martin.pantersetnosy: + martin.panter
messages: + msg332739
2013-03-13 09:23:44pitrousetmessages: + msg184073
title: ValueError: can't have unbuffered text I/O for io.open(1, 'wt', 0) -> ValueError: can't have unbuffered text I/O for io.open(1, 'wt', 0)
2013-03-13 08:16:19serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg184072
2013-03-12 19:30:00amaury.forgeotdarcsetmessages: + msg184045
2013-03-12 19:05:19rbcollinssetmessages: + msg184041
2013-03-12 18:58:36amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg184040
2013-03-12 15:03:06serhiy.storchakasetnosy: + pitrou, benjamin.peterson, stutzbach, hynek

components: + IO
versions: - Python 2.6, Python 3.1, Python 3.5
2013-03-12 14:42:40rbcollinscreate