classification
Title: io.TextIOWrapper calls buffer.read1()
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.0
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: benjamin.peterson, gvanrossum, kawai, pitrou, vstinner
Priority: normal Keywords:

Created on 2009-01-19 10:09 by kawai, last changed 2009-03-05 00:25 by vstinner. This issue is now closed.

Messages (10)
msg80152 - (view) Author: HiroakiKawai (kawai) Date: 2009-01-19 10:09
The documentation says io.TextIOWrapper wraps io.BufferedIOBase raw 
stream. In the code, io.TextIOWrapper.read(), 
io.TextIOWrapper._read_chunk() calls buffer.read1() which seems 
expecting buffer to be an instance of io.BufferedReader. I'm not sure 
which is correct right now.
msg80167 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-01-19 13:28
Indeed, read1() is not documented as a standard method of either IOBase,
RawIOBase or BufferedIOBase. I suggest that it becomes a standard method
of IOBase, with a default implementation simply calling read(n).

It also means unbuffered stdio as it was recently committed doesn't work
for stdin:

$ ./python -u
Python 3.1a0 (py3k:68756, Jan 19 2009, 01:17:26) 
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.stdin.read(1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/antoine/py3k/__svn__/Lib/io.py", line 1739, in read
    eof = not self._read_chunk()
  File "/home/antoine/py3k/__svn__/Lib/io.py", line 1565, in _read_chunk
    input_chunk = self.buffer.read1(self._CHUNK_SIZE)
AttributeError: 'FileIO' object has no attribute 'read1'
>>> 

I had been misguided by the fact that the interpreter prompt did work,
but it doesn't seem to use sys.stdin...
msg80188 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2009-01-19 16:14
IOBase doesn't even define read(), though! I think we should make it
part of BufferIOBase.
msg80193 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2009-01-19 16:57
Usage of read1():
 * _BytesIO.read1(n) (_bytesio.c): calls bytesio_read(n)
 * _BytesIO.read1(n) (io.py): return self.read(n)
 * BufferedReader._read_unlocked(n=None): 
   - if n is None or n==-1, call self.raw.read() in a "while 
True: ..." until EOF
   - else, call self.raw.read() until we get enough bytes
 * BufferedReader.read1(n): calls self._read_unlocked(...) but make 
sure that we don't call self.raw.read() more than once
 * BufferedRWPair.read1(n): return self.reader.read1(n)
 * BufferedRandom.read1(n): self.flush(); return BufferedReader.read1
(self, n)
 * TextIOWrapper._read_chunk() calls self.buffer.read1
(self._CHUNK_SIZE)

IOBase, RawIOBase, FileIO, BufferedIOBase, TextIOBase, TextIOWrapper, 
StringIO have no read1() method.
msg80198 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2009-01-19 17:59
Short example to reproduce the problem:
---
import io, os
fd = os.open("/etc/issue", os.O_RDONLY)
raw = open(fd, "rb", buffering=0)
text = io.TextIOWrapper(raw, line_buffering=False)
print(text.read(1))
---

Traceback (most recent call last):
  File "x.py", line 6, in <module>
    print(text.read(1))
  File "/home/SHARE/SVN/py3k/Lib/io.py", line 1739, in read
    eof = not self._read_chunk()
  File "/home/SHARE/SVN/py3k/Lib/io.py", line 1565, in _read_chunk
    input_chunk = self.buffer.read1(self._CHUNK_SIZE)
AttributeError: 'FileIO' object has no attribute 'read1'
msg80199 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2009-01-19 18:02
On Mon, Jan 19, 2009 at 11:59 AM, STINNER Victor <report@bugs.python.org> wrote:
>
> STINNER Victor <victor.stinner@haypocalc.com> added the comment:
>
> Short example to reproduce the problem:
> ---
> import io, os
> fd = os.open("/etc/issue", os.O_RDONLY)
> raw = open(fd, "rb", buffering=0)
> text = io.TextIOWrapper(raw, line_buffering=False)
> print(text.read(1))

You can only use TextIOWrapper over a buffered stream, so this example
is invalid anyway.
msg80201 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2009-01-19 18:17
I don't understand the motivation of having two different methods 
for "raw" streams: read1() and read(). I would prefer to have only a 
method read() (because "read" is the most common name, whereas read1() 
is only used on Python3), but read() will have to follow the read1() 
rule: at most one syscall. If someone requires a read() method with 
the "one syscall" rule, he can use a raw stream. If you don't care, 
use high level classes (TextIOWrapper, BufferedReader, etc.).

_fileio._FileIO.read() and _socket.socket.recv_into() already respect 
the syscall rule.

Since BytesIO and StringIO don't use kernel object, the syscall rule 
is meaningles.

Note: socket.SocketIO() implements RawIOBase but it has no read() nor 
write() methods ;-) That's maybe why IOBase has no read()/write() 
method.
msg82921 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2009-02-28 17:10
In the io-c branch, read1() is now a member of BufferedIOBase.
msg83138 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2009-03-04 21:32
This has been fixed in io-c branch merge. (r70152)
msg83168 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2009-03-05 00:25
> This has been fixed in io-c branch merge. r70152

Amazing, io-c is faster but also fixes many bugs!
History
Date User Action Args
2009-03-05 00:25:49vstinnersetmessages: + msg83168
2009-03-04 21:32:07benjamin.petersonsetstatus: open -> closed
resolution: fixed
messages: + msg83138
2009-02-28 17:12:32benjamin.petersonlinkissue4565 dependencies
2009-02-28 17:10:14benjamin.petersonsetmessages: + msg82921
2009-01-19 18:17:58vstinnersetmessages: + msg80201
2009-01-19 18:02:07benjamin.petersonsetmessages: + msg80199
2009-01-19 17:59:41vstinnersetmessages: + msg80198
2009-01-19 16:57:53vstinnersetmessages: + msg80193
2009-01-19 16:20:56vstinnersetnosy: + vstinner
2009-01-19 16:14:02benjamin.petersonsetnosy: + benjamin.peterson
messages: + msg80188
2009-01-19 13:29:18pitrousetmessages: - msg80166
2009-01-19 13:28:33pitrousetnosy: + gvanrossum
messages: + msg80167
2009-01-19 13:28:18pitrousetnosy: + pitrou
messages: + msg80166
2009-01-19 10:09:55kawaicreate