A prefetch() method for Buffered IO may greatly assist 3rd party buffering among other gains. If nothing else, it is worth experimenting with. 

Discussion on the topic is here: http://mail.python.org/pipermail/python-ideas/2010-September/008180.html

A summary of the method proposed (by Antoine Pitrou):

prefetch(self, buffer, skip, minread)

Skip `skip` bytes from the stream.  Then, try to read at
least `minread` bytes and write them into `buffer`. The file
pointer is advanced by at most `skip + minread`, or less if
the end of file was reached. The total number of bytes written
in `buffer` is returned, which can be more than `minread`
if additional bytes could be prefetched (but, of course,
cannot be more than `len(buffer)`).

Arguments:
- `buffer`: a writable buffer (e.g. bytearray)
- `skip`: number of bytes to skip (must be >= 0)
- `minread`: number of bytes to read (must be >= 0 and <= len(buffer))

I started a draft in python. I am attaching the _pyio version along with tests. I will continue work on the C implementation and eventually documentation if this is well received. It seems straightforward, I am interested to see what you guys think.

Also, there are now 2 places which use hasattr(self, "peek"). I was wondering if it would make sense to add peek() to BufferedIOBase and raise UnsupportedOperation or return b"".

Some benchmarks..

$ ./python -m timeit -s "from _pyio import open;f = open('LICENSE', 'rb'); b=bytearray(128)" 'while f.prefetch(b, 4, 4): pass'
_pyio.BufferedIOBase.prefetch:
100000 loops, best of 3: 10.6 usec per loop
_pyio.BufferedReader.prefetch:
100000 loops, best of 3: 6 usec per loop

$ ./python -m timeit -s "from _pyio import open;f = open('LICENSE', 'rb');b=bytearray(4);" 'while f.read(4): f.readinto(b)'
100000 loops, best of 3: 5.07 usec per loop

> I started a draft in python. I am attaching the _pyio version along
> with tests. I will continue work on the C implementation and
> eventually documentation if this is well received. It seems
> straightforward, I am interested to see what you guys think.

Thank you. I think performance measurements are prematurate until we
have an optimized C implementation anyway.

I think ultimately we also want a default implementation of read(),
peek() and read1() which uses prefetch(), so that BufferedReader
implementations only have to implement prefetch().
(care must be taken to avoid infinite loops)

That said, I think the python-dev mailing-list needs to be convinced of
the usefulness of prefetch() (if it was only me, there wouldn't be any
problem :-)). Perhaps you want to run another discussion there.

Here is an update with the C implementation. I think a working prototype will be helpful before another round on python-dev. 


I'm not sure how to handle unseekable, non-blocking streams where the read returns before `skip` bytes are exhausted. If prefetch() returns 0, then the caller would then have to use tell() to ensure subsequent reads are sane. In other words it seems prefetch() will leave the stream in an unpredictable state. Antoine, what are your thoughts?