This is an example, last assert return an error :

f = open('data.txt', 'w')
f.write("""line 1
line 2
line 3
line 4
line 5
line 6
line 7
line 8
line 9
line 10
line 11
""")
f.close()


f = open('data.txt', 'r')
assert f.readline() == 'line 1\n'

assert f.read() == """line 2
line 3
line 4
line 5
line 6
line 7
line 8
line 9
line 10
line 11
"""

f.close()

import codecs

f = codecs.open('data.txt', 'r', 'utf8')

assert f.read() == """line 1
line 2
line 3
line 4
line 5
line 6
line 7
line 8
line 9
line 10
line 11
"""

f.close()

f = codecs.open('data.txt', 'r', 'utf8')

assert f.readline() == 'line 1\n'

# this assert return a ERROR
assert f.read() == """line 2
line 3
line 4
line 5
line 6
line 7
line 8
line 9
line 10
line 11
"""

f.close()

Regards,
Stephane

Hi Stephane,

I think you're seeing different buffering behavior, which I suspect is correct according to docs.

codecs.open should default to line buffering[1], while open uses the system default[2].

The read() where the assert fails is returning the remaining buffer from the readline (which read 72 chars).

Asserting e.g. "f.read(1024) == ..." will give you the expected result.

[1] http://docs.python.org/library/codecs.html#codecs.open
[2] http://docs.python.org/library/functions.html#open

Buffering applies when writing, not when reading a file.

There is indeed a problem in codecs.py: after a readline(), read() will return the content of the internal buffer, and not more.

The "size" parameter is a hint, and should not be used to decide whether the character buffer is enough to satisfy the read() request.
Patch is attached, with test.

Amaury Forgeot d'Arc wrote:
> 
> Amaury Forgeot d'Arc <amauryfa@gmail.com> added the comment:
> 
> Buffering applies when writing, not when reading a file.
> 
> There is indeed a problem in codecs.py: after a readline(), read() will return the content of the internal buffer, and not more.
> 
> The "size" parameter is a hint, and should not be used to decide whether the character buffer is enough to satisfy the read() request.
> Patch is attached, with test.

Agreed.

The patch looks good except the if-line should read:

if chars >= 0 and len(self.charbuffer) >= chars:
  ...

Thanks,
-- 
Marc-Andre Lemburg
eGenix.com

________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

Updated patch.

[I also tried to avoid reading the underlying file if len(self.bytebuffer)>=size, but it does not work with multibytes chars when size=1]

I applied the diff to test_codecs in py3k, removed the u prefixes and ran: failure.  I applied the fix and the test passed.

Up, I think this patch isn't applied in Python 3.3a0.

According to this ticket it hasn't been applied anywhere yet (a message will be posted here when it is).

See also #12446.

I think the patch is wrong or is not optimal for case when chars is -1, but size is not.

If we want to read all data in any case, then we should call self.stream.read() without argument if chars < 0 or size < 0.

If we want to read no more than size bytes, then all loop code should be totally rewritten.

Perhaps I am wrong.

As showed in issue12446, issue14475 and issue16636 there are different methods to reproduce this bug (read(size, chars) + readlines(), readline() + readlines()). All this cases should be tested.