classification
Title: Buffered I/O inconsistent with unbuffered I/O in certain cases
Type: behavior Stage: resolved
Components: IO Versions: Python 3.1, Python 3.2, Python 3.3, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: barry, benjamin.peterson, genstein, georg.brandl, haypo, jcea, jcon, nadeem.vawda, pitrou, python-dev
Priority: release blocker Keywords: patch

Created on 2011-05-12 13:07 by genstein, last changed 2011-05-30 03:20 by jcea. This issue is now closed.

Files
File name Uploaded Description Edit
bufbug.patch pitrou, 2011-05-12 13:39 review
Messages (7)
msg135830 - (view) Author: Genstein (genstein) Date: 2011-05-12 13:07
Reporting this as requested by Antoine Pitrou: Under certain circumstances in Python 3.2 (r32:88445) it's possible for buffered I/O to lose data before it is written and/or return the wrong results when reading. I tripped over this issue whilst writing an assembler which first writes out a bunch of binary data and then goes back over it in a somewhat arbitrary order patching addresses.

The following code demonstrates the issue:

[code]
START = 0
MID = 1
LENGTH = 4

def test(buffering):
    f = open("test.bin", "w+b", buffering = buffering)
    for i in range(LENGTH):
        f.write(b'\x00')
    f.seek(MID)
    f.read(1)
    f.write(b'\x00')
    f.seek(MID)
    f.write(b'\x01')
    f.seek(START)
    f.seek(MID)
    print(f.read(1))
    f.close()

print("Buffered result: ")
test(-1)
print("Unbuffered result:")
test(0)
[end code]

Output on both Gentoo and Vista is:
    Buffered result:
    b'\x00'
    Unbuffered result:
    b'\x01'

Expected output is b'\x01' in both cases.

The issue is reproducible with larger files provided that the constants are increased ~proportionally (START 0, MID 500, LENGTH 1000 for example). Transposing the buffered/unbuffered tests and/or using different buffer sizes for the buffered test seem have no effect. 

The lengthy code at this URL also demonstrates the issue: http://pastebin.com/xqrzKr5D The above was produced from this code, which was autogenerated by intercepting my assembler's output.
msg135831 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-05-12 13:25
If you add "from _pyio import open" to the example, unbuffered and buffered tests give the same result :-)
msg135833 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-05-12 13:39
Here is a patch with assorted tests.
msg135834 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-05-12 13:45
open("test.bin", "w+b", buffering=-1) creates a 
BufferedRandom object. In the _pyio module, BufferedRandom overrides the write() method to "undo readahead", whereas the _io module reuses bufferedwriter_write() for bufferedrandom_methods and bufferedwriter_methods.

I suppose that the problem is "just" that _io.BufferedRandom.write() doesn't undo readahead.
msg135876 - (view) Author: Roundup Robot (python-dev) Date: 2011-05-12 22:28
New changeset 89f77afac947 by Antoine Pitrou in branch '3.1':
Issue #12062: Fix a flushing bug when doing a certain type of I/O sequence
http://hg.python.org/cpython/rev/89f77afac947

New changeset 47ca1244a929 by Antoine Pitrou in branch '3.2':
Issue #12062: Fix a flushing bug when doing a certain type of I/O sequence
http://hg.python.org/cpython/rev/47ca1244a929

New changeset 05e0227e3879 by Antoine Pitrou in branch 'default':
Issue #12062: Fix a flushing bug when doing a certain type of I/O sequence
http://hg.python.org/cpython/rev/05e0227e3879
msg135877 - (view) Author: Roundup Robot (python-dev) Date: 2011-05-12 22:32
New changeset 0d24d4c537a6 by Antoine Pitrou in branch '2.7':
Issue #12062: In the `io` module, fix a flushing bug when doing a certain
http://hg.python.org/cpython/rev/0d24d4c537a6
msg135878 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-05-12 22:33
Should be fixed now. Again, thank you reporting this bug!
History
Date User Action Args
2011-05-30 03:20:48jceasetnosy: + jcea
2011-05-12 22:33:22pitrousetstatus: open -> closed
resolution: fixed
messages: + msg135878

stage: patch review -> resolved
2011-05-12 22:32:30python-devsetmessages: + msg135877
2011-05-12 22:28:30python-devsetnosy: + python-dev
messages: + msg135876
2011-05-12 17:32:20jconsetnosy: + jcon
2011-05-12 14:10:30nadeem.vawdasetnosy: + nadeem.vawda
2011-05-12 13:45:20hayposetmessages: + msg135834
2011-05-12 13:39:12pitrousetfiles: + bufbug.patch
priority: normal -> release blocker

versions: + Python 3.1, Python 2.7, Python 3.3
keywords: + patch
nosy: + pitrou, barry, benjamin.peterson, georg.brandl

messages: + msg135833
stage: patch review
2011-05-12 13:25:26hayposetnosy: + haypo
messages: + msg135831
2011-05-12 13:07:27gensteincreate