This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author Yaroslav.Halchenko
Recipients Yaroslav.Halchenko
Date 2017-09-30.20:58:07
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1506805087.57.0.213398074469.issue31651@psf.upfronthosting.co.za>
In-reply-to
Content
originally detected on python 2.7, but replicated with python 3.5.3 -- apparently io.FileIO, if given a bytestring of 2GB or more, cannot write it all at once -- saves (and returns that size) only 2GB - 4096.

I found no indication for such behavior anywhere in the documentation. And it is surprising to me especially since regular file.write does it just fine!  attached is the code snippet which I list below and which demonstrates it

$> python3 --version; python3 longwrite.py
Python 3.5.3
Written 2147479552 out of 2147483648
4096 bytes were not written
Traceback (most recent call last):
  File "longwrite.py", line 28, in <module>
    assert in_digest == out_digest, "Digests do not match"
AssertionError: Digests do not match
python3 longwrite.py  7.03s user 5.80s system 99% cpu 12.848 total
1 11365 ->1.....................................:Sat 30 Sep 2017 04:56:26 PM EDT:.
smaug:/mnt/btrfs/scrap/tmp
$> cat longwrite.py
# -*- coding: utf-8 -*-
import io
import os
import hashlib

s = u' '*(256**4//2)  #+ u"перфекто"
s=s.encode('utf-8')
#s=' '*(10)

in_digest = hashlib.md5(s).hexdigest()
fname = 'outlong.dat'

if os.path.exists(fname):
    os.unlink(fname)

with io.FileIO(fname, 'wb') as f:
#with open(fname, 'wb') as f:
     n = f.write(s)

#n = os.stat(fname).st_size
print("Written %d out of %d" % (n, len(s)))
if n != len(s):
    print("%d bytes were not written" % (len(s) - n))

# checksum
with open(fname, 'rb') as f:
    out_digest = hashlib.md5(f.read()).hexdigest()
assert in_digest == out_digest, "Digests do not match"
print("all ok")
History
Date User Action Args
2017-09-30 20:58:07Yaroslav.Halchenkosetrecipients: + Yaroslav.Halchenko
2017-09-30 20:58:07Yaroslav.Halchenkosetmessageid: <1506805087.57.0.213398074469.issue31651@psf.upfronthosting.co.za>
2017-09-30 20:58:07Yaroslav.Halchenkolinkissue31651 messages
2017-09-30 20:58:07Yaroslav.Halchenkocreate