Message303429
originally detected on python 2.7, but replicated with python 3.5.3 -- apparently io.FileIO, if given a bytestring of 2GB or more, cannot write it all at once -- saves (and returns that size) only 2GB - 4096.
I found no indication for such behavior anywhere in the documentation. And it is surprising to me especially since regular file.write does it just fine! attached is the code snippet which I list below and which demonstrates it
$> python3 --version; python3 longwrite.py
Python 3.5.3
Written 2147479552 out of 2147483648
4096 bytes were not written
Traceback (most recent call last):
File "longwrite.py", line 28, in <module>
assert in_digest == out_digest, "Digests do not match"
AssertionError: Digests do not match
python3 longwrite.py 7.03s user 5.80s system 99% cpu 12.848 total
1 11365 ->1.....................................:Sat 30 Sep 2017 04:56:26 PM EDT:.
smaug:/mnt/btrfs/scrap/tmp
$> cat longwrite.py
# -*- coding: utf-8 -*-
import io
import os
import hashlib
s = u' '*(256**4//2) #+ u"перфекто"
s=s.encode('utf-8')
#s=' '*(10)
in_digest = hashlib.md5(s).hexdigest()
fname = 'outlong.dat'
if os.path.exists(fname):
os.unlink(fname)
with io.FileIO(fname, 'wb') as f:
#with open(fname, 'wb') as f:
n = f.write(s)
#n = os.stat(fname).st_size
print("Written %d out of %d" % (n, len(s)))
if n != len(s):
print("%d bytes were not written" % (len(s) - n))
# checksum
with open(fname, 'rb') as f:
out_digest = hashlib.md5(f.read()).hexdigest()
assert in_digest == out_digest, "Digests do not match"
print("all ok") |
|
Date |
User |
Action |
Args |
2017-09-30 20:58:07 | Yaroslav.Halchenko | set | recipients:
+ Yaroslav.Halchenko |
2017-09-30 20:58:07 | Yaroslav.Halchenko | set | messageid: <1506805087.57.0.213398074469.issue31651@psf.upfronthosting.co.za> |
2017-09-30 20:58:07 | Yaroslav.Halchenko | link | issue31651 messages |
2017-09-30 20:58:07 | Yaroslav.Halchenko | create | |
|