Message389652
MemoryError: null
...
File "....", line 13, in repack__file
shutil.unpack_archive(local_file_path, local_dir)
File "python3.6/shutil.py", line 983, in unpack_archive
func(filename, extract_dir, **kwargs)
File "python3.6/shutil.py", line 901, in _unpack_zipfile
data = zip.read(info.filename)
File "python3.6/zipfile.py", line 1338, in read
return fp.read()
File "python3.6/zipfile.py", line 858, in read
buf += self._read1(self.MAX_N)
File "python3.6/zipfile.py", line 948, in _read1
data = self._decompressor.decompress(data, n)
shutil.unpack_archive tries to read the whole file into memory, making use of any buffer at all. Python crashes for really large files. In my case — archive: ~1.7G, unpacked: ~10G. Interestingly zipfile.ZipFile.extractall handles this case more effective. |
|
Date |
User |
Action |
Args |
2021-03-28 21:21:23 | igorvoltaic | set | recipients:
+ igorvoltaic |
2021-03-28 21:21:23 | igorvoltaic | set | messageid: <1616966483.79.0.479664444023.issue43650@roundup.psfhosted.org> |
2021-03-28 21:21:23 | igorvoltaic | link | issue43650 messages |
2021-03-28 21:21:23 | igorvoltaic | create | |
|