Author igorvoltaic
Recipients igorvoltaic
Date 2021-03-28.21:21:23
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1616966483.79.0.479664444023.issue43650@roundup.psfhosted.org>
In-reply-to
Content
MemoryError: null
  ...
  File "....", line 13, in repack__file
    shutil.unpack_archive(local_file_path, local_dir)
  File "python3.6/shutil.py", line 983, in unpack_archive
    func(filename, extract_dir, **kwargs)
  File "python3.6/shutil.py", line 901, in _unpack_zipfile
    data = zip.read(info.filename)
  File "python3.6/zipfile.py", line 1338, in read
    return fp.read()
  File "python3.6/zipfile.py", line 858, in read
    buf += self._read1(self.MAX_N)
  File "python3.6/zipfile.py", line 948, in _read1
    data = self._decompressor.decompress(data, n)

shutil.unpack_archive tries to read the whole file into memory, making use of any buffer at all. Python crashes for really large files. In my case — archive: ~1.7G, unpacked: ~10G. Interestingly zipfile.ZipFile.extractall handles this case more effective.
History
Date User Action Args
2021-03-28 21:21:23igorvoltaicsetrecipients: + igorvoltaic
2021-03-28 21:21:23igorvoltaicsetmessageid: <1616966483.79.0.479664444023.issue43650@roundup.psfhosted.org>
2021-03-28 21:21:23igorvoltaiclinkissue43650 messages
2021-03-28 21:21:23igorvoltaiccreate