classification
Title: TarFile expose copyfileobj bufsize to improve throughput
Type: performance Stage: patch review
Components: Library (Lib) Versions: Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: lukasz.langa Nosy List: asvetlov, fried, lars.gustaebel, lukasz.langa, python-dev
Priority: normal Keywords: patch

Created on 2016-06-03 18:55 by fried, last changed 2016-09-10 02:51 by lukasz.langa. This issue is now closed.

Files
File name Uploaded Description Edit
buftest.py fried, 2016-06-03 18:55 test file to generate two random tar files and test extraction time improvements
copybufsize.patch fried, 2016-06-03 18:56 patch to expose the copy buffer size review
Messages (4)
msg267134 - (view) Author: Jason Fried (fried) * Date: 2016-06-03 18:55
The default of 16k while good for memory usage it is not well suited for all cases. if we increased this to 4MB we saw a pretty large improvement to tar file creation and extraction on linux servers.

For a 1gb tar file containing 1024 random files each of 10MB in size.
Time Delta for TarFile: 146.3240258693695
Time Delta for FastTarFile 4MB copybufsize: 102.76440262794495
Time Diff: 43.55962324142456 0.2976928975444698
msg268234 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2016-06-11 17:27
New feature -> 3.6.
msg275546 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-09-10 02:50
New changeset 0bac85e355b5 by Łukasz Langa in branch 'default':
Issue #27199: TarFile expose copyfileobj bufsize to improve throughput
https://hg.python.org/cpython/rev/0bac85e355b5
msg275547 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2016-09-10 02:51
Thanks for the patch!
History
Date User Action Args
2016-09-10 02:51:11lukasz.langasetstatus: open -> closed
resolution: fixed
messages: + msg275547
2016-09-10 02:50:52python-devsetnosy: + python-dev
messages: + msg275546
2016-06-20 17:45:06lukasz.langasetnosy: + lars.gustaebel
2016-06-11 17:27:35lukasz.langasetassignee: lukasz.langa
stage: patch review
messages: + msg268234
versions: + Python 3.6, - Python 3.5
2016-06-03 18:56:02friedsetfiles: + copybufsize.patch
keywords: + patch
2016-06-03 18:55:37friedcreate