New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
python -m gzip could use a larger buffer #87483
Comments
python -m gzip reads in chunks of 1024 bytes: Line 599 in 1f43340
This hurts performance somewhat. Using io.DEFAULT_BUFFER_SIZE will improve it. Also 'io.DEFAULT_BUFFER_SIZE' is better than: 'ARBITRARY_NUMBER_WITH_NO_COMMENT_EXPLAINING_WHY'. With 1024 blocks Compression: with io.DEFAULT_BUFFER_SIZE
Decompression:
$ hyperfine -r 10 -w 3 'cat ~/test/500000reads.fastq.gz | ./prefix/bin/python3 -m gzip -d > /dev/null'
Benchmark #1: cat ~/test/500000reads.fastq.gz | ./prefix/bin/python3 -m gzip -d > /dev/null
Time (mean ± σ): 839.9 ms ± 7.3 ms [User: 816.0 ms, System: 57.3 ms]
Range (min … max): 830.1 ms … 851.3 ms 10 runs Compression: Speedups:
It is not stellar, but it is a quite nice improvement for such a tiny change. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: