classification
Title: improve performance of BytesIO.writelines() by avoiding creation of unused PyLongs
Type: performance Stage: resolved
Components: IO Versions: Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: inada.naoki, sir-sigurd
Priority: normal Keywords: patch

Created on 2018-08-24 17:55 by sir-sigurd, last changed 2019-08-07 00:39 by inada.naoki. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 8904 merged sir-sigurd, 2018-08-24 17:59
Messages (4)
msg324011 - (view) Author: Sergey Fedoseev (sir-sigurd) * Date: 2018-08-24 17:55
Currently BytesIO.writelines() uses BytesIO.write() which returns number of written bytes as Python ints, but these ints are not used by BytesIO.writelines(), so avoiding creation of these ints can improve performance of BytesIO.writelines() especially for a large number of small lines.

Benchmarks:

python -m perf timeit --compare-to ~/tmp/cpython-master-venv/bin/python -s "from io import BytesIO; lines = [b'']*10000;" "BytesIO().writelines(lines)"                                          
/home/sergey/tmp/cpython-master-venv/bin/python: ..................... 153 us +- 3 us
/home/sergey/tmp/cpython-venv/bin/python: ..................... 126 us +- 4 us

Mean +- std dev: [/home/sergey/tmp/cpython-master-venv/bin/python] 153 us +- 3 us -> [/home/sergey/tmp/cpython-venv/bin/python] 126 us +- 4 us: 1.22x faster (-18%)

python -m perf timeit --compare-to ~/tmp/cpython-master-venv/bin/python -s "from io import BytesIO; lines = [b'b']*10000;" "BytesIO().writelines(lines)"
/home/sergey/tmp/cpython-master-venv/bin/python: ..................... 164 us +- 2 us
/home/sergey/tmp/cpython-venv/bin/python: ..................... 142 us +- 1 us

Mean +- std dev: [/home/sergey/tmp/cpython-master-venv/bin/python] 164 us +- 2 us -> [/home/sergey/tmp/cpython-venv/bin/python] 142 us +- 1 us: 1.16x faster (-13%)

python -m perf timeit --compare-to ~/tmp/cpython-master-venv/bin/python -s "from io import BytesIO; lines = [b'b'*10]*10000;" "BytesIO().writelines(lines)"
/home/sergey/tmp/cpython-master-venv/bin/python: ..................... 387 us +- 6 us
/home/sergey/tmp/cpython-venv/bin/python: ..................... 365 us +- 8 us

Mean +- std dev: [/home/sergey/tmp/cpython-master-venv/bin/python] 387 us +- 6 us -> [/home/sergey/tmp/cpython-venv/bin/python] 365 us +- 8 us: 1.06x faster (-5%)

python -m perf timeit --compare-to ~/tmp/cpython-master-venv/bin/python -s "from io import BytesIO; lines = [b'b'*100]*10000;" "BytesIO().writelines(lines)"                                                    
/home/sergey/tmp/cpython-master-venv/bin/python: ..................... 319 us +- 5 us
/home/sergey/tmp/cpython-venv/bin/python: ..................... 305 us +- 16 us

Mean +- std dev: [/home/sergey/tmp/cpython-master-venv/bin/python] 319 us +- 5 us -> [/home/sergey/tmp/cpython-venv/bin/python] 305 us +- 16 us: 1.04x faster (-4%)

python -m perf timeit --compare-to ~/tmp/cpython-master-venv/bin/python -s "from io import BytesIO; lines = [b'b'*1000]*10000;" "BytesIO().writelines(lines)"
/home/sergey/tmp/cpython-master-venv/bin/python: ..................... 1.13 ms +- 0.02 ms
/home/sergey/tmp/cpython-venv/bin/python: ..................... 988 us +- 88 us

Mean +- std dev: [/home/sergey/tmp/cpython-master-venv/bin/python] 1.13 ms +- 0.02 ms -> [/home/sergey/tmp/cpython-venv/bin/python] 988 us +- 88 us: 1.14x faster (-12%)

python -m perf timeit --compare-to ~/tmp/cpython-master-venv/bin/python -s "from io import BytesIO; lines = [b'b'*10000]*10000;" "BytesIO().writelines(lines)"
/home/sergey/tmp/cpython-master-venv/bin/python: ..................... 38.7 ms +- 0.1 ms
/home/sergey/tmp/cpython-venv/bin/python: ..................... 38.2 ms +- 0.1 ms

Mean +- std dev: [/home/sergey/tmp/cpython-master-venv/bin/python] 38.7 ms +- 0.1 ms -> [/home/sergey/tmp/cpython-venv/bin/python] 38.2 ms +- 0.1 ms: 1.01x faster (-1%)
msg339601 - (view) Author: Inada Naoki (inada.naoki) * (Python committer) Date: 2019-04-08 08:55
Hm, what happened if subclass of BytesIO overrides `write` but not `writelines`?
msg348903 - (view) Author: Sergey Fedoseev (sir-sigurd) * Date: 2019-08-02 14:31
`BytesIO.write()` and `BytesIO.writelines()` are independent of each other.
msg349142 - (view) Author: Inada Naoki (inada.naoki) * (Python committer) Date: 2019-08-07 00:38
New changeset 3e41f3cabb661824a1a197116f7f5ead64eb6ced by Inada Naoki (Sergey Fedoseev) in branch 'master':
bpo-34488: optimize BytesIO.writelines() (GH-8904)
https://github.com/python/cpython/commit/3e41f3cabb661824a1a197116f7f5ead64eb6ced
History
Date User Action Args
2019-08-07 00:39:51inada.naokisetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2019-08-07 00:38:34inada.naokisetmessages: + msg349142
2019-08-06 05:06:49inada.naokisetcomponents: + IO, - Extension Modules
versions: + Python 3.9, - Python 3.8
2019-08-02 14:31:15sir-sigurdsetmessages: + msg348903
2019-04-08 08:55:55inada.naokisetnosy: + inada.naoki
messages: + msg339601
2018-08-24 17:59:50sir-sigurdsetkeywords: + patch
stage: patch review
pull_requests: + pull_request8374
2018-08-24 17:55:50sir-sigurdcreate