Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

improve performance of BytesIO.writelines() by avoiding creation of unused PyLongs #78669

Closed
sir-sigurd mannequin opened this issue Aug 24, 2018 · 4 comments
Closed

improve performance of BytesIO.writelines() by avoiding creation of unused PyLongs #78669

sir-sigurd mannequin opened this issue Aug 24, 2018 · 4 comments
Labels
3.9 only security fixes performance Performance or resource usage topic-IO

Comments

@sir-sigurd
Copy link
Mannequin

sir-sigurd mannequin commented Aug 24, 2018

BPO 34488
Nosy @methane, @sir-sigurd
PRs
  • bpo-34488: Improve performance of BytesIO.writelines() by avoiding creation of unused ints. #8904
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2019-08-07.00:39:51.215>
created_at = <Date 2018-08-24.17:55:50.708>
labels = ['3.9', 'expert-IO', 'performance']
title = 'improve performance of BytesIO.writelines() by avoiding creation of unused PyLongs'
updated_at = <Date 2019-08-07.00:39:51.214>
user = 'https://github.com/sir-sigurd'

    bugs.python.org fields:

    activity = <Date 2019-08-07.00:39:51.214>
actor = 'methane'
assignee = 'none'
closed = True
closed_date = <Date 2019-08-07.00:39:51.215>
closer = 'methane'
components = ['IO']
creation = <Date 2018-08-24.17:55:50.708>
creator = 'sir-sigurd'
dependencies = []
files = []
hgrepos = []
issue_num = 34488
keywords = ['patch']
message_count = 4.0
messages = ['324011', '339601', '348903', '349142']
nosy_count = 2.0
nosy_names = ['methane', 'sir-sigurd']
pr_nums = ['8904']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue34488'
versions = ['Python 3.9']
    @sir-sigurd
    Copy link
    Mannequin Author

    sir-sigurd mannequin commented Aug 24, 2018

    Currently BytesIO.writelines() uses BytesIO.write() which returns number of written bytes as Python ints, but these ints are not used by BytesIO.writelines(), so avoiding creation of these ints can improve performance of BytesIO.writelines() especially for a large number of small lines.

    Benchmarks:

    python -m perf timeit --compare-to ~/tmp/cpython-master-venv/bin/python -s "from io import BytesIO; lines = [b'']*10000;" "BytesIO().writelines(lines)"
    /home/sergey/tmp/cpython-master-venv/bin/python: ..................... 153 us +- 3 us
    /home/sergey/tmp/cpython-venv/bin/python: ..................... 126 us +- 4 us

    Mean +- std dev: [/home/sergey/tmp/cpython-master-venv/bin/python] 153 us +- 3 us -> [/home/sergey/tmp/cpython-venv/bin/python] 126 us +- 4 us: 1.22x faster (-18%)

    python -m perf timeit --compare-to ~/tmp/cpython-master-venv/bin/python -s "from io import BytesIO; lines = [b'b']*10000;" "BytesIO().writelines(lines)"
    /home/sergey/tmp/cpython-master-venv/bin/python: ..................... 164 us +- 2 us
    /home/sergey/tmp/cpython-venv/bin/python: ..................... 142 us +- 1 us

    Mean +- std dev: [/home/sergey/tmp/cpython-master-venv/bin/python] 164 us +- 2 us -> [/home/sergey/tmp/cpython-venv/bin/python] 142 us +- 1 us: 1.16x faster (-13%)

    python -m perf timeit --compare-to ~/tmp/cpython-master-venv/bin/python -s "from io import BytesIO; lines = [b'b'*10]*10000;" "BytesIO().writelines(lines)"
    /home/sergey/tmp/cpython-master-venv/bin/python: ..................... 387 us +- 6 us
    /home/sergey/tmp/cpython-venv/bin/python: ..................... 365 us +- 8 us

    Mean +- std dev: [/home/sergey/tmp/cpython-master-venv/bin/python] 387 us +- 6 us -> [/home/sergey/tmp/cpython-venv/bin/python] 365 us +- 8 us: 1.06x faster (-5%)

    python -m perf timeit --compare-to ~/tmp/cpython-master-venv/bin/python -s "from io import BytesIO; lines = [b'b'*100]*10000;" "BytesIO().writelines(lines)"
    /home/sergey/tmp/cpython-master-venv/bin/python: ..................... 319 us +- 5 us
    /home/sergey/tmp/cpython-venv/bin/python: ..................... 305 us +- 16 us

    Mean +- std dev: [/home/sergey/tmp/cpython-master-venv/bin/python] 319 us +- 5 us -> [/home/sergey/tmp/cpython-venv/bin/python] 305 us +- 16 us: 1.04x faster (-4%)

    python -m perf timeit --compare-to ~/tmp/cpython-master-venv/bin/python -s "from io import BytesIO; lines = [b'b'*1000]*10000;" "BytesIO().writelines(lines)"
    /home/sergey/tmp/cpython-master-venv/bin/python: ..................... 1.13 ms +- 0.02 ms
    /home/sergey/tmp/cpython-venv/bin/python: ..................... 988 us +- 88 us

    Mean +- std dev: [/home/sergey/tmp/cpython-master-venv/bin/python] 1.13 ms +- 0.02 ms -> [/home/sergey/tmp/cpython-venv/bin/python] 988 us +- 88 us: 1.14x faster (-12%)

    python -m perf timeit --compare-to ~/tmp/cpython-master-venv/bin/python -s "from io import BytesIO; lines = [b'b'*10000]*10000;" "BytesIO().writelines(lines)"
    /home/sergey/tmp/cpython-master-venv/bin/python: ..................... 38.7 ms +- 0.1 ms
    /home/sergey/tmp/cpython-venv/bin/python: ..................... 38.2 ms +- 0.1 ms

    Mean +- std dev: [/home/sergey/tmp/cpython-master-venv/bin/python] 38.7 ms +- 0.1 ms -> [/home/sergey/tmp/cpython-venv/bin/python] 38.2 ms +- 0.1 ms: 1.01x faster (-1%)

    @sir-sigurd sir-sigurd mannequin added extension-modules C modules in the Modules dir 3.8 only security fixes performance Performance or resource usage labels Aug 24, 2018
    @methane
    Copy link
    Member

    methane commented Apr 8, 2019

    Hm, what happened if subclass of BytesIO overrides write but not writelines?

    @sir-sigurd
    Copy link
    Mannequin Author

    sir-sigurd mannequin commented Aug 2, 2019

    BytesIO.write() and BytesIO.writelines() are independent of each other.

    @methane methane added topic-IO 3.9 only security fixes and removed extension-modules C modules in the Modules dir 3.8 only security fixes labels Aug 6, 2019
    @methane
    Copy link
    Member

    methane commented Aug 7, 2019

    New changeset 3e41f3c by Inada Naoki (Sergey Fedoseev) in branch 'master':
    bpo-34488: optimize BytesIO.writelines() (GH-8904)
    3e41f3c

    @methane methane closed this as completed Aug 7, 2019
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.9 only security fixes performance Performance or resource usage topic-IO
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    1 participant
    @methane