This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: BaseSelectorEventLoop.sock_sendall() performance regression: extra copy of data
Type: resource usage Stage: resolved
Components: asyncio Versions: Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: asvetlov Nosy List: Huazuo Gao, asvetlov, miss-islington, yselivanov
Priority: normal Keywords: patch, patch, patch

Created on 2018-12-26 10:04 by Huazuo Gao, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 11418 merged asvetlov, 2019-01-02 21:29
PR 11418 merged asvetlov, 2019-01-02 21:30
PR 11418 merged asvetlov, 2019-01-02 21:30
Messages (3)
msg332534 - (view) Author: Huazuo Gao (Huazuo Gao) Date: 2018-12-26 10:04
Prior to PR 10419, sock_sendall does not make a copy of the data. PR 10419 introduced an extra copy, which may cause problem for code that send a huge chunk of data simultaneously to many peers. Relevant change is:

https://github.com/python/cpython/pull/10419/files#diff-2d64b02252335b37396e00e56fa66984R443

Bellow is a test that show the regression between 3.7.1 and 3.8-dev

---

import asyncio
import socket
import os
from subprocess import check_output

loop = asyncio.get_event_loop()

def mem_usage():
    pid = str(os.getpid())
    print(check_output(['ps', '-o', 'rss,comm'], text=True))

async def main():
    data = bytearray(10*10**6)
    data = memoryview(data)
    tasks = []
    for i in range(100):
        s1, s2 = socket.socketpair()
        s1.setblocking(False)
        s2.setblocking(False)
        tasks.append(loop.create_task(loop.sock_sendall(s1, data)))
        tasks.append(loop.create_task(loop.sock_recv(s2, 1)))
    await asyncio.sleep(0.1)
    mem_usage()
    for t in tasks:
        t.cancel()
    await asyncio.wait(tasks)

loop.run_until_complete(main())

---

result

3.7.1: 24724
3.8-dev: 979184
msg332535 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2018-12-26 10:35
Thanks for the report!
msg342643 - (view) Author: miss-islington (miss-islington) Date: 2019-05-16 13:30
New changeset 6e7890028213b30939327e7cf885bf097fc14472 by Miss Islington (bot) (Andrew Svetlov) in branch 'master':
bpo-35589: Prevent buffer copy in sock_sendall() (GH-11418)
https://github.com/python/cpython/commit/6e7890028213b30939327e7cf885bf097fc14472
History
Date User Action Args
2022-04-11 14:59:09adminsetgithub: 79770
2019-05-16 13:32:34asvetlovsetkeywords: patch, patch, patch
status: open -> closed
resolution: fixed
stage: patch review -> resolved
2019-05-16 13:30:22miss-islingtonsetnosy: + miss-islington
messages: + msg342643
2019-01-02 21:30:12asvetlovsetkeywords: + patch
stage: patch review
pull_requests: + pull_request10825
2019-01-02 21:30:05asvetlovsetkeywords: + patch
stage: (no value)
pull_requests: + pull_request10824
2019-01-02 21:29:58asvetlovsetkeywords: + patch
stage: (no value)
pull_requests: + pull_request10823
2018-12-26 10:35:51asvetlovsetassignee: asvetlov
messages: + msg332535
2018-12-26 10:04:33Huazuo Gaocreate