classification
Title: BaseSelectorEventLoop.sock_sendall() performance regression: extra copy of data
Type: resource usage Stage: patch review
Components: asyncio Versions: Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: asvetlov Nosy List: Huazuo Gao, asvetlov, yselivanov
Priority: normal Keywords: patch, patch, patch

Created on 2018-12-26 10:04 by Huazuo Gao, last changed 2019-01-02 21:30 by asvetlov.

Pull Requests
URL Status Linked Edit
PR 11418 open asvetlov, 2019-01-02 21:29
PR 11418 open asvetlov, 2019-01-02 21:30
PR 11418 open asvetlov, 2019-01-02 21:30
Messages (2)
msg332534 - (view) Author: Huazuo Gao (Huazuo Gao) Date: 2018-12-26 10:04
Prior to PR 10419, sock_sendall does not make a copy of the data. PR 10419 introduced an extra copy, which may cause problem for code that send a huge chunk of data simultaneously to many peers. Relevant change is:

https://github.com/python/cpython/pull/10419/files#diff-2d64b02252335b37396e00e56fa66984R443

Bellow is a test that show the regression between 3.7.1 and 3.8-dev

---

import asyncio
import socket
import os
from subprocess import check_output

loop = asyncio.get_event_loop()

def mem_usage():
    pid = str(os.getpid())
    print(check_output(['ps', '-o', 'rss,comm'], text=True))

async def main():
    data = bytearray(10*10**6)
    data = memoryview(data)
    tasks = []
    for i in range(100):
        s1, s2 = socket.socketpair()
        s1.setblocking(False)
        s2.setblocking(False)
        tasks.append(loop.create_task(loop.sock_sendall(s1, data)))
        tasks.append(loop.create_task(loop.sock_recv(s2, 1)))
    await asyncio.sleep(0.1)
    mem_usage()
    for t in tasks:
        t.cancel()
    await asyncio.wait(tasks)

loop.run_until_complete(main())

---

result

3.7.1: 24724
3.8-dev: 979184
msg332535 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2018-12-26 10:35
Thanks for the report!
History
Date User Action Args
2019-01-02 21:30:12asvetlovsetkeywords: + patch
stage: patch review
pull_requests: + pull_request10825
2019-01-02 21:30:05asvetlovsetkeywords: + patch
stage: (no value)
pull_requests: + pull_request10824
2019-01-02 21:29:58asvetlovsetkeywords: + patch
stage: (no value)
pull_requests: + pull_request10823
2018-12-26 10:35:51asvetlovsetassignee: asvetlov
messages: + msg332535
2018-12-26 10:04:33Huazuo Gaocreate