classification
Title: epoll: reuse epoll_event buffer instead of allocating a new one at each poll()
Type: performance Stage: resolved
Components: Extension Modules Versions: Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: christian.heimes, neologix, python-dev, sbt
Priority: normal Keywords: needs review, patch

Created on 2013-01-06 03:30 by neologix, last changed 2013-01-12 11:32 by python-dev. This issue is now closed.

Files
File name Uploaded Description Edit
epoll_realloc.diff neologix, 2013-01-06 21:30 review
Messages (4)
msg179167 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2013-01-06 03:30
Currently, epoll.poll() allocates an epoll_event buffer every time/ this is bad, because it changes an O(number of ready FDs) syscall into a O(maxevents/FD_SETSIZE) complexity.
The patch attached allocates a epoll events buffer per epoll instance, and it only gets reallocated when necessary (note that the reallocation heuristic will probably be improved in #16873 (and having a per instance maxevents count will likely be useful).

Here's a benchmark without patch:
$ ./python -m timeit -s 'import select; ep = select.epoll(); ep.register(1, select.EPOLLOUT)' 'ep.poll()'
100000 loops, best of 3: 4.25 usec per loop

With patch:
$ ./python -m timeit -s 'import select; ep = select.epoll(); ep.register(1, select.EPOLLOUT)' 'ep.poll()'
100000 loops, best of 3: 3.38 usec per loop
msg179230 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2013-01-06 21:30
Updated to reflect Richard's comment.
msg179480 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013-01-09 18:00
New changeset be8e6b81284e by Charles-François Natali in branch 'default':
Issue #16876: Optimize epoll.poll() by keeping a per-instance epoll events
http://hg.python.org/cpython/rev/be8e6b81284e
msg179786 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013-01-12 11:32
New changeset 30eb98c8afef by Charles-François Natali in branch 'default':
Issue #16876: Revert be8e6b81284e, which wasn't thread-safe: wait until a
http://hg.python.org/cpython/rev/30eb98c8afef
History
Date User Action Args
2013-01-12 11:32:53python-devsetmessages: + msg179786
2013-01-09 20:21:28neologixsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2013-01-09 18:00:54python-devsetnosy: + python-dev
messages: + msg179480
2013-01-06 21:30:48neologixsetfiles: + epoll_realloc.diff

messages: + msg179230
2013-01-06 21:30:16neologixsetfiles: - epoll_realloc.diff
2013-01-06 21:06:44neologixsetfiles: - epoll_realloc.diff
2013-01-06 21:06:33neologixsetfiles: + epoll_realloc.diff
2013-01-06 20:54:40neologixsetnosy: + sbt
2013-01-06 10:51:02neologixsetfiles: + epoll_realloc.diff
2013-01-06 10:37:18neologixsetfiles: - epoll_realloc.diff
2013-01-06 10:27:06neologixsetfiles: + epoll_realloc.diff
2013-01-06 10:26:50neologixsetfiles: - epoll_realloc.diff
2013-01-06 03:45:30christian.heimessetnosy: + christian.heimes
2013-01-06 03:44:57neologixsetfiles: + epoll_realloc.diff
2013-01-06 03:44:39neologixsetfiles: - epoll_realloc.diff
2013-01-06 03:30:02neologixcreate