classification
Title: Use select.poll instead of select.select in SocketServer.BaseServer.serve_forever
Type: performance Stage:
Components: Library (Lib) Versions:
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: giampaolo.rodola, vstinner, Беатрис Бонева
Priority: normal Keywords:

Created on 2017-09-27 11:24 by Беатрис Бонева, last changed 2017-09-28 15:05 by vstinner.

Messages (4)
msg303127 - (view) Author: Беатрис Бонева (Беатрис Бонева) Date: 2017-09-27 11:24
The select function does not work for file descriptors with number >= 1024. It has FD_SETSIZE limit (see https://stackoverflow.com/questions/7976388/increasing-limit-of-fd-setsize-and-select#7977082).
So, serve_forever won't work if more than 1024 file descriptors are opened in the process (ValueError: filedescriptor out of range in select() is raised).
Moreover, the select function is considered inefficient, hard to scale and old-fashioned nowadays. Refer to https://stackoverflow.com/questions/970979/what-are-the-differences-between-poll-and-select#3951845.
Better alternative will be to use poll or epoll even.
msg303247 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-09-28 14:38
> Better alternative will be to use poll or epoll even.

epoll requires mulitple system calls, whereas the selector seems to only be used only once in most cases, no?

Anyway, maybe selectors.DefaultSelector can be used here?

About poll(): be aware of macOS issues with poll() :-/
msg303251 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2017-09-28 14:59
I recommend to use selectors.PollSelector and fallback on selectors.SelectSelector where not available (Windows) in order to use 1 syscall only. That's what we did already in socket.py, subprocess.py and others.
msg303254 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-09-28 15:05
> I recommend to use selectors.PollSelector and fallback on selectors.SelectSelector where not available (Windows) in order to use 1 syscall only. That's what we did already in socket.py, subprocess.py and others.

It would be nice to have an API in selectors to get a "lightweight" selector, for selector and poll.

This API would be appropriate to check on macOS if poll() works or not: bpo-28087.
History
Date User Action Args
2017-09-28 15:05:48vstinnersetmessages: + msg303254
2017-09-28 14:59:52giampaolo.rodolasetnosy: + giampaolo.rodola
messages: + msg303251
2017-09-28 14:38:27vstinnersetnosy: + vstinner
messages: + msg303247
2017-09-27 11:24:44Беатрис Боневаcreate