This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author andzn
Recipients David Hirschfeld, andzn, desbma, paul.moore, pitrou, steve.dower, tim.golden, vstinner, zach.ware
Date 2019-06-05.12:42:00
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1559738520.78.0.211268208271.issue28708@roundup.psfhosted.org>
In-reply-to
Content
We would also need this limit to be raised. We are using a server that was implemented using tornado, and tornado uses select.select. 

> It looks like Modules/selectmodule.c already has the handling for when this gets large (and we really want to stop allocating the static array on the stack), which is a plus, but I'd want to see the performance cost to small selects if suddenly we're allocating 100s of KB on the heap rather than small and cheap stack allocations.

I gave this a try. The code I used for the server and the client for taken from this page: https://steelkiwi.com/blog/working-tcp-sockets/. I just added some code to measure the time it stays in the "select" call. To measure the time I used time.process_time(). I then connected/disconnected only 1 client 10.000 times in a row, discarding the measurements of the first 1000 iterations (warm-up).

Here are the results:
FD_SETSIZE = 16384 bytes
Time elapsed (avg ns):  20029.566224207087
Time elapsed (max ns):  15625000
Time elapsed (min ns):  0

FD_SETSIZE = 512 bytes
Time elapsed (avg ns):  14671.361502347418
Time elapsed (max ns):  15625000
Time elapsed (min ns):  0

> However, I see no reason why this couldn't be totally dynamic, at least on Windows. Redefining that value just changes a static array definition, but the array size never gets passed in to the OS AFAICT, so there's no reason we couldn't stack allocate for small select()s and heap allocate for big select()s. That's considerably more work than anyone had in mind, I'd wager.
I'm not sure this can be totally dynamic on Windows. The problem is the fd_set struct which is defined in Winsock2.h

typedef struct fd_set {
        u_int fd_count;               /* how many are SET? */
        SOCKET  fd_array[FD_SETSIZE];   /* an array of SOCKETs */
} fd_set;

The heap-allocated pylists are converted into fd_set structs which are always allocated on the stack btw. The problem is there's no way to create a fd_set of a desired FD_SETSIZE at runtime.

I have also submitted a PR with the change for convenience: https://github.com/python/cpython/pull/13842.
History
Date User Action Args
2019-06-05 12:42:00andznsetrecipients: + andzn, paul.moore, pitrou, vstinner, tim.golden, zach.ware, desbma, steve.dower, David Hirschfeld
2019-06-05 12:42:00andznsetmessageid: <1559738520.78.0.211268208271.issue28708@roundup.psfhosted.org>
2019-06-05 12:42:00andznlinkissue28708 messages
2019-06-05 12:42:00andzncreate