classification
Title: File object hook to modify select(ors) event mask
Type: enhancement Stage: resolved
Components: asyncio, IO Versions:
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: asvetlov, gvanrossum, martin.panter, vstinner, yselivanov, zwol
Priority: normal Keywords:

Created on 2016-03-29 15:28 by zwol, last changed 2017-12-23 19:55 by asvetlov. This issue is now closed.

Messages (6)
msg262612 - (view) Author: Zack Weinberg (zwol) * Date: 2016-03-29 15:28
This is pretty esoteric, please bear with me.  I'm attempting to enhance a transparent-SOCKS module (https://github.com/Anorov/PySocks) to support non-blocking connect().  This means, you should be able to do this:

    socks.set_default_proxy(socks.SOCKS5, "localhost")
    s = socks.socksocket(socket.AF_INET, socket.SOCK_STREAM)
    s.setblocking(False)
    err = s.connect_ex(address)
    if err == errno.EINPROGRESS:
       select.select([], [s], []) # returns when connection completes

Note the position of s in the select() arguments: the documented behavior of non-blocking connect(), at the operating system level, is that the socket will become *writable* when the connection resolves (whether successfully or not).  However, in this case, under the hood, the socket is *already connected* -- to the proxy server -- after socksocket() returns.  When connect_ex() returns EINPROGRESS, the thing we're really waiting for is a SOCKS-protocol reply message, i.e. the socket needs to become *readable* before the application can continue.  An application that used the above code (with a hypothetical version of PySocks where this was supported) would get woken up immediately, since the initial SOCKS client->server message doesn't even come close to filling up the TCP send buffer.

There's no practical way to hide this divergence with the current library.  What would be needed, I think, is a pair of new special methods on filelikes, which rewrite the set of events to listen for and the set of events to report.  Hypothetical, for my use case:

    def __preselect__(self, events):
        if not self._connecting: return events
        return selectors.EVENT_READ

    def __postselect__(self, events):
        if not self._connecting: return events
        return selectors.EVENT_WRITE

(I'm using the high-level selectors.EVENT_* constants for illustration.  This needs to happen in the low-level select-module methods, because callers can't be expected to use selectors.)

There are a bunch of awkward corner cases to worry about with this, and I'm not even sure the feature is worth it, but I thought I'd write up the problem and see what other people think.
msg262629 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-03-29 21:36
> I'm attempting to enhance a transparent-SOCKS module (https://github.com/Anorov/PySocks) to support non-blocking connect().

Obvious question: why not working on the asyncio support in this library?
msg262633 - (view) Author: Zack Weinberg (zwol) * Date: 2016-03-29 22:10
> Obvious question: why not working on the asyncio support in this library?

The whole point of a transparent SOCKS module is that it provides a function that's a *drop-in replacement* for socket.socket().  An asyncio-based SOCKS module would have a completely different API.
msg262635 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-03-29 23:09
Python’s select() call operates at a minimal OS level. I don’t completely understand your __preselect__() etc proposal, but it does sound like you are trying to invent another higher-level “asynchronous” framework like asyncio.

I think you are wrong about the proxy connection being made in the socksocket() constructor. In general, there could be both write and read events depending on the stage of the proxy connection setup. E.g. with a HTTP proxy, there would be write events to establish the underlying TCP connection, and also to send the CONNECT request, and then a read event to receive the proxy’s response.

Perhaps the SSL socket wrapper objects face a similar situation to your problem. In non-blocking mode, SSLWantRead/WriteError exceptions are raised that indicate which select() event is required.
msg262643 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-03-30 01:16
> The whole point of a transparent SOCKS module is that it provides a function that's a *drop-in replacement* for socket.socket().

Fully transparent asynchronous I/O is an old dream of developers. You may try eventlet if you like monkey-patching.

It's a deliberate choice in the asyncio design to make asynchonous I/O explicit in the code.
msg308965 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2017-12-23 19:55
Out of asynio library scope.
History
Date User Action Args
2017-12-23 19:55:53asvetlovsetstatus: open -> closed

nosy: + asvetlov
messages: + msg308965

resolution: wont fix
stage: resolved
2016-03-30 01:16:09vstinnersetmessages: + msg262643
2016-03-29 23:09:23martin.pantersetnosy: + martin.panter
messages: + msg262635
2016-03-29 22:10:35zwolsetmessages: + msg262633
2016-03-29 21:36:21vstinnersetnosy: + gvanrossum, vstinner, yselivanov
messages: + msg262629
components: + asyncio
2016-03-29 15:28:41zwolcreate