This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Asyncio BaseEventLoop can support socket types other than SOCK_STREAM
Type: enhancement Stage:
Components: asyncio Versions: Python 3.10
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: asvetlov, callumquick, jbeeler, malversan, yselivanov
Priority: normal Keywords:

Created on 2019-09-26 12:05 by malversan, last changed 2022-04-11 14:59 by admin.

Messages (20)
msg353296 - (view) Author: Malversán (malversan) Date: 2019-09-26 12:05
Currently the BaseEventLoop class in asyncio has explicit checks to raise ValueError when creating a connection if the socket argument has a type other than SOCK_STREAM:
.create_connection()
.create_server()

This is also applicable for class _UnixSelectorEventLoop:
.create_unix_connection()
.create_unix_server()

But the fact is that it actually supports other socket types, like SOCK_SEQPACKET for example.

Currently you can test this by dirty-hacking the socket class "type" property to momentarily trick the event loop into thinking that any socket is of SOCK_STREAM type.

<code>
# First create an AF_UNIX, SOCK_SEQPACKET socket.
sock = socket.socket(socket.AddressFamily.AF_UNIX, socket.SocketKind.SOCK_SEQ_PACKET)
sock.connect(path)

params = { "sock" : sock, "protocol_factory" : lambda: protocol }

# Now do the trick.
hack = (params["sock"].type != socket.SocketKind.SOCK_STREAM)

if hack:
    # Substitute class property getter with fixed value getter.
    socket_property = socket.socket.type
    socket.socket.type = property(lambda self: socket.SocketKind.SOCK_STREAM, None, None,)

# Use the socket normally to create connection and run the event loop.
loop = asyncio.new_event_loop()
coroutine = loop.create_unix_connection(**params)    # It also works with .create_connection()
transport, protocol = loop.run_until_complete(coroutine)

# Revert the trick.
if hack:
    # Restore class property getter.
    socket.socket.type = socket_property
</code>

As dirty as it looks, this works flawlessy. It just tricks the event loop .create_connection() call to bypass the explicit check of using a SOCK_STREAM socket. This done, THE EVENT LOOP SUPPORTS SOCK_SEQPACKET PERFECTLY.

This is the solution I'm currently using to communicate an application with a local daemon, but I would really prefer to have the SOCK_SEQPACKET support allowed into the event loop itself. Having in mind that it simply works with other socket types, I find that limiting the use of the event loop with an explicit SOCK_STREAM-only check is somehow artificial and unrealistic.

Thanks in advance for your attention.
msg353297 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2019-09-26 12:15
I think if we want to support SOCK_SEQPACKET by asyncio we should do it explicitly.

In the other case, we can open a can of worms: we cannot guarantee that all existing protocols are supported by asyncio seamlessly.

Anyway, this is a new feature request. It can land on Python 3.9 only.
msg353298 - (view) Author: Malversán (malversan) Date: 2019-09-26 12:38
Certainly I have only tested it with SOCK_SEQPACKET, but apparently no one has ever tested this before with a socket type other than SOCK_STREAM. It may be worth to consider the possibility that the current asyncio implementation may also support some other SocketKind sockets:
- SOCK_SEQPACKET (tested)
- SOCK_DGRAM
- SOCK_RAW
- SOCK_RDM

I agree this is an enhancement to incorporate in future releases. I do not expect previous versions to be patched.
msg353300 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2019-09-26 12:56
We can implement these using the following procedure:

1. Use only one socket type per pull request
2. Add support for, e.g. SOCK_SEQPACKET to asyncio code
3. Add test(s) that checks that SOCK_SEQPACKET works fine (./Lib/test/test_asyncio folder, perhaps test_events.py)
4. Merge the PR, repeat bullet 1) for the next socket type.

Would you work on pull requests?
msg353316 - (view) Author: Malversán (malversan) Date: 2019-09-26 15:06
In the past it took me two days to analyze asyncio code, to think up and integrate the hack I´m using for this. But I´m not kidding when I tell you that it took me two years to find a while to come here and properly report it. I'm sorry, but I never have time to dedicate to other projects (I wish I could).
msg353320 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2019-09-26 15:20
If you have no time for contribution -- that's fine, CPython is the Open Source project driven by volunteers.

The only caveat is that the issue may wait for years before we find a champion to pick it up.

For example, this particular problem is on the very end of my personal todo list because I don't use SOCK_SEQPACKET on my job (yet).
msg353322 - (view) Author: Malversán (malversan) Date: 2019-09-26 15:46
I'm sorry to read that. I thought the report could be enough to reach whoever put that SOCK_STREAM-only checks and ask him why, when the library actually works well also with other socket types.

If I ever find enough time to dive into the CPython repository I will come back here, but given my work load I would not count on it. Anyway, as long as the issue remais opened I'm confident this will be eventually fixed.
msg355945 - (view) Author: Callum Ward (callumquick) * Date: 2019-11-04 14:45
Hi Andrew, I'm a new contributor, but this sounds like a pretty cool enhancement. 

Would you be able to elaborate on what kind of things might be required to support each new socket type and test them in particular so I can see if I can take it on?
msg355948 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2019-11-04 15:07
For each type, we need at least a test that creates a socket pair and successfully transfers data through the wire.

I don't know what additional things are required. For example, on reading about SOCK_SEQ_PACKET I've found that recvmsg() is highly recommended over recv() to get messages boundaries. It obviously requires the new transport type and the new async protocol specification.

Doesn't look trivial.
msg355949 - (view) Author: Callum Ward (callumquick) * Date: 2019-11-04 15:12
A matter of creating tests to allow test enabling of new socket types I could attempt, but new protocol/transport types may be beyond me.
msg355954 - (view) Author: Malversán (malversan) Date: 2019-11-04 15:37
It has a certain logic to recommend recvmsg() in place of recv(), as SOCK_SEQ_PACKET is characterized by transmitting entire messages only. But it has to be noted that my current hack (described above) is working for SOCK_SEQ_PACKET sockets with no modification of the asyncio underlying reading logic.
msg355958 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2019-11-04 16:24
How to get the message boundary without recvmsg()? Sorry, I'm not familiar with seqpacket.
msg355960 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2019-11-04 16:25
Another question: does SSL/TLS make sense for seqpacket?
msg355962 - (view) Author: Malversán (malversan) Date: 2019-11-04 16:41
I do not have the answer about getting message boundaries at lower levels, but from a high-level point of view SOCK_SEQ_PACKET gives atomic reads, with no need to check for message boundaries yourself. Every time you read from a SOCK_SEQ_PACKET socket you get an entire message. That is the main difference with SOCK_STREAM, as far as I know.
msg355969 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2019-11-04 18:35
Can recv() get two messages at once?
What is the behavior if the buffer size passed into recv() is smaller than the message length?
msg355979 - (view) Author: Malversán (malversan) Date: 2019-11-04 19:42
In my scenario that buffer overrun never happens, maybe because I use messages that are not big enough to overflow the default recv() buffer size.

But I think I can confirm that multiple messages are never received in an atomic read, even if they are being issued intensively in short millisecond intervals. Even more, I think there is a recvmmsg() call specific for that purpose if you want to receive multiple reads at once.

As I said I do not have the answers, I rely on the high-level definitions and have little knowledge about how it works at low level.

But I think your question may be extended also to recvmsg(). What is its behaviour if it fills all the passed iovec structs?

Probably an answer can be found where you found the recommendation of using recvmsg() over recv(). There should be a reason for that recommendation.
msg356023 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2019-11-05 11:05
My point is: without a deep understanding we cannot "just enable" a new protocol.
The evidence that it works in some limited scenarios is not enough for opening the can of worms.
It is true for seqpacket, and especially true for other even not discussed protocols in general.
msg356035 - (view) Author: Malversán (malversan) Date: 2019-11-05 14:25
I agree. Your question about potential message size overflow should be tested (either for recv() and recvmsg()).

Could you please link the resource where you found the recommendation of using recvmsg() over recv() for SOCK_SEQ_PACKET?
msg384422 - (view) Author: John Beeler (jbeeler) Date: 2021-01-05 18:12
For what it's worth, a library I use currently hacks in this functionality by accessing the private method _create_connection_transport directly. This is done to allow a SOCK_RAW to be passed (which is itself required to then enable asyncio to be used for handling BLE).

This works, but as it requires the use of a private method, it's not ideal.
msg384427 - (view) Author: Malversán (malversan) Date: 2021-01-05 19:30
That means the core code also works for SOCK_RAW sockets. It's only limited by explicit socket type checks at a higher level.

As a curious note (not related to the issue), I'm also using the SOCK_SEQPACKET connection created with BaseEventLoop to access a custom daemon related to BLE functionality.
History
Date User Action Args
2022-04-11 14:59:20adminsetgithub: 82466
2021-01-05 19:30:16malversansetmessages: + msg384427
versions: + Python 3.10, - Python 3.9
2021-01-05 18:12:00jbeelersetnosy: + jbeeler
messages: + msg384422
2019-11-05 14:25:35malversansetmessages: + msg356035
2019-11-05 11:05:11asvetlovsetmessages: + msg356023
2019-11-04 19:42:47malversansetmessages: + msg355979
2019-11-04 18:35:36asvetlovsetmessages: + msg355969
2019-11-04 16:41:35malversansetmessages: + msg355962
2019-11-04 16:25:15asvetlovsetmessages: + msg355960
2019-11-04 16:24:12asvetlovsetmessages: + msg355958
2019-11-04 15:37:33malversansetmessages: + msg355954
2019-11-04 15:12:24callumquicksetmessages: + msg355949
2019-11-04 15:07:58asvetlovsetmessages: + msg355948
2019-11-04 14:45:14callumquicksetnosy: + callumquick
messages: + msg355945
2019-09-26 15:46:41malversansetmessages: + msg353322
2019-09-26 15:20:23asvetlovsetmessages: + msg353320
2019-09-26 15:06:50malversansetmessages: + msg353316
2019-09-26 12:56:40asvetlovsetmessages: + msg353300
2019-09-26 12:38:47malversansetmessages: + msg353298
2019-09-26 12:15:15asvetlovsetmessages: + msg353297
versions: - Python 3.5, Python 3.6, Python 3.7, Python 3.8
2019-09-26 12:05:11malversancreate