classification
Title: Support for read()/write()/select() on asyncio
Type: enhancement Stage: resolved
Components: asyncio Versions: Python 3.8
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: Paulo Costa, asvetlov, martin.panter, smheidrich, yselivanov
Priority: normal Keywords:

Created on 2016-02-02 22:20 by Paulo Costa, last changed 2019-05-27 20:22 by asvetlov. This issue is now closed.

Messages (8)
msg259438 - (view) Author: Paulo Costa (Paulo Costa) Date: 2016-02-02 22:20
I want to read from file descriptors from async coroutines.

I currently use `loop.add_reader(fd, callback)` and `loop.remove_reader(fd)`

It works, but IMO using callbacks feels completely alien in the middle of a coroutine.

I suggest adding new APIs to handle this. e.g.:
- asyncio.select(fd, mode)
- asyncio.read(fd, num)
- asyncio.write(fd, str)

(It would be nice to support all kinds of IO operations, but these are certainly the most important)

Using the current APIs, the async implementation of read() looks like this:

    async def async_read(fd. n):
        loop = asyncio.get_event_loop()
        future = asyncio.Future()

        def ready():
            future.set_result(os.read(fd, n))
            loop.remove_reader(fd)

        loop.add_reader(fd, ready)
        return future
msg259439 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-02-02 22:26
IMHO you misunderstood asyncio design. See examples.

Using add_reader(), lowest level API:
* https://docs.python.org/dev/library/asyncio-eventloop.html#watch-a-file-descriptor-for-read-events

Using a protocol:
* https://docs.python.org/dev/library/asyncio-protocol.html#tcp-echo-client-protocol
* https://docs.python.org/dev/library/asyncio-protocol.html#udp-echo-client-protocol

Using a stream:
* https://docs.python.org/dev/library/asyncio-stream.html#tcp-echo-client-using-streams
* https://docs.python.org/dev/library/asyncio-stream.html#get-http-headers

etc.
msg259456 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-02-03 04:52
Maybe I misunderstand the asyncio protocols, transports, streams, etc too :) but last time I looked they all seemed to buffer your write data and send it in the background. E.g. in the echo-client-using-streams example, there is a writer.write() call that does not use “yield from”.

Does asyncio have an easy way for a coroutine to write to a socket (or file descriptor) and block (suspend the coroutine) until the write has finished (or the data fits into a finite buffer)? E.g. how would you upload a multi-megabyte file?
msg259458 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-02-03 08:16
Paulo Costa:
> async def async_read(fd. n):

This method already exists:
* (TCP) https://docs.python.org/dev/library/asyncio-protocol.html#asyncio.Protocol.data_received
* (UDP) https://docs.python.org/dev/library/asyncio-protocol.html#asyncio.DatagramProtocol.datagram_received
* (TCP) https://docs.python.org/dev/library/asyncio-stream.html#asyncio.StreamReader.read


Martin Panter:
> last time I looked they all seemed to buffer your write data and send it in the background. E.g. in the echo-client-using-streams example, there is a writer.write() call that does not use “yield from”. Does asyncio have an easy way for a coroutine to write to a socket (or file descriptor) and block (suspend the coroutine) until the write has finished

Sure, you must call https://docs.python.org/dev/library/asyncio-stream.html#asyncio.StreamWriter.drain
msg261912 - (view) Author: Paulo Costa (Paulo Costa) Date: 2016-03-17 14:53
Thanks for the replies.

What I'm actually trying to do is to access from a character file from /dev/input/event*.

Streams are pretty much what I want to use but (as far as I can tell) asyncio's streams only support network and unix sockets. Files are left behind.

Anyway, despite the existence of streams and protocols, I still believe that `asyncio.read()`, `asyncio.write()` and `asyncio.select()` would be nice-to-have low-level APIs.
msg261913 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-03-17 14:57
> Anyway, despite the existence of streams and protocols, I still believe that `asyncio.read()`, `asyncio.write()` and `asyncio.select()` would be nice-to-have low-level APIs.

You can probably develop your own helper functions based on existing asyncio function.

I'm opposed to add such helper functions since it doesn't fit into the design of asyncio. Please read a little bit more about event loops and how asyncio is design. Don't hesitate to come back if you have something concrete, but IMHO this discussion should be moved to the python-tulip Google Groups rather than a *bug tracker*.

I close the issue.
msg261914 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2016-03-17 14:58
Regular are left behind because select() won't work on them. However we do support certain file descriptors (e.g. pipes) and it makes sense to support device "files". Can you come up with a new API that lets you pass a file descriptor and wrap it in a stream?
msg343682 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2019-05-27 20:12
The suggestion is using streams as a high-level API.
I see no real proposals in this issue.
select()/read()/write() is not suitable interface, sorry.

Closing, feel free to create a new issue if you come up with a concrete proposal.
History
Date User Action Args
2019-05-27 20:22:17asvetlovsetstatus: open -> closed
resolution: not a bug
stage: resolved
2019-05-27 20:12:22asvetlovsetmessages: + msg343682
2019-04-15 16:09:20vstinnersetnosy: - vstinner
2019-04-15 14:26:29cheryl.sabellasetnosy: + asvetlov

versions: + Python 3.8, - Python 3.6
2018-07-16 23:23:47gvanrossumsetnosy: - gvanrossum
2018-07-16 23:22:22smheidrichsetnosy: + smheidrich
2016-03-17 14:58:34gvanrossumsetstatus: closed -> open
resolution: not a bug -> (no value)
messages: + msg261914
2016-03-17 14:57:15vstinnersetstatus: open -> closed
resolution: not a bug
messages: + msg261913
2016-03-17 14:53:08Paulo Costasetmessages: + msg261912
2016-02-03 08:16:19vstinnersetmessages: + msg259458
2016-02-03 04:52:34martin.pantersetnosy: + martin.panter
messages: + msg259456
2016-02-02 22:26:35vstinnersetmessages: + msg259439
2016-02-02 22:20:48Paulo Costasetversions: + Python 3.6
2016-02-02 22:20:19Paulo Costacreate