Author vstinner
Recipients YoSTEALTH, benjamin.peterson, giampaolo.rodola, martin.panter, njs, pitrou, stutzbach, vstinner, xgdomingo
Date 2019-10-11.12:42:55
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1570797775.61.0.0230707707315.issue32561@roundup.psfhosted.org>
In-reply-to
Content
Here a proof-of-concept of an asynchronous io module reusing the existing blocking io module: it implements AsyncBufferedReader.readline() using existing _pyio.BufferedReader.readline().

The approach seems to work, but only if bpo-13322 is fixed first: BufferedReader, TextIOWrapper & friends must return None if the underlying object ("raw" and "buffer" objects) return None.

--

My PoC uses 3 classes:

* AsyncFileIO: similar to io.FileIO but uses "async def"
* AsyncBufferedReader: similar to io.BufferedReader but uses "async def"
* FileIOSandwich: glue between asynchronous AsyncFileIO and blocking io.BufferedReader

At the first read, FileIOSandwich.read(n) call returns None, but it stores the request read size (n).

If AsyncBufferedReader gets None, is calls FileIOSandwich._prepare_read() *asynchronously*/

Then FileIOSandwich.read(n) is called again, and this time it no longer blocks, since data has been already read.

--

Since bpo-13322 is not fixed, my PoC uses _pyio since it's easier to fix. It needs the following fix for _pyio.BufferedReader.readline():

diff --git a/Lib/_pyio.py b/Lib/_pyio.py
index c1bdac7913..e90742ec43 100644
--- a/Lib/_pyio.py
+++ b/Lib/_pyio.py
@@ -557,6 +557,8 @@ class IOBase(metaclass=abc.ABCMeta):
         res = bytearray()
         while size < 0 or len(res) < size:
             b = self.read(nreadahead())
+            if b is None and not res:
+                return None
             if not b:
                 break
             res += b


---

Example:

$ ./python poc_aio.py poc_aio.py 
data: b'import asyncio\n'

Internally, _prepare_read() reads 8 KiB.
History
Date User Action Args
2019-10-11 12:42:55vstinnersetrecipients: + vstinner, pitrou, giampaolo.rodola, benjamin.peterson, stutzbach, njs, martin.panter, YoSTEALTH, xgdomingo
2019-10-11 12:42:55vstinnersetmessageid: <1570797775.61.0.0230707707315.issue32561@roundup.psfhosted.org>
2019-10-11 12:42:55vstinnerlinkissue32561 messages
2019-10-11 12:42:55vstinnercreate