This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author skrech
Recipients asvetlov, skrech, yselivanov
Date 2022-02-21.13:03:48
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1645448629.24.0.443450493338.issue46818@roundup.psfhosted.org>
In-reply-to
Content
Hello,

Last several days I'm trying to implement an async "opener" object that can be used as Coroutine as well as an AsyncContextManager (eg. work with `await obj.open()` and `async with obj.open()`). I've researched several implementations from various python packages such as:
1. aiofiles: https://github.com/Tinche/aiofiles/blob/master/src/aiofiles/base.py#L28
2. aiohttp: https://github.com/aio-libs/aiohttp/blob/master/aiohttp/client.py#L1082

Unlike these libs though, I want my implementation to return a custom object that is a wrapper around the object returned from the underlying module I'm hiding. 

Example:
I want to implement a DataFeeder interface that has a single method `open()`. Sub-classes of this interface will support, for example, opening an file using aiofiles package. So,
AsyncFileDataFeeder.open() will call `aiofiles.open()`, but instead of returning "file-handle" from aiofiles, I want to return a custom Feed class that implements some more methods for reading -- for example:

async with async_data_feeder.open() as feed:
  async for chunk in feed.iter_chunked():
    ...

To support that I'm returning an instance of the following class from DataFeeder.open():

class ContextOpener(
    Coroutine[Any, Any, Feed],
    AbstractAsyncContextManager[Feed],
):
    __slots__ = ("_wrapped_coro", "_feed_cls", "_feed")

    def __init__(self, opener_coro: Coroutine, feed_cls: Type[Feed]):
        self._wrapped_coro = opener_coro
        self._feed_cls = feed_cls

        self._feed: Any = None

    def __await__(self) -> Generator[Any, Any, Feed]:
        print("in await", locals())
        handle = yield from self._wrapped_coro.__await__()
        return self._feed_cls(handle)

    def send(self, value: Any) -> Any:
        print("in send", locals())
        return self._wrapped_coro.send(value)

    def throw(self, *args, **kwargs) -> Any:
        print("in throw", locals())
        return self._wrapped_coro.throw(*args, **kwargs)

    def close(self) -> None:
        print("in close", locals())
        self._wrapped_coro.close()

    async def __aenter__(self) -> feeds.Feed:
        handle = await self._wrapped_coro
        self._feed = self._feed_cls(handle)
        return self._feed

    async def __aexit__(
        self,
        exc_type: Optional[Type[BaseException]],
        exc: Optional[BaseException],
        tb: Optional[TracebackType],
    ) -> None:
        await self._feed.close()
        self._feed = None


This code actually works! But I've noticed that when calling `await DataFeeder.open()` the event loop never calls my `send()` method.

if __name__ == "__main__":
    async def open_test():
        await asyncio.sleep(1)
        return 1

    async def main():
        c = ContextOpener(open_test(), feeds.AsyncFileFeed)
        ret = await c
        print("Finish:", ret, ret._handle)

The output:
in await {'self': <__main__.ContextOpener object at 0x11099cd10>}
Finish: <feeds.AsyncFileFeed object at 0x1109a9a80> 1

From then on a great thinking and reading on the Internet happened, trying to explain to myself how exactly coroutines are working. I suspect that the ContextOpener.__await__ is returning a generator instance and from then on, outer coroutines (eg. main in this case) are calling send()/throw()/close() on THAT generator, not on the ContextOpener "coroutine".
The only way to make Python call ContextOpener send() method (and friends) is when ContextOpener is the outermost coroutine that is communicating directly with the event loop:

ret = asyncio.run(ContextOpener(open_test(), feeds.AsyncFileFeed))
print("Finish:", ret)

Output:
in send {'self': <__main__.ContextOpener object at 0x10dcf47c0>, 'value': None}
in send {'self': <__main__.ContextOpener object at 0x10dcf47c0>, 'value': None}
Finish: 1

However, now I see that I have an error in my implementation that was hidden before: my send() method implementation is not complete because StopIteration case is not handled and returns 1, instead of Feed object.

Since __await__() should return iterator (by PEP492) I can't figure out a way to implement what I want unless making my coroutine class an iterator itself (actually generator) by returning `self` from __await__ and add __iter__ and __next__ methods:

    def __await__(self):
        return self

    def __iter__(self):
        return self

    def __next__(self):
        return self.send(None)

Is this the proper way to make a Coroutine out of a collections.abc.Coroutine? Why is then the documentation not explicitly saying that a Coroutine should inherit from collections.abc.Generator?

I see this as very common misconception since every such "ContextManager" similar to ContextOpener from 3rd party packages (like the aforementioned two, aiofiles and aiohttp, but there are others as well) is subclassing collections.abc.Coroutine and implements send(), throw() and close() methods that are not actually being called. I suspect, the authors of these libraries haven't noticed that because the returned value from the __await__() and send() methods is the same in their case.
History
Date User Action Args
2022-02-21 13:03:49skrechsetrecipients: + skrech, asvetlov, yselivanov
2022-02-21 13:03:49skrechsetmessageid: <1645448629.24.0.443450493338.issue46818@roundup.psfhosted.org>
2022-02-21 13:03:49skrechlinkissue46818 messages
2022-02-21 13:03:48skrechcreate