This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: asyncio.as_completed() raises TypeError when the first supplied parameter is a generator that yields awaitables
Type: behavior Stage:
Components: asyncio, Library (Lib) Versions: Python 3.10, Python 3.9, Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: alexdelorenzo, asvetlov, yselivanov
Priority: normal Keywords:

Created on 2021-05-19 05:04 by alexdelorenzo, last changed 2022-04-11 14:59 by admin.

Files
File name Uploaded Description Edit
as_completed_gen.py alexdelorenzo, 2021-05-19 05:04
Pull Requests
URL Status Linked Edit
PR 26228 open alexdelorenzo, 2021-05-19 05:04
Messages (1)
msg393923 - (view) Author: Alex DeLorenzo (alexdelorenzo) * Date: 2021-05-19 05:04
According to the documentation, asyncio.as_completed() takes a positional argument, aws, as an iterable of awaitables[1]:

    asyncio.as_completed(aws, *, loop=None, timeout=None)
      Run awaitable objects in the aws iterable concurrently.

As seen in the attached as_completed_gen.py file, built-in containers like lists, and iterators over them, are accepted by as_completed() as the first parameter without raising an error.

However, as_completed() raises TypeError if it is called with an iterable of awaitables that is also a generator. There are examples of this behavior in as_completed_gen.py, but here is a short example using a generator expression in the main() coroutine function:

    from asyncio import run, as_completed
    
    async def example(): pass
    
    async def main():
      coros = (example() for _ in range(10))
    
      for coro in as_completed(coros):  # raises TypeError
        await coro
    
    run(main())

Running that example will raise a TypeError with this message:

    TypeError: expect an iterable of futures, not generator


If we look at the first line in the body of as_completed(), we can see why this error is thrown for generators that yield awaitables:

    def as_completed(fs, *, loop=None, timeout=None):
      if futures.isfuture(fs) or coroutines.iscoroutine(fs):
        raise TypeError(f"expect an iterable of futures, not {type(fs).__name__}")
      ...


Because generators are coroutines, and the first condition in as_completed() is True, and TypeError gets raised:

    from asyncio import coroutines

    # generators and generator expressions are coroutines
    assert coroutines.iscoroutine(example() for _ in range(10))


Perhaps as_completed() can use inspect.isawaitable() instead, like so:

    from inspect import isawaitable

    def as_completed(fs, *, loop=None, timeout=None):
      if futures.isfuture(fs) or isawaitable(fs):
        ...

I made a pull request with that change here[2].


[1] https://docs.python.org/3/library/asyncio-task.html#asyncio.as_completed

[2] https://github.com/python/cpython/pull/26228
History
Date User Action Args
2022-04-11 14:59:45adminsetgithub: 88342
2021-05-19 05:53:09alexdelorenzosetcomponents: + Library (Lib)
2021-05-19 05:30:10alexdelorenzosettype: behavior
2021-05-19 05:04:33alexdelorenzocreate