This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Finalization of non-exhausted asynchronous generators is deferred
Type: resource usage Stage:
Components: Interpreter Core Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Guido.van.Rossum, Mark.Shannon, asvetlov, gvanrossum, serhiy.storchaka, yselivanov
Priority: normal Keywords:

Created on 2021-06-27 09:11 by serhiy.storchaka, last changed 2022-04-11 14:59 by admin.

Messages (6)
msg396569 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-06-27 09:11
In the following example:

def gen():
    try:
        yield 1
    finally:
        print('finalize inner')

def func():
    try:
        for x in gen():
            break
    finally:
        print('finalize outer')

func()
print('END')

the output in CPython is:

finalize inner
finalize outer
END


But in similar example for asynchronous generator:

async def gen():
    try:
        yield 1
    finally:
        print('finalize inner')

async def func():
    try:
        async for x in gen():
            break
    finally:
        print('finalize outer')

import asyncio
asyncio.run(func())
print('END')

the output in CPython is:

finalize outer
finalize inner
END

There is a strong link somewhere which prevents finalization of the asynchronous generator object until leaving the outer function.

Tested on CPython 3.7-3.11. In PyPy "finalize inner" is not printed at all. Using closing() and aclosing() is the right way to get deterministic finalization, but in any case it would be better to get rid of strong link which prevents finalization of the asynchronous generator object.
msg396610 - (view) Author: Guido van Rossum (Guido.van.Rossum) Date: 2021-06-28 00:02
Can you repro this without asyncio?
msg396611 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-06-28 05:31
Sorry, I do not understand what do you mean. The problem is that non-exhausted asynchronous generators are not finalized until asyncio.run() is finished. This differs from synchronous generators which are finalized as early as possible (in CPython).
msg396644 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2021-06-28 16:00
I am wondering whether the issue is in asyncio.run, or in the basic async generator implementation. If the latter, you should be able to demonstrate this with a manual "driver" (trampoline?) instead of asyncio.run.
msg396650 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-06-28 16:40
Inlined asyncio.run(func()) is roughly equivalent to the following code:

loop = asyncio.new_event_loop()
loop.run_until_complete(func())
loop.run_until_complete(loop.shutdown_asyncgens())
loop.close()

If comment out loop.run_until_complete(loop.shutdown_asyncgens()) I get the following output:

finalize outer
Task was destroyed but it is pending!
task: <Task pending name='Task-2' coro=<<async_generator_athrow without __name__>()>>
END

No "finalize inner" but a warning about pending task instead.

On other hand, the following code

for _ in func().__await__(): pass

produces the output in expected order:

finalize inner
finalize outer
END
msg396659 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2021-06-28 18:33
Okay, that suggests that the extra reference is being held by asyncio, right? I suppose it's in the hands of the asyncio devs then.
History
Date User Action Args
2022-04-11 14:59:47adminsetgithub: 88684
2021-06-28 18:33:50gvanrossumsetmessages: + msg396659
2021-06-28 16:40:46serhiy.storchakasetmessages: + msg396650
2021-06-28 16:00:42gvanrossumsetmessages: + msg396644
2021-06-28 05:31:27serhiy.storchakasetmessages: + msg396611
2021-06-28 00:02:14Guido.van.Rossumsetnosy: + Guido.van.Rossum
messages: + msg396610
2021-06-27 09:11:32serhiy.storchakacreate