This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Async generators are not garbage collected
Type: resource usage Stage: resolved
Components: asyncio Versions: Python 3.8
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: asvetlov, vytas, yselivanov
Priority: normal Keywords:

Created on 2020-06-15 14:27 by vytas, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
test.py vytas, 2020-06-15 14:27 Async test case
test_sync.py vytas, 2020-06-15 14:28 Sync test case
test_trio.py vytas, 2020-06-15 15:14 Async test case (w/ trio)
Messages (3)
msg371550 - (view) Author: Vytautas Liuolia (vytas) Date: 2020-06-15 14:27
Hello!
I am having issues with asynchronous generators not being garbage collected at least until the current loop has completed.

In the attached test case (test.py), one starts iterating over an asynchronous generator, then breaks and returns the first element. After each call, gc.collect() is invoked for illustration purposes.
It seems that no memory is freed until the whole test() coroutine is done.

The for-loop could obviously be extended to more iterations, or swapped out to a while-loop to easily run out of available memory.

I have then removed all async stuff, producing test_sync.py (also attached). In the sync case, everything is garbage-collected as I would expect.
msg371560 - (view) Author: Vytautas Liuolia (vytas) Date: 2020-06-15 15:14
Just for the heck of it, I've tried running my test case with Trio (FWIW, I've never used trio before), and it does seem to work as expected.
msg372277 - (view) Author: Vytautas Liuolia (vytas) Date: 2020-06-24 20:13
I've now skimmed through the relevant Trio issue https://github.com/python-trio/trio/issues/265 and the currently deferred https://www.python.org/dev/peps/pep-0533/ and, as I understand, the issue is not trivial and subject to subtle edge cases.

Getting back to the issue, it looks like a misunderstanding on my side, sorry!

As per PEP-525:

    The asyncio event loop will use sys.set_asyncgen_hooks() API to maintain a weak set of all scheduled asynchronous generators, and to schedule their aclose() coroutine methods when it is time for generators to be GCed.

It seems that my original test case just doesn't give these aclose() coroutines a chance to run. Awaiting an asyncio.sleep() "fixes" that.
History
Date User Action Args
2022-04-11 14:59:32adminsetgithub: 85158
2020-06-24 20:26:26vytassetstatus: open -> closed
resolution: not a bug
stage: resolved
2020-06-24 20:13:32vytassetmessages: + msg372277
2020-06-15 15:14:28vytassetfiles: + test_trio.py

messages: + msg371560
2020-06-15 14:28:27vytassetfiles: + test_sync.py
2020-06-15 14:27:50vytascreate