This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author achimnol
Recipients achimnol, asvetlov, gvanrossum, yselivanov
Date 2022-02-24.05:34:24
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1645680865.21.0.230450894933.issue46843@roundup.psfhosted.org>
In-reply-to
Content
I'm now tracking the recent addition and discussion of TaskGroup and cancellation scopes. It's interesting! :)

I would like to suggest to have a different mode of operation in asyncio.TaskGroup, which I named "PersistentTaskGroup".

AFAIK, TaskGroup targets to replace asyncio.gather, ensuring completion or cancellation of all tasks within the context manager scope.

I believe that a "safe" asyncio application should consist of a nested tree of task groups, which allow us to explicitly state when tasks of different purposes and contexts terminate.  For example, a task group for database transactions should be shutdown before a task group for HTTP handlers is shutdown.

To this end, in server applications with many sporadically spawned tasks throughout the whole process lifetime, there are different requirements for a task group that manages such task sets.  The tasks should *not* be cancelled upon the unhandled exceptions of sibling tasks in the task group, while we need an explicit "fallback" exception handler for those (just like "return_exceptions=True" in asyncio.gather).  The tasks belong to the task group but their references should not be kept forever to prevent memory leak (I'd suggest using weakref.WeakSet).  When terminating the task group itself, the ongoing tasks should be cancelled.  The cancellation process upon termination may happend in two phases: cancel request with initial timeout + additional limited waiting of cancellations.  (This is what Guido has mentioned in the discussion in bpo-46771.)

An initial sketch of PersistentTaskGroup is on aiotools:
https://github.com/achimnol/aiotools/blob/main/src/aiotools/ptaskgroup.py
Currently has no two-phase cancellation because it would require Python 3.11 with asyncio.Task.uncancel().

As Andrew has left a comment (https://github.com/achimnol/aiotools/issues/29#issuecomment-997437030), I think it is the time to revisit the concrete API design and whether to include PersistentTaskGroup in the stdlib or not.
History
Date User Action Args
2022-02-24 05:34:25achimnolsetrecipients: + achimnol, gvanrossum, asvetlov, yselivanov
2022-02-24 05:34:25achimnolsetmessageid: <1645680865.21.0.230450894933.issue46843@roundup.psfhosted.org>
2022-02-24 05:34:25achimnollinkissue46843 messages
2022-02-24 05:34:24achimnolcreate