This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author achimnol
Recipients achimnol, asvetlov, gvanrossum, yselivanov
Date 2022-02-25.04:45:00
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1645764301.0.0.265718415405.issue46843@roundup.psfhosted.org>
In-reply-to
Content
This particular experience, https://github.com/lablup/backend.ai-agent/pull/331, has actually motivated me to suggest PersistentTaskGroup.

The program subscribes the event stream of Docker daemon using aiohttp as an asyncio task, and this should be kept running throughout the whole application lifetime.  I first applied aiotools.TaskGroup to ensure shutdown of spawned event handler tasks, but I missed that it cancels all sibling tasks if one of the spawned tasks bubbles up an unhandled exception.  This has caused silent termination of the subscriber task and led to a bug.  We could debug this issue by inspecting aiomonitor and checking the existence of this task.  After this issue, I began to think we need a proper abstraction of a long-running task group (NOTE: the task group is long-running.  The lifetime of internal tasks does not matter).

Another case is that https://github.com/lablup/backend.ai/issues/330.

One of our customer site has suffered from excessive CPU usage by our program.  We could identify the issue by aiomonitor, and the root cause was the indefinite accumulation of peridoically created asyncio tasks to measure the disk usage of user directories, when there are too many files in them.  Since the number of tasks have exceeded 10K, it was very difficult to group and distinguish individual asyncio tasks in aiomonitor.  I thought that it would be nice if we could group such tasks into long-running groups and view task statistics separately.
History
Date User Action Args
2022-02-25 04:45:01achimnolsetrecipients: + achimnol, gvanrossum, asvetlov, yselivanov
2022-02-25 04:45:01achimnolsetmessageid: <1645764301.0.0.265718415405.issue46843@roundup.psfhosted.org>
2022-02-25 04:45:00achimnollinkissue46843 messages
2022-02-25 04:45:00achimnolcreate