Along with bpo-46843 and the new asyncio.TaskGroup API, I would like to suggest addition of context-based TaskGroup feature.

Currently asyncio.create_task() just creates a new task directly attached to the event loop, while asyncio.TaskGroup.create_task() creates a new task managed by the TaskGroup instance.

It would be ideal to all existing asyncio codes to migrate to use TaskGroup, but this is impractical.

An alternative approach is to implicitly bind asyncio.create_task() under a specific context to a specific task group, probably using contextvars.

I believe that this approach would allow more control over tasks implicitly spawned by 3rd-party libraries that cannot control.

How about your thoughts?

> I believe that this approach would allow more control over tasks implicitly spawned by 3rd-party libraries that cannot control.

Please elaborate. I'm not sure what are the benefits of this.

The main benefit is that any legacy code that I cannot modify can be upgraded to TaskGroup-based codes, which offers a better machinary for exception handling and propagation.

There may be different ways to visit this issue: allow replacing the task factory in asyncio at runtime.  Then I could just implement my own snippet to transfer the "ownership" of the task to a specific task group.

Conceptually it is similar to replace malloc using LD_PRELOAD or LD_LIBRARY_PATH manipulation.  When I cannot modify the executable/library binaries, this allows replacing the functionality of specific functions.

If we could assign a specific (persistent) task group to all asyncio tasks spawned by a black-box code (when the black-box itself does not use task groups), we could achieve the full application-level transparency on the timing of task cancellation.

It is also useful to write debugging/monitoring codes for asyncio applications.  For instance, we could "group" tasks from different libraries and count them.

-1

Now bare `create_task()` does fire-and-forget action.
After the proposed change it will fail loudly.
Even if this behavior is better it is not backward compatible.
People start blaming and asking "how to return everything back?"

My propsal is to opt-in the taskgroup binding for asyncio.create_task() under a specific context, not changing the defautl behavior.

An example would be like:

tg = asyncio.TaskGroup()
...
async with tg:
  with asyncio.TaskGroupBinder(tg):  # just a hypothetical API
    asyncio.create_task(...)         # equivalent to tg.create_task(...)
    await some_library.some_work()   # all tasks are bound to tg
  asyncio.create_task(...)           # fire-and-forget (not bound to tg)

If TaskGroup supports enumeration/counting of its own tasks and asyncio allows enumeration of TaskGroups just like asyncio.Task.all_tasks(), we could extend aiomonitor to provide per-taskgroup statistics.

In my projects, we have multiple cases to find and fix bugs in customer sites using aiomonitor and I'm willing to improve aiomonitor to support task groups as well.

Ah, and this use case also requires that TaskGroup should have an option like `return_exceptions=True` which makes it not to cancel sibling tasks upon unhandled exceptions, as I suggested in PersistentTaskGroup (bpo-46843).

I personally don't think that the described opt-in trick should be a part of asyncio.
A third-party library that patches asyncio.create_task() can be a useful thing though during the transition period.
We even cannot deprecate asyncio.create_task() right now: the minimal supported Python 3.7 thas now alternative, writing cross-version code is overcomplicated.

Ok, let me be clear: Patching asyncio.create_task() to support this opt-in contextual task group binding is not an ultimate goal of this issue.  If it becomes possible to override/extend the task factory at runtime with any event loop implementation, then it's ok to implement this feature request as a 3rd-party library.  I also don't want to bloat the stdlib with version-specific branches, if there are alternative ways to achieve the same goal.  I just wanted to check out your opinons and potential alternative approaches to implement it.

-1. Libraries that manage their own tasks should continue to do so, until they are ready to adopt TaskGroup. Trying to "own" (and wait for) tasks created by a library sounds like a recipe for disaster if the library wasn't expecting that.

Do you have a specific use case or scenario that has bitten you in the past? If you want to continue to argue for this feature we would need specifics (not a link to reams of code but a clear story telling of a problem you've encountered in real life in the past that your proposal might solve).

I have added more about my stories in bpo-46843.

I think the suggestion of implicit taskgroup binding with the current asyncio.TaskGroup has no point but it would have more meaning with PersistentTaskGroup.

So, if we treat PersistentTaskGroup as a "nested, hierarchical virtual event loop" to repeat and group shutdown procedures for different task sets separately, the point may look a little bit clearer.

It is more like assigning a virtual event loop to different modules and libraries, while keeping the behavior of asyncio.create_task() same.  The difference is that the caller controls when these virtual loops are terminated and in what order.

Does this make sense better?

Updated the title to reduce confusion.

Let's concentrate the discussion in the other issue.