classification
Title: Add stats for asyncio task usage.
Type: Stage:
Components: asyncio Versions: Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: achimnol, asvetlov, njs, thehesiod, yselivanov
Priority: normal Keywords:

Created on 2018-04-03 22:33 by asvetlov, last changed 2018-04-04 02:43 by achimnol.

Messages (6)
msg314905 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2018-04-03 22:33
I suggest adding a `Task.stats()` method.
The method should return a dict with the task usage statistics.
Dict keys:
- total_time: a time between task creation and a moment of the call (or task finishing timestamp if the task has finished). The value includes a time for `await func()` waiting.
- real_time: a cumulative time for `Task._step()` executions
- switch_count: a count of `Task._step()` calls.

It can help with tasks activity analyzing.

Yuri, what do you think about?
msg314907 - (view) Author: Alexander Mohr (thehesiod) * Date: 2018-04-03 22:39
another idea would be neat to somehow plug into the stats so people can trace (ex: via datadog tracing) "slow" tasks, similarly to things that are outputted via PYTHONASYNCIODEBUG, however in this case be able to trace slow steps and also slow accumulated steps.  In this scenario you don't know which task will be slow and don't want to track every task.
msg314908 - (view) Author: Nathaniel Smith (njs) * (Python committer) Date: 2018-04-03 23:01
FYI, here's how trio handles such things – I'm sure it's not perfect, but might be useful for inspiration: https://trio.readthedocs.io/en/latest/reference-hazmat.html#instrument-api
msg314909 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2018-04-03 23:06
> Yuri, what do you think about?

I plan to use contextvars module to introduce a full-blown tracing API to asyncio to selectively log events like tacks creations, event loop switching, IO done by transports etc.  The plan is to prototype it in uvloop first.

I'll update this issue with a link to a discussion.
msg314910 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2018-04-03 23:10
Well, event based solution is more powerful than just statistic collection.
Please keep me in the loop.
msg314913 - (view) Author: Joongi Kim (achimnol) * Date: 2018-04-04 02:43
I like trio-style instrumentation API because it could be used for more generic purposes, not only for statistics.

This stats or instrumentation API will greatly help me to utilize external monitoring services such as Datadog in my production deployments.
History
Date User Action Args
2018-04-04 02:43:26achimnolsetmessages: + msg314913
2018-04-04 01:12:23achimnolsetnosy: + achimnol
2018-04-03 23:10:32asvetlovsetmessages: + msg314910
2018-04-03 23:06:39yselivanovsetmessages: + msg314909
2018-04-03 23:01:08njssetnosy: + njs
messages: + msg314908
2018-04-03 22:39:34thehesiodsetnosy: + thehesiod
messages: + msg314907
2018-04-03 22:33:22asvetlovcreate