classification
Title: asyncio: add background task detecting reference cycles
Type: Stage: resolved
Components: asyncio Versions: Python 3.6
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: Joshua.Harlow, asvetlov, gvanrossum, martin.panter, vstinner, yselivanov
Priority: normal Keywords:

Created on 2015-07-09 14:04 by vstinner, last changed 2017-06-28 01:20 by vstinner. This issue is now closed.

Messages (9)
msg246499 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-07-09 14:04
When storing an exception in an asyncio Future object, there is a high risk of creating a reference cycle. In Python 3, exception objects store a traceback object which store frame objects. The problem is that a frame can also have a reference to the exception: we have a reference cycle (exception -> traceback -> frame -> same exception).

In debug mode, Future.set_exception() can schedule a task (ex: using loop.call_soon) to check that there is no reference cycle.

See also the issue #23587: "asyncio: use the new traceback.TracebackException class".
msg246500 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2015-07-09 14:06
Doesn't the cycle-detecting GC handle these?
msg246501 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-07-09 14:10
> Doesn't the cycle-detecting GC handle these?

Maybe you are lucky and the GC is able to break the cycle. Maybe you are unlucky and all objects part of the cycle will never be deleted. Python 3.4 is better to handle these cases, but Python 3.3 is worse to handle reference cycles.

Being aware of the cycles help the developer to control when objects are deleted. It mean breaking the cycles help to delete objects sooner.

See other asyncio issues about "reference cycles" for some examples.
msg246503 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2015-07-09 14:16
Hm. If the problem is most prominent with 3.3, why mark the issue as 3.6? Do you have an implementation already? Maybe it can be a 3rd party package rather than integrated in asyncio debug mode?
msg246504 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-07-09 14:35
> Hm. If the problem is most prominent with 3.3, why mark the issue as 3.6?

Well, I plan to implement this feature in "asyncio", so for 3.3-3.6 in fact.

> Do you have an implementation already?

Nope, it's more a TODO task for myself :-)

> Maybe it can be a 3rd party package rather than integrated in asyncio debug mode?

Hum, I'm not sure. My idea is to add code in Future.set_exception() because I will probably need a reference to the exception object (and maybe to the Future object). Currently, it's not possible to replace the Future class (whereas we have BaseEventLoop.set_task_factory and BaseEventLoop.create_task). I don't think that it's worth to make it possible to replace/hook this class. I don't expect a huge complex code to detect reference cycles.

Please give me some weeks to investigate this issue. It will be easier to discuss with a working patch.

I opened the issue as a reminder for myself, but also to colaborate on it.
msg246507 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2015-07-09 15:00
OK, no problem. (Side comment: Future is being subclassed a lot, so
parametrizing its construction may not be so easy.)
msg246543 - (view) Author: Joshua Harlow (Joshua.Harlow) Date: 2015-07-10 03:56
Out of curiosity what reference cycles can't be broken in various python versions? Is it documented/explained anywhere?
msg246545 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-07-10 06:35
Python 3.4 is able to break reference cycles even if an object part of the
cycle has a destructor. See the PEP 442.
msg297111 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-28 01:20
I lost track of this issue, so I just close it, sorry.
History
Date User Action Args
2017-06-28 01:20:27vstinnersetstatus: open -> closed
resolution: out of date
messages: + msg297111

stage: resolved
2015-09-16 17:25:00asvetlovsetnosy: + asvetlov
2015-07-10 06:35:56vstinnersetmessages: + msg246545
2015-07-10 03:56:27Joshua.Harlowsetnosy: + Joshua.Harlow
messages: + msg246543
2015-07-09 15:12:50martin.pantersetnosy: + martin.panter
2015-07-09 15:00:16gvanrossumsetmessages: + msg246507
2015-07-09 14:35:31vstinnersetmessages: + msg246504
2015-07-09 14:16:44gvanrossumsetmessages: + msg246503
2015-07-09 14:10:12vstinnersetmessages: + msg246501
2015-07-09 14:06:00gvanrossumsetmessages: + msg246500
2015-07-09 14:04:05vstinnercreate