classification
Title: reference cycle with exception state not broken by generator.close()
Type: resource usage Stage: resolved
Components: Interpreter Core Versions: Python 3.2, Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: pitrou Nosy List: benjamin.peterson, martin.panter, ncoghlan, pitrou, python-dev
Priority: normal Keywords: patch

Created on 2011-08-20 06:37 by martin.panter, last changed 2011-08-20 12:21 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
leaky_generator.py martin.panter, 2011-08-20 06:37
genexcstate.patch pitrou, 2011-08-20 12:11 review
Messages (5)
msg142515 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2011-08-20 06:37
See attached "leaky_generator.py" demo. Python doesn't appear to delete the exception variable if an exception is thrown back into it (via "throw", "close" or by deleting it). The result is a reference cycle that needs garbage collecting. This even happens when no exception variable is named. The exception variable is not leaked if the generator is called normally (via "next" or "send").

The exception variable of an exception handler is usually deleted (according to http://docs.python.org/py3k/reference/compound_stmts.html#the-try-statement) as soon as the handler is exited, even when it is not exited by simply falling through the end. So either this should also happen for "yield" inside the exception handler, or it should be documented that "yield" inside "except", even without "as", should be avoided.

I'm guessing this issue is specific to Python 3 because it exists for me in Python 3.2, 3.2.1 (both on Arch Linux) and 3.1.2 (Ubuntu) but not Python 2.7.1, 2.7.2 (Arch) nor 2.6.5 (Ubuntu).
msg142522 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-08-20 11:46
The problem is probably that the frame keeps the last execution state around, and since the exception itself has a reference to a frame, a cycle is created. Note that it doesn't happen if you catch the GeneratorExit that gets raised inside the generator at shutdown:

def leaky_generator():
    try:   
        1/0
    except:  # Exception handler with anonymous exception variable
        try:
            yield  # Yield from exception handler
        except (GeneratorExit, RuntimeError):
            pass

def throw_leaks(g):
    try:
        g.throw(RuntimeError())
    except Exception:
        pass
msg142525 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-08-20 12:11
Here is a patch.
msg142526 - (view) Author: Roundup Robot (python-dev) Date: 2011-08-20 12:20
New changeset 867ce75b885c by Antoine Pitrou in branch '3.2':
Issue #12791: Break reference cycles early when a generator exits with an exception.
http://hg.python.org/cpython/rev/867ce75b885c

New changeset 7d390c3a83c6 by Antoine Pitrou in branch 'default':
Issue #12791: Break reference cycles early when a generator exits with an exception.
http://hg.python.org/cpython/rev/7d390c3a83c6
msg142527 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-08-20 12:21
Thanks for the report, fixed.
History
Date User Action Args
2011-08-20 12:21:10pitrousetstatus: open -> closed
resolution: fixed
messages: + msg142527

stage: resolved
2011-08-20 12:20:46python-devsetnosy: + python-dev
messages: + msg142526
2011-08-20 12:11:59pitrousetfiles: + genexcstate.patch
keywords: + patch
messages: + msg142525
2011-08-20 11:46:36pitrousetnosy: + ncoghlan, benjamin.peterson
title: Yield" leaks exception being handled as garbage -> reference cycle with exception state not broken by generator.close()
messages: + msg142522

versions: + Python 3.3, - Python 3.1
2011-08-20 07:30:22georg.brandlsetassignee: pitrou

nosy: + pitrou
2011-08-20 06:37:55martin.pantercreate