classification
Title: "yield from" kills generator on re-entry
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Mark.Shannon, benjamin.peterson, ncoghlan, python-dev, scoder
Priority: normal Keywords:

Created on 2012-03-07 11:09 by scoder, last changed 2012-03-08 21:13 by Mark.Shannon. This issue is now closed.

Messages (7)
msg155072 - (view) Author: Stefan Behnel (scoder) * Date: 2012-03-07 11:09
Based on the existing "test_attempted_yield_from_loop" in Lib/test/test_pep380.py, I wrote this test and I wonder why it does not work:

"""
def test_attempted_reentry():
    """
    >>> for line in test_attempted_reentry(): print(line)
    g1: starting
    Yielded: y1
    g1: about to yield from g2
    g2: starting
    Yielded: y2
    g2: about to yield from g1
    g2: caught ValueError
    Yielded: y3
    g1: after delegating to g2
    Yielded: y4
    """
    trace = []
    def g1():
        trace.append("g1: starting")
        yield "y1"
        trace.append("g1: about to yield from g2")
        yield from g2()
        trace.append("g1: after delegating to g2")
        yield "y4"

    def g2():
        trace.append("g2: starting")
        yield "y2"
        trace.append("g2: about to yield from g1")
        try:
            yield from gi
        except ValueError:
            trace.append("g2: caught ValueError")
        else:
            trace.append("g1 did not raise ValueError on reentry")
        yield "y3"
    gi = g1()
    for y in gi:
        trace.append("Yielded: %s" % (y,))
    return trace
"""

In current CPython, I get this:

"""
Failed example:
    for line in test_attempted_reentry(): print(line)
Expected:
    g1: starting
    Yielded: y1
    g1: about to yield from g2
    g2: starting
    Yielded: y2
    g2: about to yield from g1
    g2: caught ValueError
    Yielded: y3
    g1: after delegating to g2
    Yielded: y4
Got:
    g1: starting
    Yielded: y1
    g1: about to yield from g2
    g2: starting
    Yielded: y2
    g2: about to yield from g1
    g2: caught ValueError
    Yielded: y3
"""

Even though I catch the ValueError (raised on generator reentry) at the position where I run the "yield from", the outer generator (g1) does not continue to run after the termination of g2. It shouldn't normally have an impact on the running g1 that someone attempts to jump back into it, but it clearly does here.

I noticed this while trying to adapt the implementation for Cython, because the original test was one of the few failing cases and it made the code jump through the generator support code quite wildly.
msg155112 - (view) Author: Stefan Behnel (scoder) * Date: 2012-03-07 20:06
Here is an analysis of this (less verbose) code:

    def g1():
        yield "y1"
        yield from g2()
        yield "y4"

    def g2():
        yield "y2"
        try:
            yield from gi
        except ValueError:
            pass  # catch "already running" error
        yield "y3"

    gi = g1()
    for y in gi:
        print("Yielded: %s" % (y,))

This is what it currently does:

1) g1() delegates to a new g2()

2) g2 delegates back to the g1 instance and asks for its next value

3) Python sees the active delegation in g1 and asks g2 for its next value

4) g2 sees that it's already running and throws an exception

Ok so far. Now:

5) the exception is propagated into g1 at call level 3), not at level 1)!

6) g1 undelegates and terminates by the exception

7) g2 catches the exception, yields "y3" and then terminates normally

8) g1 gets control back but has already terminated and does nothing

Effect: "y4" is not yielded anymore.

The problem is in steps 5) and 6), which are handled by g1 at the wrong call level. They shouldn't lead to undelegation and termination in g1, just to an exception being raised in g2.
msg155132 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-03-07 23:48
Added Mark Shannon to the nosy list - he's been tinkering with this area of the interpreter lately.

This definitely needs to be fixed though (even if that does mean major surgery on the implementation, up to and including the introduction of an __iter_from__ method)
msg155133 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2012-03-07 23:52
Don't worry! I'll be fixing it in a moment...
msg155134 - (view) Author: Roundup Robot (python-dev) Date: 2012-03-07 23:57
New changeset 3357eac1ba62 by Benjamin Peterson in branch 'default':
make delegating generators say they running (closes #14220)
http://hg.python.org/cpython/rev/3357eac1ba62
msg155151 - (view) Author: Stefan Behnel (scoder) * Date: 2012-03-08 06:31
Ah, yes, that should work in both implementations. I'll give it a try. Thanks!
msg155183 - (view) Author: Mark Shannon (Mark.Shannon) * Date: 2012-03-08 21:13
I've just added issue 14230 which overlaps with this issue somewhat.
History
Date User Action Args
2012-03-08 21:13:37Mark.Shannonsetmessages: + msg155183
2012-03-08 20:32:45benjamin.petersonsetstatus: open -> closed
resolution: fixed
2012-03-08 06:31:05scodersetstatus: closed -> open
resolution: fixed -> (no value)
messages: + msg155151
2012-03-07 23:57:16python-devsetstatus: open -> closed

nosy: + python-dev
messages: + msg155134

resolution: fixed
stage: resolved
2012-03-07 23:52:06benjamin.petersonsetnosy: + benjamin.peterson
messages: + msg155133
2012-03-07 23:48:21ncoghlansetnosy: + Mark.Shannon
messages: + msg155132
2012-03-07 22:37:12pitrousetnosy: + ncoghlan
2012-03-07 20:06:10scodersetmessages: + msg155112
2012-03-07 11:09:02scodercreate