Created on 2012-03-07 11:09 by scoder, last changed 2012-03-08 21:13 by Mark.Shannon. This issue is now closed.
|msg155072 - (view)||Author: Stefan Behnel (scoder) *||Date: 2012-03-07 11:09|
Based on the existing "test_attempted_yield_from_loop" in Lib/test/test_pep380.py, I wrote this test and I wonder why it does not work: """ def test_attempted_reentry(): """ >>> for line in test_attempted_reentry(): print(line) g1: starting Yielded: y1 g1: about to yield from g2 g2: starting Yielded: y2 g2: about to yield from g1 g2: caught ValueError Yielded: y3 g1: after delegating to g2 Yielded: y4 """ trace =  def g1(): trace.append("g1: starting") yield "y1" trace.append("g1: about to yield from g2") yield from g2() trace.append("g1: after delegating to g2") yield "y4" def g2(): trace.append("g2: starting") yield "y2" trace.append("g2: about to yield from g1") try: yield from gi except ValueError: trace.append("g2: caught ValueError") else: trace.append("g1 did not raise ValueError on reentry") yield "y3" gi = g1() for y in gi: trace.append("Yielded: %s" % (y,)) return trace """ In current CPython, I get this: """ Failed example: for line in test_attempted_reentry(): print(line) Expected: g1: starting Yielded: y1 g1: about to yield from g2 g2: starting Yielded: y2 g2: about to yield from g1 g2: caught ValueError Yielded: y3 g1: after delegating to g2 Yielded: y4 Got: g1: starting Yielded: y1 g1: about to yield from g2 g2: starting Yielded: y2 g2: about to yield from g1 g2: caught ValueError Yielded: y3 """ Even though I catch the ValueError (raised on generator reentry) at the position where I run the "yield from", the outer generator (g1) does not continue to run after the termination of g2. It shouldn't normally have an impact on the running g1 that someone attempts to jump back into it, but it clearly does here. I noticed this while trying to adapt the implementation for Cython, because the original test was one of the few failing cases and it made the code jump through the generator support code quite wildly.
|msg155112 - (view)||Author: Stefan Behnel (scoder) *||Date: 2012-03-07 20:06|
Here is an analysis of this (less verbose) code: def g1(): yield "y1" yield from g2() yield "y4" def g2(): yield "y2" try: yield from gi except ValueError: pass # catch "already running" error yield "y3" gi = g1() for y in gi: print("Yielded: %s" % (y,)) This is what it currently does: 1) g1() delegates to a new g2() 2) g2 delegates back to the g1 instance and asks for its next value 3) Python sees the active delegation in g1 and asks g2 for its next value 4) g2 sees that it's already running and throws an exception Ok so far. Now: 5) the exception is propagated into g1 at call level 3), not at level 1)! 6) g1 undelegates and terminates by the exception 7) g2 catches the exception, yields "y3" and then terminates normally 8) g1 gets control back but has already terminated and does nothing Effect: "y4" is not yielded anymore. The problem is in steps 5) and 6), which are handled by g1 at the wrong call level. They shouldn't lead to undelegation and termination in g1, just to an exception being raised in g2.
|msg155132 - (view)||Author: Nick Coghlan (ncoghlan) *||Date: 2012-03-07 23:48|
Added Mark Shannon to the nosy list - he's been tinkering with this area of the interpreter lately. This definitely needs to be fixed though (even if that does mean major surgery on the implementation, up to and including the introduction of an __iter_from__ method)
|msg155133 - (view)||Author: Benjamin Peterson (benjamin.peterson) *||Date: 2012-03-07 23:52|
Don't worry! I'll be fixing it in a moment...
|msg155134 - (view)||Author: Roundup Robot (python-dev)||Date: 2012-03-07 23:57|
New changeset 3357eac1ba62 by Benjamin Peterson in branch 'default': make delegating generators say they running (closes #14220) http://hg.python.org/cpython/rev/3357eac1ba62
|msg155151 - (view)||Author: Stefan Behnel (scoder) *||Date: 2012-03-08 06:31|
Ah, yes, that should work in both implementations. I'll give it a try. Thanks!
|msg155183 - (view)||Author: Mark Shannon (Mark.Shannon) *||Date: 2012-03-08 21:13|
I've just added issue 14230 which overlaps with this issue somewhat.
|2012-03-08 21:13:37||Mark.Shannon||set||messages: + msg155183|
|2012-03-08 20:32:45||benjamin.peterson||set||status: open -> closed|
|2012-03-08 06:31:05||scoder||set||status: closed -> open|
resolution: fixed -> (no value)
messages: + msg155151
|2012-03-07 23:57:16||python-dev||set||status: open -> closed|
nosy: + python-dev
messages: + msg155134
messages: + msg155133
messages: + msg155132
|2012-03-07 20:06:10||scoder||set||messages: + msg155112|