Author stestagg
Recipients stestagg, xxm
Date 2020-12-29.16:43:35
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1609260215.52.0.149559397124.issue42762@roundup.psfhosted.org>
In-reply-to
Content
Ok, so I now understand a bit more, and think it's not a bug!  But explaining it involves some fairly deep thinking about generators.

I'll try to explain my reasoning.

Let's take a simple example:

---

def foo():
    try:
        yield
    except:
        print("ERROR")

for x in foo():
    print(1)
---


It's convenient to think of it as python adding an implicit throw StopIteration() at the end of the generator function, I'll also rewrite the code to be roughly equivalent:

---
 1. def foo():
 2.     try:
 3.         yield None
 4.     except:
 5.         print("ERROR")
 6.     #throw StopIteration():
 7. 
 8. foo_gen = foo()
 9. while True:
10.     try:
11.         x = next(foo_gen)
12.    except StopIteration:
13.         break
14.    print(1)
15. del foo_gen
---

Now, if we step through how python runs the code starting at line 8.:

8. Create foo() <- this just creates a generator object
9. Enter while loop (not interesting for now)
10. try:
11. call next() on our generator to get the next value
2. try: <- we're running the generator until we get a yield now
3. yield None <- yield None to parent code
11. x = None <- assign yielded value to x
12. except StopIteration <- no exception raised so this doesn't get triggered
14. print(1) <- Print 1 to the output
9. Reached end of while loop, return to the start
10. try:
11. Re-enter the generator to get the next value, starting from where we left it..
4. except: <- there was no exception raised, so this doesn't get triggered
6. (implicit) throw StopIteration because generator finished
12. `except StopIteration` <- this is triggered because generator threw StopIteration
13. break <- break out of while loop
15. remove foo_gen variable, and clean up generator.

<- Deleting `foo_gen` causes `.close()` to be called on the generator which causes a GeneratorExit exception to be raised in the generator, BUT generator has already finished, so the GeneratorExit does nothing.

--

This is basically how the for-loop in your example works, and you can see that there's no generator exception, BUT if we change the print(1) to print(i) and try again:

8. Create foo() <- this just creates a generator object
9. Enter while loop (not interesting for now)
10. try:
11. call next() on our generator to get the next value
2. try: <- we're running the generator until we get a yield now
3. yield None <- yield None to parent code
11. x = None <- assign yielded value to x
12. except StopIteration <- no exception raised so this doesn't get triggered
*** CHANGED BIT ***
14. print(i) <- i doesn't exist, so throw NameError
14. The exception made us exit the current stack frame, so start cleaning up/deleting local variables

<- Deleting `foo_gen` causes `.close()` to be called on the generator which causes GeneratorExit() to be raised within the generator, but the generator is currently paused on line 3. so raise exception as-if we're currently running line 3:

4. except: <- this broad except catches the GeneratorExit exception because it appears to have happened on line 3.
5. print("ERROR") <- We only get here if the above steps happened.

---

So, if you don't let a generator naturally finish itself, but stop consuming the generator before it's raised its final StopIteration, then when the variable goes out-of-scope, a GeneratorExit will be raised at the point of the last yield that it ran.

If you then catch that GeneratorExit, and enter a new un-consumed loop (as in your `yield from foo()` line), then that line will also create the same situation again in a loop..

I understand that this used to "work" in previous python versions, but actually, having dug into things a lot more, I think the current behaviour is correct, and previous behaviors were not correct.

The "bug" here is in the example code that is catching GeneratorExit and then creating a new generator in the except:, rather than anything in Python
History
Date User Action Args
2020-12-29 16:43:35stestaggsetrecipients: + stestagg, xxm
2020-12-29 16:43:35stestaggsetmessageid: <1609260215.52.0.149559397124.issue42762@roundup.psfhosted.org>
2020-12-29 16:43:35stestagglinkissue42762 messages
2020-12-29 16:43:35stestaggcreate