classification
Title: yield expression inside generator expression does nothing
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Inyeol.Lee, Jim Fasarakis-Hilliard, belopolsky, benjamin.peterson, danielsh, emptysquare, erickt, esc24, georg.brandl, glyph, gvanrossum, ncoghlan, rhettinger, serhiy.storchaka, yselivanov
Priority: normal Keywords: patch

Created on 2010-11-26 19:21 by Inyeol.Lee, last changed 2018-02-04 08:53 by serhiy.storchaka. This issue is now closed.

Files
File name Uploaded Description Edit
yield-in-comprehensions.diff serhiy.storchaka, 2017-11-25 16:05
Pull Requests
URL Status Linked Edit
PR 4564 merged serhiy.storchaka, 2017-11-25 23:06
PR 4579 merged serhiy.storchaka, 2017-11-27 07:24
PR 4676 merged serhiy.storchaka, 2017-12-02 11:58
Messages (66)
msg122475 - (view) Author: Inyeol Lee (Inyeol.Lee) Date: 2010-11-26 19:21
Simple coroutine with for loop works:

>>> def pack_a():
        while True:
            L = []
            for i in range(2):
                L.append((yield))
            print(L)

>>> pa = pack_a()
>>> next(pa)
>>> pa.send(1)
>>> pa.send(2)
[1, 2]
>>>

If using list comprehension (generator expression), it fails:

>>> def pack_b():
        while True:
            L = [(yield) for i in range(2)]
            print(L)

>>> pb = pack_b()
<endless loop here>


I understand what's going on here - generator expression is converted to nested function and there's no way to either stop the execution inside nested function (since it's not started yet!) or send() a value to its yield expression. Still I think this behavior is a bug and needs fixed.

- best fix would make it behave the same as for loop.
- if it's not fixable, yield expression inside genexp should not be allowed.
msg122512 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2010-11-27 09:41
Hmm, what an interesting and unexpected side-effect of the efforts to hide the loop induction variable.
msg122645 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2010-11-28 10:14
While the behavior is confusing, I don't think yield inside comprehensions should be disallowed.  Rather, the fact that comprehensions have their own scope should be stated clearer.
msg122658 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2010-11-28 14:18
I think I can probably fix it, but it's debatable whether it should be done, since it'd make list comps more of "quasi" functions.
msg122705 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2010-11-28 19:03
This discussion should probably be moved to python-dev.  With tools like Twisted's inlineDefer or the Monocle package, there is a growing need to be able to use yield in complex expressions.  Yet, that goes against the trend toward making lists comps more like genexps and less like sugar for a simple for-loop accumulator.

Guido, do you have any thoughts on the subject?  Mark it a documentation issue or try out Benjamin's fix?
msg122759 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2010-11-29 03:13
I think it is definitely wrong the way it works in 3.x.  (Especially since it works as expected in 2.x.)

I agree with Inyeol's preference of fixes: (1) make it work properly for listcomps as well as genexps, (2) if that's not possible, forbid yield in a genexp or listcomp.

Note that even though yield in a genexp could be considered as having a well-defined meaning, that meaning is not useful and I would consider it as merely a coincidence of the specification, not an intentional effect.  So I would be fine changing its meaning.  (My assumption is that since it is not useful there is -- almost -- no code depending on that meaning.)
msg122760 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2010-11-29 03:14
PS. Wasn't there a similar issue with something inside a genexp that raises StopIteration?  Did we ever solve that?
msg122762 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-11-29 03:32
Isn't this the same issue as #3267?
msg122765 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2010-11-29 04:01
Yes it is, but I was never asked about it back then.
msg123641 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2010-12-08 20:11
#3267 did not expose endless loop possibility and was closed as won't fix.
Rather than reopen that and close this and move nosy list back, I added to nosy list here.
msg123649 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2010-12-08 20:40
FWIW, the "endless loop possibility" is not of issue here, and is simply an artifact of the specific generator function the OP uses.
msg241410 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2015-04-18 12:02
I would like to add that since the introduction of asyncio module that heavily uses "yield from" syntax, binding of yield inside comprehensions/generator expressions could lead to unexpected results/confusing behavior. See for example this question on SO: http://stackoverflow.com/questions/29334054/why-am-i-getting-different-results-when-using-a-list-comprehension-with-coroutin
msg286273 - (view) Author: Glyph Lefkowitz (glyph) (Python triager) Date: 2017-01-25 19:29
Is the fact that 'await' produces a syntax error in this context the same bug or a new one?
msg286274 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2017-01-25 19:36
> Is the fact that 'await' produces a syntax error in this context the same bug or a new one?

What kind of SyntaxError? await outside an async function is prohibited, bare await is also prohibited.
msg286280 - (view) Author: Glyph Lefkowitz (glyph) (Python triager) Date: 2017-01-25 22:15
>>> async def foo():
...     bar = [await x for x in range(10)]
  File "<input>", line 2
SyntaxError: 'await' expressions in comprehensions are not supported
msg286281 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2017-01-25 22:24
Python 3.5 does not support this, you should use Python 3.6 (plus await x will fail when you will run the coroutine, since you cannot await on int).
msg286282 - (view) Author: Glyph Lefkowitz (glyph) (Python triager) Date: 2017-01-25 22:32
OK, cool. So, long term, there will be a way to do this (suspend within a generator expression). Thanks for the pointer.
msg286283 - (view) Author: Glyph Lefkowitz (glyph) (Python triager) Date: 2017-01-25 22:32
(As far as awaiting on int, yes, I know how await works, I was focusing on the syntax.)
msg286407 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2017-01-28 11:16
Just to add my comment to this 7-years-old never-resolved issue: in PyPy 3.5, which behaves like Python 3.x in this respect, I made the following constructions give a warning.

    def wrong_listcomp():
        return [(yield 42) for i in j]
    def wrong_gencomp():
        return ((yield 42) for i in j)
    def wrong_dictcomp():
        return {(yield 42):2 for i in j}
    def wrong_setcomp():
        return {(yield 42) for i in j}

SyntaxWarning: 'yield' inside a list or generator comprehension behaves unexpectedly (http://bugs.python.org/issue10544)

The motivation is that none of the constructions above gives the "expected" result.  In more details:

- wrong_listcomp() doesn't even return a list at all.  It's possible to have a clue about why this occurs, but I would say that it is just plain wrong given the ``return [...]`` part of the syntax.  The same is true for wrong_dictcomp() and wrong_setcomp().

- wrong_gencomp() returns a generator as expected.  However, it is a generator that yields two elements for each i in j: first 42, and then whatever was ``send()`` into the generator.  I would say that it is in contradiction with the general idea that this syntax should give a generator that yields one item for each i in j.  In fact, when the user writes such code he might be expecting the "yield" to apply to the function level instead of the genexpr level---but none of the functions above end up being themselves generators.

For completeness, I think there is no problem with "await" instead of "yield" in Python 3.6.

How about fixing CPython to raise SyntaxWarning or even SyntaxError?
msg286408 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2017-01-28 11:28
> How about fixing CPython to raise SyntaxWarning or even SyntaxError?

I think it is better to just fix the issue, i.e. make comprehensions be equivalent to for-loops even if they contain `yield`. (In particular this will lead to [(yield i) for i in range(5)] be SyntaxError outside function).

The example of `await` shows that it is possible without leaking the loop variable into enclosing scope.
msg286409 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2017-01-28 11:55
Let's see if the discussion goes anywhere or if this issue remains in limbo for the next 7 years.  In the meantime, if I may humbly make a suggestion: whether the final decision is to give SyntaxError or change the semantics, one or a few intermediate versions with a SyntaxWarning might be a good idea.
msg306784 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-11-23 06:34
"Just fix the issue" is easier said than done for the same reason that comprehensions were implemented the way they are now: lambda expressions still have to work.

That is, we need to maintain the invariant that:

    [x for x in iterable]
    {x for x in iterable}
    (k:v for k, v in iterable)
    (x for x in iterable)

give the same results (respectively) as:

    [(lambda: x)() for x in iterable]
    {(lambda: x)() for x in iterable}
    ((lambda: k)():(lambda: v)() for k, v in iterable)
    ((lambda: x)() for x in iterable)

Once you work through the implications of "We need the loop variable to visible to lexically nested scopes, but invisible in the containing scope", you're going to end up with something that looks enough like a nested function that the easiest to implement and explain option is to have it *be* a nested function.

I'd be fine with a resolution that forbade yield expressions directly inside implicit scopes, though.
msg306802 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-11-23 11:07
Also see https://bugs.python.org/issue1660500 for the original Python 3.0 change to hide the iteration variable.

While the test suite already covers some interesting scoping edge cases as result of that initial patch, I think one we're currently missing is the nested comprehension case:

    >>> [[x for x in range(1, i)] for i in range(2, 5)]
    [[1], [1, 2], [1, 2, 3]]
msg306804 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-11-23 11:27
I see nothing special in `[[x for x in range(1, i)] for i in range(2, 5)]`. This should be equivalent to:

__result = []; __i = None
try:
    for __i in range(2, 5):
        __result2 = []; __x = None
        try:
            for __x in range(1, __i)
                __result2.append(__x)
            __result.append(__result2)
        finally:
            del __result2, __x
finally:
    del __i
msg306805 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-11-23 11:28
Another slight variant to that test case to make sure the inner comprehension actually generates a closure reference in the current implementation:

    >>> [[x+y for x in range(2) for y in range(1, i)] for i in range(2, 5)]
    [[1, 2], [1, 2, 2, 3], [1, 2, 3, 2, 3, 4]]

These are the kind of challenges that drove of us towards the current implementation. While the status quo definitely has its downsides, those downsides at least all flow directly from "There's an implicit nested function there that you're not expecting".

Yury took all this into account when designing the interaction between `await` and comprehensions (which is why that's in a better state), but for `yield` we inherited the existing behaviour of any other nested function.
msg306806 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-11-23 11:37
This is straightforward.

__result = []; __i = None
try:
    for __i in range(2, 5):
        __result2 = []; __x = __y = None
        try:
            for __x in range(2):
                for __y in range(1, __i):
                    __result2.append(__x + __y)
            __result.append(__result2)
        finally:
            del __result2, __x, __y
finally:
    del __i
msg306807 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-11-23 11:38
Sure, you can technically do that, but you'll have to rewrite most of the symtable pass in the compiler, since you won't be able to re-use the current recursive descent algorithms (which are designed to handle function scopes). That ran afoul of "If the implementation is hard to explain, it's a bad idea."

Where that gets particularly nasty is when you nest comprehensions with the *same* iteration variable name:

    >>> [[i for i in range(2)] for i in range(2)]
    [[0, 1], [0, 1]]

Now your name generator needs to be clever enough to account for how many levels deep it is. You also run into the lexical scoping problem anyway once a lambda expression shows up:

    >>> [[(lambda i=i: i)() for i in range(2)] for i in range(2)]
    [[0, 1], [0, 1]]
    >>> [[(lambda: i)() for i in range(2)] for i in range(2)]
    [[0, 1], [0, 1]]

As noted earlier, not impossible to resolve, but for from the "simple" option that folks seem to believe it is (and which I thought it would be before I actually tried to implement it while respecting Python's lexical scoping rules).
msg306808 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2017-11-23 11:47
Nick,

> Yury took all this into account when designing the interaction between `await` and comprehensions (which is why that's in a better state), but for `yield` we inherited the existing behaviour of any other nested function.

Actually the fact that `await` in comprehensions already works right is one of my main arguments that we can also fix `yield`. So I am totally with Serhiy here.
msg306810 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-11-23 12:00
Consider this as a sort of optimization. A function inlining. In general case we can't inline a function because it can be rebinded at any time, but in this particular case we call a just created function that doesn't have any references and live only on the stack a, and we have all information for generating an inlined code. This will allow to get rid of the overhead for creating and calling a one-time function.

The naming problem is the hardest problem. We should assign unique names for internal variables, like '.0.x'. We already do this for the single parameter of a one-time function. Or we could just use a stack (this requires more work but has additional benefits).

If you want to think about comprehensions as about calling inner functions, you can do this. And you can think about comprehensions as about loops which don't leak inner variables. The behavior should be absolutely identical until you will use "yield" which would turn your list producing function into a generator. If make "yield" turning into a generator not an implicit inner function, but an explicit enclosing function, a comprehension could be represented via an implicit inner function. Currently there is no way to express this in Python syntax, but before introducing "yield" there was no way to express its semantic.
msg306812 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-11-23 12:03
I concur with Ivan. It would be nice to me to disallow "yield" in comprehensions and generator expressions, but the fact that "await" in comprehensions already works right, and this behavior is intentional and useful, make me thinking that "yield" should work in comprehensions.
msg306829 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2017-11-23 16:21
Remind me what happens when you use `await` in a generator expression that
survives the async function's scope?

async def f(xs):
    return (await x for x in xs)
msg306831 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2017-11-23 16:26
> Remind me what happens when you use `await` in a generator expression that survives the async function's scope?

Awaiting on f([1, 2]) will result in an async generator (even though yield never appears here). Yury explained why this happens in https://bugs.python.org/issue32113
msg306832 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2017-11-23 16:29
(Of course there should be not [1, 2] in the argument, but a list of some awaitables, otherwise there will be an error later.)
msg306833 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2017-11-23 16:32
> Yury explained why this happens in https://bugs.python.org/issue32113

It happens because '(x for x in xs)' creates a synchronous generator.  So when there's an 'await' in it, it creates an asynchronous generator.
msg306834 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2017-11-23 16:33
It's also important to note, that in 3.7, this is legal:

  def foo():
    return (await x for x in xs)

Note that 'foo' is a regular function, not 'async def'.
msg306835 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2017-11-23 16:36
... but [await x for x in xs] is still valid _only_ inside async def.
msg306836 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2017-11-23 16:38
> ... but [await x for x in xs] is still valid _only_ inside async def.

Yes, because it's computed right where it is defined.

a = [x for x in xs]  # `a` is a list object
a = (x for x in xs)  # `a` is a generator

Do you understand the difference?
msg306837 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2017-11-23 16:40
Well, after all I am thinking maybe it is indeed makes sense to ban `yield` inside both sync/async and both comprehensions/generator expressions. Since we already have a smörgåsbord of intuitive behaviors.

This, _together_ with extensive clarification to the docs and error messages can fix the problem.
msg306838 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2017-11-23 16:42
I honestly think we went too far here. Asynchronous generators are still
provisional. Though the PEP is silent about this, the acceptance notice is
clear:
https://mail.python.org/pipermail/python-dev/2016-September/146267.html. I
propose to rip them out.
msg306840 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2017-11-23 16:42
> Well, after all I am thinking maybe it is indeed makes sense to ban `yield` inside both sync/async and both comprehensions/generator expressions.

I agree, as I can't imagine a real use case for

   a = ((yield a) for a in as)

which is really equivalent to

   def a():
      for a in as:
         yield (yield a)
msg306841 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2017-11-23 16:44
> Do you understand the difference?

Yury, sorry, but what is your problem? Have I said something about that this is bad. Your tone is clearly insulting, and this is not the first time. Maybe it makes sense to have some respect?
msg306842 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2017-11-23 16:47
>> Do you understand the difference?

> Yury, sorry, but what is your problem? Have I said something about that this is bad. Your tone is clearly insulting, and this is not the first time. Maybe it makes sense to have some respect?

Sorry, I didn't mean to insult anybody.  I asked an honest question with an intent to clarify if there's some misunderstanding of the topic that I'm partially responsible for in CPython.
msg306843 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2017-11-23 16:50
Yury OK, sorry then this is a misunderstanding from my side.
msg306844 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2017-11-23 16:52
> Yury OK, sorry then this is a misunderstanding from my side.

NP.  Again, sorry if I sounded that way to you.
msg306845 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2017-11-23 16:54
Guido, I am not sure about the complete removal, this is probably to radical. What I think we are missing more detailed docs that would be clear about the corner cases. (I already mentioned this in https://bugs.python.org/issue32113)
msg306846 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2017-11-23 17:09
I think we all need to calm down a bit. How about not posting about this
topic for 24 hours.
msg306847 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2017-11-23 17:10
> How about not posting about this topic for 24 hours.

OK, makes sense :-)
msg306939 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-11-25 03:57
Given the direction of the python-dev thread, should we split this question into two issues?

Issue 1: a yield expression inside a comprehension changes the type of the expression result (returning a generator-iterator instead of the expected container type)

Issue 2: a yield expression inside a generator expression interacts weirdly with the genexp's implicit yield expression

I ask, as it seems to me that issue 1 can be addressed by wrapping the affected cases in an implicit 'yield from' expression, which will both fix the return type of the expression and turn the outer function into a generator (if it isn't one already). (I'm going to put together a proof-of-concept for that idea this weekend)

By contrast, the interaction between generator expressions and explicit yield expressions seems intrinsically confusing, so I'm not sure we can do any better than declaring it a syntax error to try to combine them.
msg306940 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-11-25 04:34
I realised that even without modifying the compiler first, I could illustrate the proposed `yield from` based resolution for the comprehension case by way of explicit yield from clauses:

```
def get_gen_result(gen, inputs):
    try:
        yield_value = next(gen)
        for send_value in inputs:
            print(f"Received: {yield_value}; Sending: {send_value}")
            yield_value = gen.send(send_value)
    except StopIteration as exc:
        return exc.value
    raise RuntimeError("Failed to exhaust generator")

def example():
    comp1 = yield from [str((yield x)) for x in ('1st', '2nd')]
    comp2 = yield from [int((yield x)) for x in ('3rd', '4th')]
    return comp1, comp2

>>> result = get_gen_result(example(), range(4))
Received: 1st; Sending: 0
Received: 2nd; Sending: 1
Received: 3rd; Sending: 2
Received: 4th; Sending: 3
>>> result
(['0', '1'], [2, 3])
```

So if we decided to make yield-in-a-comprehension imply the use of yield from, we'd only need:

- DeprecationWarning in 3.7 to say "this is going to imply 'yield from (comprehension)' in 3.8+"
- making the 'yield from' implicit in 3.8 (thus ensuring that comprehensions always return the correct container type, even when they include yield expressions)
msg306963 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2017-11-25 16:01
No to both. See python-dev.
msg306964 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-11-25 16:05
Here is a sample of the implementation of the Nick's idea.
msg306983 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-11-26 02:01
Serhiy's PR now implements the "Prohibit yield & yield from in generator expressions and comprehensions" approach discussed on python-dev (currently as a hard SyntaxError, but it could be amended to be a warning instead without too much difficulty).

The PR does highlight an interesting subtlety though: the easiest way to implement this still allows yield expressions in the outermost iterator, since that gets compiled in a different scope from the rest of the comprehension body (it's evaluated eagerly and passed in to the implicit nested function).

I'm thinking we probably don't want to expose that detail to end users, and will instead want to include a second check that prohibits yield expressions in the outermost iterator as well. However, I'm not entirely sure how we could implement such a check, short of adding a new "yield expression prohibited" counter in the AST generation and/or symbol analysis pass.
msg306995 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2017-11-26 06:22
No.

On Nov 25, 2017 18:01, "Nick Coghlan" <report@bugs.python.org> wrote:

>
> Nick Coghlan <ncoghlan@gmail.com> added the comment:
>
> Serhiy's PR now implements the "Prohibit yield & yield from in generator
> expressions and comprehensions" approach discussed on python-dev (currently
> as a hard SyntaxError, but it could be amended to be a warning instead
> without too much difficulty).
>
> The PR does highlight an interesting subtlety though: the easiest way to
> implement this still allows yield expressions in the outermost iterator,
> since that gets compiled in a different scope from the rest of the
> comprehension body (it's evaluated eagerly and passed in to the implicit
> nested function).
>
> I'm thinking we probably don't want to expose that detail to end users,
> and will instead want to include a second check that prohibits yield
> expressions in the outermost iterator as well. However, I'm not entirely
> sure how we could implement such a check, short of adding a new "yield
> expression prohibited" counter in the AST generation and/or symbol analysis
> pass.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue10544>
> _______________________________________
>
msg307013 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2017-11-26 17:58
To clarify, the outermost iterator is marked here:

    [func(x, y) for x in <outermost iterator> for y in <something else>]

and it is evaluated before the comprehension proper is started. This is
just part of how the language works (it's a similar rule as "in an
assignment the RHS is evaluated before it is assigned to the LHS").

I see no benefit in banning `yield` in <outermost iterator>.

On Sat, Nov 25, 2017 at 10:22 PM, Guido van Rossum <report@bugs.python.org>
wrote:

>
> Guido van Rossum <guido@python.org> added the comment:
>
> No.
>
> On Nov 25, 2017 18:01, "Nick Coghlan" <report@bugs.python.org> wrote:
>
> >
> > Nick Coghlan <ncoghlan@gmail.com> added the comment:
> >
> > Serhiy's PR now implements the "Prohibit yield & yield from in generator
> > expressions and comprehensions" approach discussed on python-dev
> (currently
> > as a hard SyntaxError, but it could be amended to be a warning instead
> > without too much difficulty).
> >
> > The PR does highlight an interesting subtlety though: the easiest way to
> > implement this still allows yield expressions in the outermost iterator,
> > since that gets compiled in a different scope from the rest of the
> > comprehension body (it's evaluated eagerly and passed in to the implicit
> > nested function).
> >
> > I'm thinking we probably don't want to expose that detail to end users,
> > and will instead want to include a second check that prohibits yield
> > expressions in the outermost iterator as well. However, I'm not entirely
> > sure how we could implement such a check, short of adding a new "yield
> > expression prohibited" counter in the AST generation and/or symbol
> analysis
> > pass.
> >
> > ----------
> >
> > _______________________________________
> > Python tracker <report@bugs.python.org>
> > <https://bugs.python.org/issue10544>
> > _______________________________________
> >
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue10544>
> _______________________________________
>
msg307033 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-11-27 04:00
Cool, if you're OK with that behaviour, it actually makes this a lot easier, since it means:

1. Serhiy's patch is already sufficient for the final hard compatibility break
2. It can be readily adapted to emit either DeprecationWarning or SyntaxWarning for 3.7
msg307035 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2017-11-27 05:05
Great. It should be a DeprecationWarning, since we're planning to disallow
it completely, right? IIRC SyntaxWarning is for syntax that we can't
deprecate.
msg307039 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-11-27 09:25
Guido, should we write this change up as a PEP, or are you happy to just cover it as a section in the What's New document for 3.7?
msg307051 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-11-27 11:20
Note that the DeprecationWarning exception is replaced with a SyntaxError to get a more accurate error report.

$ cat yield-gen.py 
def f():
    return ((yield x) for x in range(3))

$ ./python -Wd yield-gen.py
yield-gen.py:2: DeprecationWarning: 'yield' inside generator expression
  return ((yield x) for x in range(3))

$ ./python -We yield-gen.py
  File "yield-gen.py", line 2
    return ((yield x) for x in range(3))
           ^
SyntaxError: 'yield' inside generator expression


Without this replacement the result would be:

$ ./python -We yield-gen.py
DeprecationWarning: 'yield' inside generator expression
msg307067 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2017-11-27 16:40
After the tiresome debate I am happy to see this just as a "what's new" entry rather than soliciting more debate with a PEP. (However there may be some existing PEP that suggests it should work? That PEP should be amended with a note that this is being deprecated. But I don't know if there is such a PEP.)
msg307091 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-11-27 22:01
As far as I'm aware, there's nothing that specifically promises these constructs will do anything in Py3 at all - the existing behaviour is just an accident of implementation arising from the way nested scopes and yield expressions interact in general.

Tinkering with "await" in comprehensions and generator expressions would be different, since PEP 530 actually does make promises about how we expect that to work.
msg307095 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2017-11-27 22:07
OK, great.
msg307359 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-12-01 04:54
New changeset 73a7e9b10b2ec9636e3c6396cf7b3695f8ed1856 by Nick Coghlan (Serhiy Storchaka) in branch 'master':
bpo-10544: Deprecate "yield" in comprehensions and generator expressions. (GH-4579)
https://github.com/python/cpython/commit/73a7e9b10b2ec9636e3c6396cf7b3695f8ed1856
msg307361 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-12-01 04:58
With Serhiy's patch merged, I'm marking this as resolved. Thanks all!

https://bugs.python.org/issue32189 is the follow-up issue to turn the warning into an unconditional SyntaxError in 3.8.
msg307426 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-12-02 12:14
PR 4676 backports warnings to 2.7 in Py3k mode.
msg307449 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-12-02 19:00
New changeset 65d1887170fb278c10a836e9e4319cae4707f524 by Serhiy Storchaka in branch '2.7':
[2.7] bpo-10544: Deprecate "yield" in comprehensions and generator expressions in Py3k mode. (GH-4579) (#4676)
https://github.com/python/cpython/commit/65d1887170fb278c10a836e9e4319cae4707f524
msg311593 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-02-04 08:53
New changeset 07ca9afaa8768b44baf816b4998d209ed3e0088f by Serhiy Storchaka in branch 'master':
bpo-10544: Disallow "yield" in comprehensions and generator expressions. (GH-4564)
https://github.com/python/cpython/commit/07ca9afaa8768b44baf816b4998d209ed3e0088f
History
Date User Action Args
2018-05-09 08:40:47serhiy.storchakalinkissue6673 superseder
2018-02-04 08:53:53serhiy.storchakasetmessages: + msg311593
2017-12-02 19:01:38serhiy.storchakasetstatus: open -> closed
2017-12-02 19:00:11serhiy.storchakasetmessages: + msg307449
2017-12-02 12:14:01serhiy.storchakasetstatus: closed -> open

messages: + msg307426
2017-12-02 11:58:08serhiy.storchakasetpull_requests: + pull_request4584
2017-12-01 04:58:26ncoghlansetstatus: open -> closed
versions: - Python 3.6
messages: + msg307361

resolution: fixed
stage: patch review -> resolved
2017-12-01 04:54:24ncoghlansetmessages: + msg307359
2017-11-27 22:07:18gvanrossumsetmessages: + msg307095
2017-11-27 22:01:47ncoghlansetmessages: + msg307091
2017-11-27 16:40:06gvanrossumsetmessages: + msg307067
2017-11-27 11:20:19serhiy.storchakasetmessages: + msg307051
2017-11-27 09:25:30ncoghlansetmessages: + msg307039
2017-11-27 07:24:12serhiy.storchakasetpull_requests: + pull_request4506
2017-11-27 05:05:36gvanrossumsetmessages: + msg307035
2017-11-27 04:00:32ncoghlansetmessages: + msg307033
2017-11-26 17:58:23gvanrossumsetmessages: + msg307013
2017-11-26 06:22:42gvanrossumsetmessages: + msg306995
2017-11-26 02:01:22ncoghlansetmessages: + msg306983
2017-11-25 23:06:05serhiy.storchakasetstage: needs patch -> patch review
pull_requests: + pull_request4492
2017-11-25 16:10:50levkivskyisetnosy: - levkivskyi
2017-11-25 16:05:13serhiy.storchakasetfiles: + yield-in-comprehensions.diff
keywords: + patch
messages: + msg306964
2017-11-25 16:01:53gvanrossumsetmessages: + msg306963
2017-11-25 04:34:28ncoghlansetmessages: + msg306940
2017-11-25 03:57:17ncoghlansetmessages: + msg306939
2017-11-24 13:06:18arigosetnosy: - arigo
2017-11-23 17:10:51levkivskyisetmessages: + msg306847
2017-11-23 17:09:03gvanrossumsetmessages: + msg306846
2017-11-23 16:54:38levkivskyisetmessages: + msg306845
2017-11-23 16:52:02yselivanovsetmessages: + msg306844
2017-11-23 16:50:02levkivskyisetmessages: + msg306843
2017-11-23 16:47:52yselivanovsetmessages: + msg306842
2017-11-23 16:44:07levkivskyisetmessages: + msg306841
2017-11-23 16:42:36yselivanovsetmessages: + msg306840
2017-11-23 16:42:07gvanrossumsetmessages: + msg306838
2017-11-23 16:40:00levkivskyisetmessages: + msg306837
2017-11-23 16:38:06yselivanovsetmessages: + msg306836
2017-11-23 16:36:33levkivskyisetmessages: + msg306835
2017-11-23 16:33:09yselivanovsetmessages: + msg306834
2017-11-23 16:32:06yselivanovsetmessages: + msg306833
2017-11-23 16:29:33levkivskyisetmessages: + msg306832
2017-11-23 16:26:26levkivskyisetmessages: + msg306831
2017-11-23 16:21:14gvanrossumsetmessages: + msg306829
2017-11-23 12:03:43serhiy.storchakasetmessages: + msg306812
2017-11-23 12:00:03serhiy.storchakasetmessages: + msg306810
2017-11-23 11:47:34levkivskyisetnosy: + yselivanov
messages: + msg306808
2017-11-23 11:38:01ncoghlansetmessages: + msg306807
2017-11-23 11:37:35serhiy.storchakasetmessages: + msg306806
2017-11-23 11:28:58ncoghlansetmessages: + msg306805
2017-11-23 11:27:57serhiy.storchakasetmessages: + msg306804
2017-11-23 11:07:19ncoghlansetmessages: + msg306802
2017-11-23 06:34:21ncoghlansetmessages: + msg306784
2017-11-23 01:30:53ncoghlansetnosy: + ncoghlan
2017-11-22 20:35:31levkivskyisetassignee: levkivskyi ->
2017-11-22 14:12:55serhiy.storchakasetnosy: + serhiy.storchaka

versions: + Python 3.6, Python 3.7, - Python 3.5
2017-07-20 09:09:47levkivskyisetassignee: levkivskyi
2017-07-20 08:57:50Jim Fasarakis-Hilliardsetnosy: + Jim Fasarakis-Hilliard
2017-01-28 11:55:04arigosetmessages: + msg286409
2017-01-28 11:28:57levkivskyisetmessages: + msg286408
2017-01-28 11:16:54arigosetnosy: + arigo
messages: + msg286407
2017-01-25 22:32:26glyphsetmessages: + msg286283
2017-01-25 22:32:06glyphsetmessages: + msg286282
2017-01-25 22:24:57levkivskyisetmessages: + msg286281
2017-01-25 22:15:12glyphsetmessages: + msg286280
2017-01-25 19:36:50levkivskyisetmessages: + msg286274
2017-01-25 19:29:11glyphsetnosy: + glyph
messages: + msg286273
2015-04-30 15:09:07emptysquaresetnosy: + emptysquare
2015-04-18 12:02:26levkivskyisetnosy: + levkivskyi
messages: + msg241410
2014-01-31 22:40:03yselivanovsetversions: + Python 3.5, - Python 3.3, Python 3.4
2013-11-05 03:16:14terry.reedysetnosy: - terry.reedy

versions: - Python 3.2
2013-01-15 03:32:52meador.ingesetstage: needs patch
2013-01-12 14:26:26danielshsetnosy: + danielsh
2013-01-12 00:35:42esc24setnosy: + esc24
2013-01-11 16:07:32brett.cannonsetversions: + Python 3.4, - Python 3.1
2013-01-11 16:07:15brett.cannonsetnosy: - brett.cannon

versions: + Python 3.3
2010-12-08 20:40:20georg.brandlsetmessages: + msg123649
2010-12-08 20:12:02terry.reedylinkissue3267 superseder
2010-12-08 20:11:45terry.reedysetnosy: + terry.reedy, brett.cannon, erickt
messages: + msg123641
2010-11-29 04:01:24gvanrossumsetmessages: + msg122765
2010-11-29 03:32:54belopolskysetmessages: + msg122762
2010-11-29 03:27:06belopolskysetnosy: + belopolsky
2010-11-29 03:14:41gvanrossumsetassignee: gvanrossum -> (no value)
2010-11-29 03:14:27gvanrossumsetmessages: + msg122760
2010-11-29 03:13:53gvanrossumsetmessages: + msg122759
2010-11-28 19:03:38rhettingersetversions: + Python 3.2
nosy: + gvanrossum

messages: + msg122705

assignee: gvanrossum
2010-11-28 14:18:38benjamin.petersonsetnosy: + benjamin.peterson
messages: + msg122658
2010-11-28 10:14:04georg.brandlsetnosy: + georg.brandl
messages: + msg122645
2010-11-27 09:41:46rhettingersetnosy: + rhettinger
messages: + msg122512
2010-11-26 19:21:17Inyeol.Leecreate