Simple coroutine with for loop works:

>>> def pack_a():
        while True:
            L = []
            for i in range(2):
                L.append((yield))
            print(L)

>>> pa = pack_a()
>>> next(pa)
>>> pa.send(1)
>>> pa.send(2)
[1, 2]
>>>

If using list comprehension (generator expression), it fails:

>>> def pack_b():
        while True:
            L = [(yield) for i in range(2)]
            print(L)

>>> pb = pack_b()
<endless loop here>


I understand what's going on here - generator expression is converted to nested function and there's no way to either stop the execution inside nested function (since it's not started yet!) or send() a value to its yield expression. Still I think this behavior is a bug and needs fixed.

- best fix would make it behave the same as for loop.
- if it's not fixable, yield expression inside genexp should not be allowed.

Hmm, what an interesting and unexpected side-effect of the efforts to hide the loop induction variable.

While the behavior is confusing, I don't think yield inside comprehensions should be disallowed.  Rather, the fact that comprehensions have their own scope should be stated clearer.

I think I can probably fix it, but it's debatable whether it should be done, since it'd make list comps more of "quasi" functions.

This discussion should probably be moved to python-dev.  With tools like Twisted's inlineDefer or the Monocle package, there is a growing need to be able to use yield in complex expressions.  Yet, that goes against the trend toward making lists comps more like genexps and less like sugar for a simple for-loop accumulator.

Guido, do you have any thoughts on the subject?  Mark it a documentation issue or try out Benjamin's fix?

I think it is definitely wrong the way it works in 3.x.  (Especially since it works as expected in 2.x.)

I agree with Inyeol's preference of fixes: (1) make it work properly for listcomps as well as genexps, (2) if that's not possible, forbid yield in a genexp or listcomp.

Note that even though yield in a genexp could be considered as having a well-defined meaning, that meaning is not useful and I would consider it as merely a coincidence of the specification, not an intentional effect.  So I would be fine changing its meaning.  (My assumption is that since it is not useful there is -- almost -- no code depending on that meaning.)

PS. Wasn't there a similar issue with something inside a genexp that raises StopIteration?  Did we ever solve that?

Isn't this the same issue as #3267?

Yes it is, but I was never asked about it back then.

#3267 did not expose endless loop possibility and was closed as won't fix.
Rather than reopen that and close this and move nosy list back, I added to nosy list here.

FWIW, the "endless loop possibility" is not of issue here, and is simply an artifact of the specific generator function the OP uses.

I would like to add that since the introduction of asyncio module that heavily uses "yield from" syntax, binding of yield inside comprehensions/generator expressions could lead to unexpected results/confusing behavior. See for example this question on SO: http://stackoverflow.com/questions/29334054/why-am-i-getting-different-results-when-using-a-list-comprehension-with-coroutin

Is the fact that 'await' produces a syntax error in this context the same bug or a new one?

> Is the fact that 'await' produces a syntax error in this context the same bug or a new one?

What kind of SyntaxError? await outside an async function is prohibited, bare await is also prohibited.

>>> async def foo():
...     bar = [await x for x in range(10)]
  File "<input>", line 2
SyntaxError: 'await' expressions in comprehensions are not supported

Python 3.5 does not support this, you should use Python 3.6 (plus await x will fail when you will run the coroutine, since you cannot await on int).

OK, cool. So, long term, there will be a way to do this (suspend within a generator expression). Thanks for the pointer.

(As far as awaiting on int, yes, I know how await works, I was focusing on the syntax.)

Just to add my comment to this 7-years-old never-resolved issue: in PyPy 3.5, which behaves like Python 3.x in this respect, I made the following constructions give a warning.

    def wrong_listcomp():
        return [(yield 42) for i in j]
    def wrong_gencomp():
        return ((yield 42) for i in j)
    def wrong_dictcomp():
        return {(yield 42):2 for i in j}
    def wrong_setcomp():
        return {(yield 42) for i in j}

SyntaxWarning: 'yield' inside a list or generator comprehension behaves unexpectedly (http://bugs.python.org/issue10544)

The motivation is that none of the constructions above gives the "expected" result.  In more details:

- wrong_listcomp() doesn't even return a list at all.  It's possible to have a clue about why this occurs, but I would say that it is just plain wrong given the ``return [...]`` part of the syntax.  The same is true for wrong_dictcomp() and wrong_setcomp().

- wrong_gencomp() returns a generator as expected.  However, it is a generator that yields two elements for each i in j: first 42, and then whatever was ``send()`` into the generator.  I would say that it is in contradiction with the general idea that this syntax should give a generator that yields one item for each i in j.  In fact, when the user writes such code he might be expecting the "yield" to apply to the function level instead of the genexpr level---but none of the functions above end up being themselves generators.

For completeness, I think there is no problem with "await" instead of "yield" in Python 3.6.

How about fixing CPython to raise SyntaxWarning or even SyntaxError?

> How about fixing CPython to raise SyntaxWarning or even SyntaxError?

I think it is better to just fix the issue, i.e. make comprehensions be equivalent to for-loops even if they contain `yield`. (In particular this will lead to [(yield i) for i in range(5)] be SyntaxError outside function).

The example of `await` shows that it is possible without leaking the loop variable into enclosing scope.

Let's see if the discussion goes anywhere or if this issue remains in limbo for the next 7 years.  In the meantime, if I may humbly make a suggestion: whether the final decision is to give SyntaxError or change the semantics, one or a few intermediate versions with a SyntaxWarning might be a good idea.

"Just fix the issue" is easier said than done for the same reason that comprehensions were implemented the way they are now: lambda expressions still have to work.

That is, we need to maintain the invariant that:

    [x for x in iterable]
    {x for x in iterable}
    (k:v for k, v in iterable)
    (x for x in iterable)

give the same results (respectively) as:

    [(lambda: x)() for x in iterable]
    {(lambda: x)() for x in iterable}
    ((lambda: k)():(lambda: v)() for k, v in iterable)
    ((lambda: x)() for x in iterable)

Once you work through the implications of "We need the loop variable to visible to lexically nested scopes, but invisible in the containing scope", you're going to end up with something that looks enough like a nested function that the easiest to implement and explain option is to have it *be* a nested function.

I'd be fine with a resolution that forbade yield expressions directly inside implicit scopes, though.

Also see https://bugs.python.org/issue1660500 for the original Python 3.0 change to hide the iteration variable.

While the test suite already covers some interesting scoping edge cases as result of that initial patch, I think one we're currently missing is the nested comprehension case:

    >>> [[x for x in range(1, i)] for i in range(2, 5)]
    [[1], [1, 2], [1, 2, 3]]

I see nothing special in `[[x for x in range(1, i)] for i in range(2, 5)]`. This should be equivalent to:

__result = []; __i = None
try:
    for __i in range(2, 5):
        __result2 = []; __x = None
        try:
            for __x in range(1, __i)
                __result2.append(__x)
            __result.append(__result2)
        finally:
            del __result2, __x
finally:
    del __i

Another slight variant to that test case to make sure the inner comprehension actually generates a closure reference in the current implementation:

    >>> [[x+y for x in range(2) for y in range(1, i)] for i in range(2, 5)]
    [[1, 2], [1, 2, 2, 3], [1, 2, 3, 2, 3, 4]]

These are the kind of challenges that drove of us towards the current implementation. While the status quo definitely has its downsides, those downsides at least all flow directly from "There's an implicit nested function there that you're not expecting".

Yury took all this into account when designing the interaction between `await` and comprehensions (which is why that's in a better state), but for `yield` we inherited the existing behaviour of any other nested function.

This is straightforward.

__result = []; __i = None
try:
    for __i in range(2, 5):
        __result2 = []; __x = __y = None
        try:
            for __x in range(2):
                for __y in range(1, __i):
                    __result2.append(__x + __y)
            __result.append(__result2)
        finally:
            del __result2, __x, __y
finally:
    del __i

Sure, you can technically do that, but you'll have to rewrite most of the symtable pass in the compiler, since you won't be able to re-use the current recursive descent algorithms (which are designed to handle function scopes). That ran afoul of "If the implementation is hard to explain, it's a bad idea."

Where that gets particularly nasty is when you nest comprehensions with the *same* iteration variable name:

    >>> [[i for i in range(2)] for i in range(2)]
    [[0, 1], [0, 1]]

Now your name generator needs to be clever enough to account for how many levels deep it is. You also run into the lexical scoping problem anyway once a lambda expression shows up:

    >>> [[(lambda i=i: i)() for i in range(2)] for i in range(2)]
    [[0, 1], [0, 1]]
    >>> [[(lambda: i)() for i in range(2)] for i in range(2)]
    [[0, 1], [0, 1]]

As noted earlier, not impossible to resolve, but for from the "simple" option that folks seem to believe it is (and which I thought it would be before I actually tried to implement it while respecting Python's lexical scoping rules).

Nick,

> Yury took all this into account when designing the interaction between `await` and comprehensions (which is why that's in a better state), but for `yield` we inherited the existing behaviour of any other nested function.

Actually the fact that `await` in comprehensions already works right is one of my main arguments that we can also fix `yield`. So I am totally with Serhiy here.

Consider this as a sort of optimization. A function inlining. In general case we can't inline a function because it can be rebinded at any time, but in this particular case we call a just created function that doesn't have any references and live only on the stack a, and we have all information for generating an inlined code. This will allow to get rid of the overhead for creating and calling a one-time function.

The naming problem is the hardest problem. We should assign unique names for internal variables, like '.0.x'. We already do this for the single parameter of a one-time function. Or we could just use a stack (this requires more work but has additional benefits).

If you want to think about comprehensions as about calling inner functions, you can do this. And you can think about comprehensions as about loops which don't leak inner variables. The behavior should be absolutely identical until you will use "yield" which would turn your list producing function into a generator. If make "yield" turning into a generator not an implicit inner function, but an explicit enclosing function, a comprehension could be represented via an implicit inner function. Currently there is no way to express this in Python syntax, but before introducing "yield" there was no way to express its semantic.

I concur with Ivan. It would be nice to me to disallow "yield" in comprehensions and generator expressions, but the fact that "await" in comprehensions already works right, and this behavior is intentional and useful, make me thinking that "yield" should work in comprehensions.

Remind me what happens when you use `await` in a generator expression that
survives the async function's scope?

async def f(xs):
    return (await x for x in xs)

> Remind me what happens when you use `await` in a generator expression that survives the async function's scope?

Awaiting on f([1, 2]) will result in an async generator (even though yield never appears here). Yury explained why this happens in https://bugs.python.org/issue32113

(Of course there should be not [1, 2] in the argument, but a list of some awaitables, otherwise there will be an error later.)

> Yury explained why this happens in https://bugs.python.org/issue32113

It happens because '(x for x in xs)' creates a synchronous generator.  So when there's an 'await' in it, it creates an asynchronous generator.

It's also important to note, that in 3.7, this is legal:

  def foo():
    return (await x for x in xs)

Note that 'foo' is a regular function, not 'async def'.

... but [await x for x in xs] is still valid _only_ inside async def.

> ... but [await x for x in xs] is still valid _only_ inside async def.

Yes, because it's computed right where it is defined.

a = [x for x in xs]  # `a` is a list object
a = (x for x in xs)  # `a` is a generator

Do you understand the difference?

Well, after all I am thinking maybe it is indeed makes sense to ban `yield` inside both sync/async and both comprehensions/generator expressions. Since we already have a smörgåsbord of intuitive behaviors.

This, _together_ with extensive clarification to the docs and error messages can fix the problem.

I honestly think we went too far here. Asynchronous generators are still
provisional. Though the PEP is silent about this, the acceptance notice is
clear:
https://mail.python.org/pipermail/python-dev/2016-September/146267.html. I
propose to rip them out.

> Well, after all I am thinking maybe it is indeed makes sense to ban `yield` inside both sync/async and both comprehensions/generator expressions.

I agree, as I can't imagine a real use case for

   a = ((yield a) for a in as)

which is really equivalent to

   def a():
      for a in as:
         yield (yield a)

> Do you understand the difference?

Yury, sorry, but what is your problem? Have I said something about that this is bad. Your tone is clearly insulting, and this is not the first time. Maybe it makes sense to have some respect?

>> Do you understand the difference?

> Yury, sorry, but what is your problem? Have I said something about that this is bad. Your tone is clearly insulting, and this is not the first time. Maybe it makes sense to have some respect?

Sorry, I didn't mean to insult anybody.  I asked an honest question with an intent to clarify if there's some misunderstanding of the topic that I'm partially responsible for in CPython.

Yury OK, sorry then this is a misunderstanding from my side.

> Yury OK, sorry then this is a misunderstanding from my side.

NP.  Again, sorry if I sounded that way to you.

Guido, I am not sure about the complete removal, this is probably to radical. What I think we are missing more detailed docs that would be clear about the corner cases. (I already mentioned this in https://bugs.python.org/issue32113)

I think we all need to calm down a bit. How about not posting about this
topic for 24 hours.

> How about not posting about this topic for 24 hours.

OK, makes sense :-)

Given the direction of the python-dev thread, should we split this question into two issues?

Issue 1: a yield expression inside a comprehension changes the type of the expression result (returning a generator-iterator instead of the expected container type)

Issue 2: a yield expression inside a generator expression interacts weirdly with the genexp's implicit yield expression

I ask, as it seems to me that issue 1 can be addressed by wrapping the affected cases in an implicit 'yield from' expression, which will both fix the return type of the expression and turn the outer function into a generator (if it isn't one already). (I'm going to put together a proof-of-concept for that idea this weekend)

By contrast, the interaction between generator expressions and explicit yield expressions seems intrinsically confusing, so I'm not sure we can do any better than declaring it a syntax error to try to combine them.

I realised that even without modifying the compiler first, I could illustrate the proposed `yield from` based resolution for the comprehension case by way of explicit yield from clauses:

```
def get_gen_result(gen, inputs):
    try:
        yield_value = next(gen)
        for send_value in inputs:
            print(f"Received: {yield_value}; Sending: {send_value}")
            yield_value = gen.send(send_value)
    except StopIteration as exc:
        return exc.value
    raise RuntimeError("Failed to exhaust generator")

def example():
    comp1 = yield from [str((yield x)) for x in ('1st', '2nd')]
    comp2 = yield from [int((yield x)) for x in ('3rd', '4th')]
    return comp1, comp2

>>> result = get_gen_result(example(), range(4))
Received: 1st; Sending: 0
Received: 2nd; Sending: 1
Received: 3rd; Sending: 2
Received: 4th; Sending: 3
>>> result
(['0', '1'], [2, 3])
```

So if we decided to make yield-in-a-comprehension imply the use of yield from, we'd only need:

- DeprecationWarning in 3.7 to say "this is going to imply 'yield from (comprehension)' in 3.8+"
- making the 'yield from' implicit in 3.8 (thus ensuring that comprehensions always return the correct container type, even when they include yield expressions)

No to both. See python-dev.

Here is a sample of the implementation of the Nick's idea.

Serhiy's PR now implements the "Prohibit yield & yield from in generator expressions and comprehensions" approach discussed on python-dev (currently as a hard SyntaxError, but it could be amended to be a warning instead without too much difficulty).

The PR does highlight an interesting subtlety though: the easiest way to implement this still allows yield expressions in the outermost iterator, since that gets compiled in a different scope from the rest of the comprehension body (it's evaluated eagerly and passed in to the implicit nested function).

I'm thinking we probably don't want to expose that detail to end users, and will instead want to include a second check that prohibits yield expressions in the outermost iterator as well. However, I'm not entirely sure how we could implement such a check, short of adding a new "yield expression prohibited" counter in the AST generation and/or symbol analysis pass.

No.

On Nov 25, 2017 18:01, "Nick Coghlan" <report@bugs.python.org> wrote:

>
> Nick Coghlan <ncoghlan@gmail.com> added the comment:
>
> Serhiy's PR now implements the "Prohibit yield & yield from in generator
> expressions and comprehensions" approach discussed on python-dev (currently
> as a hard SyntaxError, but it could be amended to be a warning instead
> without too much difficulty).
>
> The PR does highlight an interesting subtlety though: the easiest way to
> implement this still allows yield expressions in the outermost iterator,
> since that gets compiled in a different scope from the rest of the
> comprehension body (it's evaluated eagerly and passed in to the implicit
> nested function).
>
> I'm thinking we probably don't want to expose that detail to end users,
> and will instead want to include a second check that prohibits yield
> expressions in the outermost iterator as well. However, I'm not entirely
> sure how we could implement such a check, short of adding a new "yield
> expression prohibited" counter in the AST generation and/or symbol analysis
> pass.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue10544>
> _______________________________________
>

To clarify, the outermost iterator is marked here:

    [func(x, y) for x in <outermost iterator> for y in <something else>]

and it is evaluated before the comprehension proper is started. This is
just part of how the language works (it's a similar rule as "in an
assignment the RHS is evaluated before it is assigned to the LHS").

I see no benefit in banning `yield` in <outermost iterator>.

On Sat, Nov 25, 2017 at 10:22 PM, Guido van Rossum <report@bugs.python.org>
wrote:

>
> Guido van Rossum <guido@python.org> added the comment:
>
> No.
>
> On Nov 25, 2017 18:01, "Nick Coghlan" <report@bugs.python.org> wrote:
>
> >
> > Nick Coghlan <ncoghlan@gmail.com> added the comment:
> >
> > Serhiy's PR now implements the "Prohibit yield & yield from in generator
> > expressions and comprehensions" approach discussed on python-dev
> (currently
> > as a hard SyntaxError, but it could be amended to be a warning instead
> > without too much difficulty).
> >
> > The PR does highlight an interesting subtlety though: the easiest way to
> > implement this still allows yield expressions in the outermost iterator,
> > since that gets compiled in a different scope from the rest of the
> > comprehension body (it's evaluated eagerly and passed in to the implicit
> > nested function).
> >
> > I'm thinking we probably don't want to expose that detail to end users,
> > and will instead want to include a second check that prohibits yield
> > expressions in the outermost iterator as well. However, I'm not entirely
> > sure how we could implement such a check, short of adding a new "yield
> > expression prohibited" counter in the AST generation and/or symbol
> analysis
> > pass.
> >
> > ----------
> >
> > _______________________________________
> > Python tracker <report@bugs.python.org>
> > <https://bugs.python.org/issue10544>
> > _______________________________________
> >
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue10544>
> _______________________________________
>

Cool, if you're OK with that behaviour, it actually makes this a lot easier, since it means:

1. Serhiy's patch is already sufficient for the final hard compatibility break
2. It can be readily adapted to emit either DeprecationWarning or SyntaxWarning for 3.7

Great. It should be a DeprecationWarning, since we're planning to disallow
it completely, right? IIRC SyntaxWarning is for syntax that we can't
deprecate.

Guido, should we write this change up as a PEP, or are you happy to just cover it as a section in the What's New document for 3.7?

Note that the DeprecationWarning exception is replaced with a SyntaxError to get a more accurate error report.

$ cat yield-gen.py 
def f():
    return ((yield x) for x in range(3))

$ ./python -Wd yield-gen.py
yield-gen.py:2: DeprecationWarning: 'yield' inside generator expression
  return ((yield x) for x in range(3))

$ ./python -We yield-gen.py
  File "yield-gen.py", line 2
    return ((yield x) for x in range(3))
           ^
SyntaxError: 'yield' inside generator expression


Without this replacement the result would be:

$ ./python -We yield-gen.py
DeprecationWarning: 'yield' inside generator expression

After the tiresome debate I am happy to see this just as a "what's new" entry rather than soliciting more debate with a PEP. (However there may be some existing PEP that suggests it should work? That PEP should be amended with a note that this is being deprecated. But I don't know if there is such a PEP.)

As far as I'm aware, there's nothing that specifically promises these constructs will do anything in Py3 at all - the existing behaviour is just an accident of implementation arising from the way nested scopes and yield expressions interact in general.

Tinkering with "await" in comprehensions and generator expressions would be different, since PEP 530 actually does make promises about how we expect that to work.

OK, great.

New changeset 73a7e9b10b2ec9636e3c6396cf7b3695f8ed1856 by Nick Coghlan (Serhiy Storchaka) in branch 'master':
bpo-10544: Deprecate "yield" in comprehensions and generator expressions. (GH-4579)
https://github.com/python/cpython/commit/73a7e9b10b2ec9636e3c6396cf7b3695f8ed1856

With Serhiy's patch merged, I'm marking this as resolved. Thanks all!

https://bugs.python.org/issue32189 is the follow-up issue to turn the warning into an unconditional SyntaxError in 3.8.

PR 4676 backports warnings to 2.7 in Py3k mode.

New changeset 65d1887170fb278c10a836e9e4319cae4707f524 by Serhiy Storchaka in branch '2.7':
[2.7] bpo-10544: Deprecate "yield" in comprehensions and generator expressions in Py3k mode. (GH-4579) (#4676)
https://github.com/python/cpython/commit/65d1887170fb278c10a836e9e4319cae4707f524

New changeset 07ca9afaa8768b44baf816b4998d209ed3e0088f by Serhiy Storchaka in branch 'master':
bpo-10544: Disallow "yield" in comprehensions and generator expressions. (GH-4564)
https://github.com/python/cpython/commit/07ca9afaa8768b44baf816b4998d209ed3e0088f