Simple coroutine with for loop works:

>>> def pack_a():
        while True:
            L = []
            for i in range(2):
                L.append((yield))
            print(L)

>>> pa = pack_a()
>>> next(pa)
>>> pa.send(1)
>>> pa.send(2)
[1, 2]
>>>

If using list comprehension (generator expression), it fails:

>>> def pack_b():
        while True:
            L = [(yield) for i in range(2)]
            print(L)

>>> pb = pack_b()
<endless loop here>

I understand what's going on here - generator expression is converted to nested function and there's no way to either stop the execution inside nested function (since it's not started yet!) or send() a value to its yield expression. Still I think this behavior is a bug and needs fixed.

• best fix would make it behave the same as for loop.
• if it's not fixable, yield expression inside genexp should not be allowed.

Hmm, what an interesting and unexpected side-effect of the efforts to hide the loop induction variable.

While the behavior is confusing, I don't think yield inside comprehensions should be disallowed. Rather, the fact that comprehensions have their own scope should be stated clearer.

I think I can probably fix it, but it's debatable whether it should be done, since it'd make list comps more of "quasi" functions.

This discussion should probably be moved to python-dev. With tools like Twisted's inlineDefer or the Monocle package, there is a growing need to be able to use yield in complex expressions. Yet, that goes against the trend toward making lists comps more like genexps and less like sugar for a simple for-loop accumulator.

Guido, do you have any thoughts on the subject? Mark it a documentation issue or try out Benjamin's fix?

I think it is definitely wrong the way it works in 3.x. (Especially since it works as expected in 2.x.)

I agree with Inyeol's preference of fixes: (1) make it work properly for listcomps as well as genexps, (2) if that's not possible, forbid yield in a genexp or listcomp.

Note that even though yield in a genexp could be considered as having a well-defined meaning, that meaning is not useful and I would consider it as merely a coincidence of the specification, not an intentional effect. So I would be fine changing its meaning. (My assumption is that since it is not useful there is -- almost -- no code depending on that meaning.)

PS. Wasn't there a similar issue with something inside a genexp that raises StopIteration? Did we ever solve that?

Isn't this the same issue as bpo-3267?

Yes it is, but I was never asked about it back then.

bpo-3267 did not expose endless loop possibility and was closed as won't fix.
Rather than reopen that and close this and move nosy list back, I added to nosy list here.

FWIW, the "endless loop possibility" is not of issue here, and is simply an artifact of the specific generator function the OP uses.

I would like to add that since the introduction of asyncio module that heavily uses "yield from" syntax, binding of yield inside comprehensions/generator expressions could lead to unexpected results/confusing behavior. See for example this question on SO: http://stackoverflow.com/questions/29334054/why-am-i-getting-different-results-when-using-a-list-comprehension-with-coroutin

Is the fact that 'await' produces a syntax error in this context the same bug or a new one?

Is the fact that 'await' produces a syntax error in this context the same bug or a new one?

What kind of SyntaxError? await outside an async function is prohibited, bare await is also prohibited.

Python 3.5 does not support this, you should use Python 3.6 (plus await x will fail when you will run the coroutine, since you cannot await on int).

OK, cool. So, long term, there will be a way to do this (suspend within a generator expression). Thanks for the pointer.

(As far as awaiting on int, yes, I know how await works, I was focusing on the syntax.)

Just to add my comment to this 7-years-old never-resolved issue: in PyPy 3.5, which behaves like Python 3.x in this respect, I made the following constructions give a warning.

    def wrong_listcomp():
        return [(yield 42) for i in j]
    def wrong_gencomp():
        return ((yield 42) for i in j)
    def wrong_dictcomp():
        return {(yield 42):2 for i in j}
    def wrong_setcomp():
        return {(yield 42) for i in j}

SyntaxWarning: 'yield' inside a list or generator comprehension behaves unexpectedly (http://bugs.python.org/issue10544)

The motivation is that none of the constructions above gives the "expected" result. In more details:

• wrong_listcomp() doesn't even return a list at all. It's possible to have a clue about why this occurs, but I would say that it is just plain wrong given the return [...] part of the syntax. The same is true for wrong_dictcomp() and wrong_setcomp().

• wrong_gencomp() returns a generator as expected. However, it is a generator that yields two elements for each i in j: first 42, and then whatever was send() into the generator. I would say that it is in contradiction with the general idea that this syntax should give a generator that yields one item for each i in j. In fact, when the user writes such code he might be expecting the "yield" to apply to the function level instead of the genexpr level---but none of the functions above end up being themselves generators.

For completeness, I think there is no problem with "await" instead of "yield" in Python 3.6.

How about fixing CPython to raise SyntaxWarning or even SyntaxError?

How about fixing CPython to raise SyntaxWarning or even SyntaxError?

I think it is better to just fix the issue, i.e. make comprehensions be equivalent to for-loops even if they contain yield. (In particular this will lead to [(yield i) for i in range(5)] be SyntaxError outside function).

The example of await shows that it is possible without leaking the loop variable into enclosing scope.

Let's see if the discussion goes anywhere or if this issue remains in limbo for the next 7 years. In the meantime, if I may humbly make a suggestion: whether the final decision is to give SyntaxError or change the semantics, one or a few intermediate versions with a SyntaxWarning might be a good idea.

"Just fix the issue" is easier said than done for the same reason that comprehensions were implemented the way they are now: lambda expressions still have to work.

That is, we need to maintain the invariant that:

[x for x in iterable]
{x for x in iterable}
(k:v for k, v in iterable)
(x for x in iterable)

give the same results (respectively) as:

[(lambda: x)() for x in iterable]
{(lambda: x)() for x in iterable}
((lambda: k)():(lambda: v)() for k, v in iterable)
((lambda: x)() for x in iterable)

Once you work through the implications of "We need the loop variable to visible to lexically nested scopes, but invisible in the containing scope", you're going to end up with something that looks enough like a nested function that the easiest to implement and explain option is to have it *be* a nested function.

I'd be fine with a resolution that forbade yield expressions directly inside implicit scopes, though.

Also see https://bugs.python.org/issue1660500 for the original Python 3.0 change to hide the iteration variable.

While the test suite already covers some interesting scoping edge cases as result of that initial patch, I think one we're currently missing is the nested comprehension case:

    >>> [[x for x in range(1, i)] for i in range(2, 5)]
    [[1], [1, 2], [1, 2, 3]]

Do you understand the difference?

Yury, sorry, but what is your problem? Have I said something about that this is bad. Your tone is clearly insulting, and this is not the first time. Maybe it makes sense to have some respect?

> Do you understand the difference?

Yury, sorry, but what is your problem? Have I said something about that this is bad. Your tone is clearly insulting, and this is not the first time. Maybe it makes sense to have some respect?

Sorry, I didn't mean to insult anybody. I asked an honest question with an intent to clarify if there's some misunderstanding of the topic that I'm partially responsible for in CPython.

Yury OK, sorry then this is a misunderstanding from my side.

Yury OK, sorry then this is a misunderstanding from my side.

NP. Again, sorry if I sounded that way to you.

Guido, I am not sure about the complete removal, this is probably to radical. What I think we are missing more detailed docs that would be clear about the corner cases. (I already mentioned this in https://bugs.python.org/issue32113)

I think we all need to calm down a bit. How about not posting about this
topic for 24 hours.

How about not posting about this topic for 24 hours.

OK, makes sense :-)

Given the direction of the python-dev thread, should we split this question into two issues?

Issue 1: a yield expression inside a comprehension changes the type of the expression result (returning a generator-iterator instead of the expected container type)

Issue 2: a yield expression inside a generator expression interacts weirdly with the genexp's implicit yield expression

I ask, as it seems to me that issue 1 can be addressed by wrapping the affected cases in an implicit 'yield from' expression, which will both fix the return type of the expression and turn the outer function into a generator (if it isn't one already). (I'm going to put together a proof-of-concept for that idea this weekend)

By contrast, the interaction between generator expressions and explicit yield expressions seems intrinsically confusing, so I'm not sure we can do any better than declaring it a syntax error to try to combine them.

I realised that even without modifying the compiler first, I could illustrate the proposed yield from based resolution for the comprehension case by way of explicit yield from clauses:

def get_gen_result(gen, inputs):
    try:
        yield_value = next(gen)
        for send_value in inputs:
            print(f"Received: {yield_value}; Sending: {send_value}")
            yield_value = gen.send(send_value)
    except StopIteration as exc:
        return exc.value
    raise RuntimeError("Failed to exhaust generator")

def example():
    comp1 = yield from [str((yield x)) for x in ('1st', '2nd')]
    comp2 = yield from [int((yield x)) for x in ('3rd', '4th')]
    return comp1, comp2

>>> result = get_gen_result(example(), range(4))
Received: 1st; Sending: 0
Received: 2nd; Sending: 1
Received: 3rd; Sending: 2
Received: 4th; Sending: 3
>>> result
(['0', '1'], [2, 3])

So if we decided to make yield-in-a-comprehension imply the use of yield from, we'd only need:

• DeprecationWarning in 3.7 to say "this is going to imply 'yield from (comprehension)' in 3.8+"
• making the 'yield from' implicit in 3.8 (thus ensuring that comprehensions always return the correct container type, even when they include yield expressions)

No to both. See python-dev.

Here is a sample of the implementation of the Nick's idea.

Serhiy's PR now implements the "Prohibit yield & yield from in generator expressions and comprehensions" approach discussed on python-dev (currently as a hard SyntaxError, but it could be amended to be a warning instead without too much difficulty).

The PR does highlight an interesting subtlety though: the easiest way to implement this still allows yield expressions in the outermost iterator, since that gets compiled in a different scope from the rest of the comprehension body (it's evaluated eagerly and passed in to the implicit nested function).

I'm thinking we probably don't want to expose that detail to end users, and will instead want to include a second check that prohibits yield expressions in the outermost iterator as well. However, I'm not entirely sure how we could implement such a check, short of adding a new "yield expression prohibited" counter in the AST generation and/or symbol analysis pass.

No.

On Nov 25, 2017 18:01, "Nick Coghlan" <report@bugs.python.org> wrote:

Nick Coghlan <ncoghlan@gmail.com> added the comment:

Serhiy's PR now implements the "Prohibit yield & yield from in generator
expressions and comprehensions" approach discussed on python-dev (currently
as a hard SyntaxError, but it could be amended to be a warning instead
without too much difficulty).

The PR does highlight an interesting subtlety though: the easiest way to
implement this still allows yield expressions in the outermost iterator,
since that gets compiled in a different scope from the rest of the
comprehension body (it's evaluated eagerly and passed in to the implicit
nested function).

I'm thinking we probably don't want to expose that detail to end users,
and will instead want to include a second check that prohibits yield
expressions in the outermost iterator as well. However, I'm not entirely
sure how we could implement such a check, short of adding a new "yield
expression prohibited" counter in the AST generation and/or symbol analysis
pass.

----------

---

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue10544\>

---

To clarify, the outermost iterator is marked here:

[func(x, y) for x in <outermost iterator> for y in <something else>]

and it is evaluated before the comprehension proper is started. This is
just part of how the language works (it's a similar rule as "in an
assignment the RHS is evaluated before it is assigned to the LHS").

I see no benefit in banning yield in .

On Sat, Nov 25, 2017 at 10:22 PM, Guido van Rossum <report@bugs.python.org>
wrote:

Guido van Rossum <guido@python.org> added the comment:

No.

On Nov 25, 2017 18:01, "Nick Coghlan" <report@bugs.python.org> wrote:

>
> Nick Coghlan <ncoghlan@gmail.com> added the comment:
>
> Serhiy's PR now implements the "Prohibit yield & yield from in generator
> expressions and comprehensions" approach discussed on python-dev
(currently
> as a hard SyntaxError, but it could be amended to be a warning instead
> without too much difficulty).
>
> The PR does highlight an interesting subtlety though: the easiest way to
> implement this still allows yield expressions in the outermost iterator,
> since that gets compiled in a different scope from the rest of the
> comprehension body (it's evaluated eagerly and passed in to the implicit
> nested function).
>
> I'm thinking we probably don't want to expose that detail to end users,
> and will instead want to include a second check that prohibits yield
> expressions in the outermost iterator as well. However, I'm not entirely
> sure how we could implement such a check, short of adding a new "yield
> expression prohibited" counter in the AST generation and/or symbol
analysis
> pass.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue10544\>
> _______________________________________
>

----------

---

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue10544\>

---

Cool, if you're OK with that behaviour, it actually makes this a lot easier, since it means:

1. Serhiy's patch is already sufficient for the final hard compatibility break
2. It can be readily adapted to emit either DeprecationWarning or SyntaxWarning for 3.7

Great. It should be a DeprecationWarning, since we're planning to disallow
it completely, right? IIRC SyntaxWarning is for syntax that we can't
deprecate.

Guido, should we write this change up as a PEP, or are you happy to just cover it as a section in the What's New document for 3.7?

Note that the DeprecationWarning exception is replaced with a SyntaxError to get a more accurate error report.

$ cat yield-gen.py 
def f():
    return ((yield x) for x in range(3))

$ ./python -Wd yield-gen.py
yield-gen.py:2: DeprecationWarning: 'yield' inside generator expression
  return ((yield x) for x in range(3))

$ ./python -We yield-gen.py
  File "yield-gen.py", line 2
    return ((yield x) for x in range(3))
           ^
SyntaxError: 'yield' inside generator expression

Without this replacement the result would be:

$ ./python -We yield-gen.py
DeprecationWarning: 'yield' inside generator expression

After the tiresome debate I am happy to see this just as a "what's new" entry rather than soliciting more debate with a PEP. (However there may be some existing PEP that suggests it should work? That PEP should be amended with a note that this is being deprecated. But I don't know if there is such a PEP.)

As far as I'm aware, there's nothing that specifically promises these constructs will do anything in Py3 at all - the existing behaviour is just an accident of implementation arising from the way nested scopes and yield expressions interact in general.

Tinkering with "await" in comprehensions and generator expressions would be different, since PEP-530 actually does make promises about how we expect that to work.

OK, great.

New changeset 73a7e9b by Nick Coghlan (Serhiy Storchaka) in branch 'master':
bpo-10544: Deprecate "yield" in comprehensions and generator expressions. (GH-4579)
73a7e9b

With Serhiy's patch merged, I'm marking this as resolved. Thanks all!

https://bugs.python.org/issue32189 is the follow-up issue to turn the warning into an unconditional SyntaxError in 3.8.

PR 4676 backports warnings to 2.7 in Py3k mode.

New changeset 65d1887 by Serhiy Storchaka in branch '2.7':
[2.7] bpo-10544: Deprecate "yield" in comprehensions and generator expressions in Py3k mode. (GH-4579) (bpo-4676)
65d1887

New changeset 07ca9af by Serhiy Storchaka in branch 'master':
bpo-10544: Disallow "yield" in comprehensions and generator expressions. (GH-4564)
07ca9af