This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: inconsistency in list-generator comprehension with yield(-from)
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.4, Python 3.5
process
Status: closed Resolution: duplicate
Dependencies: Superseder: yield expression inside generator expression does nothing
View: 10544
Assigned To: Nosy List: ezio.melotti, gcewing, hakril, iritkatriel, ncoghlan, serhiy.storchaka, vstinner
Priority: normal Keywords:

Created on 2014-07-11 23:06 by hakril, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Messages (7)
msg222811 - (view) Author: Clement Rouault (hakril) * Date: 2014-07-11 23:06
Will playing with generators and `yield from` I found some inconsistency.

list comprehension with yield(-from) would return a generator.
generator comprehension would yield some None in the middle of the expected values.

Examples:
    l = ["abc", range(3)]
    g1 = [(yield from i) for i in l]
    print(g)
    <generator object <listcomp> at 0x7f5ebd58b690>
    print(list(g))
    ['a', 'b', 'c', 0, 1, 2]  # this result is super cool !

    g2 = ((yield from i) for i in l)
    print(g2)
    <generator object <genexpr> at 0x7f5ebd58b6e0>
    print(list(g2))
    ['a', 'b', 'c', None, 0, 1, 2, None]


For `g1`: it returns a generator because the listcomp contains a `yield from`.

For `g2` it append None because it yield the return value of `yield from i`.
It could be rewritten as:
    def comp(x):
        for i in x:
            yield (yield from i)
msg222877 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2014-07-12 20:08
There seem to be two issues here:

> list comprehension with yield(-from) would return a generator.

This is somewhat surprising, and I'm not sure it's expected/documented.
The example you provided seems to behave reasonably, so I don't think we should change the behavior (unless there is some actual bug in similar example).  If anything, we could document this, but I'm not sure if it's worth doing it and where could be added.

> generator comprehension would yield some None in the middle
> of the expected values.

This also seems expected, and the behavior is consistent with the equivalent generator function.
msg222898 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-07-13 01:19
It's a side effect of the hidden closure that provides the new scope for the iteration variable - that's an ordinary function object, so using yield or yield from turns it into a generator expression instead. Generator expressions are already generators, so using yield or yield from just adds more yield points beyond the implied ones.

I've never figured out a good way to document it - it's a natural consequence of the comprehension's closure. An explicit mention in the reference docs for comprehensions may be worth adding.
msg223026 - (view) Author: Clement Rouault (hakril) * Date: 2014-07-14 13:16
I found something else, I think it worth mentioning it.

This side-effect allows to create generators that return other values that None. And the CPython's behavior is not the same for all versions:

    >>> {(yield i) : i for i in range(2)}
    <generator object <dictcomp> at 0x7f0b98f41410>
    >>> list(_)
    # python3.(3,4,5)
    [0, 1]
    # python3.2
    [0, 1, {None: 1}] #  python3.2 appends the generator's return value.
msg223605 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-07-21 20:12
> For `g1`: it returns a generator because the listcomp contains a `yield from`.

IMO it's a bug: [... for ... in ...] must create a list.
msg408251 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-12-10 22:59
I think this issue was resolved by now. This is what happens on 3.11:

>>> l = ["abc", range(3)]
>>> g = [(yield from i) for i in l]
  File "<stdin>", line 1
SyntaxError: 'yield' inside list comprehension

>>> g2 = ((yield from i) for i in l)
  File "<stdin>", line 1
SyntaxError: 'yield' inside generator expression
msg408294 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-12-11 11:04
Resolved in issue10544.
History
Date User Action Args
2022-04-11 14:58:05adminsetgithub: 66163
2021-12-11 11:04:09serhiy.storchakasetstatus: pending -> closed

superseder: yield expression inside generator expression does nothing

nosy: + serhiy.storchaka
messages: + msg408294
resolution: out of date -> duplicate
stage: test needed -> resolved
2021-12-10 22:59:11iritkatrielsetstatus: open -> pending

nosy: + iritkatriel
messages: + msg408251

resolution: out of date
2014-07-21 20:12:38vstinnersetmessages: + msg223605
2014-07-21 20:11:38vstinnersetnosy: + vstinner
2014-07-19 09:07:46terry.reedysetstage: test needed
2014-07-19 09:07:37terry.reedysetversions: - Python 3.2, Python 3.3
2014-07-14 13:16:08hakrilsetmessages: + msg223026
versions: + Python 3.2, Python 3.3
2014-07-13 01:19:13ncoghlansetmessages: + msg222898
2014-07-12 20:08:22ezio.melottisetnosy: + gcewing, ncoghlan, ezio.melotti

messages: + msg222877
versions: + Python 3.4
2014-07-11 23:06:27hakrilcreate