classification
Title: inconsistency in list-generator comprehension with yield(-from)
Type: behavior Stage: test needed
Components: Interpreter Core Versions: Python 3.5, Python 3.4
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: ezio.melotti, gcewing, hakril, ncoghlan, vstinner
Priority: normal Keywords:

Created on 2014-07-11 23:06 by hakril, last changed 2014-07-21 20:12 by vstinner.

Messages (5)
msg222811 - (view) Author: Clement Rouault (hakril) * Date: 2014-07-11 23:06
Will playing with generators and `yield from` I found some inconsistency.

list comprehension with yield(-from) would return a generator.
generator comprehension would yield some None in the middle of the expected values.

Examples:
    l = ["abc", range(3)]
    g1 = [(yield from i) for i in l]
    print(g)
    <generator object <listcomp> at 0x7f5ebd58b690>
    print(list(g))
    ['a', 'b', 'c', 0, 1, 2]  # this result is super cool !

    g2 = ((yield from i) for i in l)
    print(g2)
    <generator object <genexpr> at 0x7f5ebd58b6e0>
    print(list(g2))
    ['a', 'b', 'c', None, 0, 1, 2, None]


For `g1`: it returns a generator because the listcomp contains a `yield from`.

For `g2` it append None because it yield the return value of `yield from i`.
It could be rewritten as:
    def comp(x):
        for i in x:
            yield (yield from i)
msg222877 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2014-07-12 20:08
There seem to be two issues here:

> list comprehension with yield(-from) would return a generator.

This is somewhat surprising, and I'm not sure it's expected/documented.
The example you provided seems to behave reasonably, so I don't think we should change the behavior (unless there is some actual bug in similar example).  If anything, we could document this, but I'm not sure if it's worth doing it and where could be added.

> generator comprehension would yield some None in the middle
> of the expected values.

This also seems expected, and the behavior is consistent with the equivalent generator function.
msg222898 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-07-13 01:19
It's a side effect of the hidden closure that provides the new scope for the iteration variable - that's an ordinary function object, so using yield or yield from turns it into a generator expression instead. Generator expressions are already generators, so using yield or yield from just adds more yield points beyond the implied ones.

I've never figured out a good way to document it - it's a natural consequence of the comprehension's closure. An explicit mention in the reference docs for comprehensions may be worth adding.
msg223026 - (view) Author: Clement Rouault (hakril) * Date: 2014-07-14 13:16
I found something else, I think it worth mentioning it.

This side-effect allows to create generators that return other values that None. And the CPython's behavior is not the same for all versions:

    >>> {(yield i) : i for i in range(2)}
    <generator object <dictcomp> at 0x7f0b98f41410>
    >>> list(_)
    # python3.(3,4,5)
    [0, 1]
    # python3.2
    [0, 1, {None: 1}] #  python3.2 appends the generator's return value.
msg223605 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-07-21 20:12
> For `g1`: it returns a generator because the listcomp contains a `yield from`.

IMO it's a bug: [... for ... in ...] must create a list.
History
Date User Action Args
2014-07-21 20:12:38vstinnersetmessages: + msg223605
2014-07-21 20:11:38vstinnersetnosy: + vstinner
2014-07-19 09:07:46terry.reedysetstage: test needed
2014-07-19 09:07:37terry.reedysetversions: - Python 3.2, Python 3.3
2014-07-14 13:16:08hakrilsetmessages: + msg223026
versions: + Python 3.2, Python 3.3
2014-07-13 01:19:13ncoghlansetmessages: + msg222898
2014-07-12 20:08:22ezio.melottisetnosy: + gcewing, ncoghlan, ezio.melotti

messages: + msg222877
versions: + Python 3.4
2014-07-11 23:06:27hakrilcreate