classification
Title: Possible inconsistency in behavior of list comprehensions vs. generator expressions
Type: behavior Stage:
Components: Interpreter Core Versions: Python 3.0, Python 2.5
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: carlj, georg.brandl, gvanrossum, mgiuca
Priority: normal Keywords:

Created on 2008-07-10 09:38 by carlj, last changed 2008-07-19 13:20 by georg.brandl. This issue is now closed.

Messages (4)
msg69496 - (view) Author: Carl Johnson (carlj) Date: 2008-07-10 09:38
Compare the following behaviors:

    Python 3.0a5 (r30a5:62856, May 10 2008, 10:34:28)
    [GCC 4.0.1 (Apple Inc. build 5465)] on darwin
    Type "help", "copyright", "credits" or "license" for more 
information.
     >>> def f(x):
    ...  if x > 5: raise StopIteration
    ...
     >>> [x for x in range(100) if not f(x)]
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "<stdin>", line 1, in <listcomp>
      File "<stdin>", line 2, in f
    StopIteration
     >>> list(x for x in range(100) if not f(x))
    [0, 1, 2, 3, 4, 5]

One might object that the behavior of the list comprehension is 
identical to that of a for-loop:

    >>> r = []
    >>> for x in range(100):
    ...  if not f(x):
    ...   r.append(x)
    ... 
    Traceback (most recent call last):
      File "<stdin>", line 2, in <module>
      File "<stdin>", line 2, in f
    StopIteration

However, it can be argued that in Python 3 list comprehensions should be 
thought of as "syntatic sugar" for ``list(generator expression)`` not a 
for-loop with an accumulator. (This seems to be the motivation for no 
longer "leaking" variables from list comprehensions into their enclosing 
namespace.)

One interesting question that this raises (for me at least) is whether 
the for-loop should also behave like a generator expression. Of course, 
the behavior of the generator expression can already be simulated by 
writing:

    >>> r = []
    >>> for x in range(100):
    ...  try:
    ...   if f(x):
    ...    r.append(x)
    ...  except StopIteration:
    ...   break
    ... 
    >>> r
    [0, 1, 2, 3, 4, 5]

This raises the question, do we need both a ``break`` statement and 
``raise StopIteration``? Can the former just be made into syntatic sugar 
for the later?
msg69507 - (view) Author: Matt Giuca (mgiuca) Date: 2008-07-10 16:05
You seem to be suggesting that a StopIteration raised in the body of a
for-loop should cause the loop to break. That isn't (as far as I know)
the point of StopIteration. StopIteration is only used to break the
for-loop when raised by the iterator, not the body.

Hence I think the list comprehension is behaving correctly, as the
for-loop is, in that they are both raising the StopIteration you threw,
not catching it. That's valid, because you didn't throw it in the
iterator, you threw it in the condition.

What's more strange (to me) is the fact that the generator expression
stops when it sees a StopIteration. Note that this also happens if you
do it in the head of the generator expression. eg

def f(x):
    if x > 5:
            raise StopIteration
    return x

>>> list((f(x) for x in range(0, 100)))
[0, 1, 2, 3, 4, 5]

However, if you translate that into the full generator function version:

def my_generator_expr():
    for x in range(0, 100):
        yield f(x)

You see that it is behaving correctly.

So I think you discovered an interesting quirk, but it's hard to say
anything here is misbehaving.

By the way this is not a new issue with Python 3.0. Flagging it as a
Python 2.5 issue as well.
msg69509 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2008-07-10 16:46
IMO the generator expression is wrong in interpreting the StopIteration
as a break. It should fail just like the list comprehension fails.
msg70022 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2008-07-19 13:20
This is not a bug, see this thread:
http://mail.python.org/pipermail/python-3000/2008-July/014328.html
History
Date User Action Args
2008-07-19 13:20:15georg.brandlsetstatus: open -> closed
resolution: wont fix
messages: + msg70022
nosy: + georg.brandl
2008-07-10 16:46:12gvanrossumsetnosy: + gvanrossum
messages: + msg69509
2008-07-10 16:05:10mgiucasetnosy: + mgiuca
messages: + msg69507
versions: + Python 2.5
2008-07-10 09:38:21carljcreate