classification
Title: list() != []
Type: behavior Stage: needs patch
Components: Documentation Versions: Python 3.2, Python 3.3
process
Status: pending Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Peter.Norvig, cvrebert, docs@python, ezio.melotti, mark.dickinson, r.david.murray, rhettinger, terry.reedy, wolma
Priority: low Keywords:

Created on 2012-05-17 22:47 by Peter.Norvig, last changed 2018-05-14 18:46 by cheryl.sabella.

Messages (11)
msg161023 - (view) Author: Peter Norvig (Peter.Norvig) Date: 2012-05-17 22:47
PEP 289 says "the semantic definition of a list comprehension in Python 3.0 will be equivalent to list(<generator expression>).  Here is a counterexample where they differ (tested in 3.2):

def five(x):
    "Generator yields the object x five times."
    for _ in range(5):
        yield x


# If we ask five() for 10 objects in a list comprehension,
# we get an error:

>>> F = five('x')
>>> [next(F) for _ in range(10)]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

# But if we ask five() for 10 objects in a list(generator expr),
# we get five objects, no  error:

>>> F = five('x')
>>> list(next(F) for _ in range(10))
['x', 'x', 'x', 'x', 'x']
msg161030 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-05-18 02:26
I think the behavior is correct.  next(x) is outside the for expression in the list comprehension, but 'list(x)' is an implicit 'for x in exp'.  So I believe it is the doc that needs amplification.  The PEP discussion is referring specifically to the leaking of the loop variable.
msg161033 - (view) Author: Peter Norvig (Peter.Norvig) Date: 2012-05-18 04:22
I agree with R. David Murray -- if "correct" means following the PEP 289 semantics, then

list(next(F) for _ in range(10))

should be the same as

def __gen(exp):
    for _ in exp:
        yield next(F)

list(__gen(iter(range(10))))

and indeed that is the case.  So the behvavior is correct and the documentation is both wrong, and rather informal/incomplete.
msg161084 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2012-05-18 20:50
It has been noted elsewhere (but I cannot find it) that 1) an uncaught StopIteration raised in an expression *within* a genexp is indistinguishable from the StopIteration raised *by* the genexp upon normal termination; and 2) this makes genexps different from otherwise equivalent comprehensions. The statement in PEP289 does not address this exceptional case.

Raymond, do you want to revise your PEP to mention this?

PEPs are, as their name says, proposals that express intention, not result, and especially not the result after continued patching. We do not usually usually go back and revise PEPs after they have been implemented. It would be an endless task. Hence, they are not current 'documentation' unless unless specifically referred from the docs. This one is not so referenced.

Perhaps PEP 0 should say that " Finished PEPs (done, implemented in code repository)" are historical documents and not current documentation.

I checked the entries for comprehensions and generator expressions and they do not claim equivalence, but do (briefly) describe the actual behavior. However, it is easy to miss that the entry in 5.2.8. Generator expressions implies the two facts above. Perhaps we should explicitly say something like:

If *expression* raises StopIteration, it will not be distinguished from the StopIteration raised upon normal termination of the generator, and it will make the generator expression act differently from the otherwise equivalent comprehension.
msg161187 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2012-05-20 07:14
I will be happy to clarify the PEP when I get a chance.
msg281919 - (view) Author: Wolfgang Maier (wolma) * Date: 2016-11-28 22:12
Isn't the difference between generator expressions and comprehensions what's dealt with by PEP479? So it seems this issue is outdated enough to deserve being closed?
msg281970 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2016-11-29 08:30
@wolma: I don't think PEP 479 is relevant here: we're not raising StopIteration inside a generator function, which is the situation that PEP 479 covers. The behaviour in 3.6 matches that originally reported:

Python 3.6.0b3 (default, Nov  2 2016, 08:15:32) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> def five(x):
...     for _ in range(5):
...         yield x
... 
>>> F = five('x')
>>> [next(F) for _ in range(10)]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 1, in <listcomp>
StopIteration
>>> F = five('x')
>>> list(next(F) for _ in range(10))
['x', 'x', 'x', 'x', 'x']
msg281971 - (view) Author: Wolfgang Maier (wolma) * Date: 2016-11-29 08:34
Mark, PEP479 is not fully in effect in 3.6 yet. 3.7 will raise the RuntimeError, but 3.6 still only gives a DeprecationWarning.
msg281973 - (view) Author: Wolfgang Maier (wolma) * Date: 2016-11-29 08:46
running with "-W always":

>>> def five(x):
...     for _ in range(5):
...         yield x
... 

>>> F = five('x')

>>> [next(F) for _ in range(10)]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 1, in <listcomp>
StopIteration

>>> list(next(F) for _ in range(10))
__main__:1: DeprecationWarning: generator '<genexpr>' raised StopIteration
[]
msg281974 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2016-11-29 09:06
Wolfgang: ah, thanks, that makes more sense. I misunderstood; sorry for the noise.
msg281995 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2016-11-29 15:10
OK, then this issue can be closed unless someone thinks it is worth documenting the lack of PEP 479 in 3.5 and 3.6.
History
Date User Action Args
2018-05-14 18:46:22cheryl.sabellasetstatus: open -> pending
2016-11-29 15:10:59r.david.murraysetmessages: + msg281995
2016-11-29 09:06:29mark.dickinsonsetmessages: + msg281974
2016-11-29 08:46:42wolmasetmessages: + msg281973
2016-11-29 08:34:25wolmasetmessages: + msg281971
2016-11-29 08:30:25mark.dickinsonsetnosy: + mark.dickinson
messages: + msg281970
2016-11-29 02:21:07rhettingersetassignee: rhettinger ->
2016-11-28 22:12:29wolmasetnosy: + wolma
messages: + msg281919
2012-05-20 07:14:03rhettingersetpriority: normal -> low
assignee: docs@python -> rhettinger
messages: + msg161187
2012-05-19 10:49:13cvrebertsetnosy: + cvrebert
2012-05-18 20:50:39terry.reedysetnosy: + terry.reedy, rhettinger, docs@python
messages: + msg161084

assignee: docs@python
components: + Documentation, - Interpreter Core
2012-05-18 04:22:26Peter.Norvigsetmessages: + msg161033
2012-05-18 02:26:49r.david.murraysetnosy: + r.david.murray
messages: + msg161030
2012-05-17 22:51:04ezio.melottisetnosy: + ezio.melotti
stage: needs patch

components: + Interpreter Core, - None
versions: + Python 3.3
2012-05-17 22:47:37Peter.Norvigcreate