Title: Comprehensions in a class definition mostly cannot access class variable
Type: behavior Stage: needs patch
Components: Documentation Versions: Python 3.3, Python 3.2, Python 3.1
Status: closed Resolution: duplicate
Dependencies: Superseder: exec of list comprehension fails on NameError
View: 13557
Assigned To: docs@python Nosy List: Trundle, daniel.urban, docs@python, eric.araujo, flox, jonathan.hartley, josmiley, michael.foord, mjs0, rhettinger, terry.reedy, westley.martinez
Priority: normal Keywords:

Created on 2011-04-07 12:55 by mjs0, last changed 2012-07-07 15:20 by flox. This issue is now closed.

Messages (8)
msg133213 - (view) Author: Menno Smits (mjs0) Date: 2011-04-07 12:55
A list comprehension or generator expression in a class definition fails with NameError if it has a condition that refers to another class variable. This doesn't occur if the class variable is used the "in" part of the expression.

The following works:

class Foo:
    x = range(3)
    y = [z for z in x]

but this doesn't:

class Foo:
    x = 3
    y = [z for z in range(5) if z < x]

The error reported is: NameError: global name 'x' is not defined

Both of these examples work in Python 2.

Issue3692 suggests that class variables can't be referred to inside list comprehensions and gen expressions inside class definitions and that this won't be fixed, but these examples show that it is possible to refer to an outside class variable depending on usage. This is inconsistent and confusing.

The Python 2 behaviour makes much more sense. I understand that we don't want list comprehensions to leak internal variables but they should still be able to pull names from outside scopes in a consistent way.
msg133341 - (view) Author: Jonathan Hartley (jonathan.hartley) * Date: 2011-04-08 20:53
Is also exhibited by other class variable being used in the 'output' clause of the list comprehension:

>>> class C:
...     x = 3
...     z = [z*x for z in range(4)]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in C
  File "<stdin>", line 3, in <listcomp>
msg133359 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-04-09 00:22
Devs are aware that there is an exception to the general rule for the 'for' clause. There is a technical reason why the exception is possible, though I have forgotten it.

Since you already know that changing the general behavior has been rejected, are you asking that the exception be removed (for consistency) , so that your first example would fail? If so, that will be rejected also.

I am changing this to a doc issue in case you or someone else wishes to suggest a doc improvement.

The solution to the limitation on generator expressions, of course, is to write out the generator one is trying to abbreviate.

def clipper(max):
    for i in range(5):
        if i < max:
            yield i

class Foo:
    x = 3
    y = list(clipper(x))

# [0, 1, 2]
msg133361 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-04-09 00:31
Title changed.
Generator expressions had the same limitation in 2.x.
All comprehensions have the same limitation in 3.x.
msg133362 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2011-04-09 00:42
> Devs are aware that there is an exception to the general rule
>  for the 'for' clause. There is a technical reason why the 
> exception is possible, though I have forgotten it.

It is best understood when thinking about a gexexp that
gets run long after is created:

   ge = (result_exp(loop_var) for loop_var in iter_exp)
The idea was to have the body of the iterator expression, iter_exp, fail early, before the generator is run and while the local context is still set=up and available:

   ge = (1/0 for i in pow('a', 'b'))

We want the TypeError for pow to be raised immediately.  And if a valid expression were evaluated, we would want the body's ZeroDivisionError to be raised only when the generator is invoked using next(ge()).

In the former case, the local context is still available.  In the latter case, it could be long gone.
msg133371 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-04-09 01:55
Thanks. I remember now: the initial iter_exp is evaluated immediately because it always *can* be, because it is only evaluated once and can only involve 'outside' names, whereas the result expression and any conditional expressions and further iteration expressions cannot be (evaluated immediately) because in general they usually involve the loop_var and are evaluated many times, each time with a different value of the loop_var (which are not available until the code is run).

The current 3.2 doc more or less says this already, though in fewer words.

To put it another way, the iter_exp either becomes or is treated like a parameter default value expression, so that

ge = (result_exp(loop_var) for loop_var in iter_exp)

is like

def __gf__(_it=iter_exp):
    for loop_var in _it:
        yield result_exp(loop_var)
ge = __gf__()
del __gf__

I wonder if something like this should be added to 5.2.8. Generator expressions, with that section moved up before the set/list/dict displays sections, and with the comprehension grammar included therein.

If one does *not* want the immediately evaluation of the iter_exp (which does not normally happen in generator functions) then, again, one should write a generator function. 

I guess the real lesson is that in 3.x, while comprehensions (including g.e.'s are very similar to nested loops and conditions or generator functions with the same, they are not identical in scoping behavior and need to be thought of as their own thing and not only as syntactic sugar. This is probably easier for people who start with 3.x ;-).

Along that line, here is another replacement for the not-working example 2:

class Foo:
    x = 3
    y = []
    for z in range(5):
        if z < x:

# [0, 1, 2]
msg133378 - (view) Author: Menno Smits (mjs0) Date: 2011-04-09 09:04
Thanks to everyone for the explanations.

I was hoping for behaviour along the lines of Python 2 (certainly not artificially blocking more cases in the name of consistency) but it doesn't look like that's going to happen. I think this is one slight and rare area where Python 3 has taken a step backwards when compared to Python 2 but I can live with it. As noted there's other ways to achieve the same thing.

The new Python 3 behaviour seems odd to me but perhaps that's just a bias due to many years of development with Python 2. A documentation improvement would help. 

msg164871 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2012-07-07 15:20
It looks as a duplicate of issue 13557.
I close this one, because a doc patch is attached on the other.
Date User Action Args
2012-07-07 15:20:51floxsetstatus: open -> closed
superseder: exec of list comprehension fails on NameError
resolution: duplicate
messages: + msg164871
2012-06-01 15:41:57floxsetnosy: + flox, westley.martinez, josmiley
2012-06-01 15:39:55floxlinkissue14972 superseder
2011-04-10 00:01:54michael.foordsetnosy: + michael.foord
2011-04-09 09:04:49mjs0setmessages: + msg133378
2011-04-09 01:55:30terry.reedysetmessages: + msg133371
2011-04-09 00:42:51rhettingersetmessages: + msg133362
2011-04-09 00:31:27terry.reedysetmessages: + msg133361
title: list and generator expressions in a class definition fail if expression condition refers to a class variable -> Comprehensions in a class definition mostly cannot access class variable
2011-04-09 00:22:10terry.reedysetassignee: docs@python
components: + Documentation, - Interpreter Core
versions: + Python 3.3
nosy: + terry.reedy, docs@python

messages: + msg133359
stage: needs patch
2011-04-08 20:58:49rhettingersetnosy: + rhettinger
2011-04-08 20:53:39Trundlesetnosy: + Trundle
2011-04-08 20:53:04jonathan.hartleysetnosy: + jonathan.hartley

messages: + msg133341
versions: + Python 3.1
2011-04-08 16:56:15daniel.urbansetnosy: + daniel.urban
2011-04-08 16:16:48eric.araujosetnosy: + eric.araujo
2011-04-07 12:55:20mjs0create