classification
Title: improper scope in list comprehension, when used in class declaration
Type: behavior Stage:
Components: Interpreter Core Versions: Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: John.McDonald, Mariatta, ajaksu2, charettes, georg.brandl, gvanrossum, kaizhu, levkivskyi, mark.dickinson, serhiy.storchaka
Priority: normal Keywords:

Created on 2008-08-26 23:20 by kaizhu, last changed 2018-08-27 10:46 by mark.dickinson.

Messages (13)
msg72002 - (view) Author: kai zhu (kaizhu) Date: 2008-08-26 23:20
in 3rd line, list comprehension tries to access class_attribute1 as a
global variable (code is valid in python 2.5)

>>> class Foo(object):
...   class_attribute1 = 1
...   class_attribute2 = [class_attribute1 for x in range(8)]
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in Foo
  File "<stdin>", line 3, in <listcomp>
NameError: global name 'class_attribute1' is not defined
msg72007 - (view) Author: Daniel Diniz (ajaksu2) (Python triager) Date: 2008-08-27 04:54
I believe the problem is that list comprehensions in 3.0 have scope like
that of genexprs in 2.5, but the change was deliberate (as it also
avoids leaking of temp variables).

Compare to 2.5:
>>> class Foo(object):
...    class_attribute1 = 1
...    class_attribute2 = (class_attribute1 for x in range(8))
...
>>> Foo.class_attribute2.next()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in <genexpr>
NameError: global name 'class_attribute1' is not defined
msg72011 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2008-08-27 07:32
This won't change -- in 3.0, list comprehensions and generator
expressions are both implemented using a function, so it's like the
following:

class Foo:
    attribute1 = 1
    def f():
        return attribute1
    attribute2 = f()
msg192360 - (view) Author: John McDonald (John.McDonald) Date: 2013-07-05 20:58
Could we possibly revisit this? This feels like an implementation detail, not something specified. Consider the different cases:

Type "help", "copyright", "credits" or "license" for more information.
>>> class A:
...   b = 5
...   print(locals())
...   x = [i*i for i in range(b)]
...
{'__qualname__': 'A', '__locals__': {...}, '__module__': '__main__', 'b': 5}

This case works, and shows that 'b' is clearly in locals(). 

>>> class A:
...   b = 5
...   c = b * b
...   print(locals())
...   d = []
...   for i in range(b):
...     d.append(i*i)
...   print(locals())
...   e = [i*i for i in range(b) if i*i < c]
...
{'__locals__': {...}, '__qualname__': 'A', 'b': 5, '__module__': '__main__', 'c': 25}
{'__qualname__': 'A', 'b': 5, '__module__': '__main__', 'c': 25, 'i': 4, 'd': [0, 1, 4, 9, 16], '__locals__': {...}}
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 9, in A
  File "<stdin>", line 9, in <listcomp>
NameError: global name 'c' is not defined

Again, it feels really arbitrary that the variable can be used in certain places in the list comprehension, but not others.

And of course, all of this works properly if you place the definitions either at global scope or within a function.
msg314011 - (view) Author: Simon Charette (charettes) * Date: 2018-03-17 19:13
I stumble upon this bug when porting a Python 2 codebase to 3 and suddenly got a NameError for the following code.


    class Foo:
        a = [1,2,3]
        b = [4,5,6]
        c = [x * y for x in a for y in b]

    NameError: name 'b' is not defined

Not sure what could be done at this point but I thought I'd leave some feedback given I was surprised this would break given it works just fine if not defined at the class level.

    a = [1,2,3]
    b = [4,5,6]
    c = [x * y for x in a for y in b]
msg315764 - (view) Author: Mariatta Wijaya (Mariatta) * (Python committer) Date: 2018-04-25 23:46
I'm re-opening this, since the behavior sounds like a bug to me.

While I understand why it behaves the way it is now, but it seems wrong behavior. Perhaps we should look into finding a solution.
msg316114 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2018-05-03 09:41
Mariatta,

> While I understand why it behaves the way it is now, but it seems wrong behavior. Perhaps we should look into finding a solution.

I am all in favour of dropping implicit function scope in comprehensions (mostly because of weirdness like described here, but it will also fix some `yield` issues and simplify some `await` aspects). But I was not able to convince others it worth the effort. Maybe we can make another attempt at discussing this?
msg316116 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2018-05-03 09:49
See https://bugs.python.org/issue33346 for yet another example where implicit function scope complicates life.
msg316142 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2018-05-03 22:46
There's a proposal for a fix currently in PEP 572. I'm linking to the commit here because we're likely to remove it from PEP 572, but it may be proposed as a separate PEP: https://github.com/python/peps/commit/71fea57f19231b9917ce6a558c0b763288385398

Bug or not this is not going to change without a PEP, so I'm not sure it's a good idea to keep this issue open.
msg316271 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2018-05-07 13:55
> Bug or not this is not going to change without a PEP, so I'm not sure it's a good idea to keep this issue open.

I agree this requires a PEP. I would like to keep this open as a place for pre-PEP discussions among those interested (and as a remainder to maybe write such PEP).
msg316272 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-05-07 14:09
I don't think this will need drastic changes. Just setting some flags here or there for making comprehensions using LOAD_NAME instead of LOAD_GLOBAL. The fix should lie near the fix for issue33346.
msg316274 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2018-05-07 15:30
> I don't think this will need drastic changes.

IIRC LOAD_NAME loads from the *local* scope, which will be the synthetic
function created for the comprehension (whose scope contains the loop
control variables). We may need a new opcode similar to LOAD_GLOBAL that
looks in two non-local directories rather than just one before falling back
to builtins; and the function object would need to have a link to both the
class dict and the global dict -- currently function objects gave a
__globals__ link but there's no chaining. Or perhaps __globals__ could be
set to a chainmap referencing the class dict and the globals?
msg321361 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-07-10 07:31
Indeed, this issue is more complex than it looked to me.

Technically, LOAD_FAST, which is used for reading the loop
control variables in comprehensions, uses the array f->f_localsplus, while LOAD_NAME uses f->f_locals and f->f_globals. When executing class bodies, f->f_localsplus is NULL, and if executing functions f->f_locals usually is a NULL or a cached copy of a dict created from f->f_localsplus. But we can create a hybrid frame for comprehensions in class body, which will contain independent f->f_localsplus and f->f_locals. References to the loop
control variables and to the class variables will work as expected, just the meaning of locals() will be changed.
History
Date User Action Args
2018-08-27 10:46:41mark.dickinsonsetnosy: + mark.dickinson
2018-08-27 10:01:25mark.dickinsonlinkissue34517 superseder
2018-08-27 10:00:14mark.dickinsonunlinkissue34517 superseder
2018-08-27 09:30:55serhiy.storchakalinkissue34517 superseder
2018-07-10 07:33:11serhiy.storchakalinkissue34076 superseder
2018-07-10 07:31:59serhiy.storchakasetmessages: + msg321361
2018-05-07 15:30:27gvanrossumsetmessages: + msg316274
2018-05-07 14:09:14serhiy.storchakasetmessages: + msg316272
2018-05-07 13:55:43levkivskyisetmessages: + msg316271
2018-05-03 22:46:31gvanrossumsetmessages: + msg316142
2018-05-03 09:49:28levkivskyisetmessages: + msg316116
2018-05-03 09:41:55levkivskyisetnosy: + gvanrossum, serhiy.storchaka, levkivskyi
messages: + msg316114
2018-04-25 23:46:53Mariattasetstatus: closed -> open
versions: + Python 3.8, - Python 3.0
nosy: + Mariatta

messages: + msg315764

resolution: wont fix ->
2018-03-17 19:13:28charettessetnosy: + charettes
messages: + msg314011
2013-07-05 20:58:31John.McDonaldsetnosy: + John.McDonald
messages: + msg192360
2013-06-01 03:50:15benjamin.petersonlinkissue18110 superseder
2008-08-27 07:32:02georg.brandlsetstatus: open -> closed
resolution: wont fix
messages: + msg72011
nosy: + georg.brandl
2008-08-27 04:54:23ajaksu2setnosy: + ajaksu2
messages: + msg72007
2008-08-26 23:20:13kaizhucreate