This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: erroneous behavior when creating classes inside a closure
Type: behavior Stage: needs patch
Components: Interpreter Core Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: benjamin.peterson, eric.araujo, eric.smith, gvanrossum, iritkatriel, mark.dickinson, monsanto, ncoghlan, r.david.murray, taleinat, terry.reedy
Priority: normal Keywords: patch

Created on 2010-07-11 19:56 by monsanto, last changed 2022-04-11 14:57 by admin.

Files
File name Uploaded Description Edit
test.py monsanto, 2010-07-11 19:56 Test case
obscure_corner_cases.patch benjamin.peterson, 2010-07-11 21:05
obscure_corner_cases2.patch benjamin.peterson, 2010-07-11 22:00
test3.py terry.reedy, 2010-07-23 21:08
Messages (15)
msg110037 - (view) Author: Chris Monsanto (monsanto) Date: 2010-07-11 19:56
I have a function whose closure contains a local variable that shadows a global variable (lets call it x). If I create a class as follows: 

class Test(object): x = x

Test.x will contain the value of the global x, not the local x. This ONLY happens when the names are the same, and it only happens in the class body; i.e., "class Test(object): y = x" and class "Test(object): pass; Test.x = x" work fine.

However, if there is an assignment x = x AND you make other assignments, such as y = x, in the body, the other variables will have the wrong value too.

Test case attached. Problem noticed on Python 2.6.2 on Windows and 2.6.5 on Linux.
msg110038 - (view) Author: Chris Monsanto (monsanto) Date: 2010-07-11 20:08
A friend confirmed that this was the case on 3.1.2 as well.
msg110039 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2010-07-11 20:24
I'm not sure what I correct behavior is in this case. Consider the function equivalent:

x = 3
def f(x):
    def m():
        x = x
        print x
    m()
f(4)

which gives:

Traceback (most recent call last):
  File "x.py", line 7, in <module>
    f(4)
  File "x.py", line 6, in f
    m()
  File "x.py", line 4, in m
    x = x
UnboundLocalError: local variable 'x' referenced before assignment

The class example works because name namespaces are unoptimized, so failing to find a binding in the local (class) namepsace, Python looks at the globals and finds the global definition.
msg110040 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2010-07-11 20:30
I don't see anything in

http://docs.python.org/reference/executionmodel.html#naming-and-binding

to suggest that the class should behave differently from a nested function here;  that is, I'd expect UnboundLocalError.
msg110041 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2010-07-11 20:37
Jython 2.5.1 gives the same results as Python:

newton:~ dickinsm$ cat test.py
x = "error"

def test(x):
    class Test(object):
        x = x
    print("x: ", x)
    print("Test.x: ", Test.x)

test("success")
newton:~ dickinsm$ jython2.5.1/jython test.py
('x: ', 'success')
('Test.x: ', 'error')
msg110045 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-07-11 20:53
I agree with Mark, I'd expect an UnboundLocalError.   I remembered this thread on Python-dev that may or may not be relevant, but is certainly analogous:

http://www.mail-archive.com/python-dev@python.org/msg37576.html
msg110046 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2010-07-11 21:05
Here's a patch. It raises a NameError in that case.
msg110048 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2010-07-11 21:15
I think it would be worth bringing this up on python-dev, especially since it affects alternative Python implementations.

It would also be good to have a documentation fix to the reference manual that clearly explains whatever behaviour is decided on;  it's not at all clear (to me, anyway), how to extract this information from the docs.
msg111386 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2010-07-23 21:08
Chris, when posting something like this, *please* include the output. I had to insert ()s to run this with 3.1. I will upload the py3 version as test3.py. Is your output the same as mine?

x: success
Test.x: error
Test2.y: success
Test3.x: error
Test3.y: error
Test4.x: success

There is an obvious inconsistency between Test2 and Test/Test3. This shows up also in the dis.dis(test) output. So there is definitely a bug.

To me, the Test2 result is the error. I base this on 7.7 Class Definitions: "The class’s suite is then executed in a new execution frame (see section Naming and binding), using a newly created local namespace and the original global namespace." I interpret this to mean that intermediate namespaces are not used (as was the case before 2.2). Indeed, this sentence is unchanged from the 2.1 doc (and before).
http://docs.python.org/release/2.1/ref/class.html

Of course, the intent could have changed without changing the wording, by reference to the Naming and Binding section, but then this sentence really should be changed too.

The current Naming and Binding section includes:

"A scope defines the visibility of a name within a block. If a local variable is defined in a block, its scope includes that block. If the definition occurs in a function block, the scope extends to any blocks contained within the defining one, unless a contained block introduces a different binding for the name. The scope of names defined in a class block is limited to the class block; it does not extend to the code blocks of methods."

So class blocks are an exception in propagating down, and I thought they were also an exception for propagating into, for the reason stated above. 

It is possible that this is an undefined corner of the language. Certainly, the compiler is confused as it treats one nested class (Test2) as a closure and the other three nested classes as not.

Since the name and binding design is Guido's and central to Python's operation, I personally would not touch it without his input. Hence I have added him as nosy and second the idea of pydev discussion.
msg111488 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2010-07-24 15:23
Hm. This seems an old bug, probably introduced when closures where first introduced (2.1 ISTR, by Jeremy Hylton).

Class scopes *do* behave differently from function scopes; outside a nested function, this should work:

x = 1
class C(object):
  x = x
assert X.x == 1

And I think it should work that way inside a function too.

So IMO the bug is that in classes Test and Test3, the x defined in the function scope is not used.  Test2 shows that normally, the x defined in the inner scope is accessed.

So, while for *function scopes* the rules are "if it is assigned anywhere in the function, every reference to it references the local version", for *class scopes* (outsided methods) the lookup rules are meant to be dynamic, meaning "if it isn't defined locally yet at the point of reference, use the next outer definition".

I haven't reviewed the patches.
msg111489 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2010-07-24 15:23
I meant, of course,

assert C.x == 1
msg111509 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2010-07-24 22:21
Guido clarified:
> Class scopes *do* behave differently from function scopes;
> outside a nested function, this should work:
> x = 1
> class C(object):
>   x = x
> assert C.x == 1
> And I think it should work that way inside a function too.

I take that to mean that

x = 0
def f()
  x = 1
  class C(object):
    x = x
  assert C.x == 1
f()

should work, meaning that C.x==0 and UnboundLocalError are both wrong.

That would mean to me that in "The class’s suite is then executed in a new execution frame (see section Naming and binding), using a newly created local namespace and the original global namespace." the phrase "the original global namespace" should be changed to "the surrounding namespaces".

I also think this from Guido

"So, while for *function scopes* the rules are "if it is assigned anywhere in the function, every reference to it references the local version", for *class scopes* (outsided methods) the lookup rules are meant to be dynamic, meaning "if it isn't defined locally yet at the point of reference, use the next outer definition"."

should somehow also be clearer, probably also in the class page, so that people will neither expect an UnboundLocalError.
msg111512 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2010-07-24 23:10
On Sat, Jul 24, 2010 at 3:21 PM, Terry J. Reedy <report@bugs.python.org> wrote:
>
> Terry J. Reedy <tjreedy@udel.edu> added the comment:
>
> Guido clarified:
>> Class scopes *do* behave differently from function scopes;
>> outside a nested function, this should work:
>> x = 1
>> class C(object):
>>   x = x
>> assert C.x == 1
>> And I think it should work that way inside a function too.
>
> I take that to mean that
>
> x = 0
> def f()
>  x = 1
>  class C(object):
>    x = x
>  assert C.x == 1
> f()
>
> should work, meaning that C.x==0 and UnboundLocalError are both wrong.

Indeed.

> That would mean to me that in "The class’s suite is then executed in a new execution frame (see section Naming and binding), using a newly created local namespace and the original global namespace." the phrase "the original global namespace" should be changed to "the surrounding namespaces".

Those words sound like they were never revised since I wrote them for
Python 0.9.8 or so...

> I also think this from Guido
>
> "So, while for *function scopes* the rules are "if it is assigned anywhere in the function, every reference to it references the local version", for *class scopes* (outsided methods) the lookup rules are meant to be dynamic, meaning "if it isn't defined locally yet at the point of reference, use the next outer definition"."
>
> should somehow also be clearer, probably also in the class page, so that people will neither expect an UnboundLocalError.

FWIW, unless something drastically changed recently, the language
reference is likely out of date in many areas. I would love it if a
team of anal retentive freaks started going through it with a fine
comb so as to make it describe the state of the implementation(s) more
completely.
msg322383 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2018-07-25 18:39
See additional discussion in the duplicate issue19979.
msg407468 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-12-01 16:29
Reproduced on 3.11.
History
Date User Action Args
2022-04-11 14:57:03adminsetgithub: 53472
2021-12-01 16:29:28iritkatrielsetnosy: + iritkatriel

messages: + msg407468
versions: + Python 3.9, Python 3.10, Python 3.11, - Python 2.6, Python 3.1, Python 2.7, Python 3.2
2018-07-25 18:39:24taleinatsetnosy: + taleinat
messages: + msg322383
2018-07-25 18:38:41taleinatlinkissue19979 superseder
2012-11-07 14:24:13ncoghlansetnosy: + ncoghlan
2010-07-24 23:10:10gvanrossumsetmessages: + msg111512
2010-07-24 22:21:32terry.reedysetmessages: + msg111509
2010-07-24 15:23:53gvanrossumsetmessages: + msg111489
2010-07-24 15:23:20gvanrossumsetmessages: + msg111488
2010-07-23 21:08:35terry.reedysetfiles: + test3.py
nosy: + terry.reedy, gvanrossum
messages: + msg111386

2010-07-12 01:53:29eric.smithsetnosy: + eric.smith
2010-07-11 22:00:41benjamin.petersonsetfiles: + obscure_corner_cases2.patch
2010-07-11 21:21:39eric.araujosetnosy: + eric.araujo
2010-07-11 21:15:23mark.dickinsonsetmessages: + msg110048
2010-07-11 21:05:52benjamin.petersonsetfiles: + obscure_corner_cases.patch
keywords: + patch
messages: + msg110046
2010-07-11 20:53:43r.david.murraysetnosy: + r.david.murray
messages: + msg110045
2010-07-11 20:37:47mark.dickinsonsetmessages: + msg110041
2010-07-11 20:30:21mark.dickinsonsetnosy: + mark.dickinson
messages: + msg110040
2010-07-11 20:24:43benjamin.petersonsetnosy: + benjamin.peterson
messages: + msg110039
2010-07-11 20:22:48mark.dickinsonsetversions: + Python 2.7, Python 3.2
type: behavior
components: + Interpreter Core
stage: needs patch
2010-07-11 20:11:09monsantosetversions: + Python 2.6, Python 3.1, - 3rd party
2010-07-11 20:08:31monsantosetmessages: + msg110038
versions: + 3rd party, - Python 2.6
2010-07-11 19:56:34monsantocreate