classification
Title: Need Programming FAQ entry for the behavior of closures
Type: enhancement Stage: resolved
Components: Documentation Versions: Python 3.4, Python 3.2, Python 3.3, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: ezio.melotti Nosy List: Tomáš.Dvořák, chris.jerdonek, docs@python, eric.araujo, ezio.melotti, python-dev, r.david.murray
Priority: normal Keywords: easy, patch

Created on 2011-10-03 14:39 by Tomáš.Dvořák, last changed 2013-01-04 22:52 by ezio.melotti. This issue is now closed.

Files
File name Uploaded Description Edit
issue13094.diff ezio.melotti, 2013-01-03 19:22 review
issue13094-2.diff ezio.melotti, 2013-01-04 18:26 review
Messages (14)
msg144819 - (view) Author: Tomáš Dvořák (Tomáš.Dvořák) Date: 2011-10-03 14:39
I have this python script, and run it in python 2.7.2 (installed from EPD_free 7.1-2 (32-bit), but I guess this has nothing to do with EPD.
----8<---fail.py------

class X(object):
    pass

x = X()
items = ["foo", "bar", "baz"]

for each in items:
    setattr(x, each, lambda: each)
    
print("foo", x.foo())    
print("bar", x.bar())
print("baz", x.baz())

----8<---fail.py------

I'd naively expect it to print 
('foo', 'foo')
('bar', 'bar')
('baz', 'baz')

,but it surprisingly (and annoyingly) outputs
('foo', 'baz')
('bar', 'baz')
('baz', 'baz')

Please, tell me that this is a bug :) I'd hate it if this was the intended behaviour. I spent two hours today before I found out this was the cause of my program to fail.

Best regards,
  Tomáš Dvořák
msg144820 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2011-10-03 14:45
Sorry.  It is intended behavior.  The lambda 'each' is bound to the local 'each', and by the time the lambda's execute, the value of 'each' is 'baz'.

I'm going to turn this into a doc bug, because while I'm pretty sure this is documented *somewhere*, I don't see it in the programming FAQ, and it should be there.
msg144821 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2011-10-03 15:03
To understand better what's going on, try to change the value of 'each' after the 3 prints and then call again the 3 methods: you will see that they now return the new value of each.  This is because the lambdas refer to global 'each' (that at the end of the loop is set to 'baz').
If you do setattr(x, each, lambda each=each: each), the each will be local to the lambda, and it will then work as expected.

An entry in the FAQ would be useful, I thought it was there already but apparently it's not (I'm pretty sure I saw this already somewhere in the doc, but I can't seem to find where).
msg144822 - (view) Author: Tomáš Dvořák (Tomáš.Dvořák) Date: 2011-10-03 15:13
Thank you all very much for the super-quick responses. I'm used to smalltalk, so the python variable binding behaviour is unnatural to me, but I guess there must have been some reasons for making it behave this way. 

Ezio, the 
"lambda each=each: each" 
trick works nicely, thanks a lot. But - what does it mean? :) I just don't know how to parse and understand it :-)

Best regards,
  Tomáš Dvořák
msg144823 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2011-10-03 15:22
Maybe with a different name is less confusing: lambda return_value=each: return_value
This copies the value of 'each' in a variable called 'return_value' that is local to the lambda.  Since the copy happens when the lambdas are defined, 'return_value' has then the right value.
msg178918 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-01-03 07:05
I'm having some problem at deciding what the title of the FAQ should be, and what the actual problem is.  ISTM that OP's problem is the same as:
>>> x = 1
>>> def foo(): return x
... 
>>> x = 2
>>> foo()
2

except that he has 3 lambdas in a loop that get attached to an instance rather than a simple function -- but the problem is that in both cases the function references a global variable whose value is retrieved at calling time rather that being set at definition time.
IOW the solution should be clear, but the code is complex enough that it's not easy to recognize the analogy with the simpler case.
I'm not even sure this has anything to do with closures, unless you consider the global scope a closure.

Maybe the "What are the rules for local and global variables in Python?" FAQ could be expanded with a few examples to cover this case too.
msg178938 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-01-03 12:30
The FAQ (as in, this question gets asked again and again) is something like "why do the lambdas I define in a loop all return the same result when the input value was different when each one was defined?"

The same applies to regular functions, but people almost never do that in a loop, so in that case they are more likely to think of the scoping issue.
msg178960 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-01-03 16:28
> "why do the lambdas I define in a loop all return the same result when
> the input value was different when each one was defined?"

I thought about that, but that sounds a bit too long/specific.  It also has the problem that the issue is not strictly related to lambdas or loops (even if this combination might be more common), and doesn't say where the "result" come from.
msg178968 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-01-03 17:23
The point is, it is a FAQ.  We are talking about updating the FAQ document.  It doesn't matter if the text is "too specific", if it is in fact a FAQ.  And it is.
msg178975 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-01-03 19:22
Here's a patch.
msg179064 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-01-04 18:26
Attached a new patch.
msg179071 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-01-04 19:19
Looks good to me.
msg179091 - (view) Author: Roundup Robot (python-dev) Date: 2013-01-04 22:51
New changeset fdc894d44d82 by Ezio Melotti in branch '2.7':
#13094: add Programming FAQ entry about the behavior of closures.
http://hg.python.org/cpython/rev/fdc894d44d82

New changeset 02933454b7ce by Ezio Melotti in branch '3.2':
#13094: add Programming FAQ entry about the behavior of closures.
http://hg.python.org/cpython/rev/02933454b7ce

New changeset 827ddaaa45e4 by Ezio Melotti in branch '3.3':
#13094: merge with 3.2.
http://hg.python.org/cpython/rev/827ddaaa45e4

New changeset 1bf7ae6c5324 by Ezio Melotti in branch 'default':
#13094: merge with 3.3.
http://hg.python.org/cpython/rev/1bf7ae6c5324
msg179092 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-01-04 22:52
Fixed, thanks for the review!
History
Date User Action Args
2013-01-04 22:52:45ezio.melottisetstatus: open -> closed
resolution: fixed
messages: + msg179092

stage: patch review -> resolved
2013-01-04 22:51:56python-devsetnosy: + python-dev
messages: + msg179091
2013-01-04 19:19:05r.david.murraysetmessages: + msg179071
2013-01-04 18:26:14ezio.melottisetfiles: + issue13094-2.diff

messages: + msg179064
2013-01-03 21:18:36ezio.melottisetassignee: ezio.melotti
2013-01-03 19:22:00ezio.melottisetfiles: + issue13094.diff
keywords: + patch
messages: + msg178975

stage: needs patch -> patch review
2013-01-03 17:23:05r.david.murraysetmessages: + msg178968
2013-01-03 16:28:01ezio.melottisetmessages: + msg178960
2013-01-03 12:30:39r.david.murraysetmessages: + msg178938
2013-01-03 07:05:29ezio.melottisetassignee: ezio.melotti -> (no value)

messages: + msg178918
nosy: + chris.jerdonek
2012-11-29 04:22:30ezio.melottisetassignee: docs@python -> ezio.melotti
2012-11-18 22:03:34ezio.melottisetkeywords: + easy
stage: needs patch
type: enhancement
versions: + Python 3.4
2011-10-09 09:16:13eric.araujosetnosy: + eric.araujo

assignee: docs@python
components: + Documentation, - None
versions: + Python 3.2, Python 3.3
2011-10-03 15:22:43ezio.melottisetmessages: + msg144823
2011-10-03 15:13:33Tomáš.Dvořáksetmessages: + msg144822
2011-10-03 15:03:18ezio.melottisetassignee: docs@python -> (no value)
type: behavior -> (no value)
components: + None, - Documentation
versions: - Python 3.2, Python 3.3
nosy: + ezio.melotti

messages: + msg144821
stage: needs patch -> (no value)
2011-10-03 14:46:35r.david.murraysettitle: setattr misbehaves when used with lambdas inside for loop -> Need Programming FAQ entry for the behavior of closures
2011-10-03 14:45:28r.david.murraysetassignee: docs@python
type: behavior
components: + Documentation, - None
versions: + Python 3.2, Python 3.3
nosy: + docs@python, r.david.murray

messages: + msg144820
stage: needs patch
2011-10-03 14:39:56Tomáš.Dvořákcreate