Title: locals() behaviour differs when tracing is in effect
Type: behavior Stage:
Components: Documentation, Extension Modules, Interpreter Core Versions: Python 3.3, Python 3.4, Python 2.7
Status: closed Resolution: rejected
Dependencies: Superseder: Document the circumstances where the locals() dict get updated
View: 17546
Assigned To: georg.brandl Nosy List: georg.brandl, gvanrossum, nedbat, pitrou, techtonik, terry.reedy
Priority: normal Keywords:

Created on 2009-10-08 13:49 by andbj, last changed 2013-03-28 02:07 by eric.snow. This issue is now closed.

File name Uploaded Description Edit andbj, 2009-10-08 13:49 techtonik, 2013-03-27 16:48
Messages (14)
msg93745 - (view) Author: André Bjärby (andbj) Date: 2009-10-08 13:49
Running the attached script shows that locals() behaves differently
depending on whether sys.settrace() is active.

This does not occur when calling locals() in the global scope, only when
used inside functions.
msg93747 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-10-08 15:37
The same is thing is true of the frame's f_locals attribute. This
attribute is a copy of the local variables in the frame, because the
internal storage of these variables is a raw C array for faster access.
This copy is only synchronized back when a tracing function returns, so
as to allow implementing a debugger.

>>> def f():
...   a = 1
...   l = sys._getframe().f_locals
...   b = 2
...   return l
>>> f()
{'a': 1}

The above optimization (raw C array for faster access of local
variables) is not done at the global scope, and therefore locals() at
that scope give you direct access to the variables' internal store
(which is, actually, the module's __dict__).

>>> import __main__
>>> __main__.__dict__ is locals()
msg93750 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2009-10-08 16:58
So there's no bug here right? It's even documented.
msg93751 - (view) Author: André Bjärby (andbj) Date: 2009-10-08 17:35
It's documented that "The contents of this dictionary should not be
modified; changes may not affect the values of local variables used by
the interpreter."

The script does the opposite. Perhaps an addition to the documentation?
"The contents of this dictionary may not be affected by local variables
used by the interpreter".
msg93758 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2009-10-08 19:02
I think the doc is clear enough if one reads and believes it.  The main
doc is "Update and return a dictionary representing the current local
symbol table." That pretty clearly says that subsequent changes to the
local symbol table may make the dict out of date until updated by a
subsequent call.
msg175498 - (view) Author: Ned Batchelder (nedbat) * (Python triager) Date: 2012-11-13 12:39
ITSM this could use more clarification.  It is subtle and baffling.  I'm struggling to come up with sentences to make it clearer, though.
msg175596 - (view) Author: Ned Batchelder (nedbat) * (Python triager) Date: 2012-11-14 22:45
I wrote about this here:   A reader suggested this addition to the docs, which I like:

"Multiple invocations within the scope update and return the same dictionary instance.  When a trace function is in effect, this dictionary is updated after every statement."
msg178179 - (view) Author: anatoly techtonik (techtonik) Date: 2012-12-26 00:52
I'm 3rd to vote for reopening this issue to clarify documentation or to fix (read below).

== Why it is important to fix
1. First personal user story. Until I saw the I couldn't figure out what all locals() definitions are talking about. Dictionary, that is a copy, but the copy is sporadically updated when nobody is using in background. It is a weird dictionary with inconsistent behavior placed at a rather critical part of the language.

2. Second personal story. I probably run into the same bug in issue16781. Probably, because I still don't understand what really happens. It is only a suspicion, and it is bad because with my experience I'm supposed to know what happens, but I never had the time to dig deeper until now.

3. locals() is referenced in exec/execfile() docs for the explanation of arguments. Magic of locals() multiplied by exec.execfile() gives a next level of headache when you need to resolve this. Add settrace() to the formula to awake a hater in you.

exec/execfile() are used in Python-based scripting tools (such as SCons), in trace scripts and when Python is embedded as a scripting language. Coding on this level should be easy to make more interesting tools to appear.

== Fix Idea 
1. Is it possible to rename locals() to _locals(), and leave user visible API function local() for an object that updates itself every time it is accessed?
msg178283 - (view) Author: anatoly techtonik (techtonik) Date: 2012-12-27 08:57
After a day of meditation I've come to the following definition for locals():

locals() return a dictionary that serves as a cache for current local symbol table. This cache is updated every time locals() is called or when a trace function is active.
msg178294 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2012-12-27 11:38
I agree now that the doc should be clearer (and this issue reopened). For one thing, the exact CPython behavior is an implementation issue rather than a language requirement. Which is to say, its behavior is allowed but not required. I believe an implementation *could* use dicts for function local namespaces. Synchronization would not then be an issue.

I will try to work up a concrete suggested replacement.
msg181255 - (view) Author: anatoly techtonik (techtonik) Date: 2013-02-03 09:45
Any progress on that? After a month I am inclined that this behavior should be fixed, not only documented. The correct documentation is:

NOTE: The variable returned by locals() is sporadically updated by core interpreter.
msg181298 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-02-03 20:56
Anatoly, please stop playing with the headers. It is meaningless and irritating, especially when you set them wrong. The 3.5 choice is for issues that will *not* apply before then. An example is actually making a change warned about in a deprecation warning. 3.2 will soon see its last regular bugfix release. The main purpose of the headers is to help developers develop and apply a patch. The secondary purpose is serve as history and to inform users.

As I explained in my email to you
1. The OP meant for this to be a code bug report.
2. This was properly closed as an invalid bug report. So there will be no code patch. So the headers have no use except as history.
3. I do believe the doc should be changed, *and* I believe it should be a new issue. If I do open one, I will try to remember to add it here as superseder.
msg185349 - (view) Author: anatoly techtonik (techtonik) Date: 2013-03-27 16:48
Attached where an empty locals() call changes behavior.

---        Wed Mar 27 19:48:06 2013
+++       Wed Mar 27 19:45:19 2013
@@ -3,6 +3,7 @@
 def X():
     l = locals()
     i = "foo"
+    locals()
     print("Is 'i' in stored locals()? ", ('i' in l))

 print("Running normally")
msg185353 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-03-27 17:17
Calling locals() updates the dict, just as documented.
Date User Action Args
2013-03-28 02:07:08eric.snowsetsuperseder: Document the circumstances where the locals() dict get updated
2013-03-27 17:17:51terry.reedysetmessages: + msg185353
2013-03-27 16:48:52techtoniksetfiles: +

messages: + msg185349
2013-02-03 20:56:32terry.reedysetmessages: + msg181298
2013-02-03 09:58:01ezio.melottisetversions: - Python 2.6, Python 2.5, Python 2.4, Python 3.1, Python 3.2, Python 3.5
2013-02-03 09:45:55techtoniksetmessages: + msg181255
versions: + Python 3.2, Python 3.3, Python 3.4, Python 3.5
2012-12-27 11:38:02terry.reedysetmessages: + msg178294
2012-12-27 08:57:34techtoniksetmessages: + msg178283
2012-12-26 00:52:47techtoniksetversions: + Python 2.7
nosy: + techtonik

messages: + msg178179

components: + Documentation
2012-11-14 22:45:45nedbatsetmessages: + msg175596
2012-11-13 12:39:34nedbatsetnosy: + nedbat
messages: + msg175498
2009-10-08 20:01:24andbjsetnosy: - andbj

components: + Extension Modules, Interpreter Core, - Documentation
versions: + Python 2.5, Python 2.4, - Python 2.7, Python 3.2
2009-10-08 19:02:02terry.reedysetnosy: + terry.reedy
messages: + msg93758
2009-10-08 17:37:52pitrousetassignee: georg.brandl

nosy: + georg.brandl
components: + Documentation, - Extension Modules, Interpreter Core
versions: + Python 2.7, Python 3.2, - Python 2.5, Python 2.4
2009-10-08 17:35:50andbjsetmessages: + msg93751
2009-10-08 16:58:24gvanrossumsetstatus: open -> closed

nosy: + gvanrossum
messages: + msg93750

resolution: rejected
2009-10-08 15:37:19pitrousetnosy: + pitrou
messages: + msg93747
2009-10-08 13:49:57andbjcreate