classification
Title: execfile/exec execution in other than global scope uses locals(), leading to undefined behavior
Type: behavior Stage: needs patch
Components: Documentation Versions: Python 3.2, Python 3.3, Python 3.4, Python 2.7, Python 2.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: Arfrever, benjamin.peterson, docs@python, r.david.murray, techtonik, terry.reedy
Priority: normal Keywords:

Created on 2012-12-26 00:15 by techtonik, last changed 2012-12-29 05:58 by Arfrever.

Files
File name Uploaded Description Edit
b3.py techtonik, 2012-12-26 00:15
a.py techtonik, 2012-12-26 00:22
b2.py techtonik, 2012-12-26 00:22
Messages (10)
msg178177 - (view) Author: anatoly techtonik (techtonik) Date: 2012-12-26 00:15
When a Python file is exec()uted, it magically fails to find names in imported modules. The most magical thing in the examples below (b3.py in attach for Python 3) is that first reference to wintypes.LONG in print statement is actually successfull.

--- a.py
from ctypes import wintypes

print(wintypes.LONG)

class LOGFONT(object):
  field = wintypes.LONG


--- b2.py (Python 2 version)
def main():
  execfile('a.py')
main()


--- Output
<class 'ctypes.c_long'>
Traceback (most recent call last):
  File "b2.py", line 4, in <module>
    main()
  File "b2.py", line 2, in main
    execfile('a.py')
  File "a.py", line 5, in <module>
    class LOGFONT(object):
  File "a.py", line 6, in LOGFONT
    field = wintypes.LONG
NameError: name 'wintypes' is not defined
msg178180 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-12-26 01:25
The fact that the print works should be a clue that Python is in fact finding the module and importing it.  So your problem actually has to do with namespaces, which is something you have to think about when using exec or execfile.  You can see this by replacing your import with any variable setting (say, a=1) and referencing that in the class body.

The problem here is that execfile is operating inside a function, therefore the local and global namespaces are different.  wintypes gets imported into the *local* namespace.

Now, if you inline this type of code by hand, wintypes (or a) is found in the local namespace when the class statement is executed.  But when it is done via execfile, it is not.  

I'm not clear on whether or not this is a bug, but if it isn't there is certainly missing documentation in the description of execfile.
msg178184 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2012-12-26 03:25
The best solution is to just always pass an explicit namespace to exec. That should be documented.
msg178190 - (view) Author: anatoly techtonik (techtonik) Date: 2012-12-26 07:08
The workaround with the best case is a magical knowledge, which many don't possess and don't understand (I still don't get it). It's very tempting to ask why passing explicit namespace is the best solution, but instead I'd like to concentrate on this case where execfile() is not given any arguments. The documentation says:

   If both dictionaries are omitted, the expression is executed in the environment where execfile() is called.

From this description I understand that the code should be executed just like inline code. Why it can not? What limitation of Python doesn't make this possible? Is there any secret reason under cover?
msg178232 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2012-12-26 16:00
Basically, it calls locals() in the function scope, which is undefined behavior.
msg178253 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-12-26 20:55
Do you mean that *modifying* locals() in the function scope is undefined behavior?  That makes sense, and is a documented limitation.

So we need to clarify the execfile/exec documentation to note that if called in anything other than the global scope, locals() will get used as the locals dictionary, and this will lead to undefined behavior if any operation is performed that updates the local namespace...and thus you are best recommend to always pass one or two dictionaries to execfile/exec, so that you *know* what is getting updated.

Although I have to say that the exec/execfile doing something that is specified to lead to undefined behavior smells like a bug.  (Not that we could fix it even if we agreed that it was, for backward compatibility reasons.)
msg178255 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2012-12-26 20:56
The best thing would be if we could kill the default use of locals() and globals() in execfile, but that's probably Py4 material.
msg178442 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2012-12-29 00:00
This is at most a further doc clarification issue as the code is working as documented. In a previous issue, I added the following sentence to all current versions to try to clarify this type of behavior: " If exec gets two separate objects as globals and locals, the code will be executed as if it were embedded in a class definition." (This follows " Remember that at module level, globals and locals are the same dictionary") Let's try it:

class Dummy:
    from ctypes import wintypes
    print(wintypes.LONG)

    class LOGFONT(object):
         field = wintypes.LONG
>>>
<class 'ctypes.c_long'>
Traceback (most recent call last):
...
  File "F:\Python\mypy\tem.py", line 6, in LOGFONT
    field = wintypes.LONG
NameError: name 'wintypes' is not defined

Bingo! Anatoly's result is just as documented.

The doc later says "modifications to the default locals dictionary should not be attempted." Anatoly's result is an example of why not.

Lets distill the situation:

1. The globals dict and locals mapping passed to exec are either the same object or different objects. This relation determines the execution behavior.

2. If they are the same object, the code is executed as if at module scope. They are the same if exec is called with both defaults at module scope, where globals() is locals(), or if they are explicitly made the same ("globals = locals()", "locals = globals()", or "globals=dic, locals=dic").

3. If they are different objects, the code is executed as if embedded in a (dummy) class definition. They are different if exec is called with both defaults within a class or function scope*, where globals() is not locals(), or if explicit settings leave them different ("globals = dic" where dic is not locals(), "locals=map", where map is not globals, or "globals=dic, locals=map", where dic is not map).

I believe this nails the situation#.

* In 2.x, comprehensions do not create a function scope, but in 3.x, they do. Lambda expressions always do. This is why I did not write 'within a class or function definition', as some might not see that as including comprehensions.

# The new last sentence of the second paragraph, quoted above, contradicts the older first sentence: "In all cases, if the optional parts are omitted, the code is executed in the current scope." Before 2.2, when the 'current scope' of a function was limited to global and local namespaces, that sentence was true. Indeed, it summarized points 1,2,3 above. I believe that it is not true now, and should be revised, as nonlocal namespaces cannot be seen by exec'ed code. I believe I checked that before adding the new sentence, but I would recheck before revising. I am thinking about how to perhaps rewrite the paragraph.
msg178456 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-12-29 01:26
It looks like you are correct Terry.  The problem, I think, is that the behavior of name spaces inside a class definition is probably the least intuitive aspect of python scoping, so that sentence, while technically complete, doesn't provide enough guidance.
msg178472 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2012-12-29 04:45
I suppose you could say that I kicked that particular can over to the class doc ;-).

The fundamental problem with exec is that it is at least as complicated as Python, since it executes any legal python code, and in fact is even more complicated* because there are various possible relationships with the calling context. Moreover, it always returns None, so that *any* effect is a side-effect, which tends to be 'magical'.

* For one thing, people can write and run normal python code without knowing that a = b (and other binding statements, like import) at module scope means locals()['a'] rather than globals()['a']. At module scope, there are the same because globals() is locals(). Within exec'ed code, they may not be the same thing even for 'top level' code. This is exactly what tripped up Anatoly in his example with the import.

I am thinking that a short How To Exec() might be a good idea, since a real explanation is too much for even a half-page entry in the built-ins chapter.

Note: the following doc statement "Be aware that the return and yield statements may not be used outside of function definitions" needs to have nonlocal added.

>>> nonlocal a
SyntaxError: nonlocal declaration not allowed at module level
>>> exec('nonlocal a')
Traceback (most recent call last):
  File "<pyshell#19>", line 1, in <module>
    exec('nonlocal a')
  File "<string>", line None
SyntaxError: nonlocal declaration not allowed at module level

>>> def f(): exec('nonlocal a') 

f()
...
SyntaxError: nonlocal declaration not allowed at module level

This again points to why exec can be confusing. compile() considers the string it compiles to be top-level code without any surrounding context. However, exec() enables one to run 'top level' code with different globals and locals. There is no exact precedent for this in normal operation. The closest is execution of code within a class statement (before the type(name, dic, bases) part). But even that is not absolutely the same for nonlocal (though this could be the only exception ;-).

>>> >>> class C: nonlocal a
SyntaxError: no binding for nonlocal 'a' found

A different error here (arguably not the best) -- the same as

>>> def f(): nonlocal a
SyntaxError: no binding for nonlocal 'a' found
History
Date User Action Args
2012-12-29 05:58:19Arfreversetnosy: + Arfrever
2012-12-29 04:45:39terry.reedysetmessages: + msg178472
2012-12-29 01:26:19r.david.murraysetmessages: + msg178456
2012-12-29 00:00:20terry.reedysetnosy: + terry.reedy, docs@python
messages: + msg178442

assignee: docs@python
components: + Documentation, - Interpreter Core
stage: needs patch
2012-12-26 20:56:40benjamin.petersonsetmessages: + msg178255
2012-12-26 20:55:09r.david.murraysetmessages: + msg178253
title: execfile/exec execution of class statement does not access locals() -> execfile/exec execution in other than global scope uses locals(), leading to undefined behavior
2012-12-26 16:00:10benjamin.petersonsetmessages: + msg178232
2012-12-26 07:08:18techtoniksetmessages: + msg178190
2012-12-26 03:25:40benjamin.petersonsetmessages: + msg178184
2012-12-26 01:25:04r.david.murraysetnosy: + r.david.murray, benjamin.peterson
title: execfile/exec messes up with imports in executed file -> execfile/exec execution of class statement does not access locals()
messages: + msg178180

versions: - Python 3.1
type: behavior
2012-12-26 00:22:29techtoniksetfiles: + b2.py
2012-12-26 00:22:21techtoniksetfiles: + a.py
2012-12-26 00:15:50techtonikcreate