msg248064 - (view) |
Author: Peter Eastman (Peter Eastman) |
Date: 2015-08-05 19:44 |
The following script demonstrates a bug in the exec() function in Python 3.4. (It works correctly in 2.7).
script = """
print(a)
print([a for i in range(5)])
"""
exec(script, globals(), {"a":5})
It produces the following output:
5
Traceback (most recent call last):
File "test.py", line 5, in <module>
exec(script, globals(), {"a":5})
File "<string>", line 3, in <module>
File "<string>", line 3, in <listcomp>
NameError: name 'a' is not defined
The variable "a" is getting passed to the script, as expected: printing it out works correctly. But if you use it in a comprehension, the interpreter claims it does not exist.
|
msg248068 - (view) |
Author: R. David Murray (r.david.murray) * |
Date: 2015-08-05 19:58 |
exec is subtle. See the explanation linked from issue 23087, which while not *exactly* on point explains the underlying problem (a comprehension is a new scope, and exec can't reach an intermediate scope the way a compiled function can).
As far as the difference from 2.7 goes, the scoping rules for comprehensions changed in python3: the variable you are concerned with is now part of the local scope.
|
msg248075 - (view) |
Author: Peter Eastman (Peter Eastman) |
Date: 2015-08-05 20:33 |
I don't believe that explanation is correct. You can just as easily get the same problem without explicitly passing a map to exec(). For example:
def f():
script = """
print(a)
print([a for i in range(5)])
"""
a = 5
exec(script)
f()
The documentation for exec() states, "In all cases, if the optional parts are omitted, the code is executed in the current scope." Therefore the code above should be exactly equivalent to the following:
def f():
a = 5
print(a)
print([a for i in range(5)])
f()
But the latter works and the former doesn't. Contrary to the documentation, the code is clearly not being executed in the same scope.
|
msg248076 - (view) |
Author: R. David Murray (r.david.murray) * |
Date: 2015-08-05 20:42 |
Yes it is. The comprehension is a *new* scope, within the outer scope of the exec, and it *cannot see* the variables in the outer scope of the exec. You have the same problem if you try to use a comprehension in that way in a class statement at the class level. An exec is explicitly *not* equivalent to a function body. It is equivalent to operating in a class body, if you give it two namespaces, and in a global context if you give it one. This is documented.
Please don't reopen the issue.
|
msg248077 - (view) |
Author: Peter Eastman (Peter Eastman) |
Date: 2015-08-05 20:48 |
Then fix the documentation. This behavior directly contradicts the documentation of the exec() function. The question is not what scope the comprehension runs in, it's what scope the script runs in. See my third example. A comprehension in the f() function has no problem seeing local variables defined in that function. If the script were running into the same scope as that function, then comprehensions inside the script would also see those variables. They don't, clearly demonstrating that the script does *not* run in the same scope, and contradicting the documentation.
|
msg248079 - (view) |
Author: R. David Murray (r.david.murray) * |
Date: 2015-08-05 21:06 |
OK, it looks like what the documentation of exec is missing is the fact that calling exec with no arguments in a non-global is equivalent to calling it with *two* arguments. That is, your "exec(script)" statement is equivalent to "exec(script, globals(), locals())". This is implicit but very much *not* explicit in the current documentation, and should be made explicit.
To be sure I'm explaining this fully: the documentation of exec says "If exec gets two separate objects as globals and locals, the code will be executed as if it were embedded in a class definition".
>>> class Foo:
... a = 10
... [a for x in range(5)]
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in Foo
File "<stdin>", line 3, in <listcomp>
NameError: name 'a' is not defined
|
msg248091 - (view) |
Author: Eryk Sun (eryksun) * |
Date: 2015-08-06 01:08 |
> If exec gets two separate objects as globals and locals,
> the code will be executed as if it were embedded in a
> class definition.
Probably there needs to be more clarification of the compilation context. Class definitions support lexical closures, whereas source code passed to exec is compiled at the time of the call, independent of the lexical context.
In the following example, the code objects for both the class body and the comprehension can access the free variable "a". In CPython, the class body references the free variable via the LOAD_CLASSDEREF op, and the comprehension uses the LOAD_DEREF op.
def f():
a = 5
class C:
print(a)
print([a for i in range(5)])
>>> f()
5
[5, 5, 5, 5, 5]
>>> dis.dis(f.__code__.co_consts[2])
3 0 LOAD_NAME 0 (__name__)
3 STORE_NAME 1 (__module__)
6 LOAD_CONST 0 ('f.<locals>.C')
9 STORE_NAME 2 (__qualname__)
4 12 LOAD_NAME 3 (print)
15 LOAD_CLASSDEREF 0 (a)
18 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
21 POP_TOP
5 22 LOAD_NAME 3 (print)
25 LOAD_CLOSURE 0 (a)
28 BUILD_TUPLE 1
31 LOAD_CONST 1 (<code object <listcomp> ...>)
34 LOAD_CONST 2 ('f.<locals>.C.<listcomp>')
37 MAKE_CLOSURE 0
40 LOAD_NAME 4 (range)
43 LOAD_CONST 3 (5)
46 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
49 GET_ITER
50 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
53 CALL_FUNCTION 1 (1 positional, 0 keyword pair)
56 POP_TOP
57 LOAD_CONST 4 (None)
60 RETURN_VALUE
>>> dis.dis(f.__code__.co_consts[2].co_consts[1])
5 0 BUILD_LIST 0
3 LOAD_FAST 0 (.0)
>> 6 FOR_ITER 12 (to 21)
9 STORE_FAST 1 (i)
12 LOAD_DEREF 0 (a)
15 LIST_APPEND 2
18 JUMP_ABSOLUTE 6
>> 21 RETURN_VALUE
|
msg248097 - (view) |
Author: R. David Murray (r.david.murray) * |
Date: 2015-08-06 03:14 |
OK, yes, so "a class body at global scope" or something like that :)
LOAD_CLASSDEREF is another whole level of complication to the scoping weirdness for classes; see issue 19979 and issue 24129.
|
msg388247 - (view) |
Author: Eryk Sun (eryksun) * |
Date: 2021-03-07 18:17 |
So there are a couple things to clarify here. When the documentation says "if the optional parts are omitted, the code is executed in the current scope", I think it should explicitly state that this is equivalent to calling exec(object, globals(), locals()). This should help to disabuse the reader of any assumption that the compiled code will extend the nested scoping (i.e. lexical closures) of the calling context.
When it says that if "exec gets two separate objects as globals and locals, the code will be executed as if it were embedded in a class definition", I think this can be misleading. exec() compiles top-level code. It extends module-like execution, allowing globals and locals to differ and defaulting to the current scope. This sharply contrasts to code that's compiled for a `class` statement in the same context.
|
|
Date |
User |
Action |
Args |
2022-04-11 14:58:19 | admin | set | github: 68988 |
2021-03-16 10:41:20 | mark.dickinson | link | issue43507 superseder |
2021-03-10 11:09:55 | mark.dickinson | link | issue43448 superseder |
2021-03-07 18:17:30 | eryksun | set | type: behavior -> enhancement messages:
+ msg388247 versions:
+ Python 3.8, Python 3.9, Python 3.10, - Python 3.4, Python 3.5, Python 3.6 |
2015-08-06 03:14:22 | r.david.murray | set | messages:
+ msg248097 |
2015-08-06 01:08:48 | eryksun | set | nosy:
+ eryksun messages:
+ msg248091
|
2015-08-05 21:06:48 | r.david.murray | set | versions:
+ Python 3.5, Python 3.6 |
2015-08-05 21:06:35 | r.david.murray | set | assignee: docs@python components:
+ Documentation title: Incorrect handling of local variables in comprehensions with exec() -> exec docs should note that the no argument form in a local scope is really the two argument form nosy:
+ docs@python
messages:
+ msg248079 stage: resolved -> needs patch |
2015-08-05 20:48:22 | Peter Eastman | set | status: closed -> open resolution: not a bug -> messages:
+ msg248077
|
2015-08-05 20:42:01 | r.david.murray | set | status: open -> closed resolution: not a bug messages:
+ msg248076
|
2015-08-05 20:33:59 | Peter Eastman | set | status: closed -> open resolution: not a bug -> (no value) messages:
+ msg248075
|
2015-08-05 19:58:18 | r.david.murray | set | status: open -> closed
superseder: Exec variable not found error
nosy:
+ r.david.murray messages:
+ msg248068 resolution: not a bug stage: resolved |
2015-08-05 19:44:41 | Peter Eastman | create | |