This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: exec docs should note that the no argument form in a local scope is really the two argument form
Type: enhancement Stage: needs patch
Components: Documentation Versions: Python 3.10, Python 3.9, Python 3.8
process
Status: open Resolution:
Dependencies: Superseder: Exec variable not found error
View: 23087
Assigned To: docs@python Nosy List: Peter Eastman, docs@python, eryksun, r.david.murray
Priority: normal Keywords:

Created on 2015-08-05 19:44 by Peter Eastman, last changed 2022-04-11 14:58 by admin.

Messages (9)
msg248064 - (view) Author: Peter Eastman (Peter Eastman) Date: 2015-08-05 19:44
The following script demonstrates a bug in the exec() function in Python 3.4.  (It works correctly in 2.7).

script = """
print(a)
print([a for i in range(5)])
"""
exec(script, globals(), {"a":5})

It produces the following output:

5
Traceback (most recent call last):
  File "test.py", line 5, in <module>
    exec(script, globals(), {"a":5})
  File "<string>", line 3, in <module>
  File "<string>", line 3, in <listcomp>
NameError: name 'a' is not defined

The variable "a" is getting passed to the script, as expected: printing it out works correctly.  But if you use it in a comprehension, the interpreter claims it does not exist.
msg248068 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-08-05 19:58
exec is subtle.  See the explanation linked from issue 23087, which while not *exactly* on point explains the underlying problem (a comprehension is a new scope, and exec can't reach an intermediate scope the way a compiled function can).

As far as the difference from 2.7 goes, the scoping rules for comprehensions changed in python3: the variable you are concerned with is now part of the local scope.
msg248075 - (view) Author: Peter Eastman (Peter Eastman) Date: 2015-08-05 20:33
I don't believe that explanation is correct.  You can just as easily get the same problem without explicitly passing a map to exec().  For example:

def f():
    script = """
print(a)
print([a for i in range(5)])
    """
    a = 5
    exec(script)
    
f()

The documentation for exec() states, "In all cases, if the optional parts are omitted, the code is executed in the current scope."  Therefore the code above should be exactly equivalent to the following:

def f():
    a = 5
    print(a)
    print([a for i in range(5)])
    
f()

But the latter works and the former doesn't.  Contrary to the documentation, the code is clearly not being executed in the same scope.
msg248076 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-08-05 20:42
Yes it is.  The comprehension is a *new* scope, within the outer scope of the exec, and it *cannot see* the variables in the outer scope of the exec.  You have the same problem if you try to use a comprehension in that way in a class statement at the class level.  An exec is explicitly *not* equivalent to a function body.  It is equivalent to operating in a class body, if you give it two namespaces, and in a global context if you give it one.  This is documented.

Please don't reopen the issue.
msg248077 - (view) Author: Peter Eastman (Peter Eastman) Date: 2015-08-05 20:48
Then fix the documentation.  This behavior directly contradicts the documentation of the exec() function.  The question is not what scope the comprehension runs in, it's what scope the script runs in.  See my third example.  A comprehension in the f() function has no problem seeing local variables defined in that function.  If the script were running into the same scope as that function, then comprehensions inside the script would also see those variables.  They don't, clearly demonstrating that the script does *not* run in the same scope, and contradicting the documentation.
msg248079 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-08-05 21:06
OK, it looks like what the documentation of exec is missing is the fact that calling exec with no arguments in a non-global is equivalent to calling it with *two* arguments.  That is, your "exec(script)" statement is equivalent to "exec(script, globals(), locals())".  This is implicit but very much *not* explicit in the current documentation, and should be made explicit.

To be sure I'm explaining this fully: the documentation of exec says  "If exec gets two separate objects as globals and locals, the code will be executed as if it were embedded in a class definition".

>>> class Foo:
...   a = 10
...   [a for x in range(5)]
... 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in Foo
  File "<stdin>", line 3, in <listcomp>
NameError: name 'a' is not defined
msg248091 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2015-08-06 01:08
> If exec gets two separate objects as globals and locals, 
> the code will be executed as if it were embedded in a 
> class definition.

Probably there needs to be more clarification of the compilation context. Class definitions support lexical closures, whereas source code passed to exec is compiled at the time of the call, independent of the lexical context. 

In the following example, the code objects for both the class body and the comprehension can access the free variable "a". In CPython, the class body references the free variable via the LOAD_CLASSDEREF op, and the comprehension uses the LOAD_DEREF op.

    def f():
       a = 5
       class C:
           print(a)
           print([a for i in range(5)])

    
    >>> f()
    5
    [5, 5, 5, 5, 5]
 
    >>> dis.dis(f.__code__.co_consts[2])
      3           0 LOAD_NAME                0 (__name__)
                  3 STORE_NAME               1 (__module__)
                  6 LOAD_CONST               0 ('f.<locals>.C')
                  9 STORE_NAME               2 (__qualname__)

      4          12 LOAD_NAME                3 (print)
                 15 LOAD_CLASSDEREF          0 (a)
                 18 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
                 21 POP_TOP

      5          22 LOAD_NAME                3 (print)
                 25 LOAD_CLOSURE             0 (a)
                 28 BUILD_TUPLE              1
                 31 LOAD_CONST               1 (<code object <listcomp> ...>)
                 34 LOAD_CONST               2 ('f.<locals>.C.<listcomp>')
                 37 MAKE_CLOSURE             0
                 40 LOAD_NAME                4 (range)
                 43 LOAD_CONST               3 (5)
                 46 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
                 49 GET_ITER
                 50 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
                 53 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
                 56 POP_TOP
                 57 LOAD_CONST               4 (None)
                 60 RETURN_VALUE

    >>> dis.dis(f.__code__.co_consts[2].co_consts[1])
      5           0 BUILD_LIST               0
                  3 LOAD_FAST                0 (.0)
            >>    6 FOR_ITER                12 (to 21)
                  9 STORE_FAST               1 (i)
                 12 LOAD_DEREF               0 (a)
                 15 LIST_APPEND              2
                 18 JUMP_ABSOLUTE            6
            >>   21 RETURN_VALUE
msg248097 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-08-06 03:14
OK, yes, so "a class body at global scope" or something like that :)

LOAD_CLASSDEREF is another whole level of complication to the scoping weirdness for classes; see issue 19979 and issue 24129.
msg388247 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-03-07 18:17
So there are a couple things to clarify here. When the documentation says "if the optional parts are omitted, the code is executed in the current scope", I think it should explicitly state that this is equivalent to calling exec(object, globals(), locals()). This should help to disabuse the reader of any assumption that the compiled code will extend the nested scoping (i.e. lexical closures) of the calling context.

When it says that if "exec gets two separate objects as globals and locals, the code will be executed as if it were embedded in a class definition", I think this can be misleading. exec() compiles top-level code. It extends module-like execution, allowing globals and locals to differ and defaulting to the current scope. This sharply contrasts to code that's compiled for a `class` statement in the same context.
History
Date User Action Args
2022-04-11 14:58:19adminsetgithub: 68988
2021-03-16 10:41:20mark.dickinsonlinkissue43507 superseder
2021-03-10 11:09:55mark.dickinsonlinkissue43448 superseder
2021-03-07 18:17:30eryksunsettype: behavior -> enhancement
messages: + msg388247
versions: + Python 3.8, Python 3.9, Python 3.10, - Python 3.4, Python 3.5, Python 3.6
2015-08-06 03:14:22r.david.murraysetmessages: + msg248097
2015-08-06 01:08:48eryksunsetnosy: + eryksun
messages: + msg248091
2015-08-05 21:06:48r.david.murraysetversions: + Python 3.5, Python 3.6
2015-08-05 21:06:35r.david.murraysetassignee: docs@python
components: + Documentation
title: Incorrect handling of local variables in comprehensions with exec() -> exec docs should note that the no argument form in a local scope is really the two argument form
nosy: + docs@python

messages: + msg248079
stage: resolved -> needs patch
2015-08-05 20:48:22Peter Eastmansetstatus: closed -> open
resolution: not a bug ->
messages: + msg248077
2015-08-05 20:42:01r.david.murraysetstatus: open -> closed
resolution: not a bug
messages: + msg248076
2015-08-05 20:33:59Peter Eastmansetstatus: closed -> open
resolution: not a bug -> (no value)
messages: + msg248075
2015-08-05 19:58:18r.david.murraysetstatus: open -> closed

superseder: Exec variable not found error

nosy: + r.david.murray
messages: + msg248068
resolution: not a bug
stage: resolved
2015-08-05 19:44:41Peter Eastmancreate