classification
Title: exec of function doesn't call __getitem__ or __missing__ on undefined global
Type: Stage: needs patch
Components: Documentation Versions: Python 3.2, Python 3.3, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: amaury.forgeotdarc, docs@python, johnf, mark.dickinson
Priority: normal Keywords:

Created on 2012-06-18 10:59 by johnf, last changed 2012-06-18 18:42 by mark.dickinson.

Files
File name Uploaded Description Edit
curiosity.py johnf, 2012-06-18 10:59
curiosity2.py johnf, 2012-06-18 17:45
Messages (7)
msg163095 - (view) Author: John Firestone (johnf) Date: 2012-06-18 10:59
exec(source, Dict()) doesn't call Dict().__getitem__ or Dict().__missing__ if the source string contains a function and the function references an undefined global.

class Dict1(dict):
    def __getitem__(self, key):
        print '    __getitem__', repr(key)
        if key == 's':
            return None
        return dict.__getitem__(self, key)

class Dict2(dict):
    def __missing__(self, key):
        print '    __missing__', repr(key)
        return None

source = """if 1:
    print '  1'
    s
    def f():
        print '  2'
        s
        print '  3'
    f()"""

print 'Dict1.__getitem__'
try:
    exec(source, Dict1())
except NameError as exc_value:
    print '  %s: %s' % (exc_value.__class__.__name__, exc_value)

print 'Dict2.__missing__'
try:
    exec(source, Dict2())
except NameError as exc_value:
    print '    %s: %s' % (exc_value.__class__.__name__, exc_value)


Python 2.7.3 (v2.7.3:70274d53c1dd, Apr  9 2012, 20:32:06) 
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
>>> import curiosity
Dict1.__getitem__
  1
    __getitem__ 's'
    __getitem__ 'f'
  2
    NameError: global name 's' is not defined
Dict2.__missing__
  1
    __missing__ 's'
  2
    NameError: global name 's' is not defined
>>>
msg163097 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2012-06-18 12:12
This looks like a documentation issue:  it's well documented that in the exec statement, the globals dictionary must be a dict.  What's not so clear from the documentation (AFAICT) is that it must actually have *type* dict, rather than merely being an instance of dict.  (Or, from experimentation, it *can* be an instance of a dict subclass, but the underlying C-implemented dict methods are called directly, so overloads for __getitem__ and the like don't have any effect.)
msg163098 - (view) Author: John Firestone (johnf) Date: 2012-06-18 12:34
I find the behavior inconsistent. As you can see from this example, the exec'uted code *does* call the instance's overloaded __getitem__ and __missing__ methods when outside a function, but doesn't when inside.
msg163099 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2012-06-18 12:50
> As you can see from this example, the exec'uted code *does* call the 
> instance's overloaded __getitem__ and __missing__ methods when outside a 
> function, but doesn't when inside.

Yep;  that's because the 's' and 'f' lookups at top level are *local* lookups, and the 's' lookup from inside the body of 'f' is done as a *global* lookup (as explained in the docs here: [1]).  In the exec statement, the locals can be any mapping-like object.  The behaviour's a bit clearer if you pass separate globals and locals dictionaries:

>>> source = """\
... print s
... def f():
...     print s
... f()
... """
>>> locals = {'s': 1729}
>>> globals = {'s': 31415}
>>> exec source in globals, locals
1729
31415


[1] http://docs.python.org/reference/executionmodel.html#interaction-with-dynamic-features
msg163100 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2012-06-18 13:48
With Python3 though, __getitem__ seems called though.
OTOH the 'print' symbol is not found, even though the Dict1 got a '__builtins__' entry added.
msg163110 - (view) Author: John Firestone (johnf) Date: 2012-06-18 17:45
Thank you all for the quick and interesting responses!

Here is another example, this time showing a simple
    s
sometimes behaves like
    globals()['s']
and sometimes doesn't.

class Dict(dict):
    def __getitem__(self, key):
        if key == 's':
            return 'got s'
        return dict.__getitem__(self, key)

dct = Dict()
dct['the_dict'] = dct
print 0, id(dct)

source = """if 1:
    print '1', id(globals()), globals() is the_dict
    print ' ', globals()['s']
    print ' ', s
    def f():
        print '2', id(globals()), globals() is the_dict
        print ' ', globals()['s']
        print ' ', s
        print '3'
    f()"""

exec(source, dct)


Python 2.7.3 (v2.7.3:70274d53c1dd, Apr  9 2012, 20:32:06) 
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
>>> import curiosity2
0 2459928
1 2459928 True
  got s
  got s
2 2459928 True
  got s
 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "curiosity2.py", line 22, in <module>
    exec(source, dct)
  File "<string>", line 10, in <module>
  File "<string>", line 8, in f
NameError: global name 's' is not defined
>>>
msg163113 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2012-06-18 18:42
Yes, this is definitely a dark corner of Python, and one that it seems worth trying to illuminate a bit in the documentation.
History
Date User Action Args
2012-06-18 18:42:27mark.dickinsonsetmessages: + msg163113
2012-06-18 17:45:02johnfsetfiles: + curiosity2.py

messages: + msg163110
2012-06-18 13:48:25amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg163100
2012-06-18 12:50:13mark.dickinsonsetmessages: + msg163099
2012-06-18 12:34:27johnfsetmessages: + msg163098
2012-06-18 12:12:26mark.dickinsonsetassignee: docs@python
type: behavior ->
components: + Documentation, - Interpreter Core
versions: + Python 3.2, Python 3.3
nosy: + mark.dickinson, docs@python

messages: + msg163097
stage: needs patch
2012-06-18 10:59:58johnfcreate