classification
Title: __builtins__ propagation is misleading described in exec and eval documentation
Type: behavior Stage: resolved
Components: Documentation Versions: Python 3.8, Python 3.7, Python 3.6, Python 2.7
process
Status: closed Resolution: duplicate
Dependencies: Superseder: The doc say all globals are copied on eval(), but only __builtins__ is copied
View: 22057
Assigned To: docs@python Nosy List: berker.peksag, docs@python, eryksun, martin.panter, mdk, xcombelle
Priority: normal Keywords: patch

Created on 2016-02-15 10:35 by xcombelle, last changed 2018-08-19 00:07 by berker.peksag. This issue is now closed.

Files
File name Uploaded Description Edit
issue26363.patch mdk, 2016-11-26 14:35 review
Messages (10)
msg260306 - (view) Author: Xavier Combelle (xcombelle) * Date: 2016-02-15 10:42
According to my experiment in code, the current behavior of python3.5 is different that the document says. If I understand well the purpose of this behavior is to propagate the __builtins__ global constant if globals has not one.

In
https://docs.python.org/3.6/library/functions.html#eval

it is written  "If the globals dictionary is present and lacks ‘__builtins__’, the current globals are copied into globals before expression is parsed." only the __builtins__ looks copied not all the globals


In
https://docs.python.org/3.6/library/functions.html#exec
It is written:
"If the globals dictionary does not contain a value for the key __builtins__, a reference to the dictionary of the built-in module builtins is inserted under that key." it looks like it is not a reference to the built-in module builtin, but a reference to __builtin__ global
msg281775 - (view) Author: Julien Palard (mdk) * (Python committer) Date: 2016-11-26 14:35
Hi Xavier, thanks for reporting,

Your first point is right, the implementation being:

 if (PyDict_GetItemString(globals, "__builtins__") == NULL) {
     if (PyDict_SetItemString(globals, "__builtins__",
                              PyEval_GetBuiltins()) != 0)
         return NULL;
 }

See proposed diff.


For the second point, it looks right to me in the documentation, literally: "A reference to the dictionary of the builtin module builtins is inserted", so:

It's a dict:

>>> exec('print(type(globals()["__builtins__"]))', {})
<class 'dict'>

It's the reference to the dict of the builtins module:

>>> exec('globals()["__builtins__"]["len"] = "foo"', {})
>>> len
'foo'

If you still think there's inconsistencies, please provide some code to reproduce it.
msg282055 - (view) Author: Xavier Combelle (xcombelle) * Date: 2016-11-30 00:27
It is not the dictionary of builtin module, which is inserted in , but the current __builtin__ global which happen to be normally the dictionnary of builtin. Hence in the following code, the builtins propagation works has expected.

>>> eval("""eval('spam("hello world")',{})""",{"__builtins__":{"eval":eval,"spam":print}})
hello world
msg282108 - (view) Author: Julien Palard (mdk) * (Python committer) Date: 2016-11-30 22:17
Hi Xavier,

> It is not the dictionary of builtin module, which is inserted in , but the current __builtin__ global 

It looks wrong, I'll even say the exact contrary: It _is_ the dictionary of builtin module which is inserted in, not the current __builtin__ global, with this proof:

$ ./python
Python 3.7.0a0 (default, Nov 29 2016, 11:20:17) 
[GCC 5.4.1 20161019] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> print(id(__builtins__), id(__builtins__.__dict__))
140325888797784 140325888840368
>>> eval("""print(id(__builtins__))""", {})
140325888840368


> the current __builtin__ global which happen to be normally the dictionnary of builtin.

That's not necessarily true, according to [the builtins doc](https://docs.python.org/dev/library/builtins.html):

> The value of __builtins__ is normally either this module or the value of this module’s __dict__ attribute.

Typically:
$ ./python
Python 3.7.0a0 (default, Nov 29 2016, 11:20:17) 
[GCC 5.4.1 20161019] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import builtins
>>> id(builtins), id(builtins.__dict__), id(__builtins__)
(139706743340120, 139706743382704, 139706743340120)

Here, __builtins__ is the module, not its dict.
msg282131 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2016-12-01 03:30
As shown above, exec and eval default to calling PyEval_GetBuiltins when the globals dict doesn't define '__builtins__'. PyEval_GetBuiltins uses the current frame's f_builtins. If there isn't a current frame, it defaults to the interpreter's builtins, which should be the dict of the builtins module. 

If exec and eval didn't do this, the default behavior would be to create a minimal f_builtins dict for the new frame. This dict only contains a reference to None, and it doesn't get set as '__builtins__' in globals. For example:

    from inspect import currentframe
    from ctypes import pythonapi, py_object
    g = py_object({'currentframe': currentframe})
    code = py_object(compile('currentframe()', '', 'eval'))
    frame = pythonapi.PyEval_EvalCode(code, g, g)

    >>> frame.f_builtins
    {'None': None}
    >>> frame.f_globals
    {'currentframe': <function currentframe at 0x7f2fa1d6c2f0>}

This minimalist default isn't useful in general. exec and eval are saving people from the tedium of having to manually define a useful __builtins__ when passing a new globals. The frame object uses this __builtins__ to initialize its f_builtins. Also, it knows to look for __builtins__ as a module, as used by __main__:

    g = py_object({'currentframe': currentframe, '__builtins__': __builtins__})
    frame = pythonapi.PyEval_EvalCode(code, g, g)

    >>> frame.f_builtins is vars(__builtins__)
    True
msg282165 - (view) Author: Xavier Combelle (xcombelle) * Date: 2016-12-01 12:22
Hi Julien,

You are fully right that it is the builtin module dictionnary which is inserted in eval or exec context.

However, if a "__builtins__" entry is passed to eval or exec, this builtin module dictionnary is modified hence the following work: 

>>> d={"tata":"tata"}
>>> print(eval("tata",{'__builtins__':d}))
tata
msg282206 - (view) Author: Julien Palard (mdk) * (Python committer) Date: 2016-12-01 21:17
So, is there still an inconsistency in the documentation?
msg282208 - (view) Author: Xavier Combelle (xcombelle) * Date: 2016-12-01 22:48
not an inconsisties but in the eval documentaion nothing specify that the builtins propagate between levels updates
msg282263 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-12-03 03:06
Xavier, you are welcome to propose your own version of the text, or build on Julien’s. See also Issue 22057, about copying all globals vs builtins.
msg323739 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2018-08-19 00:07
Thank you for the report, Xavier.

This is a duplicate of issue 22057.

PR 8812 clarifies the behavior when a dictionary without a "__builtins__" key is passed as *globals* to eval(). I think that makes the opposite case much easier to understand.

>>> eval("print(spam)", {'__builtins__': {'spam': 'eggs'}})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 1, in <module>
NameError: name 'print' is not defined

>>> eval("print(spam)", {'__builtins__': {'spam': 'eggs', 'print': print}})
eggs

Also, I never needed to pass a dictionary with a "__builtins__" key to eval() before, so I don't think it's an important detail to document.
History
Date User Action Args
2018-08-19 00:07:47berker.peksagsetstatus: open -> closed

superseder: The doc say all globals are copied on eval(), but only __builtins__ is copied
nosy: + berker.peksag
versions: + Python 3.8, - Python 3.5
messages: + msg323739

type: behavior
resolution: duplicate
stage: patch review -> resolved
2016-12-03 03:06:47martin.pantersetversions: + Python 2.7, Python 3.6, Python 3.7
nosy: + martin.panter

messages: + msg282263

stage: patch review
2016-12-01 22:48:23xcombellesetmessages: + msg282208
2016-12-01 21:17:10mdksetmessages: + msg282206
2016-12-01 12:22:49xcombellesetmessages: + msg282165
2016-12-01 03:30:34eryksunsetnosy: + eryksun
messages: + msg282131
2016-11-30 22:17:28mdksetmessages: + msg282108
2016-11-30 00:27:39xcombellesetmessages: + msg282055
2016-11-26 14:35:00mdksetfiles: + issue26363.patch
keywords: + patch
messages: + msg281775
2016-02-15 10:46:55mdksetnosy: + mdk
2016-02-15 10:42:15xcombellesetversions: + Python 3.5
messages: + msg260306
title: builtins propagation is misleading described in exec and eval documentation -> __builtins__ propagation is misleading described in exec and eval documentation
2016-02-15 10:35:57xcombellecreate