classification
Title: importing does not dispatch to __builtins__.__getitem__ to lookup __import__
Type: enhancement Stage:
Components: Interpreter Core Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: brett.cannon, pxeger, steven.daprano
Priority: normal Keywords:

Created on 2021-07-15 08:30 by pxeger, last changed 2021-07-15 10:51 by pxeger.

Files
File name Uploaded Description Edit
dict_lookup_demo.py pxeger, 2021-07-15 08:30 Demonstration of this behaviour
Messages (6)
msg397531 - (view) Author: Patrick Reader (pxeger) * Date: 2021-07-15 08:30
When a frame's __builtins__ is a subclass of dict with an overridden __getitem__ method, this overriden method is not used by the IMPORT_NAME instruction to lookup __import__ in the dictionary; it uses the lookup function of normal dictionaries (via _PyDict_GetItemIdWithError). This is contrary to the behaviour of the similar LOAD_BUILD_CLASS, as well as the typical name lookup semantics of LOAD_GLOBAL/LOAD_NAME, which all use PyDict_CheckExact for a "fast path" before defaulting to PyObject_GetItem, which is not the behaviour I expected.

Perhaps more seriously, if __builtins__ is not a dict at all, then it gets erroneously passed to some internal dict functions resulting in a mysterious SystemError ("Objects/dictobject.c:1440: bad argument to internal function") which, to me, indicates fragile behaviour that isn't supposed to happen.

Could this be changed, so that __builtins__ is used dynamically? I understand this is a highly specific use case and changing it would probably cause a bit of a slow-down in module importing so is perhaps not worth doing, but I've posted this issue to track it anyway.

I cannot find evidence that this behaviour has changed at all in recent history and it seems to be the same on the main branch as in 3.9.6.

Per a brief discussion with Brett Cannon on python-dev (https://mail.python.org/archives/list/python-dev@python.org/thread/ZQMF6XC76J4APJPB3X6PGATG6CV5NN44/), I have labelled this a feature request because it has never really been expected behaviour.

A short demo of these things is attached.

Links to relevant CPython code in v3.9.6:

IMPORT_NAME: https://github.com/python/cpython/blob/v3.9.6/Python/ceval.c#L5179

BUILD_CLASS: https://github.com/python/cpython/blob/v3.9.6/Python/ceval.c#L2316

LOAD_NAME: https://github.com/python/cpython/blob/v3.9.6/Python/ceval.c#L2488

LOAD_GLOBAL: https://github.com/python/cpython/blob/v3.9.6/Python/ceval.c#L2546
msg397532 - (view) Author: Steven D'Aprano (steven.daprano) * (Python committer) Date: 2021-07-15 08:35
Isn't `__builtins__` a private CPython feature? Other implementations may not have it or use it, and it is my understanding that we should not touch it.
msg397533 - (view) Author: Patrick Reader (pxeger) * Date: 2021-07-15 10:24
It may be, but in that case, why do LOAD_BUILD_CLASS and things still use it?
msg397534 - (view) Author: Patrick Reader (pxeger) * Date: 2021-07-15 10:44
Similarly, when passing a subclass of dict to exec or eval as the locals or globals, all other instructions dispatch to the correct __getitem__ method. I'm pretty sure that's not CPython-private
msg397535 - (view) Author: Steven D'Aprano (steven.daprano) * (Python committer) Date: 2021-07-15 10:45
On Thu, Jul 15, 2021 at 10:24:34AM +0000, Patrick Reader wrote:
> It may be, but in that case, why do LOAD_BUILD_CLASS and things still use it?

They're allowed to use CPython implementation details because they are 
part of the CPython implementation.
msg397536 - (view) Author: Patrick Reader (pxeger) * Date: 2021-07-15 10:51
Ok what I meant was, why does constructing a class use it when it looks up __build_class__ then?
History
Date User Action Args
2021-07-15 10:51:35pxegersetmessages: + msg397536
2021-07-15 10:45:29steven.dapranosetmessages: + msg397535
2021-07-15 10:44:02pxegersetmessages: + msg397534
2021-07-15 10:24:34pxegersetmessages: + msg397533
2021-07-15 08:35:25steven.dapranosetnosy: + steven.daprano
messages: + msg397532
2021-07-15 08:30:05pxegercreate