classification
Title: Implement _imp.exec_builtin and exec_dynamic
Type: Stage: test needed
Components: Interpreter Core Versions: Python 3.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, brett.cannon, eric.snow, ncoghlan, python-dev
Priority: critical Keywords:

Created on 2013-11-22 16:27 by brett.cannon, last changed 2014-12-16 13:54 by ncoghlan.

Messages (10)
msg203798 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-11-22 16:27
Since _imp.init_builtin and _imp.load_dynamic don't take in a module to load, need to create new functions which do. Afterwards can deprecate init_builtin, load_dynamic, and init_frozen (the latter having been worked around thanks to get_frozen_object).
msg204726 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-11-29 15:18
This is going to have to wait until Python 3.5, so I'm going to back out the exec_module() aspects of BuiltinImporter and ExtensionFileLoader. As Nick has pointed out previously, we are going to need to change the init function signature of extension modules to accept the module to initialize (or something). This also extends to built-in modules as on Windows all extension modules are compiled in, so there is no real distinction between the two scenarios, so this can't be only partially solved for built-ins but not extension modules.

This is another reason why we will need to go with a soft deprecation of load_module() in Python 3.4 and hopefully get all of this straightened out in Python 3.5 so we can introduce a hard deprecation.

And just in case we end up having to hack our way around the call signature, PyModule_Create() could be tweaked to pull modules from a cache that can be seeded with the module that is to be used for the initialization. That way call signatures don't have to change but what module gets used can be controlled.
msg204735 - (view) Author: Roundup Robot (python-dev) Date: 2013-11-29 16:00
New changeset b5bbd47a9bc4 by Brett Cannon in branch 'default':
Issue #19698: Remove exec_module() from the built-in and extension
http://hg.python.org/cpython/rev/b5bbd47a9bc4
msg205355 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2013-12-06 06:35
Is the problem here that builtins (and extension modules) don't necessarily obey the rules to the letter?  load_module() is supposed to use whatever is in sys.modules. [1]  (BuiltinImporter.load_module() is essentially a synonym for _imp.init_builtin(), right?)

So in each PyInit_*() the code should be using the module out of sys.modules...  That sounds like a bug and I expect that the builtin modules and most extension modules don't comply.

That said, I realize the spirit of the load_module() rule is to support reloading, which is mostly not applicable to builtins and extension modules.  The code you reverted would rely on enforcing the rule outside the implicit applicability.

I'll be glad when we've resolved the extension module API changes that support exec_module()!

[1] http://docs.python.org/dev/library/importlib.html#importlib.abc.Loader.load_module
msg205358 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-12-06 08:33
There are assorted shenanigans in the dynamic module loading code that make
me think we should leave the associated loaders alone for now. Running
PyInit_* more than once isn't permitted by default, so reloading is a
no-op, and "loading again" *copies* the existing module. But modules can
opt in to allowing reinitialization by declaring a module state size of
zero :)

It's a solvable problem, but not an urgent one, and hence better addressed
when we're redesigning the associated C APIs for a PEP 451 based import
system.
msg232569 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2014-12-12 17:40
I would still like to get this solved for Python 3.5. Should we hash out a solution at PyCon?
msg232612 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-12-13 14:53
Yes, if we don't get to it beforehand. I'd still like to take the draft Create/Exec C level hook design I came up with and turn it into a PEP, but I don't know when I'll get time.

Maybe I should just put that together as a (very) rough draft and lob it at import-sig? Then we'll have a concrete base for discussion at PyCon, and someone may be able to put together a draft implementation while you, me & Eric are all in the same place at the same time.
msg232623 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2014-12-13 23:24
sgtm
msg232701 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2014-12-16 02:49
Same here.
msg232743 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-12-16 13:54
Turns out I had written up a recap of the PEP 451 C extension support status back in July, so I just resent that to import-sig.
History
Date User Action Args
2014-12-16 13:54:32ncoghlansetmessages: + msg232743
2014-12-16 02:49:31eric.snowsetmessages: + msg232701
2014-12-13 23:24:13brett.cannonsetmessages: + msg232623
2014-12-13 14:53:56ncoghlansetmessages: + msg232612
2014-12-12 17:40:19brett.cannonsetmessages: + msg232569
2013-12-11 17:16:18eric.snowunlinkissue18864 dependencies
2013-12-06 08:33:33ncoghlansetmessages: + msg205358
2013-12-06 06:35:08eric.snowsetmessages: + msg205355
2013-11-29 16:00:21python-devsetnosy: + python-dev
messages: + msg204735
2013-11-29 15:18:15brett.cannonsetpriority: normal -> critical

messages: + msg204726
versions: + Python 3.5, - Python 3.4
2013-11-23 19:50:26Arfreversetnosy: + Arfrever
2013-11-22 16:28:05brett.cannonlinkissue18864 dependencies
2013-11-22 16:27:55brett.cannoncreate