This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author dabeaz
Recipients dabeaz
Date 2015-03-11.15:08:51
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1426086531.98.0.531682777516.issue23642@psf.upfronthosting.co.za>
In-reply-to
Content
I have been investigating some of the new importlib machinery and the addition of ModuleSpec objects.  I am a little curious about the intended handling of C Extension modules going forward. 

Backing up for a moment, consider a pure Python module.  It seems that I can do things like this to bring a module into existence (some steps involving sys.modules omitted).

>>> from importlib.util import find_spec, module_from_spec
>>> spec = find_spec('socket')
>>> socket = module_from_spec(spec)
>>> spec.loader.exec_module(socket)
>>>

However, it all gets "weird" with C extension modules.  For example, you can perform the first few steps:

>>> spec = find_spec('math')
>>> spec
ModuleSpec(name='math', loader=<_frozen_importlib.ExtensionFileLoader object at 0x1012122b0>, origin='/usr/local/lib/python3.5/lib-dynload/math.so')
>>> math = module_from_spec(spec)
>>> math
<module 'math' from '/usr/local/lib/python3.5/lib-dynload/math.so'>
>>> dir(math)
['__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__']

As you can see, you get a fresh "unloaded" module here.  However, if you try to bring in the module contents, things get screwy.

>>> spec.loader.exec_module(math)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'ExtensionFileLoader' object has no attribute 'exec_module'
>>>

Yes, this is the old legacy interface in action--there is no exec_module() method.   You can always fall back to load_module() like this:

>>> spec.loader.load_module(spec.name)
<module 'math' from '/usr/local/lib/python3.5/lib-dynload/math.so'>
>>>

The problem here is that it creates a brand new module and ignores the one that was previously created by module_from_spec().  That module is still empty:

>>> dir(math)
['__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__']
>>> 

I realize that I'm treading into a swamp of legacy interfaces and some pretty complex machinery here.  However, here's my question:  are C extension modules always going to be a special case that need to be considered code that interacts with the import system.  Specifically, will it need to be special-cased to use load_module() instead of the module_from_spec()/exec_module() combination?

I suppose the question might also apply to built-in and frozen modules as well (although I haven't investigated that so much). 

Mainly, I'm just trying to gain some insight from the devs as to the overall direction where the import implementation is going with this.

P.S.  ModuleSpecs are cool. +1
History
Date User Action Args
2015-03-11 15:08:52dabeazsetrecipients: + dabeaz
2015-03-11 15:08:51dabeazsetmessageid: <1426086531.98.0.531682777516.issue23642@psf.upfronthosting.co.za>
2015-03-11 15:08:51dabeazlinkissue23642 messages
2015-03-11 15:08:51dabeazcreate