This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Interaction of ModuleSpec and C Extension Modules
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: brett.cannon, dabeaz, eric.snow, ncoghlan, petr.viktorin, raulcd, scoder
Priority: normal Keywords:

Created on 2015-03-11 15:08 by dabeaz, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Messages (10)
msg237872 - (view) Author: David Beazley (dabeaz) Date: 2015-03-11 15:08
I have been investigating some of the new importlib machinery and the addition of ModuleSpec objects.  I am a little curious about the intended handling of C Extension modules going forward. 

Backing up for a moment, consider a pure Python module.  It seems that I can do things like this to bring a module into existence (some steps involving sys.modules omitted).

>>> from importlib.util import find_spec, module_from_spec
>>> spec = find_spec('socket')
>>> socket = module_from_spec(spec)
>>> spec.loader.exec_module(socket)
>>>

However, it all gets "weird" with C extension modules.  For example, you can perform the first few steps:

>>> spec = find_spec('math')
>>> spec
ModuleSpec(name='math', loader=<_frozen_importlib.ExtensionFileLoader object at 0x1012122b0>, origin='/usr/local/lib/python3.5/lib-dynload/math.so')
>>> math = module_from_spec(spec)
>>> math
<module 'math' from '/usr/local/lib/python3.5/lib-dynload/math.so'>
>>> dir(math)
['__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__']

As you can see, you get a fresh "unloaded" module here.  However, if you try to bring in the module contents, things get screwy.

>>> spec.loader.exec_module(math)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'ExtensionFileLoader' object has no attribute 'exec_module'
>>>

Yes, this is the old legacy interface in action--there is no exec_module() method.   You can always fall back to load_module() like this:

>>> spec.loader.load_module(spec.name)
<module 'math' from '/usr/local/lib/python3.5/lib-dynload/math.so'>
>>>

The problem here is that it creates a brand new module and ignores the one that was previously created by module_from_spec().  That module is still empty:

>>> dir(math)
['__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__']
>>> 

I realize that I'm treading into a swamp of legacy interfaces and some pretty complex machinery here.  However, here's my question:  are C extension modules always going to be a special case that need to be considered code that interacts with the import system.  Specifically, will it need to be special-cased to use load_module() instead of the module_from_spec()/exec_module() combination?

I suppose the question might also apply to built-in and frozen modules as well (although I haven't investigated that so much). 

Mainly, I'm just trying to gain some insight from the devs as to the overall direction where the import implementation is going with this.

P.S.  ModuleSpecs are cool. +1
msg237873 - (view) Author: David Beazley (dabeaz) Date: 2015-03-11 15:15
Note: Might be related to Issue 19713.
msg237875 - (view) Author: David Beazley (dabeaz) Date: 2015-03-11 15:28
inal comment.  It seems that one can generally avoid a lot of nastiness if importlib.reload() is used instead.  For example:

>>> mod = sys.modules[spec.name] = module_from_spec(spec)
>>> importlib.reload(mod)

This works for both source and Extension modules and completely avoids the need to worry about the exec_module()/load_module() warts.   Wouldn't say it's an obvious approach though ;-).
msg237880 - (view) Author: David Beazley (dabeaz) Date: 2015-03-11 16:13
Sorry. I take back the previous message.  It still doesn't quite do what I want.   Anyways, any insight or thoughts about this would be appreciated ;-).
msg237936 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2015-03-12 14:22
There is a proposed PEP on the import-sig: https://mail.python.org/pipermail/import-sig/2015-March/000904.html . I'm hoping to add it for the author to the PEP index on Friday.

Basically we punted on extension modules as we ran out of time in Python 3.4.
msg237938 - (view) Author: David Beazley (dabeaz) Date: 2015-03-12 14:32
This is great news.   Read the PEP draft and think this is a very good thing to be addressing. Thanks, Brett.
msg238042 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2015-03-13 17:29
The PEP is now live: https://www.python.org/dev/peps/pep-0489/ . If you want to participate in the discussion it will probably happen on the import-sig.
msg238878 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2015-03-22 06:41
Added Petr to the nosy list here as well.

Petr - this is the kind of discrepancy I'm hoping that PEP 489 can help remedy, so it may make for a good test case :)
msg244858 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2015-06-05 08:22
Since PEP 489 (Python 3.5.0b1), loaders now support create_module/exec_module. For modules which don't use the PEP's new multi-phase init mechanism (which is most of them, currently), load_module() does all the work and exec_module is a no-op.

So, this issue can be closed.




Python 3.5.0b2+ (default, Jun  2 2015, 20:48:59)                                                                               
[GCC 4.9.2 20150212 (Red Hat 4.9.2-6)] on linux                                                                                
Type "help", "copyright", "credits" or "license" for more information.                                                         
>>> from importlib.util import find_spec, module_from_spec                                                                     
>>> spec = find_spec('math')                                                                                                   
>>> spec                                                                                                                       
ModuleSpec(name='math', loader=<_frozen_importlib_external.ExtensionFileLoader object at 0x7f7d67b51588>, origin='/home/petr/dev/cpython/__installed__/lib/python3.5/lib-dynload/math.cpython-35m-x86_64-linux-gnu.so')                                       
>>> math = module_from_spec(spec)                                                                                              
>>> math                                                                                                                       
<module 'math' from '/home/petr/dev/cpython/__installed__/lib/python3.5/lib-dynload/math.cpython-35m-x86_64-linux-gnu.so'>     
>>> dir(math)
['__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', 'acos', 'acosh', 'asin', 'asinh', 'atan', 'atan2', 'atanh', 'ceil', 'copysign', 'cos', 'cosh', 'degrees', 'e', 'erf', 'erfc', 'exp', 'expm1', 'fabs', 'factorial', 'floor', 'fmod', 'frexp', 'fsum', 'gamma', 'gcd', 'hypot', 'inf', 'isclose', 'isfinite', 'isinf', 'isnan', 'ldexp', 'lgamma', 'log', 'log10', 'log1p', 'log2', 'modf', 'nan', 'pi', 'pow', 'radians', 'sin', 'sinh', 'sqrt', 'tan', 'tanh', 'trunc']
>>> spec.loader.exec_module(math)
>>> spec.loader.load_module(spec.name)
<module 'math' from '/home/petr/dev/cpython/__installed__/lib/python3.5/lib-dynload/math.cpython-35m-x86_64-linux-gnu.so'>
>>> dir(math)
['__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', 'acos', 'acosh', 'asin', 'asinh', 'atan', 'atan2', 'atanh', 'ceil', 'copysign', 'cos', 'cosh', 'degrees', 'e', 'erf', 'erfc', 'exp', 'expm1', 'fabs', 'factorial', 'floor', 'fmod', 'frexp', 'fsum', 'gamma', 'gcd', 'hypot', 'inf', 'isclose', 'isfinite', 'isinf', 'isnan', 'ldexp', 'lgamma', 'log', 'log10', 'log1p', 'log2', 'modf', 'nan', 'pi', 'pow', 'radians', 'sin', 'sinh', 'sqrt', 'tan', 'tanh', 'trunc']

The "array" module implements multi-phase init, so create_module/exec_module work more like with Python modules:

>>> spec = find_spec('array')
>>> spec
ModuleSpec(name='array', loader=<_frozen_importlib_external.ExtensionFileLoader object at 0x7f7d67b51f60>, origin='/home/petr/dev/cpython/__installed__/lib/python3.5/lib-dynload/array.cpython-35m-x86_64-linux-gnu.so')
>>> array = module_from_spec(spec)
>>> array
<module 'array' from '/home/petr/dev/cpython/__installed__/lib/python3.5/lib-dynload/array.cpython-35m-x86_64-linux-gnu.so'>
>>> dir(array)
['__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '_array_reconstructor']
>>> spec.loader.exec_module(array)
>>> dir(array)
['ArrayType', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '_array_reconstructor', 'array', 'typecodes']

(Module methods [like _array_reconstructor here] and __doc__ are initialized in load_module -- an implementation detail related to reusing the existing C structure for defining extension modules.)
msg245418 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2015-06-16 10:49
ping; this issue can be closed.
History
Date User Action Args
2022-04-11 14:58:13adminsetgithub: 67830
2018-07-28 15:34:53petr.viktorinsetstatus: open -> closed
resolution: fixed
stage: resolved
2015-06-16 10:49:43petr.viktorinsetmessages: + msg245418
2015-06-05 08:22:03petr.viktorinsetmessages: + msg244858
2015-04-13 21:44:10raulcdsetnosy: + raulcd
2015-03-22 07:17:41scodersetnosy: + scoder
2015-03-22 06:41:46ncoghlansetnosy: + petr.viktorin
messages: + msg238878
2015-03-13 17:29:54brett.cannonsetmessages: + msg238042
2015-03-12 14:32:32dabeazsetmessages: + msg237938
2015-03-12 14:22:24brett.cannonsetmessages: + msg237936
2015-03-11 16:13:18dabeazsetmessages: + msg237880
2015-03-11 15:28:09dabeazsetmessages: + msg237875
2015-03-11 15:21:35berker.peksagsetnosy: + brett.cannon, ncoghlan, eric.snow
2015-03-11 15:15:40dabeazsetmessages: + msg237873
2015-03-11 15:08:51dabeazcreate