Issue23642
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2015-03-11 15:08 by dabeaz, last changed 2022-04-11 14:58 by admin. This issue is now closed.
Messages (10) | |||
---|---|---|---|
msg237872 - (view) | Author: David Beazley (dabeaz) | Date: 2015-03-11 15:08 | |
I have been investigating some of the new importlib machinery and the addition of ModuleSpec objects. I am a little curious about the intended handling of C Extension modules going forward. Backing up for a moment, consider a pure Python module. It seems that I can do things like this to bring a module into existence (some steps involving sys.modules omitted). >>> from importlib.util import find_spec, module_from_spec >>> spec = find_spec('socket') >>> socket = module_from_spec(spec) >>> spec.loader.exec_module(socket) >>> However, it all gets "weird" with C extension modules. For example, you can perform the first few steps: >>> spec = find_spec('math') >>> spec ModuleSpec(name='math', loader=<_frozen_importlib.ExtensionFileLoader object at 0x1012122b0>, origin='/usr/local/lib/python3.5/lib-dynload/math.so') >>> math = module_from_spec(spec) >>> math <module 'math' from '/usr/local/lib/python3.5/lib-dynload/math.so'> >>> dir(math) ['__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__'] As you can see, you get a fresh "unloaded" module here. However, if you try to bring in the module contents, things get screwy. >>> spec.loader.exec_module(math) Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'ExtensionFileLoader' object has no attribute 'exec_module' >>> Yes, this is the old legacy interface in action--there is no exec_module() method. You can always fall back to load_module() like this: >>> spec.loader.load_module(spec.name) <module 'math' from '/usr/local/lib/python3.5/lib-dynload/math.so'> >>> The problem here is that it creates a brand new module and ignores the one that was previously created by module_from_spec(). That module is still empty: >>> dir(math) ['__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__'] >>> I realize that I'm treading into a swamp of legacy interfaces and some pretty complex machinery here. However, here's my question: are C extension modules always going to be a special case that need to be considered code that interacts with the import system. Specifically, will it need to be special-cased to use load_module() instead of the module_from_spec()/exec_module() combination? I suppose the question might also apply to built-in and frozen modules as well (although I haven't investigated that so much). Mainly, I'm just trying to gain some insight from the devs as to the overall direction where the import implementation is going with this. P.S. ModuleSpecs are cool. +1 |
|||
msg237873 - (view) | Author: David Beazley (dabeaz) | Date: 2015-03-11 15:15 | |
Note: Might be related to Issue 19713. |
|||
msg237875 - (view) | Author: David Beazley (dabeaz) | Date: 2015-03-11 15:28 | |
inal comment. It seems that one can generally avoid a lot of nastiness if importlib.reload() is used instead. For example: >>> mod = sys.modules[spec.name] = module_from_spec(spec) >>> importlib.reload(mod) This works for both source and Extension modules and completely avoids the need to worry about the exec_module()/load_module() warts. Wouldn't say it's an obvious approach though ;-). |
|||
msg237880 - (view) | Author: David Beazley (dabeaz) | Date: 2015-03-11 16:13 | |
Sorry. I take back the previous message. It still doesn't quite do what I want. Anyways, any insight or thoughts about this would be appreciated ;-). |
|||
msg237936 - (view) | Author: Brett Cannon (brett.cannon) * | Date: 2015-03-12 14:22 | |
There is a proposed PEP on the import-sig: https://mail.python.org/pipermail/import-sig/2015-March/000904.html . I'm hoping to add it for the author to the PEP index on Friday. Basically we punted on extension modules as we ran out of time in Python 3.4. |
|||
msg237938 - (view) | Author: David Beazley (dabeaz) | Date: 2015-03-12 14:32 | |
This is great news. Read the PEP draft and think this is a very good thing to be addressing. Thanks, Brett. |
|||
msg238042 - (view) | Author: Brett Cannon (brett.cannon) * | Date: 2015-03-13 17:29 | |
The PEP is now live: https://www.python.org/dev/peps/pep-0489/ . If you want to participate in the discussion it will probably happen on the import-sig. |
|||
msg238878 - (view) | Author: Nick Coghlan (ncoghlan) * | Date: 2015-03-22 06:41 | |
Added Petr to the nosy list here as well. Petr - this is the kind of discrepancy I'm hoping that PEP 489 can help remedy, so it may make for a good test case :) |
|||
msg244858 - (view) | Author: Petr Viktorin (petr.viktorin) * | Date: 2015-06-05 08:22 | |
Since PEP 489 (Python 3.5.0b1), loaders now support create_module/exec_module. For modules which don't use the PEP's new multi-phase init mechanism (which is most of them, currently), load_module() does all the work and exec_module is a no-op. So, this issue can be closed. Python 3.5.0b2+ (default, Jun 2 2015, 20:48:59) [GCC 4.9.2 20150212 (Red Hat 4.9.2-6)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> from importlib.util import find_spec, module_from_spec >>> spec = find_spec('math') >>> spec ModuleSpec(name='math', loader=<_frozen_importlib_external.ExtensionFileLoader object at 0x7f7d67b51588>, origin='/home/petr/dev/cpython/__installed__/lib/python3.5/lib-dynload/math.cpython-35m-x86_64-linux-gnu.so') >>> math = module_from_spec(spec) >>> math <module 'math' from '/home/petr/dev/cpython/__installed__/lib/python3.5/lib-dynload/math.cpython-35m-x86_64-linux-gnu.so'> >>> dir(math) ['__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', 'acos', 'acosh', 'asin', 'asinh', 'atan', 'atan2', 'atanh', 'ceil', 'copysign', 'cos', 'cosh', 'degrees', 'e', 'erf', 'erfc', 'exp', 'expm1', 'fabs', 'factorial', 'floor', 'fmod', 'frexp', 'fsum', 'gamma', 'gcd', 'hypot', 'inf', 'isclose', 'isfinite', 'isinf', 'isnan', 'ldexp', 'lgamma', 'log', 'log10', 'log1p', 'log2', 'modf', 'nan', 'pi', 'pow', 'radians', 'sin', 'sinh', 'sqrt', 'tan', 'tanh', 'trunc'] >>> spec.loader.exec_module(math) >>> spec.loader.load_module(spec.name) <module 'math' from '/home/petr/dev/cpython/__installed__/lib/python3.5/lib-dynload/math.cpython-35m-x86_64-linux-gnu.so'> >>> dir(math) ['__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', 'acos', 'acosh', 'asin', 'asinh', 'atan', 'atan2', 'atanh', 'ceil', 'copysign', 'cos', 'cosh', 'degrees', 'e', 'erf', 'erfc', 'exp', 'expm1', 'fabs', 'factorial', 'floor', 'fmod', 'frexp', 'fsum', 'gamma', 'gcd', 'hypot', 'inf', 'isclose', 'isfinite', 'isinf', 'isnan', 'ldexp', 'lgamma', 'log', 'log10', 'log1p', 'log2', 'modf', 'nan', 'pi', 'pow', 'radians', 'sin', 'sinh', 'sqrt', 'tan', 'tanh', 'trunc'] The "array" module implements multi-phase init, so create_module/exec_module work more like with Python modules: >>> spec = find_spec('array') >>> spec ModuleSpec(name='array', loader=<_frozen_importlib_external.ExtensionFileLoader object at 0x7f7d67b51f60>, origin='/home/petr/dev/cpython/__installed__/lib/python3.5/lib-dynload/array.cpython-35m-x86_64-linux-gnu.so') >>> array = module_from_spec(spec) >>> array <module 'array' from '/home/petr/dev/cpython/__installed__/lib/python3.5/lib-dynload/array.cpython-35m-x86_64-linux-gnu.so'> >>> dir(array) ['__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '_array_reconstructor'] >>> spec.loader.exec_module(array) >>> dir(array) ['ArrayType', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '_array_reconstructor', 'array', 'typecodes'] (Module methods [like _array_reconstructor here] and __doc__ are initialized in load_module -- an implementation detail related to reusing the existing C structure for defining extension modules.) |
|||
msg245418 - (view) | Author: Petr Viktorin (petr.viktorin) * | Date: 2015-06-16 10:49 | |
ping; this issue can be closed. |
History | |||
---|---|---|---|
Date | User | Action | Args |
2022-04-11 14:58:13 | admin | set | github: 67830 |
2018-07-28 15:34:53 | petr.viktorin | set | status: open -> closed resolution: fixed stage: resolved |
2015-06-16 10:49:43 | petr.viktorin | set | messages: + msg245418 |
2015-06-05 08:22:03 | petr.viktorin | set | messages: + msg244858 |
2015-04-13 21:44:10 | raulcd | set | nosy:
+ raulcd |
2015-03-22 07:17:41 | scoder | set | nosy:
+ scoder |
2015-03-22 06:41:46 | ncoghlan | set | nosy:
+ petr.viktorin messages: + msg238878 |
2015-03-13 17:29:54 | brett.cannon | set | messages: + msg238042 |
2015-03-12 14:32:32 | dabeaz | set | messages: + msg237938 |
2015-03-12 14:22:24 | brett.cannon | set | messages: + msg237936 |
2015-03-11 16:13:18 | dabeaz | set | messages: + msg237880 |
2015-03-11 15:28:09 | dabeaz | set | messages: + msg237875 |
2015-03-11 15:21:35 | berker.peksag | set | nosy:
+ brett.cannon, ncoghlan, eric.snow |
2015-03-11 15:15:40 | dabeaz | set | messages: + msg237873 |
2015-03-11 15:08:51 | dabeaz | create |