classification
Title: submodule of c-extension module is quirky
Type: Stage:
Components: Versions:
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: YannickJadoul, mattip, skoslowski
Priority: normal Keywords:

Created on 2021-03-02 11:41 by mattip, last changed 2021-03-02 14:54 by skoslowski.

Files
File name Uploaded Description Edit
test.c mattip, 2021-03-02 11:41 Complete C module to reproduce the issue
Messages (2)
msg387917 - (view) Author: mattip (mattip) * Date: 2021-03-02 11:41
If I have a module "parent", and I add another module "child" with a method "f" to it:

child = PyModule_Create(...);
PyModule_AddObject(parent, "child", child);

then I can call 

import parent
parent.child.f()

but importing like this

from parent.child import f

raises a ModuleNotFoundError: ... 'parent' is not a package


This came up in PyTorch https://github.com/pytorch/pytorch/issues/38137 
and in pybind11 https://github.com/pybind/pybind11/issues/2639, 
and in various other places like stackoverflow https://stackoverflow.com/questions/38454852/importerror-with-error-is-not-a-package

A complete example is attached

If this is intentional, it might be nice to emit a warning when calling PyModule_AddObject with a module.
msg387935 - (view) Author: Sebastian Koslowski (skoslowski) * Date: 2021-03-02 14:54
>>> import parent.child

first imports "parent" (successfully) but then fails, because the import code has no knowledge of were to find ".child". This is because 
a) the module "parent" is not marked as a package (hence the error message) 
b) even if it were a package, there is no (ext) module file to locate and load.

If you instead run

>> from parent import child

only "parent" is imported, and "child" is retrieved as an attribute - which it actually is. The import code itself will add such an attribute, too [1]. However, that is after the submodule was located and loaded. Attribute lookup on the parent is not part of the submodule import itself.

A (hacky) work-around would be to add an entry for "parent.child" in sys.modules. This prevents the import machinery from running. 

A (more) proper solution would be to mark "parent" as a package and install some importlib hooks. See [2] for an example from Cython-land. 

Finally there is PyImport_AppendInittab() [3], which could possibly be used to register "parent.child". I have never tried that. Even if this worked, it would be unsupported and probably not without side-effects.

[1] https://docs.python.org/3/reference/import.html#submodules
[2] https://stackoverflow.com/questions/30157363/collapse-multiple-submodules-to-one-cython-extension
[3] https://docs.python.org/3/c-api/import.html?highlight=inittab#c.PyImport_AppendInittab
History
Date User Action Args
2021-03-02 14:54:05skoslowskisetnosy: + skoslowski
messages: + msg387935
2021-03-02 13:14:30YannickJadoulsetnosy: + YannickJadoul
2021-03-02 11:41:23mattipcreate