Title: submodule of c-extension module is quirky
Type: Stage:
Components: Versions:
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: YannickJadoul, mattip, skoslowski
Priority: normal Keywords:

Created on 2021-03-02 11:41 by mattip, last changed 2021-03-02 14:54 by skoslowski.

File name Uploaded Description Edit
test.c mattip, 2021-03-02 11:41 Complete C module to reproduce the issue
Messages (2)
msg387917 - (view) Author: mattip (mattip) * Date: 2021-03-02 11:41
If I have a module "parent", and I add another module "child" with a method "f" to it:

child = PyModule_Create(...);
PyModule_AddObject(parent, "child", child);

then I can call 

import parent

but importing like this

from parent.child import f

raises a ModuleNotFoundError: ... 'parent' is not a package

This came up in PyTorch 
and in pybind11, 
and in various other places like stackoverflow

A complete example is attached

If this is intentional, it might be nice to emit a warning when calling PyModule_AddObject with a module.
msg387935 - (view) Author: Sebastian Koslowski (skoslowski) * Date: 2021-03-02 14:54
>>> import parent.child

first imports "parent" (successfully) but then fails, because the import code has no knowledge of were to find ".child". This is because 
a) the module "parent" is not marked as a package (hence the error message) 
b) even if it were a package, there is no (ext) module file to locate and load.

If you instead run

>> from parent import child

only "parent" is imported, and "child" is retrieved as an attribute - which it actually is. The import code itself will add such an attribute, too [1]. However, that is after the submodule was located and loaded. Attribute lookup on the parent is not part of the submodule import itself.

A (hacky) work-around would be to add an entry for "parent.child" in sys.modules. This prevents the import machinery from running. 

A (more) proper solution would be to mark "parent" as a package and install some importlib hooks. See [2] for an example from Cython-land. 

Finally there is PyImport_AppendInittab() [3], which could possibly be used to register "parent.child". I have never tried that. Even if this worked, it would be unsupported and probably not without side-effects.

Date User Action Args
2021-03-02 14:54:05skoslowskisetnosy: + skoslowski
messages: + msg387935
2021-03-02 13:14:30YannickJadoulsetnosy: + YannickJadoul
2021-03-02 11:41:23mattipcreate