Title: Add option to disallow > 1 instance of an extension module
Type: Stage: test needed
Components: Extension Modules, Subinterpreters Versions: Python 3.9
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: corona10, eric.snow, koubaa, shihai1991, terry.reedy, vstinner
Priority: normal Keywords:

Created on 2020-05-11 22:12 by vstinner, last changed 2020-11-19 11:48 by shihai1991.

Messages (6)
msg368665 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-05-11 22:12
When a C extension module is created with PyModuleDef_Init(), it becomes possible to create more than one instance of the module.

It would take significant effort to modify some extensions to make their code fully ready to have two isolated module.

For example, the atexit module calls _Py_PyAtExit() to register itself into the PyInterpreterState. If the module is created more than once, the most recently created module wins, and calls registered on other atexit instances are ignore: see bpo-40288.

One simple option would be to simply disallow loading the module more than once per interpreter.

Also, some extensions are not fully compatible with subinterpreters. It may be interesting to allow to load them in a subinterpreter if it's not already loaded in another interpreter, like another subinterpreter or the main interpreter. It would be only load it once per Python *process*. For example, numpy would be a good candidate for such option.

I'm not sure fow a module should announced in its definition that it should not be loaded more than once.
msg368994 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2020-05-16 01:10
Title clarified.  Leaving subinterpreters aside, only one instance, AFAIK, is true for stdlib and python modules unless imported with different names, as can happen with main module (which is a nuisance if not a bug).  So only once per interpreter seems like a bugfix.  Only once per process, if once per interpreter otherwise becomes normal, seems like an enhancement, even if a necessary option.
msg376524 - (view) Author: mohamed koubaa (koubaa) * Date: 2020-09-07 19:07
What about a new PyModuleDef_Slot function?

static int my_can_create(/*need any arg??, InterpreterState, etc?*/) {
    if (test_global_condition()) {
        return -1; //Don't allow creation
    return 0; //Allow creation

static PyModuleDef_Slot signal_slots[] = {
    {Py_mod_exec, my_exec},
    {Py_mod_can_create, my_can_create},

msg376674 - (view) Author: Dong-hee Na (corona10) * (Python committer) Date: 2020-09-10 08:58
One of my opinions is that

Since module object supports detecting error during Py_mod_exec,
The only thing we have to do is return -1 when the module has already existed.

we should define new exception type and don't have to define new slots.
The pros of this method is that we don't have to modify module object and no needs to write the new PEP

but the cons of this method is that we should promise the standard exception when try to create multiple instances which is not allowed.


On the other side, defining a Py_mod_exec_once that supports execution for just once can be a way.
Although the usage is little, it will be fine because the use case will exist.

Please point out what I missed :)
msg376676 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-09-10 10:00
One option is to get the behavior before multi-phase initialization. We store extensions in a list. Once it's load, it cannot be unloaded before we exit Python. See _PyState_AddModule() and _PyState_AddModule().

Calling PyInit_xxx() the second time would simply return the existing module object.

When we exit Python, the clear and/or free function of the module is called.
msg376685 - (view) Author: mohamed koubaa (koubaa) * Date: 2020-09-10 13:48
Something like this?

static PyObject *def;

    if (def == NULL) {
       def = PyModuleDef_Init(&mymod);
    return def;

Then add a flag to PyModuleDef to indicate it is already exec?
Date User Action Args
2020-11-19 11:48:30shihai1991setnosy: + shihai1991
2020-09-10 13:48:12koubaasetmessages: + msg376685
2020-09-10 10:00:12vstinnersetmessages: + msg376676
2020-09-10 08:58:38corona10setmessages: + msg376674
2020-09-07 19:07:03koubaasetnosy: + koubaa
messages: + msg376524
2020-05-18 14:03:05vstinnersetcomponents: + Subinterpreters
2020-05-16 01:10:16terry.reedysetnosy: + terry.reedy
title: Add an option to disallow creating more than one instance of a module -> Add option to disallow > 1 instance of an extension module
messages: + msg368994

stage: test needed
2020-05-11 22:12:29vstinnercreate