This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: warning extension module inited twice in python3.9
Type: enhancement Stage: resolved
Components: Extension Modules Versions: Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: brett.cannon, shihai1991, vstinner
Priority: normal Keywords: patch

Created on 2020-02-29 13:03 by shihai1991, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 18700 closed shihai1991, 2020-02-29 13:07
PR 18739 merged vstinner, 2020-03-02 12:07
Messages (6)
msg362980 - (view) Author: Hai Shi (shihai1991) * (Python triager) Date: 2020-02-29 13:03
In master branch, `_PyWarnings_Init()` have been called twice.
```
(gdb) bt
#0  _PyWarnings_Init () at Python/_warnings.c:1338
#1  0x0000000000525df3 in pycore_init_import_warnings (tstate=tstate@entry=0x9a19c0, sysmod=0x7ffff7f7e5f0) at Python/pylifecycle.c:680
...
Breakpoint 1, _PyWarnings_Init () at Python/_warnings.c:1338
1338    {
(gdb) bt
#0  _PyWarnings_Init () at Python/_warnings.c:1338
#1  0x0000000000511aac in _imp_create_builtin (module=<optimized out>, spec=0x7ffff7f2e7d0) at Python/import.c:1293
```

but in 2.7 branch, '_PyWarnings_Init()':
```
Breakpoint 1, _PyWarnings_Init() at Python/_warnings.c:886
886         m = Py_InitModule3(MODULE_NAME, warnings_functions, warnings__doc__);
Missing separate debuginfos, use: debuginfo-install glibc-2.17-260.el7_6.4.x86_64
(gdb) bt
#0  _PyWarnings_Init () at Python/_warnings.c:886
#1  0x00000000004fc4db in Py_InitializeEx (install_sigs=1) at Python/pythonrun.c:242
#2  0x00000000004fcb03 in Py_Initialize () at Python/pythonrun.c:370
#3  0x00000000004154fd in Py_Main (argc=1, argv=0x7fffffffe428) at Modules/main.c:505
#4  0x00000000004145f0 in main (argc=1, argv=0x7fffffffe428) at ./Modules/python.c:23
```

Why? because pylifecycle.c and _imp extension module will call `_PyWarnings_Init()`.

isn't it?
msg363164 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-03-02 11:56
_PyWarnings_Init() initializes tstate->interp->warnings which is the module state of the warnings module:

    WarningsState *st = _Warnings_GetState();
    if (st == NULL) {
        goto error;
    }
    if (_Warnings_InitState(st) < 0) {
        goto error;
    }

In Python 2, _PyWarnings_Init() called Py_InitModule3() which immediately stored the _warnings module to sys.modules['_warnings'].

In Python 3, _PyWarnings_Init() calls PyModule_Create() which creates a module but doesn't add it to sys.modules.

I don't think that removing _PyWarnings_Init() call from pylifecycle.c is correct. We want the _warnings module to be ready as early as possible.

The problem is that pylifecycle.c creates a module object which is not stored anywhere:

        /* Initialize _warnings. */
        if (_PyWarnings_Init() == NULL) {
            return _PyStatus_ERR("can't initialize warnings");
        }

pylifecycle.c should only call _Warnings_InitState() without creating a module.

There are C functions which access the module state (tstate->interp->warnings) without going through the module object, like PyErr_WarnFormat().
msg363169 - (view) Author: Hai Shi (shihai1991) * (Python triager) Date: 2020-03-02 12:49
Wow, Thanks, victor, much useful infos.

>In Python 3, _PyWarnings_Init() calls PyModule_Create() which creates a module but doesn't add it to sys.modules.

this operation will be executed in _bootstrap_external.

> We want the _warnings module to be ready as early as possible.

if we don't load _warnings module as soon as possible, what possible risks will be raise? 

> There are C functions which access the module state (tstate->interp->warnings) without going through the module object, like PyErr_WarnFormat().

This is important info.
msg363174 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-03-02 14:02
New changeset 66b7973c1b2e6aa6a2462c6b13971a08cd665af2 by Victor Stinner in branch 'master':
bpo-39796: Fix _warnings module initialization (GH-18739)
https://github.com/python/cpython/commit/66b7973c1b2e6aa6a2462c6b13971a08cd665af2
msg363175 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-03-02 14:04
> if we don't load _warnings module as soon as possible, what possible risks will be raise?

There are different risks:

* Worst case: using the _warnings module state before it's initialized may lead to crash.
* Best case: some warnings are not emitted.

Anyway, the initial issue should now be fixed by my commit.
msg363183 - (view) Author: Hai Shi (shihai1991) * (Python triager) Date: 2020-03-02 15:10
copy that, thanks for your explanation, victor.
History
Date User Action Args
2022-04-11 14:59:27adminsetgithub: 83977
2020-03-02 15:10:54shihai1991setmessages: + msg363183
2020-03-02 14:04:16vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg363175

stage: patch review -> resolved
2020-03-02 14:02:24vstinnersetmessages: + msg363174
2020-03-02 12:49:09shihai1991setmessages: + msg363169
2020-03-02 12:07:21vstinnersetpull_requests: + pull_request18094
2020-03-02 11:56:10vstinnersetnosy: + vstinner
messages: + msg363164
2020-02-29 13:07:28shihai1991setkeywords: + patch
stage: patch review
pull_requests: + pull_request18062
2020-02-29 13:03:37shihai1991create