classification
Title: regression: PySys_SetArgvEx(0, NULL, 0): SystemError: Python-3.8.0b3/Objects/unicodeobject.c:2089: bad argument to internal function
Type: Stage: resolved
Components: Extension Modules Versions: Python 3.9, Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: hroncok, vstinner
Priority: normal Keywords: patch

Created on 2019-08-23 07:39 by hroncok, last changed 2019-09-05 14:43 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 15415 merged vstinner, 2019-08-23 09:35
PR 15420 merged vstinner, 2019-08-23 10:21
Messages (6)
msg350262 - (view) Author: Miro Hrončok (hroncok) * Date: 2019-08-23 07:39
There is a regression between Python 3.7 and 3.8 when using PySys_SetArgvEx(0, NULL, 0).

Consider this example:

#include <Python.h>

int main() {
    Py_Initialize();
    PySys_SetArgvEx(0, NULL, 0);  /* HERE */
    PyRun_SimpleString("from time import time,ctime\n"
                       "print('Today is', ctime(time()))\n");
    Py_FinalizeEx();
    return 0;
}


This works in 3.7 but no longer does in 3.8:

$ gcc $(python3.7-config --cflags --ldflags) example.c 
$ ./a.out 
Today is Fri Aug 23 07:59:52 2019

$ gcc $(python3.8-config --cflags --ldflags --embed) example.c 
$ ./a.out 
Fatal Python error: no mem for sys.argv
SystemError: /builddir/build/BUILD/Python-3.8.0b3/Objects/unicodeobject.c:2089: bad argument to internal function

Current thread 0x00007f12c78b9740 (most recent call first):
Aborted (core dumped)


The documentation https://docs.python.org/3/c-api/init.html#c.PySys_SetArgvEx explicitly mentions passing 0 to PySys_SetArgvEx:

"if argc is 0..."

So I guess this is not something you shouldn't do.
msg350268 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-08-23 09:39
Oops, it's my fault! PR 15415 fix the crash.

The regression was introduced by:

commit 74f6568bbd3e70806ea3219e8bacb386ad802ccf
Author: Victor Stinner <vstinner@redhat.com>
Date:   Fri Mar 15 15:08:05 2019 +0100

    bpo-36301: Add _PyWstrList structure (GH-12343)

Simplified change:

-        static wchar_t *empty_argv[1] = {L""};
+        wchar_t* empty_argv[1] = {L""};

It's a deliberate change to not waste memory: PySys_SetArgvEx() (make_sys_argv()) is only called once, whereas static keeps the memory for the whole lifecycle of the process.

But I didn't notice a subtle issue with memory allocated on the stack: the code works well with gcc -O0! The bug only triggers with gcc -O3.
msg350269 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-08-23 10:04
New changeset c48682509dc49b43fe914fe6c502bc390345d1c2 by Victor Stinner in branch 'master':
bpo-37926: Fix PySys_SetArgvEx(0, NULL, 0) crash (GH-15415)
https://github.com/python/cpython/commit/c48682509dc49b43fe914fe6c502bc390345d1c2
msg350275 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-08-23 11:22
New changeset ca9ae94a2aba35d94ac1ec081f9bcac3a13aebd3 by Victor Stinner in branch '3.8':
bpo-37926: Fix PySys_SetArgvEx(0, NULL, 0) crash (GH-15415) (GH-15420)
https://github.com/python/cpython/commit/ca9ae94a2aba35d94ac1ec081f9bcac3a13aebd3
msg350276 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-08-23 11:23
I tested my fix manually using example from msg350262 and gcc -O3 (./configure && make): I can reproduce the crash without the change, and I confirm that my change fix the crash.

Thanks Miro for the bug report: the fix will be included in next Python 3.8 beta release. I fixed the bug in 3.8 and master branches (3.7 is not affected).
msg351201 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-05 14:43
FYI this bug was found in paraview in Fedora Rawhide: https://bugzilla.redhat.com/show_bug.cgi?id=1743896
History
Date User Action Args
2019-09-05 14:43:15vstinnersetmessages: + msg351201
2019-08-23 11:23:48vstinnersetstatus: open -> closed
versions: + Python 3.9
messages: + msg350276

resolution: fixed
stage: patch review -> resolved
2019-08-23 11:22:20vstinnersetmessages: + msg350275
2019-08-23 10:21:02vstinnersetpull_requests: + pull_request15120
2019-08-23 10:04:19vstinnersetmessages: + msg350269
2019-08-23 09:39:47vstinnersetmessages: + msg350268
2019-08-23 09:35:21vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request15117
2019-08-23 07:39:09hroncokcreate