classification
Title: Python crashes if Py_Initialize/Py_Finalize are called multiple times
Type: crash Stage: resolved
Components: Interpreter Core Versions: Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: amaury.forgeotdarc, lemburg, r37c, vstinner
Priority: normal Keywords: patch

Created on 2013-05-14 21:04 by r37c, last changed 2019-10-22 22:48 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
pythonrun.c.patch r37c, 2013-05-14 21:04 Patch for Python/pythonrun.c
Messages (5)
msg189250 - (view) Author: Rômulo (r37c) Date: 2013-05-14 21:04
I have patched (see attachment) Python 2.7.4 (as available for download at python.org/download) to disable initialization of Unicode (an embeded system requirement) and now it segfaults with the following program:

    #include <Python.h>
    
    int main(int argc, char** argv)
    {
      int i;
      Py_NoSiteFlag = 1;
    
      Py_SetProgramName(argv[0]);
    
      for (i = 0; i < 3; i++)
      {
        printf("run no. %d\n", i);
    
        Py_Initialize();
        Py_Finalize();
      }
    
      return 0;
    }

The problem appears to be related with the reference count of the empty tuple. I've also applied the following patch in Objects/tupleobject.c to help diagnose the problem:

    @@ -928,6 +928,8 @@ PyTuple_Fini(void)
     #if PyTuple_MAXSAVESIZE > 0
         /* empty tuples are used all over the place and applications may
          * rely on the fact that an empty tuple is a singleton. */
    +    printf("free_list[0]->ob_refcnt before XDECREF: %d\n",
    +        free_list[0]->ob_refcnt);
         Py_XDECREF(free_list[0]);
         free_list[0] = NULL;

*Without* the patch for Python/pythonrun.c the program produces the following results under Ubuntu 13.04 x64:

    run no. 0
    free_list[0]->ob_refcnt before XDECREF: 58
    run no. 1
    free_list[0]->ob_refcnt before XDECREF: 57
    run no. 2
    free_list[0]->ob_refcnt before XDECREF: 57

Note the strange ref count of the empty tuple (free_list[0]). Now, *with* the patch, the application will not hold so many references to the empty tuple and the finalization code ends up trying to deallocate it (what, from my limited understading of the code, is not supposed to happen):

    run no. 0
    free_list[0]->ob_refcnt before XDECREF: 2
    run no. 1
    free_list[0]->ob_refcnt before XDECREF: 1
    Segmentation fault (core dumped)

The actual patch I'm using is much more complicated. This is just the minimal patch able to reproduce the problem. I tried undefining Py_USING_UNICODE but then the build doesn't succeed.
msg189256 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2013-05-14 23:28
The official way to build without unicode is
  ./configure --enable-unicode=no
But see issue17979.
msg189260 - (view) Author: Rômulo (r37c) Date: 2013-05-15 02:00
I've made some further investigation on this issue. Here's the output and the stack trace using --with-pydebug (and the attached patch applied):

    run no. 0
    [8766 refs]
    free_list[0]->ob_refcnt before XDECREF: 2
    run no. 1
    [8844 refs]
    free_list[0]->ob_refcnt before XDECREF: 1
    Fatal Python error: PyThreadState_Get: no current thread
    Aborted (core dumped)

    Thread 1 (Thread 0x7ffff7fed700 (LWP 32572)):
    #0  0x00007ffff7131425 in raise () from /lib/x86_64-linux-gnu/libc.so.6
    #1  0x00007ffff7134b8b in abort () from /lib/x86_64-linux-gnu/libc.so.6
    #2  0x00000000004086f4 in Py_FatalError (
        msg=0x5a0378 "PyThreadState_Get: no current thread")
        at Python/pythonrun.c:1631
    #3  0x000000000054120c in PyThreadState_Get () at Python/pystate.c:330
    #4  0x000000000049a13b in tupledealloc (op=0x84e3a0)
        at Objects/tupleobject.c:218
    #5  0x000000000047adf1 in _Py_Dealloc (op=0x84e3a0) at Objects/object.c:2262
    #6  0x000000000049bba9 in PyTuple_Fini () at Objects/tupleobject.c:933
    #7  0x000000000040519b in Py_Finalize () at Python/pythonrun.c:466
    #8  0x00000000004047b6 in main ()

A build of Python 2.7.3 using the same steps *does not* crash, although it shows the same curious behavior for free_list[0]->ob_refcnt:

    [8676 refs]
    free_list[0]->ob_refcnt before XDECREF: 2
    run no. 1
    [8754 refs]
    free_list[0]->ob_refcnt before XDECREF: 1
    run no. 2
    [8832 refs]
    free_list[0]->ob_refcnt before XDECREF: 1

I have no idea why it behaves like that, but it seems that the default configuration with Unicode enabled is covering some bug (because of the additional references held).

Regarding the official way to disable Unicode, I cannot build Python using --enable-unicode=no on Ubuntu 13.04. In 2.7.4, 2.7.3, 2.7.2 and 2.6.8 ./configure aborts with:

    checking what type to use for unicode... configure: error:
    invalid value for --enable-unicode. Use either ucs2 or ucs4
    (lowercase).

I'm afraid I'm missing something really obvious. With 2.5.6 I'm able to get past the configure part but then it fails when building the modules.
msg189272 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2013-05-15 07:26
On 15.05.2013 01:28, Amaury Forgeot d'Arc wrote:
> 
> Amaury Forgeot d'Arc added the comment:
> 
> The official way to build without unicode is
>   ./configure --enable-unicode=no
> But see issue17979.

The official way to build without Unicode is:

./configure --disable-unicode

See http://bugs.python.org/issue8767 for the most recent set of fixes that
were supplied to make it work again in Python 2.7.4.
msg355166 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-10-22 22:48
Program described in msg189250 does not crash on the master branch of Python. The PEP 587 deeply reworked Python initialization code. There is now a preinitialization phase, for example.

Moreover, Programs/_testembed.c has a test calling Py_Initialize()+Py_Finalize() multiple times, and this test calls Py_SetProgramName() as well.

Upgrade to Python 3.8 to get the fix. (I didn't check if Python 3.7 is also fixed.)
History
Date User Action Args
2019-10-22 22:48:36vstinnersetstatus: open -> closed

nosy: + vstinner
messages: + msg355166

resolution: fixed
stage: resolved
2013-05-15 07:26:31lemburgsetnosy: + lemburg
messages: + msg189272
2013-05-15 02:00:34r37csetmessages: + msg189260
2013-05-14 23:28:29amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg189256
2013-05-14 21:04:28r37ccreate