classification
Title: Crash with subinterpreters and Py_NewInterpreter()
Type: crash Stage:
Components: Interpreter Core Versions: Python 2.7
process
Status: pending Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: eric.snow, steveire, vstinner
Priority: normal Keywords:

Created on 2017-04-12 18:01 by steveire, last changed 2019-11-22 22:28 by eric.snow.

Messages (4)
msg291568 - (view) Author: Stephen Kelly (steveire) Date: 2017-04-12 18:01
When attempting to use PyImport_ImportModule("os") (or to import many other libraries), there is a crash on Py_Finalize if Py_NoSiteFlag is set. The issue appears to be the use of frozenset() as a result of importing the module.

I reproduced this on Windows after building 2.7.13 with VS 2015 by applying the following patch:


Python changes.

--- Include\\fileobject.h
+++ Include\\fileobject.h
@@ -70,7 +70,7 @@
 */
 int _PyFile_SanitizeMode(char *mode);
 
-#if defined _MSC_VER && _MSC_VER >= 1400
+#if defined _MSC_VER && _MSC_VER >= 1400 && _MSC_VER < 1900
 /* A routine to check if a file descriptor is valid on Windows.  Returns 0
  * and sets errno to EBADF if it isn't.  This is to avoid Assertions
  * from various functions in the Windows CRT beginning with
--- Modules\\posixmodule.c
+++ Modules\\posixmodule.c
@@ -529,7 +529,7 @@
 #endif
 
 
-#if defined _MSC_VER && _MSC_VER >= 1400
+#if defined _MSC_VER && _MSC_VER >= 1400 && _MSC_VER < 1900
 /* Microsoft CRT in VS2005 and higher will verify that a filehandle is
  * valid and raise an assertion if it isn't.
  * Normally, an invalid fd is likely to be a C program error and therefore
--- Modules\\timemodule.c
+++ Modules\\timemodule.c
@@ -68,6 +70,9 @@
 #if defined(MS_WINDOWS) && !defined(__BORLANDC__)
 /* Win32 has better clock replacement; we have our own version below. */
 #undef HAVE_CLOCK
+#define timezone _timezone
+#define tzname _tzname
+#define daylight _daylight
 #endif /* MS_WINDOWS && !defined(__BORLANDC__) */
 
 #if defined(PYOS_OS2)



Backtrace:


 	KernelBase.dll!00007ff963466142()	Unknown
>	python27_d.dll!Py_FatalError(const char * msg) Line 1700	C
 	python27_d.dll!PyThreadState_Get() Line 332	C
 	python27_d.dll!set_dealloc(_setobject * so) Line 553	C
 	python27_d.dll!_Py_Dealloc(_object * op) Line 2263	C
 	python27_d.dll!PySet_Fini() Line 1084	C
 	python27_d.dll!Py_Finalize() Line 526	C
 	mn.exe!main(int argc, char * * argv) Line 40	C
 	[External Code]	



Reproducing code:


#include <Python.h>

int main(int argc, char** argv)
{
    // http://www.awasu.com/weblog/embedding-python/threads

    // #### Comment this to avoid crash
    Py_NoSiteFlag = 1;

    Py_Initialize();
    PyEval_InitThreads(); // nb: creates and locks the GIL
                        // NOTE: We save the current thread state, and restore it when we unload,
                        // so that we can clean up properly.
    PyThreadState* pMainThreadState = PyEval_SaveThread(); // nb: this also releases the GIL

    PyEval_AcquireLock(); // nb: get the GIL

    PyThreadState* pThreadState = Py_NewInterpreter();
    assert(pThreadState != NULL);
    PyEval_ReleaseThread(pThreadState); // nb: this also releases the GIL

    PyEval_AcquireThread(pThreadState);

    // Can reproduce by importing the os module, but the issue actually appears 
    // because of the use of frozenset, so simplify to that.
#if 0
    PyObject* osModule = PyImport_ImportModule("os");
    Py_DECREF(osModule);
#endif

    // As in abc.py ABCMeta class
    PyRun_SimpleString("abstractmethods = frozenset(set())");
    
    PyEval_ReleaseThread(pThreadState);

    // release the interpreter 
    PyEval_AcquireThread(pThreadState); // nb: this also locks the GIL
    Py_EndInterpreter(pThreadState);
    PyEval_ReleaseLock(); // nb: release the GIL
  
    // clean up
    PyEval_RestoreThread(pMainThreadState); // nb: this also locks the GIL
    Py_Finalize();
}
msg291592 - (view) Author: Stephen Kelly (steveire) Date: 2017-04-13 08:33
I found that if I build and run this code with Python 3, then I get a very different backtrace.

 	KernelBase.dll!00007ff963466142()	Unknown
 	python36_d.dll!Py_FatalError(const char * msg) Line 1457	C
 	python36_d.dll!PyEval_AcquireLock() Line 253	C
	mn.exe!main(int argc, char * * argv) Line 22	C
 	[External Code]	

The failure is

 Py_FatalError("PyEval_AcquireLock: current thread state is NULL");

So, it is not clear to me what is incorrect about this code, but presumably I have done something incorrect here?
msg291599 - (view) Author: Stephen Kelly (steveire) Date: 2017-04-13 10:54
The issue

 http://bugs.python.org/issue17978 

has a quite similar backtrace and there is discussion in

 http://bugs.python.org/issue17703#msg241412

about changing the TRASHCAN macro to access the _PyThreadState_Current directly instead of calling PyThreadState_Get (which is fatal for a nullptr). The very next thing the TRASHCAN macro does is check if the thread state is nullptr in a non-fatal way. 

Is there a reason not to apply that patch?
msg357330 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2019-11-22 22:28
Victor's comment [1] on that related issue, implies that this may no longer be a problem (on 3.8).  Please check.  Thanks!


[1] https://bugs.python.org/issue17978#msg355166
History
Date User Action Args
2019-11-22 22:28:27eric.snowsetstatus: open -> pending

messages: + msg357330
2019-10-22 23:40:58vstinnersetnosy: + eric.snow

title: Crash on Py_Finalize if Py_NoSiteFlag is used -> Crash with subinterpreters and Py_NewInterpreter()
2017-04-21 17:04:47vstinnersetnosy: + vstinner
2017-04-13 10:54:46steveiresetmessages: + msg291599
2017-04-13 08:33:29steveiresetmessages: + msg291592
2017-04-12 18:01:11steveirecreate