classification
Title: Repeated Py_Initialize and Py_Finalize usage brings fatal error?
Type: Stage: resolved
Components: Interpreter Core Versions: Python 3.2
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: pitrou Nosy List: ocean-city, pitrou
Priority: normal Keywords: patch

Created on 2010-09-11 04:37 by ocean-city, last changed 2010-09-13 14:17 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
gildestroy.patch pitrou, 2010-09-12 16:51
Messages (8)
msg116075 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2010-09-11 04:37
When I ran the code in #6869 (msg95444) on py3k, I noticed
Fatal Python error occurs.

# Actually, I need to copy created executable file into
# PC/VC6 directoly where python32_d.dll exists. Otherwise,
# the error "encoding.utf-8 was not found" occured even if
# PC/VC6 was added to environment variable PATH. But probably
# it is another issue.

E:\python-dev\py3k\PC\VC6>foo
--- 0 ---
[34859 refs]
--- 1 ---
Fatal Python error: drop_gil: wrong thread state

... And here is stacktrace.

_NMSG_WRITE(int 10) line 221
abort() line 44 + 7 bytes
Py_FatalError(const char * 0x1e371878) line 2109
drop_gil(_ts * 0x00920d90) line 321 + 10 bytes
PyEval_SaveThread() line 424 + 6 bytes
0012ff30()
PyCFunction_Call(_object * 0x00b99378, _object * 0x00afcc40, _object * 0x00000000) line 89 + 9 bytes
PyObject_Call(_object * 0x00b99378, _object * 0x00afcc40, _object * 0x00000000) line 2149 + 15 bytes
call_function_tail(_object * 0x00b99378, _object * 0x00afcc40) line 2181 + 15 bytes
_PyObject_CallMethod_SizeT(_object * 0x00bf6838, char * 0x1e332898, char * 0x00000000) line 2297 + 13 bytes
io_open(_object * 0x00c17f78, _object * 0x00ae0f88, _object * 0x00000000) line 423 + 16 bytes
PyCFunction_Call(_object * 0x00b0cab8, _object * 0x00ae0f88, _object * 0x00000000) line 84 + 15 bytes
PyObject_Call(_object * 0x00b0cab8, _object * 0x00ae0f88, _object * 0x00000000) line 2149 + 15 bytes
call_function_tail(_object * 0x00b0cab8, _object * 0x00ae0f88) line 2181 + 15 bytes
PyObject_CallMethod(_object * 0x00bf6138, char * 0x1e3d6abc, char * 0x1e3d6ab4) line 2258 + 13 bytes
create_stdio(_object * 0x00bf6138, int 0, int 0, char * 0x1e3d6968, char * 0x00000000, char * 0x00000000) line 808 + 48 bytes
initstdio() line 931 + 28 bytes
Py_InitializeEx(int 1) line 303 + 5 bytes
Py_Initialize() line 314 + 7 bytes
main() line 10 + 8 bytes
mainCRTStartup() line 206 + 25 bytes
KERNEL32! 77e789d5()

Because first run succeeded, Py_Finalize might not do enough
work to reset thread state, I guess.

I didn't try this on other platforms. (thread_nt.h was used here)
msg116079 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2010-09-11 05:36
Sorry, msg92444
msg116174 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-09-12 11:35
I can't reproduce under Linux. Is 3.1 also affected?
Can you try the following patch:

Index: Python/pythonrun.c
===================================================================
--- Python/pythonrun.c	(révision 84718)
+++ Python/pythonrun.c	(copie de travail)
@@ -219,6 +219,8 @@
 
     /* auto-thread-state API, if available */
 #ifdef WITH_THREAD
+    if (PyEval_ThreadsInitialized())
+        PyEval_ReInitThreads();
     _PyGILState_Init(interp, tstate);
 #endif /* WITH_THREAD */
msg116194 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-09-12 13:50
It should be noted that repeatedly calling Py_Initialize / Py_Finalize leaks a lot (even without the "import _ctypes"):

--- 0 ---
[34543 refs]
--- 1 ---
[37540 refs]
--- 2 ---
[40504 refs]
--- 3 ---
[43468 refs]
--- 4 ---
[46432 refs]
--- 5 ---
[49396 refs]
--- 6 ---
[52360 refs]
--- 7 ---
[55324 refs]
--- 8 ---
[58288 refs]
--- 9 ---
[61252 refs]
--- 10 ---
[64216 refs]

And it also translates into memory consumption...
msg116208 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2010-09-12 16:00
> Is 3.1 also affected?
Sorry, 3.1 is not affected.

> Can you try the following patch:
Your patch works. :-)

E:\python-dev\py3k\PC\VC6>ctypes_crash.exe
--- 0 ---
[36242 refs]
--- 1 ---
[39140 refs]
--- 2 ---
[41538 refs]
--- 3 ---
[43930 refs]
--- 4 ---
[46322 refs]
--- 5 ---
[48714 refs]
--- 6 ---
[51106 refs]
--- 7 ---
[53498 refs]
--- 8 ---
[55890 refs]
--- 9 ---
[58282 refs]
msg116217 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-09-12 16:51
Ok. How about this patch now?
(I'm trying to solve this in a clean way :))
msg116241 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2010-09-12 22:44
gildestroy.patch also works.

E:\PYTHON~1\py3k\PC\VC6>ctypes_crash.exe
--- 0 ---
[36303 refs]
--- 1 ---
[39184 refs]
--- 2 ---
[41582 refs]
--- 3 ---
[43974 refs]
--- 4 ---
[46366 refs]
--- 5 ---
[48758 refs]
--- 6 ---
[51150 refs]
--- 7 ---
[53542 refs]
--- 8 ---
[55934 refs]
--- 9 ---
[58326 refs]
msg116308 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-09-13 14:17
Fix committed in r84774. Thank you!
History
Date User Action Args
2010-09-13 14:17:26pitrousetstatus: open -> closed
resolution: fixed
messages: + msg116308

stage: resolved
2010-09-12 22:44:03ocean-citysetmessages: + msg116241
2010-09-12 16:51:16pitrousetfiles: + gildestroy.patch
keywords: + patch
messages: + msg116217
2010-09-12 16:00:25ocean-citysetmessages: + msg116208
versions: - Python 3.1
2010-09-12 13:50:53pitrousetmessages: + msg116194
2010-09-12 11:35:03pitrousetmessages: + msg116174
2010-09-12 10:54:03georg.brandlsetassignee: pitrou

nosy: + pitrou
2010-09-11 05:36:35ocean-citysetmessages: + msg116079
2010-09-11 04:37:45ocean-citycreate