classification
Title: Call _PyGILState_Init() earlier in Py_InitializeEx()
Type: crash Stage:
Components: Versions: Python 3.2, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: amaury.forgeotdarc, haypo, loewis, pitrou, tim.peters
Priority: normal Keywords: patch

Created on 2010-03-04 23:09 by haypo, last changed 2010-08-17 22:28 by haypo. This issue is now closed.

Files
File name Uploaded Description Edit
gil_state_init-py3k.patch haypo, 2010-08-17 22:14
Messages (11)
msg100432 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2010-03-04 23:09
_PyGILState_Init() initialize autoInterpreterState variable. This variable have to be set before the first call to PyGILState_Ensure().

The problem is that _PyGILState_Init() is called late: at the end of Py_InitializeEx(). It's called after initsite(), whereas initsite() may need to call _PyGILState_Init().

Example: add the following lines at the end of site.py:
---
import readline
import rlcompleter
readline.parse_and_bind("tab: complete")
raw_input("press TAB")
---
Run an interpreter and press TAB:
---
$ ./python
press TABpython: Python/pystate.c:595: PyGILState_Ensure: Assertion `autoInterpreterState' failed.
Abandon
---

Other example of functiions using _PyGILState_Init(): _PyObject_Dump() (useful for debug), sqlite module, os.popen*(), ctypes Python callbacks, etc.

I don't know the right place for _PyGILState_Init() in Py_InitializeEx(). _PyGILState_Init() calls the following functions:
 - PyThread_get_thread_ident()
 - PyThread_allocate_lock()
 - PyThread_acquire_lock()
 - PyThread_release_lock()
 - Py_FatalError() (on error)

None of these function use Python functions, so _PyGILState_Init() can be called very early. The earliest position is just after the call to PyThreadState_New(), before PyThreadState_Swap(). It tested it and it works correctly.

If _PyGILState_Init() is called before PyThreadState_Swap(), PyThreadState_Swap() can be simplified (it doesn't need to check if GIL is initialized or not). Attached patch implement that.

--

I hit this bug when I was debuging Python: I added a call to _PyObject_Dump() which stopped Python (assertion error) because the GIL was not initialized :-/

I already hitted this bug some weeks ago when I was working on the thread state preallocation when creating a new thread: #7544.
msg100433 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2010-03-04 23:12
haypo> I already hitted this bug some weeks ago when I was working 
haypo> on the thread state preallocation when creating a new thread:
haypo> #7544.

According to msg97715: it was indirectly the same issue, call _PyObject_Dump() while the GIL is not initialized.
msg100443 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-03-05 00:08
Bootstrap issues are always quite delicate, and I'd suggest being careful in this case.
msg100618 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2010-03-08 00:10
See also #3137.
msg100620 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2010-03-08 00:38
I would keep the call to PyThreadState_Swap() next to PyThreadState_New(): create the thread state and install it. _PyGILState_Init() may come after.
msg100622 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2010-03-08 00:44
haypo, it seems you removed the initial message...
msg100623 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2010-03-08 00:45
> haypo, it seems you removed the initial message...

I know... My mouse clicked on [remove] button, it wasn't me! I don't know how to restore it. The message: msg100432.
msg100624 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2010-03-08 01:02
I found the following issue in Roundup tracker to restore a deleted message:
http://psf.upfronthosting.co.za/roundup/meta/issue267

I tried to write the URL, but it doesn't work:
http://bugs.python.org/issue8063?@action=edit&@add@messages=100432

I tried also using Poster Firefox extension (to post a POST request, not a GET request), but it doesn't work. My URL should be wrong :-/
msg100668 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2010-03-08 21:21
My SIGINT.patch for #3137 moves the call to _PyGILState_Init() just before initsite(), so it doesn't change too much Py_InitializeEx() and it's enough for me :-)
msg114183 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2010-08-17 22:14
While working on #9425, I usually hit two annoying issues:
 - _PyObject_Dump() crashs (assertion error) if I call it (with gdb) in Py_InitializeEx()
 - because of python-gdb.py, gdb does segfault (I don't know yet where it does come from)

So I'm back on the GIL topic: I still would like to initialize the GIL earlier in Py_InitializeEx(). As Amaury wrote, I think that the right place is just after "(void) PyThreadState_Swap(tstate);". This is exactly what does my new patch (for py3k).

I think that only python 3.2 should be patched.
msg114185 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2010-08-17 22:28
Commited as r84163 to 3.2. Don't backport because it is not really a bug and I prefer to avoid touching stable branches with such minor details.
History
Date User Action Args
2010-08-17 22:28:14hayposetstatus: open -> closed
resolution: fixed
messages: + msg114185
2010-08-17 22:14:19hayposetstatus: closed -> open
resolution: fixed -> (no value)
2010-08-17 22:14:13hayposetfiles: - gil_state_init-trunk.patch
2010-08-17 22:14:05hayposetfiles: + gil_state_init-py3k.patch

messages: + msg114183
2010-03-21 14:06:38hayposetstatus: open -> closed
resolution: fixed
2010-03-10 20:17:36adminsetmessages: + msg100432
2010-03-08 21:21:14hayposetmessages: + msg100668
2010-03-08 01:02:27hayposetmessages: + msg100624
2010-03-08 00:45:45hayposetmessages: + msg100623
2010-03-08 00:44:09amaury.forgeotdarcsetmessages: + msg100622
2010-03-08 00:39:56hayposetmessages: - msg100432
2010-03-08 00:38:17amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg100620
2010-03-08 00:10:51hayposetmessages: + msg100618
2010-03-05 00:09:00pitrousetnosy: + pitrou
messages: + msg100443
2010-03-05 00:08:35pitrousetnosy: + tim.peters, loewis
2010-03-04 23:12:07hayposetmessages: + msg100433
2010-03-04 23:09:55haypocreate