Title: Race condition using PyGILState_Ensure on a new thread
Type: Stage: resolved
Components: Interpreter Core Versions: Python 3.1, Python 3.2, Python 3.3, Python 2.7, Python 2.6
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: neologix, pitrou, syeberman, vstinner
Priority: normal Keywords:

Created on 2011-05-25 19:58 by syeberman, last changed 2019-10-23 00:26 by vstinner. This issue is now closed.

Messages (3)
msg136888 - (view) Author: Sye van der Veen (syeberman) * Date: 2011-05-25 19:58
I'm wanting to call PyThreadState_SetAsyncExc from a function registered with SetConsoleCtrlHandler.  To do so, I need to call PyGILState_Ensure, which asserts that Python is initialized, so I need to check for that.  However, I noticed a race condition with the code:

  if( Py_IsInitialized( ) ) {
    // XXX What if another thread calls Py_Finalize here?
    gstate = PyGILState_Ensure( );
    PyThreadState_SetAsyncExc( MainThreadId, PyExc_SystemExit );
    PyGILState_Release( gstate );

What I need is to be able to hold the GIL around the entire block of code, potentially before Py_Initialize is called for the first time.  Now, 3.2 deprecated PyEval_AcquireLock, and PyEval_InitThreads can no longer be called before Py_Initialize.  Thankfully, I'm on 2.6.4, so I was able to write this code:

  PyEval_AcquireLock( );
  if( Py_IsInitialized( ) ) {
    gstate = PyGILState_Ensure( );
    PyThreadState_SetAsyncExc( MainThreadId, PyExc_SystemExit );
    PyGILState_Release( gstate );
  PyEval_ReleaseLock( );

The problem in 2.6.4 is that PyGILState_Ensure deadlocks because the GIL is already held, so that doesn't solve my problem.  (Incidentally, the PyGILState_Ensure docs say it works "regardless of the current state of the GIL", which is incorrect.)

The 3.2 docs say to use PyEval_AcquireThread or PyEval_RestoreThread, which both require an existing PyThreadState.  To get that, I would need to call PyThreadState_New, which needs a PyInterpreterState.  To get _that_ I could use PyInterpreterState_Head, since I know I only use one interpreter.  Now I'm getting into "advanced debugger" territory, but it's no use anyway; it's possible that Py_Finalize could sneak in between the time I get this interpreter and when I acquire the GIL, causing me to access a free'd interpreter.

I believe the best fix for this would be to have a version of PyGILState_Ensure that works even when Python is not initialized.  It would not be able to create a thread, and thus I would not expect to be able to call any Python API, but it would always ensure the GIL is acquired.  This _may_ have to be a new "PyGILState_EnsureEx" function, because existing code expects PyGILState_Ensure to always allow them to call the Python API.  The resulting code would be:

  gstate = PyGILState_EnsureEx( );
  if( Py_IsInitialized( ) ) {
    PyThreadState_SetAsyncExc( MainThreadId, PyExc_SystemExit );
  PyGILState_Release( gstate );

This would require that Py_Initialize itself acquires the GIL.

The above problem was found on 2.6.4, but I've consulted the 3.2 docs and 3.3 code (via the online source) and it looks like the situation would be exactly the same.  In the meantime, I'm going to stick with the first piece of code and hope nobody hits CTRL-BREAK during program clean-up.
msg221433 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2014-06-24 08:14
Can one of you on the nosy list pick this up please, it's way out of my league.
msg355197 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-10-23 00:26
Bug report in 2011. I close this old issue.
Date User Action Args
2019-10-23 00:26:38vstinnersetstatus: open -> closed
resolution: out of date
messages: + msg355197

stage: resolved
2019-03-15 23:40:09BreamoreBoysetnosy: - BreamoreBoy
2014-06-24 08:14:33BreamoreBoysetnosy: + BreamoreBoy
messages: + msg221433
2011-05-25 20:02:45vstinnersetnosy: + pitrou, vstinner, neologix
2011-05-25 19:58:14syebermancreate