classification
Title: Issues with PyEval_InitThreads and PyGILState_Ensure
Type: Stage: resolved
Components: Interpreter Core Versions: Python 3.6, Python 2.7
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: iritkatriel, pitrou, tzickel, vstinner
Priority: normal Keywords:

Created on 2016-01-03 19:04 by tzickel, last changed 2021-01-20 11:15 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
signalexample.py tzickel, 2016-01-03 19:04 an example for part D
Messages (3)
msg257425 - (view) Author: (tzickel) * Date: 2016-01-03 19:04
A few issues regarding threads:
A. (Python 2 & 3) The documentation (https://docs.python.org/3/c-api/init.html) about initializing the GIL/Threading system does not specify that calling PyEval_InitThreads actually binds the calling thread as the main_thread in the ceval.c, meaning that the thread will be in charge till the process goes down for handling Py_AddPendingCall calls, and if it ends/dies, they won't be handled anymore.

This ceval.c's main_thread is different BTW from the one in signalmodule.c which is bound to the thread that called Py_InitializeEx.

Maybe there is sense for both main_thread to be the same one and initialized in the same time ? (even without a GIL existing)

B. (Python 3) Besides the bad documentation regarding this, python 3.4 added issue #19576 which actually hides the call for PyEval_InitThreads inside PyGILState_Ensure. Without careful care and knowledge by the programmer, this might cause for a short lived thread created in C to actually bind the ceval.c's main_thread and when the thread dies main_thread will never be changed again.

The reason this is important is beforehand, the programmer needed to think about PyEval_InitThreads now it's hidden and not even mentioned in the documentation.

C. (Python 2 & 3) In PyEval_InitThreads documentation it's written "It is not safe to call this function when it is unknown which thread (if any) currently has the global interpreter lock." Thus it should be mentioned that PyGILState_Ensure is now calling it in the documentation ?

Also I believe the reason this quote exists is because a potential race condition between thread A which might be running code in PyEval_EvalFrameEx (before PyEval_InitThreads is called, and thus is not GIL aware), and thread B which calls PyEval_InitThreads then calls PyGILState_Ensure, then running Python code, while thread A is still running python code as well. I think it should be explained more clearly in the documentation the implications (race condition).

I think there might be a way to make an PyEval_InitThreads variant which can overcome this race condition. Basically it involves using Py_AddPendingCall to a C function which calls PyEval_InitThreads, and notifies the calling command/thread when it's done. This way we can be sure that the GIL is taken by one thread, and all the others are blocked. (maybe a signal should be sent as well, in case the main_thread is blocked on an I/O operation).

D. (Python 2) If the main_thread finishes it's job, while other python threads are still alive, signal handling isn't processed anymore (Example will be added as a file).
msg383400 - (view) Author: Irit Katriel (iritkatriel) * (Python triager) Date: 2020-12-19 20:26
Since PyEval_InitThreads is now an empty function and to be removed, this issue seems out of date. A I right?
msg385333 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-01-20 11:15
The GIL is now created by _PyEval_InitGIL() which is called by Py_Initialize(): at Python startup.

_PyRuntime.main_thread is set by _PyRuntime_Initialize() which is also called by Py_Initialize(): at Python startup.

The code to handle signals and pending calls changed a lot since 2016. I close the issue.
History
Date User Action Args
2021-01-20 11:15:43vstinnersetstatus: pending -> closed

messages: + msg385333
stage: resolved
2020-12-19 20:26:32iritkatrielsetstatus: open -> pending

nosy: + iritkatriel
messages: + msg383400

resolution: out of date
2016-01-15 21:29:12SilentGhostsetnosy: + vstinner
2016-01-03 19:07:00tzickelsetnosy: + pitrou
2016-01-03 19:04:26tzickelcreate