This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Race condition and crash when embedding multi-thread script
Type: crash Stage: resolved
Components: Extension Modules Versions: Python 2.7
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: Oleksiy Markovets, iritkatriel
Priority: normal Keywords:

Created on 2015-07-01 13:05 by Oleksiy Markovets, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
main.cpp Oleksiy Markovets, 2015-07-01 13:05 c++ part
main.py Oleksiy Markovets, 2015-07-01 13:06 python part
Messages (3)
msg246035 - (view) Author: Oleksiy Markovets (Oleksiy Markovets) Date: 2015-07-01 13:05
INTRODUCTION

While embedding python script in c++ application I faced random crashes 
with error:

Fatal Python error: Py_EndInterpreter: not the last thread                                                                                                                                                                                                                     
 Aborted  

Even though I explicitly join each thread created with *threading* 
module. Please see attachment, this is simplest python script +
c++ code which demonstrates problem.

By using threading.setprofiler I was able to make problem appear each run.
If you uncomment  time.sleep(1) problem won't reproduce.

Note:
In main.cpp I manually create separate interpreter because in real-life
application is multi-threaded and different thread uses different interpreters
for at least some sand-boxing.

INVESTIGATION

I did some investigation, and here is what I found out:

* Each new thread is started by calling *thread_PyThread_start_new_thread* from threadmodule.c.
Basically this function creates new thread which executes *t_bootstrap* and adds this
thread to interpreter's thread list.

* *t_bootstrap* executes python function *Thread.__bootstrap* and when it's done, removes
thread from interpreter's thread list.

* *Thread.__bootstrap* runs *Thread.run()* (actually python code which should be executed
in separate thread) and when it's done calls *Thread.__stop* (which by mean of condition
variable sets Boolean flag Thread.__stopped)

* *Thread.join* doesn't wait thread to exit, it only waits for *Thread.__stopped* flag
to be set.


So here is race condition:
When main thread finished *Thread.join* it's only guaranteed that *Thread.run*
is finished, but *t_bootstrap* still may be running (interpreter's thread list
is not cleared).

POSSIBLE SOLUTION

To call Thread.__stop from *t_bootstrap* (instead of Thread.__bootstrap) after
removing thread from interpreter's thread list.

Or do not use detached threads and call something like *pthread_join* in *Thread.join*.
Which is IMHO much clearer approach but also requires much more efforts.
msg246036 - (view) Author: Oleksiy Markovets (Oleksiy Markovets) Date: 2015-07-01 13:06
attached file
msg401508 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-09-09 18:31
Python 2.7 is no longer maintained. Please create a new issue if you are having problems with this in a current version (>= 3.9).
History
Date User Action Args
2022-04-11 14:58:18adminsetgithub: 68732
2021-09-09 18:31:46iritkatrielsetstatus: open -> closed

nosy: + iritkatriel
messages: + msg401508

resolution: out of date
stage: resolved
2015-07-01 13:06:00Oleksiy Markovetssetfiles: + main.py

messages: + msg246036
2015-07-01 13:05:29Oleksiy Markovetscreate