classification
Title: PyThreadState_Next not thread safe
Type: Stage:
Components: Interpreter Core Versions: Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: ajaksu2, eric.snow, jd, jpe, pitrou, serhiy.storchaka, vstinner
Priority: normal Keywords:

Created on 2004-09-02 16:56 by jpe, last changed 2020-05-15 01:09 by vstinner.

Messages (14)
msg60562 - (view) Author: John Ehresman (jpe) * Date: 2004-09-02 16:56
I can't convince myself that PyThreadState_Next is
thread safe even if the GIL is held because
PyGILState_Release can delete a thread state after
releasing the GIL if the gilstate_counter reaches 0. 
So another thread could be deleting a thread state my
thread is referencing.

If the GIL was held when the thread state was deleted,
the I think the next function would be thread safe
because new thread states are always added to the front
of the list.

If these functions should work when the GIL is not
held, the head_mutex in pystate.c probably needs to be
exposed in some manner so code can lock it, iterate
through the threads, and then unlock it.
msg82072 - (view) Author: Daniel Diniz (ajaksu2) (Python triager) Date: 2009-02-14 14:43
Can someone take a look a this and either close or propose what to do?
msg114376 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2010-08-19 16:22
No reply to msg82072.
msg358187 - (view) Author: Julien Danjou (jd) * Date: 2019-12-10 13:37
This is actually true and it's quite easy to produce a bug that triggers a segmentation fault when iterating over a large list of threads create dynamically.

That makes PyThreadState_Next barely usable for any program that has a few threads. The head mutex needs to be locked to be able to iterate safely over the list of thread.
msg358188 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-12-10 13:38
> This is actually true and it's quite easy to produce a bug that triggers a segmentation fault when iterating over a large list of threads create dynamically.

This issue is closed. Would you be able to write a reproducer script? If yes, maybe open a new issue, or I can reopen this one.
msg358189 - (view) Author: Julien Danjou (jd) * Date: 2019-12-10 13:42
It'd be great if you could reopen it. I'm interested in fixing it properly.

It's impossible to "solve" in Python 2 since the head mutex is not accessible (it's static in pystate.c) but this is doable with Python 3 (yay).

We'd simply need to provide a new API to lock/unlock the interpreter's mutex.

How hard to you need the script to reproduce? I can try to spend some time to extract some of my code to have one if that's really needed (I'm using Cython to generate some of the C code).
msg358190 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-12-10 13:44
I reopened the issue and changed the version to 3.9.
msg358191 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-12-10 13:46
> We'd simply need to provide a new API to lock/unlock the interpreter's mutex.

Are you talking about this lock?

#define HEAD_LOCK(runtime) \
    PyThread_acquire_lock((runtime)->interpreters.mutex, WAIT_LOCK)
#define HEAD_UNLOCK(runtime) \
    PyThread_release_lock((runtime)->interpreters.mutex)
msg358192 - (view) Author: Julien Danjou (jd) * Date: 2019-12-10 13:50
Yes, that's the one 👍
msg358261 - (view) Author: Julien Danjou (jd) * Date: 2019-12-11 13:46
Victor, do you have an idea of how you would like this mutex to be exposed?

I see it only recently changed in Python 3.7 to be part of a structure, though this structure is not public AFAIU.

Is adding 2 new lock/unlock C functions good enough?
msg358262 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-12-11 14:02
I'm not sure of what is the best approach. Currently, bpo-10915 is blocking different projects: "Make the PyGILState API compatible with multiple interpreters". We may have to fix this API first, and clarify the scope of the different locks.
msg358347 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2019-12-13 17:50
On Wed, Dec 11, 2019 at 7:02 AM STINNER Victor <report@bugs.python.org> wrote:
> We may have to fix this API first, and clarify the scope of the different locks.

+1
msg364094 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-03-13 13:46
See also bpo-35370: "Add _PyEval_SetTrace(tstate, func, arg) function".
msg368895 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-05-15 01:09
I marked bpo-26461 as a duplicate of this issue.
History
Date User Action Args
2020-05-15 01:09:40vstinnersetmessages: + msg368895
2020-05-15 01:09:24vstinnerlinkissue26461 superseder
2020-03-13 13:46:53vstinnersetmessages: + msg364094
2019-12-13 17:50:40eric.snowsetmessages: + msg358347
2019-12-11 16:43:01BreamoreBoysetnosy: - BreamoreBoy
2019-12-11 14:02:09vstinnersetmessages: + msg358262
2019-12-11 13:46:21jdsetmessages: + msg358261
2019-12-10 13:50:04jdsetmessages: + msg358192
2019-12-10 13:46:53vstinnersetmessages: + msg358191
2019-12-10 13:44:49vstinnersetnosy: + pitrou, eric.snow, serhiy.storchaka
2019-12-10 13:44:37vstinnersetstatus: closed -> open
resolution: out of date ->
messages: + msg358190

versions: - Python 2.7, Python 3.5, Python 3.6, Python 3.7, Python 3.8
2019-12-10 13:42:39jdsetmessages: + msg358189
2019-12-10 13:38:20vstinnersetnosy: + vstinner
messages: + msg358188
2019-12-10 13:37:06jdsetversions: + Python 2.7, Python 3.5, Python 3.6, Python 3.7, Python 3.8, Python 3.9
nosy: + jd
title: PyThreadState_Next not thread safe? -> PyThreadState_Next not thread safe
messages: + msg358187

2010-08-19 16:22:08BreamoreBoysetstatus: open -> closed

nosy: + BreamoreBoy
messages: + msg114376

resolution: out of date
2009-02-14 14:43:19ajaksu2setnosy: + ajaksu2
messages: + msg82072
2004-09-02 16:56:47jpecreate