At fork, Python calls PyOS_AfterFork_Child() in the child process which indirectly calls _PyThreadState_DeleteExcept() whichs calls release_sentinel() of the thread which releases the thread state lock (threading.Thread._tstate_lock).

Problem: using a lock after fork is unsafe and can crash.

That's exactly what happens randomly on AIX when stressing ThreadJoinOnShutdown.test_reinit_tls_after_fork() of test_threading:
https://bugs.python.org/issue40068#msg365031

There are different options to solve this issue:

* Reset _tstate_lock before using it... not sure that it's worth it, since we are going to delete the threading.Thread object with its _tstate_lock object anymore. After calling fork, the child process has exactly 1 thread: all other threads have been removed.

* Modify release_sentinel() to not use the lock: avoid PyThread_release_lock() call.

See also bpo-40089: "Add _at_fork_reinit() method to locks".

>since we are going to delete the threading.Thread object with its _tstate_lock object anymore.

Do we have any pep or discuss record about this plan?

> * Modify release_sentinel() to not use the lock: avoid PyThread_release_lock() call.

Hm, I am not sure about it. Looks like we can not remove it directly.

> Do we have any pep or discuss record about this plan?

"since we are going to delete the threading.Thread object with its _tstate_lock object anyway" sentence is a description of the current _PyThreadState_DeleteExcept() implementation.

> "since we are going to delete the threading.Thread object with its _tstate_lock object anyway" sentence is a description of the current _PyThreadState_DeleteExcept() implementation.

Oh, got it. thanks for your explanation :)

I marked bpo-43665 "AIX: test_importlib regression (ENV change)" as a duplicate of this issue.

test_multiprocessing_pool_circular_import() of test_importlib now triggers this crash as well.

Adding 3.10.

While a (sort of) duplicate I also would like to add that before revision "7cb033c423b65def1632d6c3c747111543b342a2" this was not showing up as an issue with test_importlib.

my issue was with test_importlib suddenly going into error.

As this seem to be a long-standing issue is it perhaps a possibility to change the test so that the bot can go green again?

> As this seem to be a long-standing issue is it perhaps a possibility to change the test so that the bot can go green again?

This issue is a real crash and it seems quite easy to get it. I prefer to not hide the bug in the test suite. The bug must be fixed.

OK: further.

Two options are suggested:

There are different options to solve this issue:

* Reset _tstate_lock before using it... not sure that it's worth it, since we are going to delete the threading.Thread object with its _tstate_lock object anymore. After calling fork, the child process has exactly 1 thread: all other threads have been removed.

* Modify release_sentinel() to not use the lock: avoid PyThread_release_lock() call.

** as to option 1 - it is 'worth it' if it stops the crashes

** This is deeper than I usually go in Python code - but I'll make an effort - help is appreciated.

OK. Please explain. Looking at tstate assignment

In posixmodule.c:PyOSAfterFork_Child()
    PyStatus status;
    _PyRuntimeState *runtime = &_PyRuntime;

...
    PyThreadState *tstate = _PyThreadState_GET();

and later calls
    status = _PyRuntimeState_ReInitThreads(runtime);

Yet in Posix/ceval.c
PyStatus
_PyEval_ReInitThreads(PyThreadState *tstate)
{
    _PyRuntimeState *runtime = tstate->interp->runtime;

** this looks like runtime->interp->runtime

And then we get down to:
    /* Destroy all threads except the current one */
    _PyThreadState_DeleteExcept(runtime, tstate);

Is this correct - as it looks like:
_PyThreadState_DeleteExcept(runtime->interp->runtime, runtime) -- where runtime == &_PyRuntime;

Willing to spend more time on this - but the variable names chosen blind me - looks like a circle.

And, thinking about the address in the core dump starting with 0xd (segment 13) - confuses me somewhat - as from memory - I thought the shared library code/data converged on segment 14 (ie, addresses would begin with 0xe).

So, perhaps it is the 0xd* address that is killing everything.

See: https://bugs.python.org/issue43665#msg389923 -- as I am also still confused re: this line - looks suspect: with all arguments at 0xdddddddd

PyVectorcall_Call(callable = 0xdddddddd, tuple = 0xdddddddd, kwargs = 

0xdddddddd), line 255 in "call.c"

And also this line, and lines like it - notice the value for what looks like a variable for argument_count:

_PyEval_Vector(tstate = 0x22023cf0, con = 0x2000120c, locals = 
0x22023960, args = 0x220239f0, argcount = 3508213100, kwnames = 
0x22023cf0), line 46 in "pycore_ceval.h"
_PyFunction_Vectorcall(func = 0x220239f0, stack = (nil), nargsf = 
570572016, kwnames = 0x220239f0), line 361 in "call.c"
method_vectorcall(method = 0x10103bdc, args = (nil), nargsf = 0, kwnames 
= 0x20045994), line 119 in "abstract.h"

How could release_sentinel() be structured to not call PyThread_release_lock()? This seems to be a situation where _PyThreadState_DeleteExcept() is deleting all thread states.  thread__set_sentinel() sets release_sentinel() as its on_delete hook. The thread state is being deleted, release_sentinel() is called and discovers that the lock is still held.

        if (lock->locked) {
            PyThread_release_lock(lock->lock_lock);

Do you suggest something like _PyThread_at_fork_reinit() and leak the memory? To not release the lock in the thread of the child process?

It seems that PyOS_AfterFork_Child() needs to do something like

PyThreadState *tstate = PyThreadState_Get();
PyObject *wr = _PyObject_CAST(tstate->on_delete_data);
PyObject *obj = PyWeakref_GET_OBJECT(wr);
lockobject *lock;
if (obj != Py_None) {
    lock = (lockobject *) obj;
    if (lock->locked) {
        /* Leak memory on purpose. */
        lock->locked = 0;
    }
}

before the call to _PyEval_ReInitThreads.

Maybe encapsulate that as PyThread_ReInitThreads().

The locks in the threads in the child need to be cleared before _PyThreadState_DeleteExcept() so that Python does not try to release the locks in the child.