classification
Title: Document that PyEval_RestoreThread and PyGILState_Ensure can terminate the calling thread
Type: Stage: resolved
Components: Interpreter Core Versions: Python 3.8, Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: eric.snow, pablogsal, pitrou, vstinner
Priority: normal Keywords: patch

Created on 2019-03-25 22:14 by pablogsal, last changed 2019-04-14 02:49 by pablogsal. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 12541 merged pablogsal, 2019-03-25 22:15
PR 12820 merged pablogsal, 2019-04-13 16:30
Messages (11)
msg338826 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-03-25 22:14
Currently PyEval_RestoreThread and its callers (mainly PyGILState_Ensure) can terminate the thread if the interpreter is finalizing:

PyEval_RestoreThread(PyThreadState *tstate)
{
    if (tstate == NULL)
        Py_FatalError("PyEval_RestoreThread: NULL tstate");
    assert(gil_created());

    int err = errno;
    take_gil(tstate);
    /* _Py_Finalizing is protected by the GIL */
    if (_Py_IsFinalizing() && !_Py_CURRENTLY_FINALIZING(tstate)) {
        drop_gil(tstate);
        PyThread_exit_thread();
        Py_UNREACHABLE();
    }
    errno = err;

    PyThreadState_Swap(tstate);
}

This behaviour that protects against problems due to daemon threads registered with the interpreter can be *very* surprising for C-extensions that are using these functions to implement callbacks that can call into Python. These callbacks threads are not owned by the interpreter and are usually joined by someone else, ending in deadlocks in many situations.

I propose to add a warning to the documentation to inform users about this situation.
msg338827 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-03-25 22:16
> if (_Py_IsFinalizing() && !_Py_CURRENTLY_FINALIZING(tstate))

_Py_IsFinalizing() check is redundant :-)
msg338828 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-03-25 22:17
> This behaviour that protects against problems due to daemon threads registered with the interpreter can be *very* surprising for C-extensions that are using these functions to implement callbacks that can call into Python.

I really dislike the design of daemon threads. If it would be only me, I would prefer this nasty feature causing so much issues at Python shutdown.
msg338830 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-03-25 22:28
> I really dislike the design of daemon threads. If it would be only me, I would prefer this nasty feature causing so much issues at Python shutdown.

Well, given that this happens as well in Python3.7 and before, at least we should document it and we can think about changing it in the future. But for now, but I suggest keeping both PRs separated (in case we really want to change anything).
msg338831 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-03-25 22:28
> Well, given that this happens as well in Python3.7 and before, at least we should document it and we can think about changing it in the future. But for now, but I suggest keeping both PRs separated (in case we really want to change anything).

Right. It was just a general comment :-)
msg338832 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-03-25 22:30
> Right. It was just a general comment :-)
Yep, daemon threads are evil :)
msg339153 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2019-03-29 21:26
Related:

* #36475: "PyEval_AcquireLock() and PyEval_AcquireThread() do not handle runtime finalization properly."
* #36476: "Runtime finalization assumes all other threads have exited."
msg339154 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2019-03-29 21:30
> > if (_Py_IsFinalizing() && !_Py_CURRENTLY_FINALIZING(tstate))
>
> _Py_IsFinalizing() check is redundant :-)

Not really:

* _Py_IsFinalizing() checks if the runtime is finalizing
* _Py_CURRENTLY_FINALIZING checks if the current thread is the one
running finalization
msg339156 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2019-03-29 22:01
> Currently PyEval_RestoreThread and its callers (mainly PyGILState_Ensure)
> can terminate the thread if the interpreter is finalizing:

s/interpreter/runtime/

Most likely (guaranteed?) it will be in the main interpreter, but it is actually not triggered by interpreter finalization (though it probably should be; I've opened #36479).
msg340167 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-04-13 16:23
New changeset fde9b33dfeedd4a4ed723b12d2330979dc684760 by Pablo Galindo in branch 'master':
bpo-36427: Document that PyEval_RestoreThread and PyGILState_Ensure can terminate the calling thread (GH-12541)
https://github.com/python/cpython/commit/fde9b33dfeedd4a4ed723b12d2330979dc684760
msg340178 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-04-14 02:49
New changeset 7723d0545c3369e1b2601b207c250c70ce90b75e by Pablo Galindo in branch '3.7':
[3.7] bpo-36427: Document that PyEval_RestoreThread and PyGILState_Ensure can terminate the calling thread (GH-12541) (GH-12820)
https://github.com/python/cpython/commit/7723d0545c3369e1b2601b207c250c70ce90b75e
History
Date User Action Args
2019-04-14 02:49:46pablogsalsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2019-04-14 02:49:32pablogsalsetmessages: + msg340178
2019-04-13 16:30:49pablogsalsetpull_requests: + pull_request12745
2019-04-13 16:23:27pablogsalsetmessages: + msg340167
2019-03-29 22:01:16eric.snowsetmessages: + msg339156
2019-03-29 21:30:35eric.snowsetmessages: + msg339154
2019-03-29 21:26:09eric.snowsetmessages: + msg339153
2019-03-25 22:30:00pablogsalsetmessages: + msg338832
2019-03-25 22:28:55vstinnersetmessages: + msg338831
2019-03-25 22:28:05pablogsalsetmessages: + msg338830
2019-03-25 22:17:33vstinnersetmessages: + msg338828
2019-03-25 22:16:24vstinnersetmessages: + msg338827
2019-03-25 22:15:27pablogsalsetkeywords: + patch
stage: patch review
pull_requests: + pull_request12490
2019-03-25 22:14:35pablogsalcreate