classification
Title: Inefficient signal handling in multithreaded applications
Type: performance Stage: resolved
Components: Interpreter Core Versions: Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: eric.snow, vstinner
Priority: normal Keywords: patch

Created on 2020-03-19 00:57 by vstinner, last changed 2020-05-05 15:27 by eric.snow. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 19067 merged vstinner, 2020-03-19 01:31
PR 19077 merged vstinner, 2020-03-19 17:38
PR 19080 merged vstinner, 2020-03-19 18:28
PR 19087 merged vstinner, 2020-03-20 08:38
PR 19091 merged vstinner, 2020-03-20 13:13
Messages (8)
msg364580 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-03-19 00:57
When a thread gets a signal, SIGNAL_PENDING_SIGNALS() sets signals_pending to 1 and eval_breaker to 1. In this case, _PyEval_EvalFrameDefault() calls handle_signals(), but since it's not the main thread, it does nothing and signals_pending value remains 1. Moreover, eval_breaker value remains 1 which means that the following code will be called before executing *each* bytecode instruction.

    if (_Py_atomic_load_relaxed(eval_breaker)) {
        (...)
        opcode = _Py_OPCODE(*next_instr);
        if (opcode == SETUP_FINALLY || ...) {
            ...
        }
        if (_Py_atomic_load_relaxed(&ceval->signals_pending)) {
            if (handle_signals(tstate) != 0) {
                goto error;
            }
        }
        if (_Py_atomic_load_relaxed(&ceval->pending.calls_to_do)) {
            ...
        }

        if (_Py_atomic_load_relaxed(&ceval->gil_drop_request)) {
            ...
        }
        if (tstate->async_exc != NULL) {
            ...
        }
    }

This is inefficient.

I'm working on a PR modifying SIGNAL_PENDING_SIGNALS() to not set eval_breaker to 1 if the current thread is not the main thread.
msg364583 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-03-19 01:35
I found this issue while working on PR 19066 of bpo-39984.
msg364611 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-03-19 16:40
New changeset 5a3a71dddbe80edc75a3a74ce3b7978cf8635901 by Victor Stinner in branch 'master':
bpo-40010: Optimize signal handling in multithreaded applications (GH-19067)
https://github.com/python/cpython/commit/5a3a71dddbe80edc75a3a74ce3b7978cf8635901
msg364628 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-03-19 18:48
New changeset a36adfa6bbf5e612a4d4639124502135690899b8 by Victor Stinner in branch 'master':
bpo-39877: 4th take_gil() fix for daemon threads (GH-19080)
https://github.com/python/cpython/commit/a36adfa6bbf5e612a4d4639124502135690899b8
msg364659 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-03-20 08:29
New changeset da2914db4b6f786a1e9f0b424efeeb6ca9418912 by Victor Stinner in branch 'master':
bpo-40010: Pass tstate to ceval GIL functions (GH-19077)
https://github.com/python/cpython/commit/da2914db4b6f786a1e9f0b424efeeb6ca9418912
msg364666 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-03-20 12:39
New changeset d2a8e5b42c5e9c4e745a0589043a8aebb49f8ca2 by Victor Stinner in branch 'master':
bpo-40010: COMPUTE_EVAL_BREAKER() checks for subinterpreter (GH-19087)
https://github.com/python/cpython/commit/d2a8e5b42c5e9c4e745a0589043a8aebb49f8ca2
msg364667 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-03-20 13:50
New changeset d83168854e19d0381fa57db25fca6c622917624f by Victor Stinner in branch 'master':
bpo-40010: Optimize pending calls in multithreaded applications (GH-19091)
https://github.com/python/cpython/commit/d83168854e19d0381fa57db25fca6c622917624f
msg368171 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2020-05-05 15:27
Good catch on this, Victor.  Thanks for doing it.
History
Date User Action Args
2020-05-05 15:27:40eric.snowsetnosy: + eric.snow
messages: + msg368171
2020-03-25 17:33:17vstinnersetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2020-03-20 14:18:47vstinnersettitle: Inefficient sigal handling in multithreaded applications -> Inefficient signal handling in multithreaded applications
2020-03-20 13:50:38vstinnersetmessages: + msg364667
2020-03-20 13:13:01vstinnersetpull_requests: + pull_request18451
2020-03-20 12:39:06vstinnersetmessages: + msg364666
2020-03-20 08:38:42vstinnersetpull_requests: + pull_request18447
2020-03-20 08:29:20vstinnersetmessages: + msg364659
2020-03-19 18:48:34vstinnersetmessages: + msg364628
2020-03-19 18:28:08vstinnersetpull_requests: + pull_request18438
2020-03-19 17:38:09vstinnersetstage: patch review
pull_requests: + pull_request18433
2020-03-19 16:40:16vstinnersetmessages: + msg364611
2020-03-19 01:35:28vstinnersetmessages: + msg364583
stage: patch review -> (no value)
2020-03-19 01:31:35vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request18421
2020-03-19 00:57:25vstinnersettype: performance
2020-03-19 00:57:19vstinnercreate