Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Inefficient signal handling in multithreaded applications #84191

Closed
vstinner opened this issue Mar 19, 2020 · 8 comments
Closed

Inefficient signal handling in multithreaded applications #84191

vstinner opened this issue Mar 19, 2020 · 8 comments
Labels
3.9 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) performance Performance or resource usage

Comments

@vstinner
Copy link
Member

BPO 40010
Nosy @vstinner, @ericsnowcurrently
PRs
  • bpo-40010: Optimize signal handling in multithreaded applications #19067
  • bpo-40010: Pass tstate to ceval GIL functions #19077
  • bpo-39877: 4th take_gil() fix for daemon threads #19080
  • bpo-40010: COMPUTE_EVAL_BREAKER() checks for subinterpreter #19087
  • bpo-40010: Optimize pending calls in multithreaded applications #19091
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2020-03-25.17:33:17.981>
created_at = <Date 2020-03-19.00:57:19.125>
labels = ['interpreter-core', '3.9', 'performance']
title = 'Inefficient signal handling in multithreaded applications'
updated_at = <Date 2020-05-05.15:27:40.795>
user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2020-05-05.15:27:40.795>
actor = 'eric.snow'
assignee = 'none'
closed = True
closed_date = <Date 2020-03-25.17:33:17.981>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2020-03-19.00:57:19.125>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 40010
keywords = ['patch']
message_count = 8.0
messages = ['364580', '364583', '364611', '364628', '364659', '364666', '364667', '368171']
nosy_count = 2.0
nosy_names = ['vstinner', 'eric.snow']
pr_nums = ['19067', '19077', '19080', '19087', '19091']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue40010'
versions = ['Python 3.9']
    @vstinner
    Copy link
    Member Author

    When a thread gets a signal, SIGNAL_PENDING_SIGNALS() sets signals_pending to 1 and eval_breaker to 1. In this case, _PyEval_EvalFrameDefault() calls handle_signals(), but since it's not the main thread, it does nothing and signals_pending value remains 1. Moreover, eval_breaker value remains 1 which means that the following code will be called before executing *each* bytecode instruction.

        if (_Py_atomic_load_relaxed(eval_breaker)) {
        (...)
        opcode = _Py_OPCODE(*next_instr);
        if (opcode == SETUP_FINALLY || ...) {
            ...
        }
        if (_Py_atomic_load_relaxed(&ceval->signals_pending)) {
            if (handle_signals(tstate) != 0) {
                goto error;
            }
        }
        if (_Py_atomic_load_relaxed(&ceval->pending.calls_to_do)) {
            ...
        }

        if (_Py_atomic_load_relaxed(&ceval->gil_drop_request)) {
            ...
        }
        if (tstate->async_exc != NULL) {
            ...
        }
    }

    This is inefficient.

    I'm working on a PR modifying SIGNAL_PENDING_SIGNALS() to not set eval_breaker to 1 if the current thread is not the main thread.

    @vstinner vstinner added 3.9 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) performance Performance or resource usage labels Mar 19, 2020
    @vstinner
    Copy link
    Member Author

    I found this issue while working on PR 19066 of bpo-39984.

    @vstinner
    Copy link
    Member Author

    New changeset 5a3a71d by Victor Stinner in branch 'master':
    bpo-40010: Optimize signal handling in multithreaded applications (GH-19067)
    5a3a71d

    @vstinner
    Copy link
    Member Author

    New changeset a36adfa by Victor Stinner in branch 'master':
    bpo-39877: 4th take_gil() fix for daemon threads (GH-19080)
    a36adfa

    @vstinner
    Copy link
    Member Author

    New changeset da2914d by Victor Stinner in branch 'master':
    bpo-40010: Pass tstate to ceval GIL functions (GH-19077)
    da2914d

    @vstinner
    Copy link
    Member Author

    New changeset d2a8e5b by Victor Stinner in branch 'master':
    bpo-40010: COMPUTE_EVAL_BREAKER() checks for subinterpreter (GH-19087)
    d2a8e5b

    @vstinner
    Copy link
    Member Author

    New changeset d831688 by Victor Stinner in branch 'master':
    bpo-40010: Optimize pending calls in multithreaded applications (GH-19091)
    d831688

    @vstinner vstinner changed the title Inefficient sigal handling in multithreaded applications Inefficient signal handling in multithreaded applications Mar 20, 2020
    @vstinner vstinner changed the title Inefficient sigal handling in multithreaded applications Inefficient signal handling in multithreaded applications Mar 20, 2020
    @ericsnowcurrently
    Copy link
    Member

    Good catch on this, Victor. Thanks for doing it.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    @vstinner vstinner mentioned this issue Sep 7, 2022
    Open
    @ericsnowcurrently ericsnowcurrently mentioned this issue May 23, 2023
    Closed
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.9 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) performance Performance or resource usage
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    2 participants
    @vstinner @ericsnowcurrently