This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Rewrite asyncio signal handler
Type: enhancement Stage: needs patch
Components: asyncio Versions: Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: asvetlov, njs, sourcejedi, vstinner, yselivanov
Priority: normal Keywords: patch, patch, patch

Created on 2019-01-16 08:17 by asvetlov, last changed 2022-04-11 14:59 by admin.

Pull Requests
URL Status Linked Edit
PR 11577 closed asvetlov, 2019-01-16 09:46
PR 11577 closed asvetlov, 2019-01-16 09:46
PR 11577 closed asvetlov, 2019-01-16 09:46
Messages (5)
msg333751 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2019-01-16 08:17
Asyncio uses a pipe to wakeup event loop in cases of
1. Signal handlers (set_wakeup_fd)
2. Calling asyncio code from another thread

In both cases, it sends b'\0' to the pipe to wake up a loop.
If the pipe is full OSError is raised.
asyncio logs these exceptions in debug mode.
The logging can be omitted because if the pipe is full the loop wakes up and drains the pipe anyway.
msg333768 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-01-16 14:14
In short, attached PR reverts the following commit:

commit fe5649c7b7bf52147480d6b1124a3d8e3597aee3
Author: Victor Stinner <victor.stinner@gmail.com>
Date:   Thu Jul 17 22:43:40 2014 +0200

    Python issue #21645, Tulip issue 192: Rewrite signal handling
    
    Since Python 3.3, the C signal handler writes the signal number into the wakeup
    file descriptor and then schedules the Python call using Py_AddPendingCall().
    
    asyncio uses the wakeup file descriptor to wake up the event loop, and relies
    on Py_AddPendingCall() to schedule the final callback with call_soon().
    
    If the C signal handler is called in a thread different than the thread of the
    event loop, the loop is awaken but Py_AddPendingCall() was not called yet. In
    this case, the event loop has nothing to do and go to sleep again.
    Py_AddPendingCall() is called while the event loop is sleeping again and so the
    final callback is not scheduled immediatly.
    
    This patch changes how asyncio handles signals. Instead of relying on
    Py_AddPendingCall() and the wakeup file descriptor, asyncio now only relies on
    the wakeup file descriptor. asyncio reads signal numbers from the wakeup file
    descriptor to call its signal handler.

=> https://github.com/python/asyncio/issues/192


Since this change, the C signal handler of Python has been reworked in bpo-30038 by:

commit 4ae01496971624c75080431806ed1c08e00f22c7
Author: Nathaniel J. Smith <njs@pobox.com>
Date:   Tue May 16 14:12:11 2017 -0700

    bpo-30038: fix race condition in signal delivery + wakeup fd (#1082)
    
    Before, it was possible to get the following sequence of
    events (especially on Windows, where the C-level signal handler for
    SIGINT is run in a separate thread):
    
    - SIGINT arrives
    - trip_signal is called
    - trip_signal writes to the wakeup fd
    - the main thread wakes up from select()-or-equivalent
    - the main thread checks for pending signals, but doesn't see any
    - the main thread drains the wakeup fd
    - the main thread goes back to sleep
    - trip_signal sets is_tripped=1 and calls Py_AddPendingCall to notify
      the main thread the it should run the Python-level signal handler
    - the main thread doesn't notice because it's asleep
    
    This has been causing repeated failures in the Trio test suite:
      https://github.com/python-trio/trio/issues/119


I am very scared by signals. These things are very complex and it's hard to handle them proplery on all platforms :-( So I'm not excited by changing the code.


> If the pipe is full OSError is raised.

Are you able to reproduce this issue, or is it a theorical issue?

What is the pipe buffer size on Linux? See bpo-33733.
msg333783 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2019-01-16 19:06
Victor, thanks for the message.
I forgot mentioned changes.
You are right, the issue is "theoretical", I didn't see problems with current implementation.
The self-pipe buffer is pretty big, a chance to reach its limit is relatively low.
But a change to break asyncio on some OS with the patch is much higher.
Moreover, the patch assumes that python signal handler will be called *before* reading from self-pipe. 
Otherwise, a signal callback will be postponed up to next writing to the pipe, which looks like a hidden bug.

Let's close the issue.
msg333784 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2019-01-16 19:25
> You are right, the issue is "theoretical", I didn't see problems with current implementation.
> The self-pipe buffer is pretty big, a chance to reach its limit is relatively low.

Actually, I disagree.  The problem manifests itself easily when you orchestrate a big number of subprocesses (think 1000s).

I've implemented a different strategy for signals handling in uvloop [1] and I think we can easily employ similar approach in asyncio.

Please keep this open. 

[1] https://github.com/MagicStack/uvloop/commit/6e03e513625eb544a5f60c9bf9922019fc3a0b29
msg379220 - (view) Author: Alan Jenkins (sourcejedi) * Date: 2020-10-21 16:43
Here's a simple test case that fails on the main branch.

https://github.com/sourcejedi/cpython/commit/50184ea3b354fd775866d036ccee058ec6734927

> the patch assumes that python signal handler will be called *before* reading from self-pipe. 
> Otherwise, a signal callback will be postponed up to next writing to the pipe, which looks like a hidden bug.

I think it will be?  I can imagine it being a concern for future changes though.  I thought it could break if anyone takes seriously the comment "XXX Signals should be recorded per thread, now we have thread state." :-).

But if we're not confident, surely all we have to do is make sure to call _write_to_self() at the end of the python signal handler (_UnixSelectorEventLoop._sig_handler()).

It seems a pity not to have example code we can point to, that handles signals correctly.  Even if it's too tedious to explain the correct way in the official docs.
History
Date User Action Args
2022-04-11 14:59:10adminsetgithub: 79930
2020-10-21 16:43:27sourcejedisetnosy: + sourcejedi
messages: + msg379220
2019-01-16 19:25:24yselivanovsetkeywords: patch, patch, patch
stage: resolved -> needs patch
2019-01-16 19:25:10yselivanovsetkeywords: patch, patch, patch
status: closed -> open
resolution: rejected ->
messages: + msg333784
2019-01-16 19:06:55asvetlovsetkeywords: patch, patch, patch
status: open -> closed
resolution: rejected
stage: patch review -> resolved
2019-01-16 19:06:29asvetlovsetkeywords: patch, patch, patch

messages: + msg333783
2019-01-16 14:14:16vstinnersettitle: Ignore exception if event loop wakeup pipe is full -> Rewrite asyncio signal handler
nosy: + vstinner, njs

messages: + msg333768

keywords: patch, patch, patch
2019-01-16 09:46:33asvetlovsetkeywords: + patch
stage: patch review
pull_requests: + pull_request11256
2019-01-16 09:46:28asvetlovsetkeywords: + patch
stage: (no value)
pull_requests: + pull_request11255
2019-01-16 09:46:23asvetlovsetkeywords: + patch
stage: (no value)
pull_requests: + pull_request11254
2019-01-16 08:18:10asvetlovsetnosy: + yselivanov

type: enhancement
components: + asyncio
versions: + Python 3.7, Python 3.8
2019-01-16 08:17:26asvetlovcreate