classification
Title: asyncio.add_signal_handler call fails if not on main thread
Type: behavior Stage: resolved
Components: asyncio Versions: Python 3.9, Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: asvetlov Nosy List: aeros, asvetlov, epiphyte, haxor, jnwatson, lukasz.langa, miss-islington, ned.deily, vstinner, xtreak, yselivanov
Priority: release blocker Keywords: needs review, patch

Created on 2018-09-14 15:27 by jnwatson, last changed 2019-10-23 15:44 by miss-islington. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 15477 closed aeros, 2019-08-24 21:40
PR 15492 merged asvetlov, 2019-08-25 15:16
PR 15512 merged miss-islington, 2019-08-26 09:52
PR 16901 merged vstinner, 2019-10-23 14:27
PR 16902 merged miss-islington, 2019-10-23 15:25
Messages (16)
msg325350 - (view) Author: Nic Watson (jnwatson) Date: 2018-09-14 15:27
Summary:  essentially asyncio.add_signal_handler doesn't work when called off the main thread.  One might consider this a documentation failure.

(Note: there's also a bigger issue with cleanup, which I'll submit separately)

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/home/nic/.pyenv/versions/3.6.4/lib/python3.6/asyncio/unix_events.py", line 91, in add_signal_handler
    signal.set_wakeup_fd(self._csock.fileno())
ValueError: set_wakeup_fd only works in main thread

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/nic/.pyenv/versions/3.6.4/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/home/nic/.pyenv/versions/3.6.4/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/home/nic/tmp/signal_event_loop_bug.py", line 14, in do_loop
    loop.add_signal_handler(signal.SIGINT, mysighandler)
  File "/home/nic/.pyenv/versions/3.6.4/lib/python3.6/asyncio/unix_events.py", line 93, in add_signal_handler
    raise RuntimeError(str(exc))
RuntimeError: set_wakeup_fd only works in main thread

Code:

import asyncio
import signal
import threading

def mysighandler():
    pass

def do_loop():
    loop = asyncio.new_event_loop()
    # This will fail with RuntimeError: set_wakeup_fd only works in main thread
    loop.add_signal_handler(signal.SIGINT, mysighandler)
    loop.close()

t = threading.Thread(target=do_loop)
t.start()
t.join()
msg326426 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2018-09-26 08:01
In my opinion, we should don't change the behavior.
Signal handlers should be processed in main thread at the end, that's how Python works for regular synchronous API.

Mentioning it in the doc makes sense.

Would you make a pull request?
msg347525 - (view) Author: Brett Slatkin (haxor) Date: 2019-07-09 07:24
I believe the scope of this bug may be larger than it originally seemed.

Now that ProactorEventLoop is the default for Python 3.8 (https://bugs.python.org/issue34687), I'm seeing this same problem on Windows when you try to call asyncio.new_event_loop() from within a thread. It breaks with the ProactorEventLoop (snippet #1 below). It works fine with the SelectorEventLoop (snippet #2 below).

Am I wrong to expect to be able to create a unique event loop for each thread from within the thread itself?

I worked around this problem by manually forcing the event loop policy (https://bugs.python.org/issue33792).

=== Snippet #1 (breaks) ===

import asyncio
from threading import Thread

def my_func():
    asyncio.new_event_loop()

t = Thread(target=my_func)
t.start()
t.join()

=== Output from snippet #1 ===

PS G:\> python .\repro.py
Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Users\User\AppData\Local\Programs\Python\Python38\lib\threading.py", line 932, in _bootstrap_inner
    self.run()
  File "C:\Users\User\AppData\Local\Programs\Python\Python38\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File ".\repro.py", line 6, in my_func
    asyncio.new_event_loop()
  File "C:\Users\User\AppData\Local\Programs\Python\Python38\lib\asyncio\events.py", line 758, in new_event_loop
    return get_event_loop_policy().new_event_loop()
  File "C:\Users\User\AppData\Local\Programs\Python\Python38\lib\asyncio\events.py", line 656, in new_event_loop
    return self._loop_factory()
  File "C:\Users\User\AppData\Local\Programs\Python\Python38\lib\asyncio\windows_events.py", line 310, in __init__
    super().__init__(proactor)
  File "C:\Users\User\AppData\Local\Programs\Python\Python38\lib\asyncio\proactor_events.py", line 630, in __init__
    signal.set_wakeup_fd(self_no)
ValueError: set_wakeup_fd only works in main thread

=== Snippet #2 (works) ===

import asyncio
from threading import Thread

# Work-around from https://bugs.python.org/issue34679
policy = asyncio.get_event_loop_policy()
policy._loop_factory = asyncio.SelectorEventLoop

def my_func():
    asyncio.new_event_loop()

t = Thread(target=my_func)
t.start()
t.join()

=== More details ===

My version of Python:

3.8.0b2 (tags/v3.8.0b2:21dd01d, Jul  4 2019, 16:00:09) [MSC v.1916 64 bit (AMD64)]

My version of Windows (it's a developer VM from https://developer.microsoft.com/en-us/windows/downloads/virtual-machines):

PS G:\> [System.Environment]::OSVersion.Version

Major  Minor  Build  Revision
-----  -----  -----  --------
10     0      17763  0
msg347546 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2019-07-09 11:08
Good point.
For proactor event loop set_wakeup_fd() is used for Ctrl+C handling.
Skipping this call for non-main thread in proactor implementation makes sense.
msg350288 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2019-08-23 14:03
Please fix this ASAP, last 3.8 beta is scheduled for Monday.
msg350298 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2019-08-23 14:32
Andrew, can you fix ctrl-c/windows issue? I'm basically afk until monday. 

And we're not going to do anything about add_signal_handler in 3.8.
msg350304 - (view) Author: Brett Slatkin (haxor) Date: 2019-08-23 15:05
Maybe we should just roll back https://bugs.python.org/issue34687 to fix both issues? Otherwise asyncio will be broken on Windows in 3.8. Is general API stability more important than performance?
msg350399 - (view) Author: Kyle Stanley (aeros) * (Python triager) Date: 2019-08-24 20:17
> Skipping this call for non-main thread in proactor implementation makes sense.

How do we identify whether or not set_wakeup_fd() is being called from a non-main thread?
msg350401 - (view) Author: Kyle Stanley (aeros) * (Python triager) Date: 2019-08-24 21:27
> How do we identify whether or not set_wakeup_fd() is being called from a non-main thread?

Never mind, I think I found the answer to my own question and tested a patch locally, I'll open a PR.
msg350463 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2019-08-25 15:14
The issue is related to Python 3.8 and master only. 3.6-3.7 are not affected
msg350464 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2019-08-25 15:15
Kyle, thanks for the fix.
I have basically the same change in my PR but with test and news note.
msg350472 - (view) Author: Kyle Stanley (aeros) * (Python triager) Date: 2019-08-25 20:47
> Kyle, thanks for the fix.
> I have basically the same change in my PR but with test and news note.

No problem, that works for me. I was mostly just trying to help with resolving some of the release blockers for 3.8b4.
msg350515 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2019-08-26 09:51
New changeset 1c0600998681295735a18690fae184b0c9a4ca51 by Andrew Svetlov in branch 'master':
bpo-34679: Restore instantiation Windows IOCP event loop from non-main thread (#15492)
https://github.com/python/cpython/commit/1c0600998681295735a18690fae184b0c9a4ca51
msg350518 - (view) Author: miss-islington (miss-islington) Date: 2019-08-26 10:14
New changeset 69d22b8fee442c12829a1032a72489c8133de271 by Miss Islington (bot) in branch '3.8':
bpo-34679: Restore instantiation Windows IOCP event loop from non-main thread (GH-15492)
https://github.com/python/cpython/commit/69d22b8fee442c12829a1032a72489c8133de271
msg355230 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-10-23 15:25
New changeset 1b53a24fb4417c764dd5933bce505f5c94249ca6 by Victor Stinner in branch 'master':
bpo-34679: ProactorEventLoop only uses set_wakeup_fd() in main thread (GH-16901)
https://github.com/python/cpython/commit/1b53a24fb4417c764dd5933bce505f5c94249ca6
msg355233 - (view) Author: miss-islington (miss-islington) Date: 2019-10-23 15:44
New changeset cbf474c98e702d12c97cd16a1e44ede10ea52b5b by Miss Skeleton (bot) in branch '3.8':
bpo-34679: ProactorEventLoop only uses set_wakeup_fd() in main thread (GH-16901)
https://github.com/python/cpython/commit/cbf474c98e702d12c97cd16a1e44ede10ea52b5b
History
Date User Action Args
2019-10-23 15:44:03miss-islingtonsetmessages: + msg355233
2019-10-23 15:25:55miss-islingtonsetpull_requests: + pull_request16439
2019-10-23 15:25:37vstinnersetnosy: + vstinner
messages: + msg355230
2019-10-23 14:27:17vstinnersetpull_requests: + pull_request16437
2019-08-26 10:14:58miss-islingtonsetnosy: + miss-islington
messages: + msg350518
2019-08-26 09:53:15asvetlovsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2019-08-26 09:52:01miss-islingtonsetpull_requests: + pull_request15198
2019-08-26 09:51:17asvetlovsetmessages: + msg350515
2019-08-25 20:47:24aerossetmessages: + msg350472
2019-08-25 15:16:39asvetlovsetkeywords: + patch
pull_requests: + pull_request15180
2019-08-25 15:15:38asvetlovsetmessages: + msg350464
2019-08-25 15:14:55asvetlovsetmessages: + msg350463
versions: + Python 3.9, - Python 3.6, Python 3.7
2019-08-24 22:22:47aerossetkeywords: + needs review, - patch
2019-08-24 21:40:45aerossetkeywords: + patch
stage: patch review
pull_requests: + pull_request15163
2019-08-24 21:27:58aerossetmessages: + msg350401
2019-08-24 20:17:48aerossetnosy: + aeros
messages: + msg350399
2019-08-23 15:05:54haxorsetmessages: + msg350304
2019-08-23 14:32:29yselivanovsetmessages: + msg350298
2019-08-23 14:03:56lukasz.langasetmessages: + msg350288
2019-08-07 14:54:43lukasz.langasetpriority: normal -> release blocker
nosy: + ned.deily, lukasz.langa
2019-07-09 11:08:51asvetlovsetassignee: asvetlov
messages: + msg347546
2019-07-09 07:24:44haxorsetnosy: + haxor
messages: + msg347525
2018-09-26 08:01:58asvetlovsetversions: + Python 3.8
2018-09-26 08:01:41asvetlovsetmessages: + msg326426
2018-09-25 10:14:05xtreaksetnosy: + xtreak
2018-09-14 15:37:26epiphytesetnosy: + epiphyte
2018-09-14 15:27:45jnwatsoncreate