This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Interpreter hang when interrupting a loop.run_in_executor() future
Type: behavior Stage: resolved
Components: asyncio Versions: Python 3.6, Python 3.4, Python 3.5
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: RemiCardona, gvanrossum, iritkatriel, rsebille, vstinner, yselivanov
Priority: normal Keywords:

Created on 2017-01-18 14:58 by rsebille, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
run_in_executor.py rsebille, 2017-01-18 14:58
Messages (4)
msg285732 - (view) Author: Romain Sébille (rsebille) Date: 2017-01-18 14:58
Hi,

I stumble on this today and can't find any reasons or previous issue as to why this happen so I thought of submitting the issue here.

I attached a script (run_in_executor.py) to reproduce the problem, the original problem arise when waiting to read on a socket but I have the same result with time.sleep().

The output:
$ python3.6 run_in_executor.py
Start
Going to sleep
^CClosing loop
Traceback (most recent call last):
  File "run_in_executor.py", line 15, in <module>
    loop.run_until_complete(sleep_in_executor(loop))
  File "/usr/local/lib/python3.6/asyncio/base_events.py", line 454, in run_until_complete
    self.run_forever()
  File "/usr/local/lib/python3.6/asyncio/base_events.py", line 421, in run_forever
    self._run_once()
  File "/usr/local/lib/python3.6/asyncio/base_events.py", line 1389, in _run_once
    event_list = self._selector.select(timeout)
  File "/usr/local/lib/python3.6/selectors.py", line 445, in select
    fd_event_list = self._epoll.poll(timeout, max_ev)
KeyboardInterrupt
^CError in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 39, in _python_exit
    t.join()
  File "/usr/local/lib/python3.6/threading.py", line 1056, in join
    self._wait_for_tstate_lock()
  File "/usr/local/lib/python3.6/threading.py", line 1072, in _wait_for_tstate_lock
    elif lock.acquire(block, timeout):
KeyboardInterrupt

Current behavior:
We have to send two ^C before we go back to the shell.

Expected behavior:
Only one ^C to go back to the shell.

Python versions tested:
- Python 3.4.2
- Python 3.4.6
- Python 3.5.3
- Python 3.6.0

From what I have gathered the error is within the ThreadPoolExecutor cleaning function but I don't see why there are still living workers, the Executor.shutdown() take care of that.
Unfortunately I don't really know what to look, is a AsyncIO bug ? A ThreadPoolExecutor bug ? Or something else.
msg285734 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-01-18 15:04
Sending SIGTERM (CTRL+c, KeyboardInterrupt) while Python is waiting in threading.Thread.join() is not supported.

You have a thread running time.sleep(1). On Linux, this function calls the select() syscall. So you have the main thread waiting for the thread state lock, and a thread waiting in select(). If SIGTERM is received in the main thread, you enter a weird state of the threading module.

Python has a signal.pthread_kill() which might help to only send a signal to a specific thread, but I don't know exactly how Python behaves, since Python wants to execute Python signal handles in the main thread...
msg285748 - (view) Author: Romain Sébille (rsebille) Date: 2017-01-18 17:22
Ok, thanks for the explanation!

One thing still bother me, run_in_executor() allow us to launch blocking operations but without blocking the main thread, so there is a high chance that we wait inside one of the pool's worker.
Does this mean run_in_executor() shouldn't be used for long/indefinite blocking operation?
msg407819 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-12-06 14:47
@asyncio.coroutine was removed in 3.10, and the run_in_executor.py doesn't run with async def because that does not allow yield.

I'll close this as out of date, please create a new issue if you are still having a related problem with async def.
History
Date User Action Args
2022-04-11 14:58:42adminsetgithub: 73495
2021-12-06 14:47:45iritkatrielsetstatus: open -> closed

nosy: + iritkatriel
messages: + msg407819

resolution: out of date
stage: resolved
2017-01-19 09:25:21RemiCardonasetnosy: + RemiCardona
2017-01-18 17:22:06rsebillesetmessages: + msg285748
2017-01-18 15:04:53vstinnersetnosy: + vstinner
messages: + msg285734
2017-01-18 14:58:33rsebillecreate