Issue31489
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2017-09-16 05:25 by Ilya.Kulakov, last changed 2022-04-11 14:58 by admin.
Messages (7) | |||
---|---|---|---|
msg302321 - (view) | Author: Ilya Kulakov (Ilya.Kulakov) * | Date: 2017-09-16 05:25 | |
It looks like a signal delivered to multiprocessing's process implicitly created by ProcessPoolExecutor triggers signal handler in the parent: ``` from concurrent.futures import ProcessPoolExecutor import asyncio import os import signal import sys import time def background_task(): print(f"Running background task {os.getpid()}", file=sys.stderr) time.sleep(15) print("Exiting background task", file=sys.stderr) async def task(): print("Running task") executor = ProcessPoolExecutor() with executor: loop = asyncio.get_event_loop() try: await loop.run_in_executor(executor, background_task) except asyncio.CancelledError: print("Cancelling task 1", file=sys.stderr) try: await asyncio.sleep(15) except asyncio.CancelledError: print("Cancelling task 2", file=sys.stderr) raise raise def main(): def terminate(coro): print(f"Terminating {os.getpid()}") coro.cancel() loop = asyncio.get_event_loop() t = asyncio.ensure_future(task()) loop.add_signal_handler(signal.SIGTERM, terminate, t) try: print(f"Running {os.getpid()}", file=sys.stderr) loop.run_until_complete(t) finally: loop.run_until_complete(loop.shutdown_asyncgens()) loop.close() if __name__ == '__main__': main() ``` 1. Normal execution: > Running 1000 > Running task > Running background task 9999 > Exiting background task 2. Sending SIGTERM to parent once: > Running 1000 > Running task > Running background task 9999 < kill -s SIGTERM 9999 > Terminating 1000 > Cancelling task 1 > Exiting background task 3. Sending SIGTERM to parent twice: > Running 1000 > Running task > Running background task 9999 < kill -s SIGTERM 1000 > Terminating 1000 > Cancelling task 1 < kill -s SIGTERM 1000 > Terminating 1000 > Cancelling task 2 > Exiting background task 4. Sending SIGTERM to once to parent and once to child: > Running 1000 > Running task > Running background task 9999 < kill -s SIGTERM 1000 > Terminating 1000 > Cancelling task 1 < kill -s SIGTERM 9999 > Terminating 1000 > Cancelling task 2 > Exiting background task As you can see, sending SIGTERM into a subprocess somehow triggered a signal handler inside a parent. This is unexpected. |
|||
msg302323 - (view) | Author: Ilya Kulakov (Ilya.Kulakov) * | Date: 2017-09-16 05:37 | |
I think either loop's signal handler should not be called from a subprocess or at the very least, os.getpid / os.getpgrp should report correctly. |
|||
msg308652 - (view) | Author: Antoine Pitrou (pitrou) * | Date: 2017-12-19 15:01 | |
I get the feeling (without actually investigating) that this is because a fork()-created process inherits all the parent's configuration, including (in this case) signal handlers and whatever file descriptor was configured to receive signal events using signal.set_wakeup_fd(). So the child process, when it receives a signal, also writes on that file descriptor which happens to be the same underlying self-pipe as in the parent. In Python 3.6 it seems there isn't much you can't do against this (well, nothing obvious, in any case). In Python 3.7, you'll have two fixes available in ProcessPoolExecutor (*): * either pass an initializer function that resets signal configuration to a sane default state * or pass a "forkserver" multiprocessing context that will avoid inheritance issues in the process pool workers (*) see docs at https://docs.python.org/3.7/library/concurrent.futures.html#concurrent.futures.ProcessPoolExecutor I would generally recommend using "forkserver" whenever possible, since it eliminates all those inheritance issues by design. |
|||
msg308670 - (view) | Author: Nathaniel Smith (njs) * | Date: 2017-12-19 18:59 | |
Ouch, yes, that's a tricky bug. This is definitely caused by the way that asyncio internally converts signals into messages along a pipe (well, socket, but same thing), and then after a fork-without-exec the child keeps writing into that pipe. It's exacerbated by asyncio's choice to use the self-pipe as its source of truth about which signals have arrived vs just a wake-up pipe (see [1]), but that's not really the main issue; even without this we'd get spurious wakeups and other unpleasantness. In addition to the workarounds Antoine suggested, it would possibly make sense for forked children to disable any wakeup_fd, perhaps in PyOS_AfterFork or by adding a getpid() check to the C level signal handler. I can't think of any cases where you actually want to processes to share the same wake-up fd. And even if this isn't fixed at that level, it would make sense for asyncio to use the new atfork module to do something similar for asyncio specifically. Also relevant: https://github.com/python/asyncio/issues/347 [1] https://github.com/dabeaz/curio/issues/118 |
|||
msg308677 - (view) | Author: Ilya Kulakov (Ilya.Kulakov) * | Date: 2017-12-19 20:05 | |
Can you suggest an alternative to ProcessPoolExecutor for 3.6? |
|||
msg308678 - (view) | Author: Antoine Pitrou (pitrou) * | Date: 2017-12-19 20:06 | |
You may switch to multiprocessing.Pool (with the "forkserver" method). Otherwise, you could workaround it by executing a function on all workers that will reset the signal configuration. To maximize the chances that it does get executed on all workers, you could add a sleep() call inside it... |
|||
msg308696 - (view) | Author: Nathaniel Smith (njs) * | Date: 2017-12-20 00:25 | |
It might be possible to create ProcessPoolExecutor and get it to spawn all the workers *before* you start the asyncio loop. It looks like ProcessPoolExecutor delays spawning workers until the first piece of work is submitted, but at that point it spawns all of them immediately, so something like this might work: executor = ProcessPoolExecutor(...) executor.submit(lambda: None).wait() with asyncio.get_event_loop() as loop: loop.run_until_complete(...) |
History | |||
---|---|---|---|
Date | User | Action | Args |
2022-04-11 14:58:52 | admin | set | github: 75670 |
2017-12-20 00:25:03 | njs | set | messages: + msg308696 |
2017-12-19 20:06:55 | pitrou | set | messages: + msg308678 |
2017-12-19 20:05:35 | Ilya.Kulakov | set | messages: + msg308677 |
2017-12-19 18:59:42 | njs | set | messages: + msg308670 |
2017-12-19 15:01:12 | pitrou | set | nosy:
+ njs, pitrou messages: + msg308652 versions: + Python 3.7 |
2017-09-16 05:37:43 | Ilya.Kulakov | set | messages: + msg302323 |
2017-09-16 05:25:54 | Ilya.Kulakov | create |