classification
Title: Signal delivered to a subprocess triggers parent's handler
Type: behavior Stage:
Components: asyncio Versions: Python 3.7, Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Ilya.Kulakov, njs, pitrou, yselivanov
Priority: normal Keywords:

Created on 2017-09-16 05:25 by Ilya.Kulakov, last changed 2017-12-20 00:25 by njs.

Messages (7)
msg302321 - (view) Author: Ilya Kulakov (Ilya.Kulakov) * Date: 2017-09-16 05:25
It looks like a signal delivered to multiprocessing's process implicitly created by ProcessPoolExecutor triggers signal handler in the parent:

```
from concurrent.futures import ProcessPoolExecutor
import asyncio
import os
import signal
import sys
import time


def background_task():
    print(f"Running background task {os.getpid()}", file=sys.stderr)
    time.sleep(15)
    print("Exiting background task", file=sys.stderr)


async def task():
    print("Running task")

    executor = ProcessPoolExecutor()
    with executor:
        loop = asyncio.get_event_loop()

        try:
            await loop.run_in_executor(executor, background_task)
        except asyncio.CancelledError:
            print("Cancelling task 1", file=sys.stderr)

            try:
                await asyncio.sleep(15)
            except asyncio.CancelledError:
                print("Cancelling task 2", file=sys.stderr)
                raise

            raise

def main():
    def terminate(coro):
        print(f"Terminating {os.getpid()}")
        coro.cancel()

    loop = asyncio.get_event_loop()
    t = asyncio.ensure_future(task())
    loop.add_signal_handler(signal.SIGTERM, terminate, t)

    try:
        print(f"Running {os.getpid()}", file=sys.stderr)
        loop.run_until_complete(t)
    finally:
        loop.run_until_complete(loop.shutdown_asyncgens())
        loop.close()


if __name__ == '__main__':
    main()
```

1. Normal execution:

> Running 1000
> Running task
> Running background task 9999
> Exiting background task

2. Sending SIGTERM to parent once:

> Running 1000
> Running task
> Running background task 9999

< kill -s SIGTERM 9999

> Terminating 1000
> Cancelling task 1
> Exiting background task


3. Sending SIGTERM to parent twice:

> Running 1000
> Running task
> Running background task 9999

< kill -s SIGTERM 1000

> Terminating 1000
> Cancelling task 1

< kill -s SIGTERM 1000

> Terminating 1000
> Cancelling task 2
> Exiting background task


4. Sending SIGTERM to once to parent and once to child:

> Running 1000
> Running task
> Running background task 9999

< kill -s SIGTERM 1000

> Terminating 1000
> Cancelling task 1

< kill -s SIGTERM 9999

> Terminating 1000
> Cancelling task 2
> Exiting background task


As you can see, sending SIGTERM into a subprocess somehow triggered a signal handler inside a parent. This is unexpected.
msg302323 - (view) Author: Ilya Kulakov (Ilya.Kulakov) * Date: 2017-09-16 05:37
I think either loop's signal handler should not be called from a subprocess or at the very least, os.getpid / os.getpgrp should report correctly.
msg308652 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-12-19 15:01
I get the feeling (without actually investigating) that this is because a fork()-created process inherits all the parent's configuration, including (in this case) signal handlers and whatever file descriptor was configured to receive signal events using signal.set_wakeup_fd().  So the child process, when it receives a signal, also writes on that file descriptor which happens to be the same  underlying self-pipe as in the parent.

In Python 3.6 it seems there isn't much you can't do against this (well, nothing obvious, in any case). In Python 3.7, you'll have two fixes available in ProcessPoolExecutor (*):

* either pass an initializer function that resets signal configuration to a sane default state
* or pass a "forkserver" multiprocessing context that will avoid inheritance issues in the process pool workers

(*) see docs at https://docs.python.org/3.7/library/concurrent.futures.html#concurrent.futures.ProcessPoolExecutor

I would generally recommend using "forkserver" whenever possible, since it eliminates all those inheritance issues by design.
msg308670 - (view) Author: Nathaniel Smith (njs) * (Python committer) Date: 2017-12-19 18:59
Ouch, yes, that's a tricky bug. This is definitely caused by the way that asyncio internally converts signals into messages along a pipe (well, socket, but same thing), and then after a fork-without-exec the child keeps writing into that pipe. It's exacerbated by asyncio's choice to use the self-pipe as its source of truth about which signals have arrived vs just a wake-up pipe (see [1]), but that's not really the main issue; even without this we'd get spurious wakeups and other unpleasantness.

In addition to the workarounds Antoine suggested, it would possibly make sense for forked children to disable any wakeup_fd, perhaps in PyOS_AfterFork or by adding a getpid() check to the C level signal handler. I can't think of any cases where you actually want to processes to share the same wake-up fd. And even if this isn't fixed at that level, it would make sense for asyncio to use the new atfork module to do something similar for asyncio specifically.

Also relevant: https://github.com/python/asyncio/issues/347

[1] https://github.com/dabeaz/curio/issues/118
msg308677 - (view) Author: Ilya Kulakov (Ilya.Kulakov) * Date: 2017-12-19 20:05
Can you suggest an alternative to ProcessPoolExecutor for 3.6?
msg308678 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-12-19 20:06
You may switch to multiprocessing.Pool (with the "forkserver" method).

Otherwise, you could workaround it by executing a function on all workers that will reset the signal configuration. To maximize the chances that it does get executed on all workers, you could add a sleep() call inside it...
msg308696 - (view) Author: Nathaniel Smith (njs) * (Python committer) Date: 2017-12-20 00:25
It might be possible to create ProcessPoolExecutor and get it to spawn all the workers *before* you start the asyncio loop. It looks like ProcessPoolExecutor delays spawning workers until the first piece of work is submitted, but at that point it spawns all of them immediately, so something like this might work:

executor = ProcessPoolExecutor(...)
executor.submit(lambda: None).wait()
with asyncio.get_event_loop() as loop:
    loop.run_until_complete(...)
History
Date User Action Args
2017-12-20 00:25:03njssetmessages: + msg308696
2017-12-19 20:06:55pitrousetmessages: + msg308678
2017-12-19 20:05:35Ilya.Kulakovsetmessages: + msg308677
2017-12-19 18:59:42njssetmessages: + msg308670
2017-12-19 15:01:12pitrousetnosy: + njs, pitrou

messages: + msg308652
versions: + Python 3.7
2017-09-16 05:37:43Ilya.Kulakovsetmessages: + msg302323
2017-09-16 05:25:54Ilya.Kulakovcreate