classification
Title: File descriptor error when subprocess call is used with event loop enabled in main thread
Type: crash Stage: resolved
Components: asyncio Versions: Python 3.6
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: Manoj C, asvetlov, yselivanov
Priority: normal Keywords:

Created on 2019-09-11 10:12 by Manoj C, last changed 2019-09-13 11:57 by asvetlov. This issue is now closed.

Messages (2)
msg351795 - (view) Author: Manoj C (Manoj C) Date: 2019-09-11 10:12
I have been using asyncio to run subprocess calls in a separate thread. For this purpose I start an event loop in my main thread as per the recommendation - https://docs.python.org/3/library/asyncio-subprocess.html#subprocess-and-threads .

Now when I use a normal subprocess call in the main thread, I start getting following file descriptor error after few iterations:

```
Exception ignored when trying to write to the signal wakeup fd:
BlockingIOError: [Errno 11] Resource temporarily unavailable
```

I have reproduced the problem in a small script and am seeing that the error goes away if I do not start the even loop in the main thread.

```
import asyncio
import subprocess
import time


def try_error():
    for i in range(1,500):
        print(i)
        try:
            subprocess.run(["ls", "-l"], check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        except subprocess.CalledProcessError as e:
            print(f"Exception raised {e.stderr}\nOutput {e.stdout}")



def definite_error():
    w = asyncio.get_child_watcher()
    l = asyncio.get_event_loop()
    try_error()


if __name__ == "__main__":
    definite_error()
```

This is the smallest subset of the code which can reproduce the error. In the original code, I run a asyncio.create_subprocess_exec in a parallel thread. The normal subprocess call is part of third party code which call from the main thread and hence cannot modify it.
msg352301 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2019-09-13 11:57
You spawn too many subprocesses that finish virtually at the same time.
It leads to wakeup_fd overrun.

Python 3.6 is in security mode, sorry (and the fix is impossible).
Python 3.7 has warn_on_full_buffer=False flag for  https://docs.python.org/3/library/signal.html#signal.set_wakeup_fd but asyncio doesn't use it for reasons.

Python 3.8 doesn't subscribe for SIGCHLD by default, thus the provided example finishes without any warning.

I'm going to closing the issue as won't fix.
History
Date User Action Args
2019-09-13 11:57:12asvetlovsetstatus: open -> closed
resolution: wont fix
messages: + msg352301

stage: resolved
2019-09-11 10:12:22Manoj Ccreate