classification
Title: ProactorEventLoop raises unhandled ConnectionResetError
Type: Stage: patch review
Components: asyncio Versions: Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Ben.Darnell, Burak Yiğit Kaya, Jonathan Slenders, PeterL777, Rustam S., Segev Finer, asvetlov, cjrh, cmeyer, yselivanov
Priority: normal Keywords: patch

Created on 2019-12-09 20:47 by Jonathan Slenders, last changed 2020-05-30 00:41 by cmeyer.

Pull Requests
URL Status Linked Edit
PR 20525 open cmeyer, 2020-05-30 00:41
Messages (9)
msg358140 - (view) Author: Jonathan Slenders (Jonathan Slenders) Date: 2019-12-09 20:47
We have a snippet of code that runs perfectly fine using the `SelectorEventLoop`, but crashes *sometimes* using the `ProactorEventLoop`.

The traceback is the following. The exception cannot be caught within the asyncio application itself (e.g., it is not attached to any Future or propagated in a coroutine). It probably propagates in `run_until_complete()`.

  File "C:\Python38\lib\asyncio\proactor_events.py", line 768, in _loop_self_reading
    f.result()  # may raise
  File "C:\Python38\lib\asyncio\windows_events.py", line 808, in _poll
    value = callback(transferred, key, ov)
  File "C:\Python38\lib\asyncio\windows_events.py", line 457, in finish_recv
    raise ConnectionResetError(*exc.args)

I can see that in `IocpProactor._poll`, `OSError` is caught and attached to the future, but not `ConnectionResetError`. I would expect that `ConnectionResetError` too will be attached to the future.

In order to reproduce, run the following snippet on Python 3.8:

from prompt_toolkit import prompt  # pip install prompt_toolkit
while 1:
    prompt('>')

Hold down the enter key, and it'll trigger quickly.

See also: https://github.com/prompt-toolkit/python-prompt-toolkit/issues/1023
msg358141 - (view) Author: Jonathan Slenders (Jonathan Slenders) Date: 2019-12-09 20:57
Suppressing `ConnectionResetError` in `BaseProactorEventLoop._loop_self_reading`, like we do with `CancelledError` seems to fix it.

Although I'm not sure what it causing the error, and whether we need to handle it somehow.
msg358144 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-12-09 21:34
ConnectionResetError means that the pipe socket is closed. Was the event loop closed? Can you provide a reproducer? Can you try to get debug logs to see what's going on?
https://docs.python.org/dev/library/asyncio-dev.html#debug-mode
msg358199 - (view) Author: Jonathan Slenders (Jonathan Slenders) Date: 2019-12-10 16:18
Thanks Victor for the reply.

It looks like it's the self-socket in the BaseProactorEventLoop that gets closed. It's exactly this FD for which the exception is raised.

We don't close the event loop anywhere. I also don't see `_close_self_pipe` being called anywhere.

Debug logs don't provide any help. I'm looking into a reproducer.
msg358208 - (view) Author: Jonathan Slenders (Jonathan Slenders) Date: 2019-12-10 17:49
It looks like the following code will reproduce the issue:

```
import asyncio
import threading

loop = asyncio.get_event_loop()

while True:
    def test():
        loop.call_soon_threadsafe(loop.stop)

    threading.Thread(target=test).start()
    loop.run_forever()

```

Leave it running on Windows, in Python 3.8 for a few seconds, then it starts spawning `ConnectionResetError`s.
msg358226 - (view) Author: Jonathan Slenders (Jonathan Slenders) Date: 2019-12-10 21:18
Even simpler, the following code will crash after so many iterations:


```
import asyncio
loop = asyncio.get_event_loop()

while True:
    loop.call_soon_threadsafe(loop.stop)
    loop.run_forever()
```

Adding a little sleep of 0.01s after `run_forever()` prevents the issue.

So, to me it looks like the cancellation of the `_OverlappedFuture` that wraps around the `_recv()` call from the self-pipe did not complete before we start `_recv()` again in the next `run_forever()` call. No idea if that makes sense...
msg362076 - (view) Author: Ben Darnell (Ben.Darnell) * Date: 2020-02-16 15:05
I just spent some time digging into this. Each call to `run_forever` starts a call to `_loop_self_reading`, then attempts to cancel it before returning:

https://github.com/python/cpython/blob/1ed61617a4a6632905ad6a0b440cd2cafb8b6414/Lib/asyncio/windows_events.py#L312-L325

The comment at line 321 is not entirely accurate: the future will not resolve in the future, but it may have *already* resolved, and added its callback to the call_soon queue. This callback will run if the event loop is restarted again. Since `_loop_self_reading` calls itself, this results in two copies of the "loop" running concurrently and stepping on each other's `_self_reading_futures`. 

This appears to be fairly harmless except for noise in the logs when only one of the loops is stopped cleanly.

I believe the simplest fix is for `_loop_self_reading` to compare its argument to `self._self_reading_future` to determine if it is the "current" loop and if not, don't reschedule anything.
msg366002 - (view) Author: Rustam S. (Rustam S.) Date: 2020-04-08 19:12
Please take a look at this as well:
(ipython #12049 'Unhandled exception in event loop' (WinError 995))
https://github.com/ipython/ipython/issues/12049#issuecomment-586544339 and below
msg368692 - (view) Author: Chris Meyer (cmeyer) * Date: 2020-05-12 01:20
Here is another way to reproduce this (or an extremely similar) error without a loop. Since may be a race condition, I'm not sure this works 100% of the time on all machines - but it did on several machines I tried.

```
import asyncio

loop = asyncio.get_event_loop()

def func():
    pass

f = loop.run_in_executor(None, func)
loop.stop()
loop.run_forever()
loop.stop()
loop.run_forever()
loop.stop()
loop.run_forever()
```

```
Error on reading from the event loop self pipe
loop: <ProactorEventLoop running=True closed=False debug=False>
Traceback (most recent call last):
  File "C:\Miniconda3\envs\py38\lib\asyncio\windows_events.py", line 453, in finish_recv
    return ov.getresult()
OSError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Miniconda3\envs\py38\lib\asyncio\proactor_events.py", line 768, in _loop_self_reading
    f.result()  # may raise
  File "C:\Miniconda3\envs\py38\lib\asyncio\windows_events.py", line 808, in _poll
    value = callback(transferred, key, ov)
  File "C:\Miniconda3\envs\py38\lib\asyncio\windows_events.py", line 457, in finish_recv
    raise ConnectionResetError(*exc.args)
ConnectionResetError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request
```
History
Date User Action Args
2020-05-30 00:41:18cmeyersetkeywords: + patch
stage: patch review
pull_requests: + pull_request19769
2020-05-22 17:49:40Burak Yiğit Kayasetnosy: + Burak Yiğit Kaya
2020-05-12 01:20:02cmeyersetmessages: + msg368692
2020-04-26 16:42:13Segev Finersetnosy: + Segev Finer
2020-04-20 15:07:59cmeyersetnosy: + cmeyer
2020-04-09 00:42:21PeterL777setnosy: + PeterL777
2020-04-08 19:12:07Rustam S.setnosy: + Rustam S.
messages: + msg366002
2020-02-16 15:05:06Ben.Darnellsetnosy: + Ben.Darnell
messages: + msg362076
2020-02-02 01:36:57cjrhsetnosy: + cjrh
2020-01-15 21:54:41vstinnersetnosy: - vstinner
2019-12-10 21:18:18Jonathan Slenderssetmessages: + msg358226
2019-12-10 17:49:24Jonathan Slenderssetmessages: + msg358208
2019-12-10 16:18:23Jonathan Slenderssetmessages: + msg358199
2019-12-09 21:34:59vstinnersetnosy: + vstinner
messages: + msg358144
2019-12-09 20:57:39Jonathan Slenderssetmessages: + msg358141
2019-12-09 20:47:38Jonathan Slenderscreate