This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: asyncio crashes when tearing down the proactor event loop
Type: crash Stage:
Components: asyncio Versions: Python 3.10, Python 3.9, Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: asvetlov, cjrh, cmeyer, jack1142, lawsonjl.ornl, machine.gw, mikeshardmind, pepoluan, rmawatson, steve.dower, yselivanov
Priority: normal Keywords:

Created on 2020-01-06 18:29 by mikeshardmind, last changed 2022-04-11 14:59 by admin.

Files
File name Uploaded Description Edit
example.py mikeshardmind, 2020-01-06 18:29 toy code demonstrating the difference.
Messages (7)
msg359448 - (view) Author: Michael Hall (mikeshardmind) * Date: 2020-01-06 18:29
When using asyncio.run for an asynchronous application utilizing ssl, on windows using the proactor event loop the application crashes when the loop is closed, completely skipping a finally block in the process.

This appears to be due to a __del__ method on transports used.

Manual handling of the event loop close while including a brief sleep appears to work as intended.

Both versions work fine with the selector event loop on linux.

This appears to be a somewhat known issue already, as it's been reported to aiohttp, however both the traceback, and the differing behavior seem to indicate this is an issue with the proactor event loop.

(On linux this still emits a resource warning without the sleep)

While I don't mind handling the loop cleanup, it seems like this case should also emit a resource warning rather than crashing.

If it's decided in which way this should be handled, I'm willing to contribute to or help test whatever direction the resolution for this should go. 

Traceback included below, toy version of the problem attached as code.

Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x0000026463039820>
Traceback (most recent call last):
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python38\lib\asyncio\proactor_events.py", line 116, in __del__
    self.close()
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python38\lib\asyncio\proactor_events.py", line 108, in close
    self._loop.call_soon(self._call_connection_lost, None)
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python38\lib\asyncio\base_events.py", line 715, in call_soon
    self._check_closed()
  File "C:\Users\Michael\AppData\Local\Programs\Python\Python38\lib\asyncio\base_events.py", line 508, in _check_closed       
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
msg359515 - (view) Author: Michael Hall (mikeshardmind) * Date: 2020-01-07 15:56
I don't know if it would be feasible to add this to asyncio, but having a way to mark a resource as needing to be deterministically cleaned up at loop close could probably solve this as well as the underlying reasons why the transports are leaning on __del__ behavior which is an implementation detail (differing on, for example, pypy) and probably improve the overall usefulness of asyncio.run as well.

An addition like that probably requires more discussion than fixing this crash though.
msg362174 - (view) Author: Michael Hall (mikeshardmind) * Date: 2020-02-18 03:49
Linking out to a real-world example where this still manages to happen after running the event loop for an entire 2 seconds waiting for transports to close themselves after finishing everything else:

https://github.com/Cog-Creators/Red-DiscordBot/issues/3560

As well as what we're currently looking at for a temporary solution for this at this point:

https://github.com/Cog-Creators/Red-DiscordBot/pull/3566


I looked into what would need to change to handle this in CPython, but am not confident in my ability to make such a PR after doing so, at least not without more discussion about it.

The best solution I considered involves making the only public way to make transports be tied to an event loop which hasn't been closed yet, and ensuring the event loop keeps a reference to each of these so that it can deterministically close them at loop finalization. Searching GitHub alone found that this would break way too many things.

If this can't be fully fixed, a solution which at least ensures this can't cause an uncatchable exception would be appreciated.
msg367932 - (view) Author: rmawatson (rmawatson) Date: 2020-05-02 14:30
This is also happening in other libraries besides aiohttp. Specifically xhttp on windows. There is an open issue at https://github.com/encode/httpx/issues/914
msg392261 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-04-28 22:03
I'm also seeing (and annoyed by) this.

Do we need to do anything more than suppress the error? If we're just failing to notify the event loop that we're closing because the loop is gone, does the notification even matter?
msg396339 - (view) Author: Joe (lawsonjl.ornl) Date: 2021-06-22 14:16
We are running into this all the time, ever since the Proactor became the default on Windows in 3.8. 

Usually it comes up when the program terminates due to an unhandled exception during a highly concurrent operation. The resulting cascade of RuntimeErrors often obscures the real reason for failure, which makes debugging more painful than it should be. But sometimes this cascade of RuntimeErrors will occur even when the program otherwise executes successfully. So it can be difficult to know if the program actually blew up or if it's just benign event loop vomit.

Converting this particular error to a warning would be great, but eliminating the call to the event loop in __del__ would be even better. I understand that's easier said than done, or else ProactorBasePipeTransport wouldn't be leaning on __del__ in the first place.
msg400132 - (view) Author: Shane Mai (machine.gw) Date: 2021-08-23 12:27
I added some code to wait for all tasks completion before exit: 
    currentTask = asyncio.current_task()
    for t in asyncio.all_tasks():
        if currentTask != t:
            await t
and still got the exception


Then I think it created additional thread somewhere and created an event_loop inside it. To dig it out I sub-classed threading.Thread. I also wait for all tasks in all threads before exiting:

loops: list[asyncio.AbstractEventLoop] = []

class MyThread(threading.Thread):
    def start(self):
        global loops
        loops.append(asyncio.get_event_loop())
        super().start()

async def func1():
    async with aiohttp.ClientSession() as session:
        async with session.get('https://www.okex.com/api/v5/public/time') as resp:
            print(resp.status)
            print(await resp.json())

threading.Thread = MyThread
import aiohttp

async def main():
    global loops
    loops.append(asyncio.get_running_loop())
    print(sys.argv)
    task = asyncio.create_task(func1())
    await task
    print('done.')

    currentTask = asyncio.current_task()
    for loop in loops:
        for t in asyncio.all_tasks(loop):
            if currentTask != t:
                await t

    print('done2.')
    #await asyncio.sleep(1)

#if __file__ == '__main__':
asyncio.run(main())


Then I found out the thread is created inside _loop.getaddrinfo: (files are from python 3.9.6)
    File aiohttp\resolver.py, line 31, in ThreadedResolver.resolve
    FILE asyncio\base_events.py, line 856, in BaseEventLoop(ProactorEventLoop).getaddrinfo

And it is strange that another thread is created when program exit:
    FILE asyncio\base_events.py, line 563, in BaseEventLoop(ProactorEventLoop).shutdown_default_executor

But sad it seems vscode cannot break a __del__ call. If I break somewhere  else first then it would not crash:(
History
Date User Action Args
2022-04-11 14:59:25adminsetgithub: 83413
2021-08-23 12:27:41machine.gwsetnosy: + machine.gw
messages: + msg400132
2021-06-22 14:16:37lawsonjl.ornlsetnosy: + lawsonjl.ornl
messages: + msg396339
2021-04-28 22:03:54steve.dowersetnosy: + steve.dower
messages: + msg392261
2021-04-13 18:44:34mikeshardmindsetversions: + Python 3.9, Python 3.10
2021-01-29 15:24:20pepoluansetnosy: + pepoluan
2020-09-13 02:52:45cjrhsetnosy: + cjrh
2020-05-04 17:11:06cmeyersetnosy: + cmeyer
2020-05-02 14:30:06rmawatsonsetnosy: + rmawatson
messages: + msg367932
2020-03-14 19:21:31jack1142setnosy: + jack1142
2020-02-18 03:49:55mikeshardmindsetmessages: + msg362174
2020-01-07 15:56:48mikeshardmindsetmessages: + msg359515
2020-01-06 18:29:20mikeshardmindcreate