Issue36709
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2019-04-24 08:48 by tomchristie, last changed 2022-04-11 14:59 by admin. This issue is now closed.
Messages (14) | |||
---|---|---|---|
msg340764 - (view) | Author: Tom Christie (tomchristie) * | Date: 2019-04-24 08:48 | |
If an asyncio SSL connection is left open (eg. any kind of keep-alive connection) then after closing the event loop, an exception will be raised... Python: ``` import asyncio import ssl import certifi async def f(): ssl_context = ssl.create_default_context() ssl_context.load_verify_locations(cafile=certifi.where()) await asyncio.open_connection('example.org', 443, ssl=ssl_context) loop = asyncio.get_event_loop() loop.run_until_complete(f()) loop.close() ``` Traceback: ``` $ python example.py Fatal write error on socket transport protocol: <asyncio.sslproto.SSLProtocol object at 0x10e7874a8> transport: <_SelectorSocketTransport fd=8> Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/selector_events.py", line 868, in write n = self._sock.send(data) OSError: [Errno 9] Bad file descriptor Fatal error on SSL transport protocol: <asyncio.sslproto.SSLProtocol object at 0x10e7874a8> transport: <_SelectorSocketTransport closing fd=8> Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/selector_events.py", line 868, in write n = self._sock.send(data) OSError: [Errno 9] Bad file descriptor During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/sslproto.py", line 676, in _process_write_backlog self._transport.write(chunk) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/selector_events.py", line 872, in write self._fatal_error(exc, 'Fatal write error on socket transport') File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/selector_events.py", line 681, in _fatal_error self._force_close(exc) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/selector_events.py", line 693, in _force_close self._loop.call_soon(self._call_connection_lost, exc) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 677, in call_soon self._check_closed() File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 469, in _check_closed raise RuntimeError('Event loop is closed') RuntimeError: Event loop is closed ``` It looks to me like the original "OSError: [Errno 9] Bad file descriptor" probably shouldn't be raised in any case - if when attempting to tear down the SSL connection, then we should probably pass silently in the case that the socket has already been closed uncleanly. Bought to my attention via: https://github.com/encode/httpcore/issues/16 |
|||
msg340765 - (view) | Author: Tom Christie (tomchristie) * | Date: 2019-04-24 09:10 | |
This appears somewhat related: https://bugs.python.org/issue34506 As it *also* logs exceptions occuring during `_fatal_error` and `_force_close`. |
|||
msg343776 - (view) | Author: Andrew Svetlov (asvetlov) * | Date: 2019-05-28 13:11 | |
From my understanding, the correct code should close all transports and wait for their connection_lost() callbacks before closing the loop. |
|||
msg343780 - (view) | Author: Tom Christie (tomchristie) * | Date: 2019-05-28 14:04 | |
> From my understanding, the correct code should close all transports and wait for their connection_lost() callbacks before closing the loop. Ideally, yes, although we should be able to expect that an SSL connection that hasn't been gracefully closed wouldn't loudly error on teardown like that. In standard sync code, the equivelent would running something like this... ```python session = requests.Session() session.get('https://example.com/') ``` We wouldn't expect a traceback to be raised on exiting. (Even though the user *hasn't* explicitly closed the session, and even though a keep alive SSL connection will be open at the point of exit.) |
|||
msg343883 - (view) | Author: Andrew Svetlov (asvetlov) * | Date: 2019-05-29 13:13 | |
I would say that if requests a designed from scratch more idiomatic way could be with requests.Session() as session: session.get('https://example.com/') or session = requests.Session() session.get('https://example.com/') session.close() Like the recommended way to handle files. |
|||
msg343977 - (view) | Author: Tom Christie (tomchristie) * | Date: 2019-05-30 14:00 | |
Right, and `requests` *does* provide both those styles. The point more being that *not* having closed the transport at the point of exit shouldn't end up raising a hard error. It doesn't raise errors in sync-land, and it shouldn't do so in async-land. Similarly, we wouldn't expect an open file resource to cause errors to be raised at the point of exit. |
|||
msg343978 - (view) | Author: Andrew Svetlov (asvetlov) * | Date: 2019-05-30 14:10 | |
The difference is that socket.close() is an instant call. After socket.close() the socket is done. But transport.close() doesn't close the transport instantly. asyncio requires at least one loop iteration for calling protocol.connection_lost() and actual socket closing. In case of SSL it may take much longer. Sorry, that's how asyncio is designed. |
|||
msg343979 - (view) | Author: Yury Selivanov (yselivanov) * | Date: 2019-05-30 14:42 | |
> Sorry, that's how asyncio is designed. Andrew, couldn't we provide a "stream.terminate()" method (not a coroutine) that would do the following: * close the transport * set a flag in the protocol that the stream has been terminated. When the flag is set, the protocol simply ignores all errors (i.e. they are never shown to the user or logged) This way Tom could have a weakref to the stream object from his high-level wrapper, and whenever the wrapper object is dereferenced it could terminate its stream. > Sorry, that's how asyncio is designed. I think it's a real problem, let's try to find out if we can provide a solution. |
|||
msg343980 - (view) | Author: Andrew Svetlov (asvetlov) * | Date: 2019-05-30 14:52 | |
It's not about streams only. The stream protocol can have such flag, sure. But transport emits a warning like "unclosed transport ..." Not sure if we have to drop this warning, it enforces writing good code that controls all created resources lifecycle. |
|||
msg343981 - (view) | Author: Yury Selivanov (yselivanov) * | Date: 2019-05-30 14:57 | |
> Not sure if we have to drop this warning, it enforces writing good code that controls all created resources lifecycle. Right, maybe we add "transport.terminate()" as well? Synchronously & immediately closing a transport is a valid use case. TBH I don't see why we absolutely must wait the "connection_lost" callback call. I mean it would be reasonable to allow users to terminate the connection (even if it means that in some cases, like SSL, it won't be correctly closed) and not care about what happens to it next. |
|||
msg343984 - (view) | Author: Andrew Svetlov (asvetlov) * | Date: 2019-05-30 15:54 | |
Sorry, I'm not comfortable with such change just before the feature freeze. The idea is maybe good but let's discuss and implement it later. |
|||
msg379222 - (view) | Author: Christian Heimes (christian.heimes) * | Date: 2020-10-21 16:46 | |
This seems to be an asyncio problem. |
|||
msg414425 - (view) | Author: Kumar Aditya (kumaraditya) * | Date: 2022-03-03 11:22 | |
The original issue is fixed on main branch with bpo-44011 GH-31275, It now only raises warnings but no exceptions: ----------------------------------------------------------- (env) @kumaraditya303 ➜ /workspaces/cpython (latin1 ✗) $ python main.py /workspaces/cpython/main.py:12: DeprecationWarning: There is no current event loop loop = asyncio.get_event_loop() /workspaces/cpython/env/lib/python3.11/site-packages/certifi/core.py:36: DeprecationWarning: path is deprecated. Use files() instead. Refer to https://importlib-resources.readthedocs.io/en/latest/using.html#migrating-from-legacy for migration advice. _CACERT_CTX = get_path("certifi", "cacert.pem") sys:1: ResourceWarning: unclosed <socket.socket fd=7, family=2, type=1, proto=6, laddr=('172.16.5.4', 49202), raddr=('93.184.216.34', 443)> ResourceWarning: Enable tracemalloc to get the object allocation traceback /workspaces/cpython/Lib/asyncio/sslproto.py:116: ResourceWarning: unclosed transport <asyncio._SSLProtocolTransport object> _warnings.warn( /workspaces/cpython/Lib/asyncio/selector_events.py:710: ResourceWarning: unclosed transport <_SelectorSocketTransport fd=7> _warn(f"unclosed transport {self!r}", ResourceWarning, source=self) ResourceWarning: Enable tracemalloc to get the object allocation traceback ----------------------------------------------------------- @asvetlvo This can be closed now. |
|||
msg414825 - (view) | Author: Andrew Svetlov (asvetlov) * | Date: 2022-03-10 02:07 | |
Thanks! |
History | |||
---|---|---|---|
Date | User | Action | Args |
2022-04-11 14:59:14 | admin | set | github: 80890 |
2022-03-10 02:07:51 | asvetlov | set | status: open -> closed resolution: fixed messages: + msg414825 stage: resolved |
2022-03-03 11:22:48 | kumaraditya | set | versions:
+ Python 3.11, - Python 3.7 nosy: + kumaraditya messages: + msg414425 type: behavior |
2020-10-21 16:46:37 | christian.heimes | set | assignee: christian.heimes -> messages: + msg379222 nosy: christian.heimes, asvetlov, tomchristie, yselivanov |
2019-05-30 15:54:49 | asvetlov | set | messages: + msg343984 |
2019-05-30 14:57:32 | yselivanov | set | messages: + msg343981 |
2019-05-30 14:52:05 | asvetlov | set | messages: + msg343980 |
2019-05-30 14:42:53 | yselivanov | set | messages: + msg343979 |
2019-05-30 14:10:43 | asvetlov | set | messages: + msg343978 |
2019-05-30 14:00:41 | tomchristie | set | messages: + msg343977 |
2019-05-29 13:13:55 | asvetlov | set | messages: + msg343883 |
2019-05-28 14:04:19 | tomchristie | set | messages: + msg343780 |
2019-05-28 13:11:30 | asvetlov | set | messages: + msg343776 |
2019-04-24 09:10:09 | tomchristie | set | messages: + msg340765 |
2019-04-24 08:48:27 | tomchristie | create |