> Is there a real-world situation where it's specifically necessary or even beneficial to utilize ThreadPoolExecutor at this point after thread finalization rather than earlier in the program? Not that it doesn't exist, but to me it intuitively seems very odd to utilize an executor within an atexit hook, which are intended to just be resource finalization/cleanup functions called at interpreter shutdown. Assuming there is a genuine use case I'm not seeing, it may be worth weighing against the decision to convert the executors to not use daemon threads, as I presently don't think there's a way to (safely) allow that behavior without reverting back to using daemon threads.
To put that in perspective, here is the original issue that trigged this bug for me:
Traceback (most recent call last):
File "/root/project/ddtrace/profiling/scheduler.py", line 50, in flush
exp.export(events, start, self._last_export)
File "/root/project/ddtrace/profiling/exporter/http.py", line 186, in export
self._upload(client, self.endpoint_path, body, headers)
File "/root/project/ddtrace/profiling/exporter/http.py", line 189, in _upload
self._retry_upload(self._upload_once, client, path, body, headers)
File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/tenacity/__init__.py", line 423, in __call__
do = self.iter(retry_state=retry_state)
File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/tenacity/__init__.py", line 360, in iter
return fut.result()
File "/root/.pyenv/versions/3.9.0/lib/python3.9/concurrent/futures/_base.py", line 433, in result
return self.__get_result()
File "/root/.pyenv/versions/3.9.0/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/tenacity/__init__.py", line 426, in __call__
result = fn(*args, **kwargs)
File "/root/project/ddtrace/profiling/exporter/http.py", line 193, in _upload_once
client.request("POST", path, body=body, headers=headers)
File "/root/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 1255, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/root/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 1301, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/root/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 1250, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/root/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 1010, in _send_output
self.send(msg)
File "/root/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 950, in send
self.connect()
File "/root/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 921, in connect
self.sock = self._create_connection(
File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/socket.py", line 88, in create_connection
addrs = list(getaddrinfo(host, port, 0, SOCK_STREAM))
File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/_socketcommon.py", line 247, in getaddrinfo
addrlist = get_hub().resolver.getaddrinfo(host, port, family, type, proto, flags)
File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/hub.py", line 841, in _get_resolver
self._resolver = self.resolver_class(hub=self) # pylint:disable=not-callable
File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/resolver/thread.py", line 39, in __init__
self.pool = hub.threadpool
File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/hub.py", line 865, in _get_threadpool
self._threadpool = self.threadpool_class(self.threadpool_size, hub=self)
File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/hub.py", line 860, in threadpool_class
return GEVENT_CONFIG.threadpool
File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/_config.py", line 50, in getter
return self.settings[setting_name].get()
File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/_config.py", line 146, in get
self.value = self.validate(self._default())
File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/_config.py", line 248, in validate
return self._import_one_of([self.shortname_map.get(x, x) for x in value])
File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/_config.py", line 223, in _import_one_of
return self._import_one(candidates[-1])
File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/_config.py", line 237, in _import_one
module = importlib.import_module(module)
File "/root/.pyenv/versions/3.9.0/lib/python3.9/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 790, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/threadpool.py", line 748, in <module>
class ThreadPoolExecutor(concurrent.futures.ThreadPoolExecutor):
File "/root/.pyenv/versions/3.9.0/lib/python3.9/concurrent/futures/__init__.py", line 49, in __getattr__
from .thread import ThreadPoolExecutor as te
File "/root/.pyenv/versions/3.9.0/lib/python3.9/concurrent/futures/thread.py", line 37, in <module>
threading._register_atexit(_python_exit)
File "/root/.pyenv/versions/3.9.0/lib/python3.9/threading.py", line 1370, in _register_atexit
raise RuntimeError("can't register atexit after shutdown")
RuntimeError: can't register atexit after shutdown
What's happening is that the ddtrace library registers an atexit hook that does an HTTP call. As the application runs using gevent, some gevent modules that were not loaded are loaded during the atexit() hook and the loading of `concurrent.futures.thread` is done very late, at the point where the interpreter is shutting down.
I'm totally fine blaming gevent here if you prefer. The problem is that there's nothing preventing any library call to be made in an atexit() hook, and any library could decide to use `concurent.futures` without the library user being able to do anything about it, except maybe, making sure `concurent.futures` is loaded very early in the program. However, having to load this library even if you don't use it to be sure it does not break would break the separation of principles.
At at least, at this stage, it might be the responsibility of Python to make sure all `threading._register_at_exit` calls are done whatever if the library is used or not (i.e. that'd mean loading `concurent.futures` with `threading` unconditionally). |