Issue42647
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2020-12-15 13:59 by jd, last changed 2022-04-11 14:59 by admin.
Messages (4) | |||
---|---|---|---|
msg383058 - (view) | Author: Julien Danjou (jd) * | Date: 2020-12-15 13:59 | |
Python 3.9 introduced a regression with concurrent.futures. The following program works fine on Python < 3.8 but raises an error on 3.9: ``` import atexit import concurrent.futures def spawn(): with concurrent.futures.ThreadPoolExecutor() as t: pass atexit.register(spawn) ``` ``` $ python3.9 rep.py Error in atexit._run_exitfuncs: Traceback (most recent call last): File "<frozen importlib._bootstrap>", line 1007, in _find_and_load File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 680, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 790, in exec_module File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed File "/usr/local/Cellar/python@3.9/3.9.0_3/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py", line 37, in <module> threading._register_atexit(_python_exit) File "/usr/local/Cellar/python@3.9/3.9.0_3/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 1370, in _register_atexit raise RuntimeError("can't register atexit after shutdown") RuntimeError: can't register atexit after shutdown ``` |
|||
msg385584 - (view) | Author: Irit Katriel (iritkatriel) * | Date: 2021-01-24 16:24 | |
b61b818d916942aad1f8f3e33181801c4a1ed14b is the first bad commit commit b61b818d916942aad1f8f3e33181801c4a1ed14b Author: Kyle Stanley <aeros167@gmail.com> Date: Fri Mar 27 15:31:22 2020 -0400 bpo-39812: Remove daemon threads in concurrent.futures (GH-19149) Remove daemon threads from :mod:`concurrent.futures` by adding an internal `threading._register_atexit()`, which calls registered functions prior to joining all non-daemon threads. This allows for compatibility with subinterpreters, which don't support daemon threads. |
|||
msg385972 - (view) | Author: Kyle Stanley (aeros) * | Date: 2021-01-30 07:11 | |
Thanks for bringing attention to this, Julien. While the regression is definitely unfortunate, I'm uncertain about whether the behavior is *correct* to allow in the first place. Specifically, allowing the registration of an atexit hook which uses a ThreadPoolExecutor within it means that the finalization of the executor will be done *after* thread finalization occurs, leaving dangling threads which will have to be abruptly killed upon interpreter exit instead of being safely joined. From my perspective at least, this doesn't seem like something to be encouraged. Is there a real-world situation where it's specifically necessary or even beneficial to utilize ThreadPoolExecutor at this point after thread finalization rather than earlier in the program? Not that it doesn't exist, but to me it intuitively seems very odd to utilize an executor within an atexit hook, which are intended to just be resource finalization/cleanup functions called at interpreter shutdown. Assuming there is a genuine use case I'm not seeing, it may be worth weighing against the decision to convert the executors to not use daemon threads, as I presently don't think there's a way to (safely) allow that behavior without reverting back to using daemon threads. That said, I'll admit that I'm more than a bit biased as the author of the commit which introduced the regression, so I'll CC Antoine Pitrou (active expert for threading and concurrent.futures) to help make the final decision. |
|||
msg386067 - (view) | Author: Julien Danjou (jd) * | Date: 2021-02-01 13:28 | |
> Is there a real-world situation where it's specifically necessary or even beneficial to utilize ThreadPoolExecutor at this point after thread finalization rather than earlier in the program? Not that it doesn't exist, but to me it intuitively seems very odd to utilize an executor within an atexit hook, which are intended to just be resource finalization/cleanup functions called at interpreter shutdown. Assuming there is a genuine use case I'm not seeing, it may be worth weighing against the decision to convert the executors to not use daemon threads, as I presently don't think there's a way to (safely) allow that behavior without reverting back to using daemon threads. To put that in perspective, here is the original issue that trigged this bug for me: Traceback (most recent call last): File "/root/project/ddtrace/profiling/scheduler.py", line 50, in flush exp.export(events, start, self._last_export) File "/root/project/ddtrace/profiling/exporter/http.py", line 186, in export self._upload(client, self.endpoint_path, body, headers) File "/root/project/ddtrace/profiling/exporter/http.py", line 189, in _upload self._retry_upload(self._upload_once, client, path, body, headers) File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/tenacity/__init__.py", line 423, in __call__ do = self.iter(retry_state=retry_state) File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/tenacity/__init__.py", line 360, in iter return fut.result() File "/root/.pyenv/versions/3.9.0/lib/python3.9/concurrent/futures/_base.py", line 433, in result return self.__get_result() File "/root/.pyenv/versions/3.9.0/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result raise self._exception File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/tenacity/__init__.py", line 426, in __call__ result = fn(*args, **kwargs) File "/root/project/ddtrace/profiling/exporter/http.py", line 193, in _upload_once client.request("POST", path, body=body, headers=headers) File "/root/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 1255, in request self._send_request(method, url, body, headers, encode_chunked) File "/root/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 1301, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/root/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 1250, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/root/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 1010, in _send_output self.send(msg) File "/root/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 950, in send self.connect() File "/root/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 921, in connect self.sock = self._create_connection( File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/socket.py", line 88, in create_connection addrs = list(getaddrinfo(host, port, 0, SOCK_STREAM)) File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/_socketcommon.py", line 247, in getaddrinfo addrlist = get_hub().resolver.getaddrinfo(host, port, family, type, proto, flags) File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/hub.py", line 841, in _get_resolver self._resolver = self.resolver_class(hub=self) # pylint:disable=not-callable File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/resolver/thread.py", line 39, in __init__ self.pool = hub.threadpool File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/hub.py", line 865, in _get_threadpool self._threadpool = self.threadpool_class(self.threadpool_size, hub=self) File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/hub.py", line 860, in threadpool_class return GEVENT_CONFIG.threadpool File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/_config.py", line 50, in getter return self.settings[setting_name].get() File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/_config.py", line 146, in get self.value = self.validate(self._default()) File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/_config.py", line 248, in validate return self._import_one_of([self.shortname_map.get(x, x) for x in value]) File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/_config.py", line 223, in _import_one_of return self._import_one(candidates[-1]) File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/_config.py", line 237, in _import_one module = importlib.import_module(module) File "/root/.pyenv/versions/3.9.0/lib/python3.9/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1030, in _gcd_import File "<frozen importlib._bootstrap>", line 1007, in _find_and_load File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 680, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 790, in exec_module File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/threadpool.py", line 748, in <module> class ThreadPoolExecutor(concurrent.futures.ThreadPoolExecutor): File "/root/.pyenv/versions/3.9.0/lib/python3.9/concurrent/futures/__init__.py", line 49, in __getattr__ from .thread import ThreadPoolExecutor as te File "/root/.pyenv/versions/3.9.0/lib/python3.9/concurrent/futures/thread.py", line 37, in <module> threading._register_atexit(_python_exit) File "/root/.pyenv/versions/3.9.0/lib/python3.9/threading.py", line 1370, in _register_atexit raise RuntimeError("can't register atexit after shutdown") RuntimeError: can't register atexit after shutdown What's happening is that the ddtrace library registers an atexit hook that does an HTTP call. As the application runs using gevent, some gevent modules that were not loaded are loaded during the atexit() hook and the loading of `concurrent.futures.thread` is done very late, at the point where the interpreter is shutting down. I'm totally fine blaming gevent here if you prefer. The problem is that there's nothing preventing any library call to be made in an atexit() hook, and any library could decide to use `concurent.futures` without the library user being able to do anything about it, except maybe, making sure `concurent.futures` is loaded very early in the program. However, having to load this library even if you don't use it to be sure it does not break would break the separation of principles. At at least, at this stage, it might be the responsibility of Python to make sure all `threading._register_at_exit` calls are done whatever if the library is used or not (i.e. that'd mean loading `concurent.futures` with `threading` unconditionally). |
History | |||
---|---|---|---|
Date | User | Action | Args |
2022-04-11 14:59:39 | admin | set | github: 86813 |
2021-02-01 13:41:43 | corona10 | set | nosy:
+ vstinner |
2021-02-01 13:28:41 | jd | set | messages: + msg386067 |
2021-01-30 07:11:23 | aeros | set | nosy:
+ pitrou messages: + msg385972 |
2021-01-24 16:25:34 | iritkatriel | set | nosy:
+ aeros |
2021-01-24 16:24:51 | iritkatriel | set | nosy:
+ iritkatriel messages: + msg385584 |
2020-12-15 13:59:10 | jd | create |