Issue39074
Created on 2019-12-17 15:51 by krypticus, last changed 2020-01-06 22:51 by vstinner. This issue is now closed.
Messages (3) | |||
---|---|---|---|
msg358551 - (view) | Author: Adam (krypticus) | Date: 2019-12-17 15:51 | |
When running 3.7, we noticed a memory leak in threading._shutdown_locks when non-deamon threads are started but "join()" or "is_alive()" is never called. Here's a test to illustrate the growth: ========= import threading import time import tracemalloc def test_leaking_locks(): tracemalloc.start(10) snap1 = tracemalloc.take_snapshot() def print_things(): print('.', end='') for x in range(500): t = threading.Thread(target=print_things) t.start() time.sleep(5) print('') gc.collect() snap2 = tracemalloc.take_snapshot() filters = [] for stat in snap2.filter_traces(filters).compare_to(snap1.filter_traces(filters), 'traceback')[:10]: print("New Bytes: {}\tTotal Bytes {}\tNew blocks: {}\tTotal blocks: {}: ".format(stat.size_diff, stat.size, stat.count_diff ,stat.count)) for line in stat.traceback.format(): print(line) ========= ========= Output in v3.6.8: New Bytes: 840 Total Bytes 840 New blocks: 1 Total blocks: 1: File "/usr/local/lib/python3.6/threading.py", line 884 self._bootstrap_inner() New Bytes: 608 Total Bytes 608 New blocks: 4 Total blocks: 4: File "/usr/local/lib/python3.6/tracemalloc.py", line 387 self.traces = _Traces(traces) File "/usr/local/lib/python3.6/tracemalloc.py", line 524 return Snapshot(traces, traceback_limit) File "/gems/tests/integration/endpoint_connection_test.py", line 856 snap1 = tracemalloc.take_snapshot() File "/usr/local/lib/python3.6/site-packages/_pytest/python.py", line 198 testfunction(**testargs) File "/usr/local/lib/python3.6/site-packages/pluggy/callers.py", line 187 res = hook_impl.function(*args) File "/usr/local/lib/python3.6/site-packages/pluggy/manager.py", line 87 firstresult=hook.spec.opts.get("firstresult") if hook.spec else False, File "/usr/local/lib/python3.6/site-packages/pluggy/manager.py", line 93 return self._inner_hookexec(hook, methods, kwargs) File "/usr/local/lib/python3.6/site-packages/pluggy/hooks.py", line 286 return self._hookexec(self, self.get_hookimpls(), kwargs) File "/usr/local/lib/python3.6/site-packages/_pytest/python.py", line 1459 self.ihook.pytest_pyfunc_call(pyfuncitem=self) File "/usr/local/lib/python3.6/site-packages/_pytest/runner.py", line 111 item.runtest() ========== Output in v3.7.4: New Bytes: 36000 Total Bytes 36000 New blocks: 1000 Total blocks: 1000: File "/usr/local/lib/python3.7/threading.py", line 890 self._bootstrap_inner() File "/usr/local/lib/python3.7/threading.py", line 914 self._set_tstate_lock() File "/usr/local/lib/python3.7/threading.py", line 904 self._tstate_lock = _set_sentinel() New Bytes: 32768 Total Bytes 32768 New blocks: 1 Total blocks: 1: File "/usr/local/lib/python3.7/threading.py", line 890 self._bootstrap_inner() File "/usr/local/lib/python3.7/threading.py", line 914 self._set_tstate_lock() File "/usr/local/lib/python3.7/threading.py", line 909 _shutdown_locks.add(self._tstate_lock) ================= It looks like this commit didn't take into account the tstate_lock cleanup that happens in the C code, and it's not removing the _tstate_lock of completed threads from the _shutdown_locks once the thread finishes, unless the code manually calls "join()" or "is_alive()" on the thread: https://github.com/python/cpython/commit/468e5fec8a2f534f1685d59da3ca4fad425c38dd Let me know if I can provide more clarity on this! |
|||
msg358597 - (view) | Author: Adam (krypticus) | Date: 2019-12-17 22:56 | |
Looks like this issue might be a duplicate of https://bugs.python.org/issue37788 |
|||
msg359476 - (view) | Author: STINNER Victor (vstinner) * ![]() |
Date: 2020-01-06 22:51 | |
Ok, I close this issue as a duplicate of bpo-37788. |
History | |||
---|---|---|---|
Date | User | Action | Args |
2020-01-06 22:51:08 | vstinner | set | status: open -> closed superseder: fix for bpo-36402 (threading._shutdown() race condition) causes reference leak nosy: + vstinner messages: + msg359476 resolution: duplicate stage: resolved |
2019-12-17 22:56:31 | krypticus | set | messages: + msg358597 |
2019-12-17 22:45:58 | krypticus | set | type: resource usage -> security |
2019-12-17 15:51:34 | krypticus | create |