classification
Title: Threading memory leak in _shutdown_locks for non-daemon threads
Type: security Stage: resolved
Components: Versions: Python 3.9, Python 3.8, Python 3.7
process
Status: closed Resolution: duplicate
Dependencies: Superseder: fix for bpo-36402 (threading._shutdown() race condition) causes reference leak
View: 37788
Assigned To: Nosy List: krypticus, vstinner
Priority: normal Keywords:

Created on 2019-12-17 15:51 by krypticus, last changed 2020-01-06 22:51 by vstinner. This issue is now closed.

Messages (3)
msg358551 - (view) Author: Adam (krypticus) Date: 2019-12-17 15:51
When running 3.7, we noticed a memory leak in threading._shutdown_locks when non-deamon threads are started but "join()" or "is_alive()" is never called. Here's a test to illustrate the growth:

=========
import threading
import time
import tracemalloc

def test_leaking_locks():
    tracemalloc.start(10)
    snap1 = tracemalloc.take_snapshot()
    def print_things():
        print('.', end='')

    for x in range(500):
        t = threading.Thread(target=print_things)
        t.start()

    time.sleep(5)
    print('')
    gc.collect()
    snap2 = tracemalloc.take_snapshot()
    filters = []
    for stat in snap2.filter_traces(filters).compare_to(snap1.filter_traces(filters), 'traceback')[:10]:
        print("New Bytes: {}\tTotal Bytes {}\tNew blocks: {}\tTotal blocks: {}: ".format(stat.size_diff, stat.size, stat.count_diff ,stat.count))
        for line in stat.traceback.format():
            print(line)

=========

=========
Output in v3.6.8:

New Bytes: 840	Total Bytes 840	New blocks: 1	Total blocks: 1: 
  File "/usr/local/lib/python3.6/threading.py", line 884
    self._bootstrap_inner()
New Bytes: 608	Total Bytes 608	New blocks: 4	Total blocks: 4: 
  File "/usr/local/lib/python3.6/tracemalloc.py", line 387
    self.traces = _Traces(traces)
  File "/usr/local/lib/python3.6/tracemalloc.py", line 524
    return Snapshot(traces, traceback_limit)
  File "/gems/tests/integration/endpoint_connection_test.py", line 856
    snap1 = tracemalloc.take_snapshot()
  File "/usr/local/lib/python3.6/site-packages/_pytest/python.py", line 198
    testfunction(**testargs)
  File "/usr/local/lib/python3.6/site-packages/pluggy/callers.py", line 187
    res = hook_impl.function(*args)
  File "/usr/local/lib/python3.6/site-packages/pluggy/manager.py", line 87
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/usr/local/lib/python3.6/site-packages/pluggy/manager.py", line 93
    return self._inner_hookexec(hook, methods, kwargs)
  File "/usr/local/lib/python3.6/site-packages/pluggy/hooks.py", line 286
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/usr/local/lib/python3.6/site-packages/_pytest/python.py", line 1459
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  File "/usr/local/lib/python3.6/site-packages/_pytest/runner.py", line 111
    item.runtest()
==========
Output in v3.7.4:

New Bytes: 36000	Total Bytes 36000	New blocks: 1000	Total blocks: 1000: 
  File "/usr/local/lib/python3.7/threading.py", line 890
    self._bootstrap_inner()
  File "/usr/local/lib/python3.7/threading.py", line 914
    self._set_tstate_lock()
  File "/usr/local/lib/python3.7/threading.py", line 904
    self._tstate_lock = _set_sentinel()
New Bytes: 32768	Total Bytes 32768	New blocks: 1	Total blocks: 1: 
  File "/usr/local/lib/python3.7/threading.py", line 890
    self._bootstrap_inner()
  File "/usr/local/lib/python3.7/threading.py", line 914
    self._set_tstate_lock()
  File "/usr/local/lib/python3.7/threading.py", line 909
    _shutdown_locks.add(self._tstate_lock)
=================

It looks like this commit didn't take into account the tstate_lock cleanup that happens in the C code, and it's not removing the _tstate_lock of completed threads from the _shutdown_locks once the thread finishes, unless the code manually calls "join()" or "is_alive()" on the thread:

https://github.com/python/cpython/commit/468e5fec8a2f534f1685d59da3ca4fad425c38dd

Let me know if I can provide more clarity on this!
msg358597 - (view) Author: Adam (krypticus) Date: 2019-12-17 22:56
Looks like this issue might be a duplicate of https://bugs.python.org/issue37788
msg359476 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-01-06 22:51
Ok, I close this issue as a duplicate of bpo-37788.
History
Date User Action Args
2020-01-06 22:51:08vstinnersetstatus: open -> closed

superseder: fix for bpo-36402 (threading._shutdown() race condition) causes reference leak

nosy: + vstinner
messages: + msg359476
resolution: duplicate
stage: resolved
2019-12-17 22:56:31krypticussetmessages: + msg358597
2019-12-17 22:45:58krypticussettype: resource usage -> security
2019-12-17 15:51:34krypticuscreate