This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author krypticus
Recipients krypticus
Date 2019-12-17.15:51:33
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1576597894.7.0.119079561961.issue39074@roundup.psfhosted.org>
In-reply-to
Content
When running 3.7, we noticed a memory leak in threading._shutdown_locks when non-deamon threads are started but "join()" or "is_alive()" is never called. Here's a test to illustrate the growth:

=========
import threading
import time
import tracemalloc

def test_leaking_locks():
    tracemalloc.start(10)
    snap1 = tracemalloc.take_snapshot()
    def print_things():
        print('.', end='')

    for x in range(500):
        t = threading.Thread(target=print_things)
        t.start()

    time.sleep(5)
    print('')
    gc.collect()
    snap2 = tracemalloc.take_snapshot()
    filters = []
    for stat in snap2.filter_traces(filters).compare_to(snap1.filter_traces(filters), 'traceback')[:10]:
        print("New Bytes: {}\tTotal Bytes {}\tNew blocks: {}\tTotal blocks: {}: ".format(stat.size_diff, stat.size, stat.count_diff ,stat.count))
        for line in stat.traceback.format():
            print(line)

=========

=========
Output in v3.6.8:

New Bytes: 840	Total Bytes 840	New blocks: 1	Total blocks: 1: 
  File "/usr/local/lib/python3.6/threading.py", line 884
    self._bootstrap_inner()
New Bytes: 608	Total Bytes 608	New blocks: 4	Total blocks: 4: 
  File "/usr/local/lib/python3.6/tracemalloc.py", line 387
    self.traces = _Traces(traces)
  File "/usr/local/lib/python3.6/tracemalloc.py", line 524
    return Snapshot(traces, traceback_limit)
  File "/gems/tests/integration/endpoint_connection_test.py", line 856
    snap1 = tracemalloc.take_snapshot()
  File "/usr/local/lib/python3.6/site-packages/_pytest/python.py", line 198
    testfunction(**testargs)
  File "/usr/local/lib/python3.6/site-packages/pluggy/callers.py", line 187
    res = hook_impl.function(*args)
  File "/usr/local/lib/python3.6/site-packages/pluggy/manager.py", line 87
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/usr/local/lib/python3.6/site-packages/pluggy/manager.py", line 93
    return self._inner_hookexec(hook, methods, kwargs)
  File "/usr/local/lib/python3.6/site-packages/pluggy/hooks.py", line 286
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/usr/local/lib/python3.6/site-packages/_pytest/python.py", line 1459
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  File "/usr/local/lib/python3.6/site-packages/_pytest/runner.py", line 111
    item.runtest()
==========
Output in v3.7.4:

New Bytes: 36000	Total Bytes 36000	New blocks: 1000	Total blocks: 1000: 
  File "/usr/local/lib/python3.7/threading.py", line 890
    self._bootstrap_inner()
  File "/usr/local/lib/python3.7/threading.py", line 914
    self._set_tstate_lock()
  File "/usr/local/lib/python3.7/threading.py", line 904
    self._tstate_lock = _set_sentinel()
New Bytes: 32768	Total Bytes 32768	New blocks: 1	Total blocks: 1: 
  File "/usr/local/lib/python3.7/threading.py", line 890
    self._bootstrap_inner()
  File "/usr/local/lib/python3.7/threading.py", line 914
    self._set_tstate_lock()
  File "/usr/local/lib/python3.7/threading.py", line 909
    _shutdown_locks.add(self._tstate_lock)
=================

It looks like this commit didn't take into account the tstate_lock cleanup that happens in the C code, and it's not removing the _tstate_lock of completed threads from the _shutdown_locks once the thread finishes, unless the code manually calls "join()" or "is_alive()" on the thread:

https://github.com/python/cpython/commit/468e5fec8a2f534f1685d59da3ca4fad425c38dd

Let me know if I can provide more clarity on this!
History
Date User Action Args
2019-12-17 15:51:34krypticussetrecipients: + krypticus
2019-12-17 15:51:34krypticussetmessageid: <1576597894.7.0.119079561961.issue39074@roundup.psfhosted.org>
2019-12-17 15:51:34krypticuslinkissue39074 messages
2019-12-17 15:51:33krypticuscreate