classification
Title: Threading.timer leaks memory in 3.8.0/3.8.1
Type: Stage: resolved
Components: Library (Lib) Versions: Python 3.8
process
Status: closed Resolution: duplicate
Dependencies: Superseder: fix for bpo-36402 (threading._shutdown() race condition) causes reference leak
View: 37788
Assigned To: Nosy List: krypticus, mneerup, pitrou, vstinner, xtreak
Priority: normal Keywords:

Created on 2020-01-03 15:32 by mneerup, last changed 2020-01-06 22:53 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
images_code.tar mneerup, 2020-01-03 15:32 Snippet that consumes memory in 3.8.0/3.8.1 and images from mprof
Messages (5)
msg359239 - (view) Author: Mathias (mneerup) Date: 2020-01-03 15:32
Hi,

I think there is an issue with memory allocating with threading.Timer in 3.8.0/3.8.1.

When I run the attached code in Python 3.7.3 I get a memory consumption of approx. 10 MB. If I run the same code with python 3.8.0 or 3.8.1, it keeps consuming memory (several hundreds of MB). 

I've attached 3 images where I run the attached script under mprof.

By inspect of the output from tracemalloc, I can see that 
/usr/lib/python3.8/threading.py:908 keeps allocating memory.

I have tested this with python 3.8.0 from ubuntu 16.04 repository and python 3.8.1 from source. Both versions suffers from the memory consumption issue.
msg359240 - (view) Author: Matthew Smith (odinsbane) Date: 2020-01-03 15:57
This issue also affects me, pyth. I tried it out with various combinations of sleep, and durations of tasks, but what I noticed is that threading_shutdown_locks continues to grow at each iteration.

for i in range(100000):
    for j in range(10):
        timer = threading.Thread(target = callback)
        timer.start()
    time.sleep(1)
    print(len(threading._shutdown_locks))

If I accumulate the Timer/Thread s and join them, then the _shutdown_locks will be cleared.
msg359241 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2020-01-03 16:03
Please include the script as an attachment or text so that it will be easy to view them. The script as below from the attachment : 

import threading
import time
import tracemalloc

def callback():
    pass

tracemalloc.start()
for i in range(100000):
    snapshot1 = tracemalloc.take_snapshot()
    for i in range(100000):
        timer = threading.Timer(1, callback)
        timer.start()
    snapshot2 = tracemalloc.take_snapshot()
    top_stats = snapshot2.compare_to(snapshot1, 'lineno')
    print("[ Top 10 differences ]")
	
    for stat in top_stats[:10]:
        print(stat)
msg359244 - (view) Author: Adam (krypticus) Date: 2020-01-03 17:05
I filed a bug for this a few weeks ago, and then found another ticket about the same issue before:

https://bugs.python.org/issue37788

My ticket: 
https://bugs.python.org/issue39074

The memory leak was from a change introduced about 6 months ago:

https://github.com/python/cpython/commit/468e5fec8a2f534f1685d59da3ca4fad425c38dd
msg359478 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-01-06 22:53
I close this issue as a duplicate of bpo-37788.
History
Date User Action Args
2020-01-06 22:53:50vstinnersetstatus: open -> closed
superseder: fix for bpo-36402 (threading._shutdown() race condition) causes reference leak
messages: + msg359478

resolution: duplicate
stage: resolved
2020-01-03 17:47:16pablogsalsetnosy: + vstinner
2020-01-03 17:05:49krypticussetnosy: + krypticus
messages: + msg359244
2020-01-03 16:03:10xtreaksetnosy: + xtreak, pitrou, - odinsbane
messages: + msg359241
2020-01-03 15:57:33odinsbanesetnosy: + odinsbane
messages: + msg359240
2020-01-03 15:32:28mneerupcreate