classification
Title: memory leak in threading ?
Type: behavior Stage: resolved
Components: Extension Modules Versions: Python 3.8
process
Status: closed Resolution: duplicate
Dependencies: Superseder: fix for bpo-36402 (threading._shutdown() race condition) causes reference leak
View: 37788
Assigned To: Nosy List: igorvm, mark.dickinson, martin.panter
Priority: normal Keywords:

Created on 2021-03-02 18:10 by igorvm, last changed 2021-03-09 13:03 by mark.dickinson. This issue is now closed.

Files
File name Uploaded Description Edit
memleak.py igorvm, 2021-03-02 18:10 python code illustrating the leak
Messages (7)
msg387948 - (view) Author: Igor Mandrichenko (igorvm) Date: 2021-03-02 18:10
There is an apparent memory leak in threading. It looks like memory grows when I create, run and destroy threads. The memory keeps adding at the rate of about 100 bytes per thread.

I am attaching the code, which works for Linux. getMemory() function is Linux-specific, it gets current process memory utilization
msg387950 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2021-03-02 18:32
I'm not sure that this counts as a bug.

Each non-daemon thread adds a lock to the threading._shutdown_locks set: https://github.com/python/cpython/blob/8b795ab5541d8a4e69be4137dfdc207714270b77/Lib/threading.py#L933-L935.

That lock is removed when the thread is "join"ed: https://github.com/python/cpython/blob/8b795ab5541d8a4e69be4137dfdc207714270b77/Lib/threading.py#L988-L990

So the simple solution is to make sure that you always join your threads (which I'd expect normal code to do for non-daemon threads anyway), or to make your threads daemon threads.
msg387952 - (view) Author: Igor Mandrichenko (igorvm) Date: 2021-03-02 18:51
You are right. When I add join(), the memory does not grow any more.
Thanks !
msg388051 - (view) Author: Igor Mandrichenko (igorvm) Date: 2021-03-03 20:01
Since Timer thread is never joined, should not it be a daemon ?
msg388125 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2021-03-04 19:46
> Since Timer thread is never joined, should not it be a daemon?

Possibly, but I think that's a new question/issue. :-) And presumably there's no reason you can't make your Timer threads into daemon threads if you have a need to.
msg388357 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2021-03-09 11:48
Sounds the same as Issue 37788, which is still open.
msg388362 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2021-03-09 13:02
Thanks; I missed that one. I'll update the state of this one to mark it as a duplicate of #37788.
History
Date User Action Args
2021-03-09 13:03:03mark.dickinsonsetsuperseder: fix for bpo-36402 (threading._shutdown() race condition) causes reference leak
resolution: duplicate
2021-03-09 13:02:48mark.dickinsonsetmessages: + msg388362
2021-03-09 11:48:15martin.pantersetnosy: + martin.panter
messages: + msg388357
2021-03-04 19:46:39mark.dickinsonsetmessages: + msg388125
2021-03-03 20:01:28igorvmsetmessages: + msg388051
2021-03-02 18:51:37igorvmsetstatus: open -> closed
stage: resolved
2021-03-02 18:51:07igorvmsetmessages: + msg387952
2021-03-02 18:32:56mark.dickinsonsetnosy: + mark.dickinson
messages: + msg387950
2021-03-02 18:10:58igorvmcreate