Title: multiprocessing.pool.ThreadPool eats up memories
Type: Stage: resolved
Components: Library (Lib) Versions: Python 3.5, Python 2.7
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: davin, pitrou, renlifeng
Priority: normal Keywords:

Created on 2016-02-23 09:47 by renlifeng, last changed 2017-03-21 07:50 by pitrou. This issue is now closed.

File name Uploaded Description Edit renlifeng, 2016-02-23 09:47 code to reproduce the symptom
Messages (4)
msg260716 - (view) Author: renlifeng (renlifeng) Date: 2016-02-23 09:47
If func creates lots objects and appends them to a list, and runs over and over, will eventually eat up all memories. Cleaning the list at the end of func does not help.

One can reproduce by running the attached file.

By contrast, after replacing ThreadPool with the theading module (see the commented out lines), memory usage will not grow continuously.

By the way, I used what's in Debian stretch, i.e. python 2.7.11 and 3.5.1.
msg289893 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-03-20 18:11
Your script works fine here.  I see:
iter 30, 30040
iter 29, 269848
iter 28, 269848
iter 27, 271996
msg289909 - (view) Author: renlifeng (renlifeng) Date: 2017-03-21 03:22
I confirm that 3.5.3 and 2.7.13 have fixed this problem. Now the memory usage will stop growing after using 28% of physical memory.

In other words, this problem can not be reproduced with the latest version of python, Thanks.

It's OK for me to set it to any of out of date, fixed, or closed.
msg289926 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-03-21 07:50
Thanks for the feedback!  I'm closing the issue then.
Date User Action Args
2017-03-21 07:50:57pitrousetstatus: open -> closed
resolution: fixed
messages: + msg289926

stage: resolved
2017-03-21 03:22:29renlifengsetmessages: + msg289909
2017-03-20 18:11:05pitrousetnosy: + pitrou, davin
messages: + msg289893
2016-02-23 09:47:20renlifengcreate