classification
Title: multiprocessing.pool.ThreadPool eats up memories
Type: Stage: resolved
Components: Library (Lib) Versions: Python 3.5, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: davin, pitrou, renlifeng
Priority: normal Keywords:

Created on 2016-02-23 09:47 by renlifeng, last changed 2017-03-21 07:50 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
cvtest5.py renlifeng, 2016-02-23 09:47 code to reproduce the symptom
Messages (4)
msg260716 - (view) Author: renlifeng (renlifeng) Date: 2016-02-23 09:47
If func creates lots objects and appends them to a list, and runs over and over, pool.map(func...) will eventually eat up all memories. Cleaning the list at the end of func does not help.

One can reproduce by running the attached file.

By contrast, after replacing ThreadPool with the theading module (see the commented out lines), memory usage will not grow continuously.

By the way, I used what's in Debian stretch, i.e. python 2.7.11 and 3.5.1.
msg289893 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-03-20 18:11
Your script works fine here.  I see:
iter 30, 30040
iter 29, 269848
iter 28, 269848
iter 27, 271996
[...]
msg289909 - (view) Author: renlifeng (renlifeng) Date: 2017-03-21 03:22
I confirm that 3.5.3 and 2.7.13 have fixed this problem. Now the memory usage will stop growing after using 28% of physical memory.

In other words, this problem can not be reproduced with the latest version of python, Thanks.

It's OK for me to set it to any of out of date, fixed, or closed.
msg289926 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-03-21 07:50
Thanks for the feedback!  I'm closing the issue then.
History
Date User Action Args
2017-03-21 07:50:57pitrousetstatus: open -> closed
resolution: fixed
messages: + msg289926

stage: resolved
2017-03-21 03:22:29renlifengsetmessages: + msg289909
2017-03-20 18:11:05pitrousetnosy: + pitrou, davin
messages: + msg289893
2016-02-23 09:47:20renlifengcreate