This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Memory leak in multiprocessing.pool
Type: resource usage Stage: resolved
Components: Library (Lib) Versions: Python 3.2, Python 3.3, Python 2.7
process
Status: closed Resolution: duplicate
Dependencies: Superseder: join method of multiprocessing Pool object hangs if iterable argument of pool.map is empty
View: 12157
Assigned To: Nosy List: jnoller, neologix, yang
Priority: normal Keywords:

Created on 2011-12-06 20:28 by yang, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Messages (3)
msg148937 - (view) Author: Yang Zhang (yang) Date: 2011-12-06 20:28
Calling Pool.map (and friends) on empty lists [] causes Pool._cache to hang on to the MapResults forever:

tp = ThreadPool(5)
xs = tp.map(lambda x: 'a' * int(10e6), range(100))
# now using 1GB mem = 100 * 10MB strs
del xs
gc.collect()
# still using 1GB mem
tp.close(); tp.join()
# now using ~0GB mem
msg148938 - (view) Author: Yang Zhang (yang) Date: 2011-12-06 20:39
Oops, sorry - pasted a wrong example.


In [38]: tp = ThreadPool(5)

In [39]: xs = tp.map(lambda x: x, [])

In [40]: xs
Out[40]: []

In [41]: tp._cache
Out[41]: {3: <multiprocessing.pool.MapResult at 0x244ab50>}
msg148980 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2011-12-07 17:49
It's a duplicate of #12157.
History
Date User Action Args
2022-04-11 14:57:24adminsetgithub: 57751
2011-12-07 17:49:26neologixsetstatus: open -> closed

superseder: join method of multiprocessing Pool object hangs if iterable argument of pool.map is empty

nosy: + neologix
messages: + msg148980
resolution: duplicate
stage: needs patch -> resolved
2011-12-07 08:42:21ezio.melottisetnosy: + jnoller
stage: needs patch

versions: - Python 2.6, Python 3.1, Python 3.4
2011-12-06 20:39:25yangsetmessages: + msg148938
2011-12-06 20:28:29yangcreate