This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author or12
Recipients or12
Date 2020-08-19.13:55:13
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1597845315.75.0.199249722825.issue41588@roundup.psfhosted.org>
In-reply-to
Content
I've been debugging a high memory consumption in one of my scripts and traced it back to the `concurrent.futures.ThreadPoolExecutor`.

When further investigating and playing around, I found out that when using `concurrent.futures.ThreadPoolExecutor` with the map function, and passing a dictionary to the map's function as an argument, the memory used by the pool won't be freed and as a result the total memory consumption will continue to rise. (Seems like it also happens when passing a list and maybe even other types).

Here is an example of a code to recreate this issue:

```
#!/usr/bin/env python3

import os
import time
import psutil
import random
import concurrent.futures

from memory_profiler import profile as mem_profile

p = psutil.Process(os.getpid())

def do_magic(values):
    return None

@mem_profile
def foo():
    a = {i: chr(i) for i in range(1024)}
    with concurrent.futures.ThreadPoolExecutor(max_workers=10) as pool:
        proccessed_data = pool.map(do_magic, a)

def fooer():
    while True:
        foo()
        time.sleep(1)

fooer()
```
History
Date User Action Args
2020-08-19 13:55:16or12setrecipients: + or12
2020-08-19 13:55:15or12setmessageid: <1597845315.75.0.199249722825.issue41588@roundup.psfhosted.org>
2020-08-19 13:55:15or12linkissue41588 messages
2020-08-19 13:55:13or12create