Message375647
I've been debugging a high memory consumption in one of my scripts and traced it back to the `concurrent.futures.ThreadPoolExecutor`.
When further investigating and playing around, I found out that when using `concurrent.futures.ThreadPoolExecutor` with the map function, and passing a dictionary to the map's function as an argument, the memory used by the pool won't be freed and as a result the total memory consumption will continue to rise. (Seems like it also happens when passing a list and maybe even other types).
Here is an example of a code to recreate this issue:
```
#!/usr/bin/env python3
import os
import time
import psutil
import random
import concurrent.futures
from memory_profiler import profile as mem_profile
p = psutil.Process(os.getpid())
def do_magic(values):
return None
@mem_profile
def foo():
a = {i: chr(i) for i in range(1024)}
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as pool:
proccessed_data = pool.map(do_magic, a)
def fooer():
while True:
foo()
time.sleep(1)
fooer()
``` |
|
Date |
User |
Action |
Args |
2020-08-19 13:55:16 | or12 | set | recipients:
+ or12 |
2020-08-19 13:55:15 | or12 | set | messageid: <1597845315.75.0.199249722825.issue41588@roundup.psfhosted.org> |
2020-08-19 13:55:15 | or12 | link | issue41588 messages |
2020-08-19 13:55:13 | or12 | create | |
|