Message391586
I use the following code to produce what looks like a memory leak after emptying a queue with the get() method.
import queue
import os
import psutil
def run(del_after_puts, del_after_gets, n_puts, process):
mem = queue.Queue()
for msg in range(n_puts):
msg_put = f'{msg}_0000000000000000000000000000000000000000000000000000000000000333333333333331111111111'
if msg % 1000000 == 0:
print(f'puting {msg} qsize {len(mem.queue)}')
mem.put(msg_put)
print(f'------ put done ----- qsize {len(mem.queue)}')
print(f'mem_pct {round(process.memory_percent(), 2)}% ')
if del_after_puts:
print(f'deleting queue after puts {mem}')
del mem
print(f'mem_pct {round(process.memory_percent(), 2)}% ')
return
for _ in range(n_puts):
msg_get = mem.get()
msg = int(msg_get.split('_')[0])
if msg % 1000000 == 0:
print(f'getting_q {msg} qsize {len(mem.queue)} ')
mem.task_done()
print(f'------ gets done ----- qsize {len(mem.queue)}')
print(f'mem_pct {round(process.memory_percent(), 2)}% ')
if del_after_gets:
print(f'deleting queue after gets {mem}')
del mem
print(f'mem_pct {round(process.memory_percent(), 2)}% ')
return
if __name__ == '__main__':
del_after_puts = False
del_after_gets = False
n_puts = 20_000_000
print()
print('#########')
print(f'del_after_puts {del_after_puts} del_after_gets {del_after_gets} n_puts {n_puts}')
process = psutil.Process(os.getpid())
print('before run')
print(f'mem_pct {round(process.memory_percent(), 2)}% ')
run(del_after_puts, del_after_gets, n_puts, process)
print(f'after run return')
print(f'mem_pct {round(process.memory_percent(), 2)}% ')
This script can be run in 3 ways:
1. Add n_puts elements into the queue and then empty it.
2. Add n_puts elements into the queue and then delete the queue object
3. Add n_puts elements into the queue and then empty it and then delete the queue object.
For the 1st and 3rd case, the script seems to produce a memory leak as in the following:
1st case, before putting elements into the queue mem used is 0.15%, after emptying it 2.22%:
> #########
>
> del_after_puts False del_after_gets False n_puts 20000000
>
> before run
>
> mem_pct 0.15%
>
> ------ put done ----- qsize 20000000
>
> mem_pct 37.61%
>
> ------ gets done ----- qsize 0
>
> mem_pct 2.22%
3rd case, before putting elements into the queue mem used is 0.15%, after emptying it 2.22%, after deleting the object, 2.22%:
> #########
>
> del_after_puts False del_after_gets True n_puts 20000000
>
> before run
>
> mem_pct 0.15%
>
> ------ put done ----- qsize 20000000
>
> mem_pct 37.61%
>
> ------ gets done ----- qsize 0
>
> mem_pct 2.22%
>
> deleting queue after gets <queue.Queue object at 0x7fbd87295a10>
>
> mem_pct 2.22%
For the 2nd case, mem_pct at the start is 0.15%, after putting all elements into the queue and just deleting it, 0.16%, which is almost the same.
> #########
>
> del_after_puts True del_after_gets False n_puts 20000000
>
> before run
>
> mem_pct 0.15%
>
> ------ put done ----- qsize 20000000
>
> mem_pct 37.61%
>
> deleting queue after puts <queue.Queue object at 0x7f29084eca10>
>
> mem_pct 0.16%
As it can be seen, memory returns to the start level only in the first case when only `queue.put()` is invoked, hence it seems that `queue.get()` produces a memory leak.
This is persistent across python 3.7, 3.8, as well as 3.9. |
|
Date |
User |
Action |
Args |
2021-04-22 09:13:22 | multiks2200 | set | recipients:
+ multiks2200 |
2021-04-22 09:13:22 | multiks2200 | set | messageid: <1619082802.82.0.467723490819.issue43911@roundup.psfhosted.org> |
2021-04-22 09:13:22 | multiks2200 | link | issue43911 messages |
2021-04-22 09:13:21 | multiks2200 | create | |
|