This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author multiks2200
Recipients multiks2200
Date 2021-04-22.09:13:21
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1619082802.82.0.467723490819.issue43911@roundup.psfhosted.org>
In-reply-to
Content
I use the following code to produce what looks like a memory leak after emptying a queue with the get() method.


    import queue
    import os
    import psutil
    
    def run(del_after_puts, del_after_gets, n_puts, process):
    
            mem = queue.Queue()
        
            for msg in range(n_puts):
                msg_put = f'{msg}_0000000000000000000000000000000000000000000000000000000000000333333333333331111111111'
                if msg % 1000000 == 0:
                    print(f'puting  {msg} qsize {len(mem.queue)}')
                mem.put(msg_put)
        
        
            print(f'------ put done  ----- qsize {len(mem.queue)}')
            print(f'mem_pct {round(process.memory_percent(), 2)}% ')
        
            if del_after_puts:
                print(f'deleting queue after puts {mem}')
                del mem
                print(f'mem_pct {round(process.memory_percent(), 2)}% ')
                return
        
            for _ in range(n_puts):
                msg_get = mem.get()
                msg = int(msg_get.split('_')[0])
                if msg % 1000000 == 0:
                    print(f'getting_q {msg} qsize {len(mem.queue)} ')
                    mem.task_done()
                
            print(f'------ gets done  ----- qsize {len(mem.queue)}')
            print(f'mem_pct {round(process.memory_percent(), 2)}% ')
        
            if del_after_gets:
                print(f'deleting queue after gets {mem}')
                del mem
                print(f'mem_pct {round(process.memory_percent(), 2)}% ')
                return
        
        if __name__ == '__main__':
            del_after_puts = False
            del_after_gets = False
            n_puts = 20_000_000
            print()
            print('#########')
            print(f'del_after_puts {del_after_puts} del_after_gets {del_after_gets} n_puts {n_puts}')
        
            process = psutil.Process(os.getpid())
            print('before run')
            print(f'mem_pct {round(process.memory_percent(), 2)}% ')
        
            run(del_after_puts, del_after_gets, n_puts, process)
        
            print(f'after run return')
            print(f'mem_pct {round(process.memory_percent(), 2)}% ')

This script can be run in 3 ways:
1. Add n_puts elements into the queue and then empty it.
2. Add n_puts elements into the queue and then delete the queue object
3. Add n_puts elements into the queue and then empty it and then delete the queue object.

For the 1st and 3rd case, the script seems to produce a memory leak as in the following:

1st case, before putting elements into the queue mem used is 0.15%, after emptying it  2.22%:

> #########
> 
> del_after_puts False del_after_gets False n_puts 20000000
> 
> before run
> 
> mem_pct 0.15% 
> 
> ------ put done  ----- qsize 20000000
> 
> mem_pct 37.61% 
> 
> ------ gets done  ----- qsize 0
> 
> mem_pct 2.22%

3rd case, before putting elements into the queue mem used is 0.15%, after emptying it  2.22%, after deleting the object, 2.22%:

> #########
> 
> del_after_puts False del_after_gets True n_puts 20000000
> 
> before run
> 
> mem_pct 0.15% 
> 
> ------ put done  ----- qsize 20000000
> 
> mem_pct 37.61% 
> 
> ------ gets done  ----- qsize 0
> 
> mem_pct 2.22%
> 
> deleting queue after gets <queue.Queue object at 0x7fbd87295a10>
> 
> mem_pct 2.22%

For the 2nd case, mem_pct at the start is 0.15%, after putting all elements into the queue and just deleting it, 0.16%, which is almost the same.

> #########
> 
> del_after_puts True del_after_gets False n_puts 20000000
> 
> before run
> 
> mem_pct 0.15% 
> 
> ------ put done  ----- qsize 20000000
> 
> mem_pct 37.61% 
> 
> deleting queue after puts <queue.Queue object at 0x7f29084eca10>
> 
> mem_pct 0.16%

As it can be seen, memory returns to the start level only in the first case when only `queue.put()` is invoked, hence it seems that `queue.get()` produces a memory leak. 

This is persistent across python 3.7, 3.8, as well as 3.9.
History
Date User Action Args
2021-04-22 09:13:22multiks2200setrecipients: + multiks2200
2021-04-22 09:13:22multiks2200setmessageid: <1619082802.82.0.467723490819.issue43911@roundup.psfhosted.org>
2021-04-22 09:13:22multiks2200linkissue43911 messages
2021-04-22 09:13:21multiks2200create