Author Brunsgaard
Recipients Brunsgaard, gvanrossum, vstinner, yselivanov
Date 2016-02-01.16:06:19
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1454342780.58.0.210365622359.issue26259@psf.upfronthosting.co.za>
In-reply-to
Content
When making repeated calls to queue.get, memory is building up and is not freed until queue.push is called.

I wrote this little program to show my findings. The program will perform a lot of calls to queue.get and once every 60 seconds a queue.push is performed. Every 15 seconds the memory usage of dictionaries is printet to the console. You can find the output below the program

```
import asyncio
from pympler import muppy
from pympler import summary


q = asyncio.Queue()
loop = asyncio.get_event_loop()
closing = False


async def get_with_timeout():
    while not closing:
        try:
            task = asyncio.ensure_future(q.get())
            await asyncio.wait_for(task, 0.2)
        except asyncio.TimeoutError:
            pass

def mem_profiling():
    if not closing:
        types_ = muppy.filter(muppy.get_objects(), Type=dict)
        summary.print_(summary.summarize(types_))
        loop.call_later(15, mem_profiling)

def put():
    q.put_nowait(None)
    loop.call_later(60, put)

put()
tasks = [asyncio.ensure_future(get_with_timeout()) for _ in range(10000)]
mem_profiling()

try:
    loop.run_forever()
except KeyboardInterrupt:
    closing = True
    loop.run_until_complete(
        asyncio.ensure_future(asyncio.wait(tasks)))
finally:
    loop.close()
```

Output:

                                   types |   # objects |   total size
======================================== | =========== | ============
                            <class 'dict |       11886 |      3.20 MB
         <class 'collections.OrderedDict |          11 |     16.84 KB
  <class 'email._encoded_words._QByteMap |           1 |    288     B
                                   types |   # objects |   total size
======================================== | =========== | ============
                            <class 'dict |      224835 |     62.44 MB
         <class 'collections.OrderedDict |          11 |     16.84 KB
  <class 'email._encoded_words._QByteMap |           1 |    288     B
                                   types |   # objects |   total size
======================================== | =========== | ============
                            <class 'dict |      401487 |    111.00 MB
         <class 'collections.OrderedDict |          11 |     16.84 KB
  <class 'email._encoded_words._QByteMap |           1 |    288     B
                                   types |   # objects |   total size
======================================== | =========== | ============
                            <class 'dict |      576621 |    158.86 MB
         <class 'collections.OrderedDict |          11 |     16.84 KB
  <class 'email._encoded_words._QByteMap |           1 |    288     B
                                   types |   # objects |   total size
======================================== | =========== | ============
                            <class 'dict |      219745 |     58.99 MB
         <class 'collections.OrderedDict |          11 |     16.84 KB
  <class 'email._encoded_words._QByteMap |           1 |    288     B
                                   types |   # objects |   total size
======================================== | =========== | ============
                            <class 'dict |      390061 |    105.72 MB
         <class 'collections.OrderedDict |          11 |     16.84 KB
  <class 'email._encoded_words._QByteMap |           1 |    288     B
                                   types |   # objects |   total size
======================================== | =========== | ============
                            <class 'dict |      560623 |    152.65 MB
         <class 'collections.OrderedDict |          11 |     16.84 KB
  <class 'email._encoded_words._QByteMap |           1 |    288     B
                                   types |   # objects |   total size
======================================== | =========== | ============
                            <class 'dict |      220372 |     59.11 MB
         <class 'collections.OrderedDict |          11 |     16.84 KB
  <class 'email._encoded_words._QByteMap |           1 |    288     B
                                   types |   # objects |   total size
======================================== | =========== | ============

What we see is that memory builds up ~3mb/s, and when the push method is called the memory usage returns to normal.

Is this the expected behavior or is this a bug? If it is expected I think we should update the documentation, to let people know about this behavior.

--
Jonas Brunsgaard
History
Date User Action Args
2016-02-01 16:06:20Brunsgaardsetrecipients: + Brunsgaard, gvanrossum, vstinner, yselivanov
2016-02-01 16:06:20Brunsgaardsetmessageid: <1454342780.58.0.210365622359.issue26259@psf.upfronthosting.co.za>
2016-02-01 16:06:20Brunsgaardlinkissue26259 messages
2016-02-01 16:06:19Brunsgaardcreate