Message259328
I think this is expected behavior when you do a lot of get() calls
that are cancelled by a timeout. Each call leaves a cancelled Future
in the deque self._getters; these are removed only when _wakeup_next()
is called (by put or put_nowait).
Why is your app doing this?
I suppose we could band-aid this by removing cancelled Futures when
next calling get(), but that won't help if you start a load of get()
calls and then cancel them all.
Note that I don't want to complicate this code -- in the past it has
been very difficult to prove it was correct under all circumstances,
and now I am sure it is correct (barring memory leaks, as you've
shown). I want to make sure any fix for this issue doesn't complicate
the code, making it harder to understand or prove its correctness.
On Mon, Feb 1, 2016 at 8:06 AM, Jonas Brunsgaard <report@bugs.python.org> wrote:
>
> New submission from Jonas Brunsgaard:
>
> When making repeated calls to queue.get, memory is building up and is not freed until queue.push is called.
>
> I wrote this little program to show my findings. The program will perform a lot of calls to queue.get and once every 60 seconds a queue.push is performed. Every 15 seconds the memory usage of dictionaries is printet to the console. You can find the output below the program
>
> ```
> import asyncio
> from pympler import muppy
> from pympler import summary
>
>
> q = asyncio.Queue()
> loop = asyncio.get_event_loop()
> closing = False
>
>
> async def get_with_timeout():
> while not closing:
> try:
> task = asyncio.ensure_future(q.get())
> await asyncio.wait_for(task, 0.2)
> except asyncio.TimeoutError:
> pass
>
> def mem_profiling():
> if not closing:
> types_ = muppy.filter(muppy.get_objects(), Type=dict)
> summary.print_(summary.summarize(types_))
> loop.call_later(15, mem_profiling)
>
> def put():
> q.put_nowait(None)
> loop.call_later(60, put)
>
> put()
> tasks = [asyncio.ensure_future(get_with_timeout()) for _ in range(10000)]
> mem_profiling()
>
> try:
> loop.run_forever()
> except KeyboardInterrupt:
> closing = True
> loop.run_until_complete(
> asyncio.ensure_future(asyncio.wait(tasks)))
> finally:
> loop.close()
> ```
>
> Output:
>
> types | # objects | total size
> ======================================== | =========== | ============
> <class 'dict | 11886 | 3.20 MB
> <class 'collections.OrderedDict | 11 | 16.84 KB
> <class 'email._encoded_words._QByteMap | 1 | 288 B
> types | # objects | total size
> ======================================== | =========== | ============
> <class 'dict | 224835 | 62.44 MB
> <class 'collections.OrderedDict | 11 | 16.84 KB
> <class 'email._encoded_words._QByteMap | 1 | 288 B
> types | # objects | total size
> ======================================== | =========== | ============
> <class 'dict | 401487 | 111.00 MB
> <class 'collections.OrderedDict | 11 | 16.84 KB
> <class 'email._encoded_words._QByteMap | 1 | 288 B
> types | # objects | total size
> ======================================== | =========== | ============
> <class 'dict | 576621 | 158.86 MB
> <class 'collections.OrderedDict | 11 | 16.84 KB
> <class 'email._encoded_words._QByteMap | 1 | 288 B
> types | # objects | total size
> ======================================== | =========== | ============
> <class 'dict | 219745 | 58.99 MB
> <class 'collections.OrderedDict | 11 | 16.84 KB
> <class 'email._encoded_words._QByteMap | 1 | 288 B
> types | # objects | total size
> ======================================== | =========== | ============
> <class 'dict | 390061 | 105.72 MB
> <class 'collections.OrderedDict | 11 | 16.84 KB
> <class 'email._encoded_words._QByteMap | 1 | 288 B
> types | # objects | total size
> ======================================== | =========== | ============
> <class 'dict | 560623 | 152.65 MB
> <class 'collections.OrderedDict | 11 | 16.84 KB
> <class 'email._encoded_words._QByteMap | 1 | 288 B
> types | # objects | total size
> ======================================== | =========== | ============
> <class 'dict | 220372 | 59.11 MB
> <class 'collections.OrderedDict | 11 | 16.84 KB
> <class 'email._encoded_words._QByteMap | 1 | 288 B
> types | # objects | total size
> ======================================== | =========== | ============
>
> What we see is that memory builds up ~3mb/s, and when the push method is called the memory usage returns to normal.
>
> Is this the expected behavior or is this a bug? If it is expected I think we should update the documentation, to let people know about this behavior.
>
> --
> Jonas Brunsgaard
>
> ----------
> components: asyncio
> files: poc.py
> messages: 259327
> nosy: Jonas Brunsgaard, gvanrossum, haypo, yselivanov
> priority: normal
> severity: normal
> status: open
> title: Memleak when repeated calls to asyncio.queue.Queue.get is performed, without push to queue.
> type: resource usage
> versions: Python 3.5
> Added file: http://bugs.python.org/file41772/poc.py
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue26259>
> _______________________________________ |
|
Date |
User |
Action |
Args |
2016-02-01 16:35:23 | gvanrossum | set | recipients:
+ gvanrossum, vstinner, yselivanov, Brunsgaard |
2016-02-01 16:35:23 | gvanrossum | link | issue26259 messages |
2016-02-01 16:35:23 | gvanrossum | create | |
|