classification
Title: asyncio.Queue leaks memory if the queue is empty and consumers poll it frequently
Type: resource usage Stage: resolved
Components: asyncio Versions: Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: asvetlov, cjrh, snihalani, yselivanov, zackelan
Priority: normal Keywords: patch

Created on 2017-09-28 17:27 by zackelan, last changed 2017-12-13 19:16 by asvetlov. This issue is now closed.

Files
File name Uploaded Description Edit
test.py zackelan, 2017-09-28 17:27
Pull Requests
URL Status Linked Edit
PR 3813 merged snihalani, 2017-09-29 05:30
PR 4326 merged python-dev, 2017-11-07 17:36
Messages (6)
msg303264 - (view) Author: Zack Elan (zackelan) Date: 2017-09-28 17:27
Repro: Call asyncio.wait_for(some_queue.get(), some_timeout) repeatedly, with no items in the queue, so that the call times out each time.

Expected: No increase in memory while polling an empty queue

Actual: The queue holds on to pending "getter" futures until a item passes through the queue, which clears the pending tasks out.

Use case:

I have producer and consumer asyncio.Tasks, linked by an asyncio.Queue. The producer is idle most of the time and pushes messages very infrequently. The consumer task polls the queue with wait_for and a timeout so that it's able to record "yep, I'm still idle" when the wait_for times out.

Attached script has a minimal repro. The producer emits an item every 60 seconds and the consumer polls every second. At the 59th second the _getters member of the queue is holding on to 59 pending futures. By varying the intervals this can leak an arbitrary amount of memory, for example producer emitting an item once a day vs. consumer polling once a second will leak 86,400 futures.

From the attached script:

2017-09-28 10:09:12,699 Queue <Queue maxsize=100 _getters[1]> is idle
...
2017-09-28 10:10:09,784 Queue <Queue maxsize=100 _getters[58]> is idle
2017-09-28 10:10:10,785 Queue <Queue maxsize=100 _getters[59]> is idle
2017-09-28 10:10:11,699 Received 0 from <Queue maxsize=100 tasks=1>
2017-09-28 10:10:12,700 Queue <Queue maxsize=100 _getters[1]> is idle
2017-09-28 10:10:13,702 Queue <Queue maxsize=100 _getters[2]> is idle

Noticed this in 3.6.1, though based on no changes to https://github.com/python/cpython/blob/master/Lib/asyncio/queues.py between the 3.6 branch and master I suspect it also affects 3.6.x and 3.7.
msg303290 - (view) Author: Suren Nihalani (snihalani) * Date: 2017-09-29 00:37
I am working on fixing this.
msg303371 - (view) Author: Caleb Hattingh (cjrh) * Date: 2017-09-29 23:24
This looks like a dupe, or at least quite closely related to https://bugs.python.org/issue26259. If the PR resolves both issues that one should be closed too.
msg305787 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2017-11-07 17:35
New changeset c62f0cb3b1f6f9ca4ce463b1c99b0543bdfa38d6 by Andrew Svetlov (Suren Nihalani) in branch 'master':
bpo-31620: have asyncio/queues not leak memory when you've exceptions during waiting (#3813)
https://github.com/python/cpython/commit/c62f0cb3b1f6f9ca4ce463b1c99b0543bdfa38d6
msg305789 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2017-11-07 19:08
New changeset ac4f6d4448fb6f9affb817bafb8357450fe43349 by Andrew Svetlov (Miss Islington (bot)) in branch '3.6':
bpo-31620: have asyncio/queues not leak memory when you've exceptions during waiting (GH-3813) (#4326)
https://github.com/python/cpython/commit/ac4f6d4448fb6f9affb817bafb8357450fe43349
msg308234 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2017-12-13 19:16
Sorry for late closing the issue, PR was merged a month ago.
History
Date User Action Args
2017-12-13 19:18:26asvetlovlinkissue26259 superseder
2017-12-13 19:16:34asvetlovsetstatus: open -> closed
versions: + Python 3.7
messages: + msg308234

resolution: fixed
stage: patch review -> resolved
2017-11-07 19:08:17asvetlovsetmessages: + msg305789
2017-11-07 17:36:31python-devsetpull_requests: + pull_request4283
2017-11-07 17:35:25asvetlovsetnosy: + asvetlov
messages: + msg305787
2017-09-29 23:24:37cjrhsetnosy: + cjrh
messages: + msg303371
2017-09-29 05:30:38snihalanisetkeywords: + patch
stage: patch review
pull_requests: + pull_request3797
2017-09-29 00:37:56snihalanisetnosy: + snihalani
messages: + msg303290
2017-09-28 17:27:49zackelancreate