Author zackelan
Recipients yselivanov, zackelan
Date 2017-09-28.17:27:48
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1506619669.24.0.466225441844.issue31620@psf.upfronthosting.co.za>
In-reply-to
Content
Repro: Call asyncio.wait_for(some_queue.get(), some_timeout) repeatedly, with no items in the queue, so that the call times out each time.

Expected: No increase in memory while polling an empty queue

Actual: The queue holds on to pending "getter" futures until a item passes through the queue, which clears the pending tasks out.

Use case:

I have producer and consumer asyncio.Tasks, linked by an asyncio.Queue. The producer is idle most of the time and pushes messages very infrequently. The consumer task polls the queue with wait_for and a timeout so that it's able to record "yep, I'm still idle" when the wait_for times out.

Attached script has a minimal repro. The producer emits an item every 60 seconds and the consumer polls every second. At the 59th second the _getters member of the queue is holding on to 59 pending futures. By varying the intervals this can leak an arbitrary amount of memory, for example producer emitting an item once a day vs. consumer polling once a second will leak 86,400 futures.

From the attached script:

2017-09-28 10:09:12,699 Queue <Queue maxsize=100 _getters[1]> is idle
...
2017-09-28 10:10:09,784 Queue <Queue maxsize=100 _getters[58]> is idle
2017-09-28 10:10:10,785 Queue <Queue maxsize=100 _getters[59]> is idle
2017-09-28 10:10:11,699 Received 0 from <Queue maxsize=100 tasks=1>
2017-09-28 10:10:12,700 Queue <Queue maxsize=100 _getters[1]> is idle
2017-09-28 10:10:13,702 Queue <Queue maxsize=100 _getters[2]> is idle

Noticed this in 3.6.1, though based on no changes to https://github.com/python/cpython/blob/master/Lib/asyncio/queues.py between the 3.6 branch and master I suspect it also affects 3.6.x and 3.7.
History
Date User Action Args
2017-09-28 17:27:49zackelansetrecipients: + zackelan, yselivanov
2017-09-28 17:27:49zackelansetmessageid: <1506619669.24.0.466225441844.issue31620@psf.upfronthosting.co.za>
2017-09-28 17:27:49zackelanlinkissue31620 messages
2017-09-28 17:27:48zackelancreate