Author spenczar
Recipients asvetlov, spenczar, yselivanov
Date 2021-02-03.19:41:21
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1612381282.49.0.371079618361.issue43119@roundup.psfhosted.org>
In-reply-to
Content
I am writing some software that reads records from a very large file (~hundreds of GB), putting them in an `asyncio.Queue` as it goes, and a chain of consumers handle each record and do stuff over the network.

To my surprise, my program runs out of memory because the Queue producer coroutine never yields control. I think (but am not sure) that the asyncio.Queue.put method has no preemption point if the queue is not full; I was using an unbounded Queue, so it was _never_ full, so my coroutine was never unscheduled.

I have attached a file with a minimal reproducer. It creates an unbounded queue. A 'publish' task calls `queue.put` from an infinite sequence. A 'subscribe' task calls `queue.get`, and prints each item. Nothing gets printed if I run this, because the `queue.put` never blocks.

I would expect that `await queue.put` would occasionally cede execution to any other runnable coroutines, even if the queue is unbounded.
History
Date User Action Args
2021-02-03 19:41:22spenczarsetrecipients: + spenczar, asvetlov, yselivanov
2021-02-03 19:41:22spenczarsetmessageid: <1612381282.49.0.371079618361.issue43119@roundup.psfhosted.org>
2021-02-03 19:41:22spenczarlinkissue43119 messages
2021-02-03 19:41:22spenczarcreate