Message386454
I am writing some software that reads records from a very large file (~hundreds of GB), putting them in an `asyncio.Queue` as it goes, and a chain of consumers handle each record and do stuff over the network.
To my surprise, my program runs out of memory because the Queue producer coroutine never yields control. I think (but am not sure) that the asyncio.Queue.put method has no preemption point if the queue is not full; I was using an unbounded Queue, so it was _never_ full, so my coroutine was never unscheduled.
I have attached a file with a minimal reproducer. It creates an unbounded queue. A 'publish' task calls `queue.put` from an infinite sequence. A 'subscribe' task calls `queue.get`, and prints each item. Nothing gets printed if I run this, because the `queue.put` never blocks.
I would expect that `await queue.put` would occasionally cede execution to any other runnable coroutines, even if the queue is unbounded. |
|
Date |
User |
Action |
Args |
2021-02-03 19:41:22 | spenczar | set | recipients:
+ spenczar, asvetlov, yselivanov |
2021-02-03 19:41:22 | spenczar | set | messageid: <1612381282.49.0.371079618361.issue43119@roundup.psfhosted.org> |
2021-02-03 19:41:22 | spenczar | link | issue43119 messages |
2021-02-03 19:41:22 | spenczar | create | |
|