Title: make queue.Queue objects iterable
Type: enhancement Stage: resolved
Components: Library (Lib) Versions:
Status: closed Resolution: later
Dependencies: Superseder:
Assigned To: Nosy List: alaniwi, rhettinger
Priority: normal Keywords:

Created on 2020-07-19 10:03 by alaniwi, last changed 2020-07-19 19:11 by rhettinger. This issue is now closed.

Messages (2)
msg373950 - (view) Author: Alan Iwi (alaniwi) Date: 2020-07-19 10:03
It is possible to make `queue.Queue` and `queue.SimpleQueue` objects iterable by adding simple `__iter__` and `__next__` methods.  This is to suggest adding these methods to the existing `Queue` and `SimpleQueue` so that they are iterable by default.

class Fifo(SimpleQueue):

    def __iter__(self):
        return self

    def __next__(self):
        if not self.empty():
            return self.get()
        raise StopIteration
msg373971 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2020-07-19 19:11
I prefer the current API because it makes the blocking step explicit.  That isn't the kind of thing that should be hidden behind a "for" or buried in another tool that accepts an iterable argument, sorted(q), for example. 

Also, the get() method supports two arguments "block" and "timeout" which would be inaccessible by a iterator.

The typical use case for queue objects is to process the gets one-at-a-time, blocking until a job arrives.  Queues typically start out empty.  And if they become empty later, it doesn't usually mean that there will be no more job requests.  This would make it awkward to use the proposed feature in real code without introducing an unnecessary extra level of nesting.

    -- Code with current API --
    while True:
        job = q.get()

    -- Code with proposed API --
    while True:          # Needed to restart when the queue is empty.
        for job in q:    # Consumes all currently available jobs.
            work(job)    # Now nested two layers deep.
        sleep(0.1)       # Prevent spin-wait bug.

For most users, the proposed feature would likely be a distraction that leads them away from the cleaner, lower overhead solution with an explicit get().  Also, the spin-wait bug is hard to see during code review or debugging.

FWIW, there is a race-condition in your definition of __next__().  The LBYL logic should probably be replaced with EAFP logic putting a get_nowait() inside a try/except.

I suggest taking this to the python-ideas mail list to discuss the pros and cons.  There are cases where a for-loop would look nicer than what we have now, so people might find the proposal acceptable despite all the problems listed above.

I'll mark this as closed.  If the idea gains traction on the mail list, feel free to re-open.
Date User Action Args
2020-07-19 19:11:47rhettingersetstatus: open -> closed
resolution: later
messages: + msg373971

stage: resolved
2020-07-19 12:05:01xtreaksetnosy: + rhettinger
2020-07-19 10:03:55alaniwicreate