This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Add lossy queue to queue library module
Type: enhancement Stage:
Components: Library (Lib) Versions: Python 3.2
process
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: rhettinger Nosy List: bpb, rhettinger
Priority: normal Keywords:

Created on 2009-11-17 10:40 by bpb, last changed 2022-04-11 14:56 by admin. This issue is now closed.

Messages (4)
msg95374 - (view) Author: Ben Bass (bpb) Date: 2009-11-17 10:40
Many applications would benefit from 'connectionless' queues, i.e. they 
don't want to care whether anything is reading from the other end.  
Using current queue module classes this is not practical, because there 
is a choice between unbounded memory consumption or blocking. I propose 
adding a 'LossyQueue' class in the queue module which would allow 
bounded memory consumption without blocking on put.  (i.e. items are 
dropped in fifo manner beyond a certain limit).  In my view this is at 
least as natural as the PriorityQueue and LifoQueue extensions in that 
module.

Outline as follows:

class LossyQueue(Queue):
    "Queue subclass which drops items on overflow"
    def _init(self, maxsize):
        if maxsize > 0:
            # build the deque with maxsize limit
            self.queue = deque(maxlen=maxsize)
        else:
            # same as normal Queue instance
            self.queue = collections.deque()
        # deque alone handles maxsize,
        # so we pretend we have none
        self.maxsize = 0

if there is interest in this I will offer a proper patch with docs and 
tests.
msg95394 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2009-11-17 19:39
I'm curious about your "many applications would benefit from
'connectionless' queues".  What do you have in mind?  Is there any
reason those apps cannot use collections.deque() directly?
msg95415 - (view) Author: Ben Bass (bpb) Date: 2009-11-18 09:44
'connectionless' is from how I see it as an analogy with UDP (vs TCP); 
why not just use a deque is primarily about having the same API - a 
client (getter) of the queue shouldn't know or care whether it is a 
'lossy' queue or a normal queue.  I guess most uses of a normal queue 
(excepting the 'task' functions) could just use a deque, but it wouldn't 
feel natural.

Use cases: non-critical event/status reporting is my canonical example.
Specific examples:
 - a program which executes a long running process in a thread. It wants 
to update a GUI progress bar or similar, which must occur in a different 
thread because of the GUI model. By using a LossyQueue, the server 
thread is simplified; it doesn't have to care whether anything is 
listening on the other end, allowing greater decoupling (e.g. no changes 
required if there isn't a GUI). LossyQueues become part of the interface 
which can be used or not as required.
 - emulating/providing wrapper around UDP sockets
 - many application protocols support a get/set/report type interface 
with the addition of asynchronous events (e.g. SNMP, Netconf, SCPI).  In 
these type of applications a suitable abstraction might be a normal 
Queue(s) for the standard commands and a LossyQueue for the events 
(which some applications might not care about).  The point is that to 
the user of this abstraction, these two interfaces look the same.

The 'server' doesn't care if a client is listening or not (it won't 
block and it won't use unlimited memory)
The 'client' (if it wants to use it) doesn't know that it isn't a normal 
queue (same API).
-> decouples server and client tasks.
msg102217 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2010-04-03 03:17
I think it would be better to put this in the ASPN recipes cookbook to let it mature and gather a following.  Right now, it is not at all clear that this is the right thing to do.
History
Date User Action Args
2022-04-11 14:56:54adminsetgithub: 51586
2010-04-03 03:17:50rhettingersetstatus: open -> closed
resolution: rejected
messages: + msg102217

versions: - Python 2.7
2009-11-18 09:44:46bpbsetmessages: + msg95415
2009-11-17 19:39:31rhettingersetassignee: rhettinger

messages: + msg95394
nosy: + rhettinger
2009-11-17 10:40:39bpbcreate