Title: Suggestion to imporve queue.full reliability
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.6
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: Sudharsan R, rhettinger
Priority: normal Keywords:

Created on 2016-05-04 19:37 by Sudharsan R, last changed 2016-05-04 23:48 by rhettinger. This issue is now closed.

Messages (3)
msg264851 - (view) Author: Sudharsan R (Sudharsan R) Date: 2016-05-04 19:37
I was browsing through the code for queue.full and queue.put methods. The queue.full method claims it is not a reliable one. But, the same internal implementation - _qsize is used for put as well as full. "put" is thread safe and reliable and "full" is not.
Why can't the full method acquire self.not_full instead of self.mutex, which will make both of the methods accurate at that instance of time?

    def full(self):
        """Return True if the queue is full, False otherwise (not reliable!)."""
        n = 0 < self.maxsize == self._qsize()
        return n
    def put(self, item, block=True, timeout=None):
        """Put an item into the queue.

        If optional args 'block' is true and 'timeout' is None (the default),
        block if necessary until a free slot is available. If 'timeout' is
        a positive number, it blocks at most 'timeout' seconds and raises
        the Full exception if no free slot was available within that time.
        Otherwise ('block' is false), put an item on the queue if a free slot
        is immediately available, else raise the Full exception ('timeout'
        is ignored in that case).
            if self.maxsize > 0:
                if not block:
                    if self._qsize() == self.maxsize:
                        raise Full
msg264853 - (view) Author: Sudharsan R (Sudharsan R) Date: 2016-05-04 19:50
Just add on to it..
with q.not_full:

this will hang. So if we acquire the not_full while computing size, all puts will wait. Same is the case for q.empty() and q.not_empty method and condition respectively.
msg264865 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2016-05-04 23:48
The reliability issue for qsize(), empty(), and full() has nothing to do with their implementations.  The problem is that any information obtained by those methods is potentially out-of-date by the time you try to use it (the LBYL problem).  This is why multi-threaded programmers prefer (try: os.remove(fn) except OSError: pass) over (if os.path.exists(fn): os.remove(fn)) where the latter has an intrinsic race condition regardless of the implementation of os.path.exists().

One other note, self.not_full uses self.mutex internally.  There is only one underlying lock for the whole queue implementation.

Lastly, despite not having an underscore prefix in its name, q.not_full() is not part of the public API.  Your "it hangs" example is part of the reason why ;-)

In general, don't use full and empty tests to decide whether or not to do a put or get.  Instead, just try the put or get directly and catch the exception if queue turns out to be full or empty.
Date User Action Args
2016-05-04 23:48:23rhettingersetstatus: open -> closed
resolution: not a bug
messages: + msg264865
2016-05-04 19:50:25Sudharsan Rsetmessages: + msg264853
2016-05-04 19:47:26SilentGhostsetnosy: + rhettinger
components: + Library (Lib)
2016-05-04 19:37:12Sudharsan Rcreate