This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Queue doesn't recognize it is full after shrinking maxsize
Type: Stage: patch review
Components: Versions: Python 3.2
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: rhettinger Nosy List: eric.araujo, jaraco, rhettinger
Priority: low Keywords: patch

Created on 2010-10-15 01:25 by jaraco, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
shrinking_queue_not_full.patch jaraco, 2010-10-15 01:25
Messages (7)
msg118736 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2010-10-15 01:25
The Queue object has a maxsize parameter and attribute, but due to the test for a full queue, shrinking the maxsize could result in the Queue not recognizing that it is full.

The attached patch (against the Python 3 trunk) demonstrates this limitation with a unit test and fixes the failing test case.
msg118828 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2010-10-15 20:29
I should point out that in Python 2.5, it was possible for a subclass to override the _full method to account for this situation, but with Python 2.6 and later, the calculation in _full was hand-inlined... so it's not readily possible for a subclass to correct for this behavior. It might be desirable to refactor that calculation into a _full method, though I suspect this would have performance implications, which is why I stuck with just adjusting the comparison in its various uses.
msg118834 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2010-10-15 21:50
I've briefly looked at the patch and it seems reasonable.  Will look it in more detail by 3.2 goes out. 

I'm curious about your use case for wanting to change the maxsize of an existing Queue that is already in use.  That seems like a odd design.
msg118835 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2010-10-15 22:00
I'm not sure what our use case is. I discovered this when I was looking at our project's util library, and we have a Queue subclass that overrides _full to handle the scenario where the queue shrinks. I'm guessing it's being used to dynamically adjust the queue size for performance reasons. Consider where you have a queue of items, but depending on the run time environment, that queue might need to be more or less limiting. Maybe the queue is larger during the day when requests are heavier and shrinks at night (so it's not accepting more items than it can handle when day breaks).

Surely that's a contrived example, and I suspect our actual usage is more concrete. I'll see if I can learn where we're using that functionality.

If shrinking maxsize is not allowed, that should be enforced in the interface (such as with a read-only/increase-only property) and documented. If you would rather proceed that way, I'll draft the patch for that instead.
msg118839 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2010-10-15 22:30
That won't be necessary.  The change from == to <= is innocuous.

There's no need to lock-up maxsize in a read-only property.  We're consenting adults.  Besides, it would probably break someone-else's odd use case.  I don't want to expand the API, nor do I want to cripple anyone's ability to do weird stuff with it.

FWIW, the full() and empty() methods are usually not a good idea.  It's better to catch a Full exception.  Otherwise, the information can be out of date by the time you try to use it.

I'm going to mark this as a 3.2 only change.  There were no guarantees about the behavior when maxsize is changed, nor should we make such guarantees.
msg118840 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2010-10-15 22:43
Thanks!
msg120073 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2010-10-31 17:58
Applied in r86049.
History
Date User Action Args
2022-04-11 14:57:07adminsetgithub: 54319
2010-10-31 17:58:21rhettingersetstatus: open -> closed
resolution: fixed
messages: + msg120073
2010-10-15 22:43:40jaracosetmessages: + msg118840
2010-10-15 22:30:50rhettingersetmessages: + msg118839
versions: - Python 3.1, Python 2.7
2010-10-15 22:00:28jaracosetmessages: + msg118835
2010-10-15 21:50:52rhettingersetmessages: + msg118834
2010-10-15 20:29:50jaracosetmessages: + msg118828
2010-10-15 15:49:57rhettingersetpriority: normal -> low
assignee: rhettinger
2010-10-15 14:20:46eric.araujosetnosy: + eric.araujo
2010-10-15 07:43:53ned.deilysetnosy: + rhettinger
stage: patch review

versions: - Python 2.6, Python 2.5, Python 3.3
2010-10-15 01:25:04jaracocreate