This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author torsten
Recipients torsten
Date 2014-01-06.15:30:08
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1389022210.55.0.898891193549.issue20147@psf.upfronthosting.co.za>
In-reply-to
Content
The behaviour of multiprocessing.Queue surprised me today in that Queue.get() may raise an exception even if an item is immediately available. I tried to flush entries without blocking by using the timeout=0 keyword argument:
$ /opt/python3/bin/python3
Python 3.4.0b1 (default:247f12fecf2b, Jan  6 2014, 14:50:23) 
[GCC 4.6.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from multiprocessing import Queue
>>> q = Queue()
>>> q.put("hi")
>>> q.get(timeout=0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/python3/lib/python3.4/multiprocessing/queues.py", line 107, in get
    raise Empty
queue.Empty

Actually even passing a small non-zero timeout will not give me my queue entry:
>>> q.get(timeout=1e-6)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/torsten/opensrc/cpython/Lib/multiprocessing/queues.py", line 107, in get
    raise Empty
queue.Empty

Expected behaviour for me would be to return the item that is in the queue. I know that there is a kwarg *block* which gives me the desired behaviour:
>>> q.get(block=False)
'hi'
In my case the get call is embedded in my own module which does not currently expose the block parameter. My local solution is of course to update the wrapper:

if timeout == 0:
    timeout = None
    block = False

However I see a few smells here in the python standard library. First, everything else seems to accept timeout=0 as nonblocking:

>>> import threading
>>> lock = threading.Lock()
>>> lock.acquire(timeout=0)
True
>>> from queue import Queue
>>> q = Queue()
>>> q.put("hi")
>>> q.get(timeout=0)
'hi'
Of special note is that queue.Queue behaves as I would have expected. IMHO it should be consistent with multiprocessing.Queue.

Also note that queue.Queue.get() and queue.Queue.put() name their blocking flag "block", while everybody else uses "blocking".

As a side note, I think the current approach is flawed in computing the deadline. Basically it does the following:

    deadline = time.time() + timeout
    if not self._rlock.acquire(block, timeout):
        raise Empty
    timeout = deadline - time.time()
    if timeout < 0 or not self._poll(timeout):
        raise Empty

On my system, just taking the time twice and computing the delta takes 2 microseconds:

>>> import time
>>> t0 = time.time(); time.time() - t0
2.384185791015625e-06

Therefore calling Queue.get(block, timeout) with 0 < timeout < 2e-6 will never return anything from the queue even though Queue.get(block=False) would do that. This contradicts the idea that Queue.get(block=False) will return faster than with block=True with any timeout > 0.

Apart from that, as Python does not currently support waiting on multiple sources, we currently often check a queue with a small timeout concurrently with doing other stuff. In case the system get really loaded, I would expect this to cause problems because the updated timeout may fall below zero.

Suggested patch attached.
History
Date User Action Args
2014-01-06 15:30:10torstensetrecipients: + torsten
2014-01-06 15:30:10torstensetmessageid: <1389022210.55.0.898891193549.issue20147@psf.upfronthosting.co.za>
2014-01-06 15:30:10torstenlinkissue20147 messages
2014-01-06 15:30:09torstencreate