classification
Title: multiprocessing Queue.qsize() and Queue.empty() with different results
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 2.7
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: aod, sbt
Priority: normal Keywords:

Created on 2013-05-15 20:41 by aod, last changed 2013-06-24 18:25 by sbt. This issue is now closed.

Messages (7)
msg189302 - (view) Author: Andre Dias (aod) Date: 2013-05-15 20:41
The problem is that Queue.empty() is True even if Queue.qsize()>0!


#!/usr/bin/python

from multiprocessing import Queue

numbers=Queue()
for i in range (0,10):
    numbers.put(i)

if numbers.qsize()>0 and numbers.empty():
    print "BUG?!"
msg189305 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013-05-15 21:19
From the docs:

 qsize()
    Return the approximate size of the queue. Because of
    multithreading/multiprocessing semantics, this number
    is not reliable.

Adding a short sleep before calling qsize() and empty() should make things appear to work.

But really, there are no good reasons for using qsize() except for debugging.  The same applies to queue.Queue.
msg189309 - (view) Author: Andre Dias (aod) Date: 2013-05-15 21:25
But qsize() is working. what is not working is empty()

2013/5/15 Richard Oudkerk <report@bugs.python.org>

>
> Richard Oudkerk added the comment:
>
> >From the docs:
>
>  qsize()
>     Return the approximate size of the queue. Because of
>     multithreading/multiprocessing semantics, this number
>     is not reliable.
>
> Adding a short sleep before calling qsize() and empty() should make things
> appear to work.
>
> But really, there are no good reasons for using qsize() except for
> debugging.  The same applies to queue.Queue.
>
> ----------
> nosy: +sbt
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue17985>
> _______________________________________
>
msg189315 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013-05-15 22:24
On 15/05/2013 10:25pm, Andre Dias wrote:
> But qsize() is working. what is not working is empty()

empty() returns False when there is data in the underlying pipe.  But 
the data does not enter the pipe until a background thread has written 
it to the pipe.  This should not cause any problems.

Using Queue.empty() is always subject to races.  It is just that the 
multiprocessing version has an additional type of race compared to the 
normal one.
msg189318 - (view) Author: Andre Dias (aod) Date: 2013-05-15 22:33
RIchard,

But the example program has no races, no threads, nothing.
empty() is returning TRUE even though qsize() is >0 (which actually is)
And it happens almost every time I run that small example.
I had read the module doc, and I know its an unreliable method, but man,
the example program is too simple to fail
Truth is I decided qsize() is more reliable and im using it in my programs,
but man empty() problem is so ridiculous that I decided to submit here

2013/5/15 Richard Oudkerk <report@bugs.python.org>

>
> Richard Oudkerk added the comment:
>
> On 15/05/2013 10:25pm, Andre Dias wrote:
> > But qsize() is working. what is not working is empty()
>
> empty() returns False when there is data in the underlying pipe.  But
> the data does not enter the pipe until a background thread has written
> it to the pipe.  This should not cause any problems.
>
> Using Queue.empty() is always subject to races.  It is just that the
> multiprocessing version has an additional type of race compared to the
> normal one.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue17985>
> _______________________________________
>
msg189322 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013-05-15 22:46
On 15/05/2013 11:33pm, Andre Dias wrote:
> But the example program has no races, no threads, nothing.
> empty() is returning TRUE even though qsize() is >0 (which actually is)
> And it happens almost every time I run that small example.
> I had read the module doc, and I know its an unreliable method, but man,
> the example program is too simple to fail
> Truth is I decided qsize() is more reliable and im using it in my programs,
> but man empty() problem is so ridiculous that I decided to submit here

If you assume everything is single-threaded then you would assume that 
this is too simple to fail.  But there *is* more than thread involved.

empty() and qsize() are basically redundant.  You would almost certainly 
be better off using get_noblock()/put_noblock() and catching Empty/Full.
msg191794 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013-06-24 18:25
This is really a documentation issue.  The doc fix for #18277 covers this.
History
Date User Action Args
2013-06-24 18:25:02sbtsetstatus: open -> closed
type: behavior
messages: + msg191794

components: + Library (Lib), - Extension Modules
resolution: wont fix
stage: resolved
2013-05-15 22:46:52sbtsetmessages: + msg189322
2013-05-15 22:33:12aodsetmessages: + msg189318
2013-05-15 22:24:06sbtsetmessages: + msg189315
2013-05-15 21:25:36aodsetmessages: + msg189309
2013-05-15 21:19:00sbtsetnosy: + sbt
messages: + msg189305
2013-05-15 20:41:12aodcreate