classification
Title: multiprocessing.Queue documentation is lacking important details
Type: Stage: resolved
Components: Documentation Versions: Python 2.7, Python 2.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: docs@python Nosy List: Garrett.Moore, docs@python, python-dev, sbt
Priority: normal Keywords:

Created on 2012-03-06 00:32 by Garrett.Moore, last changed 2013-07-02 12:04 by sbt. This issue is now closed.

Messages (3)
msg154995 - (view) Author: Garrett Moore (Garrett.Moore) Date: 2012-03-06 00:32
1) If cancel_join_thread() is called, data may be lost. This is not explicitly stated. I had multiple writers put() data in a Queue, and wanted to have the workers finish before I began consuming the data. This caused a deadlock because my Queue was not empty, and it seemed like the a way to force my workers finish was to use cancel_join_thread(). This caused data loss.

2) multiprocessing.Queue states "The Queue class is a near clone of Queue.Queue."

Queue.Queue states "If maxsize is less than or equal to zero, the queue size is infinite."

mp.Queue provides no information on queue size. It is reasonable to assume then that it inherits the property of Queue.Queue.

After discussion on IRC, it seems that mp.Queue maximum size is implementation-dependent and likely relies on how much data Pipes can hold on your platform. If this is the case there should be some mention of the fact that mp.Queue does NOT function like Queue.Queue does for maximum size.
msg155011 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-03-06 12:28
What you were told on IRC was wrong.  By default the queue *does* have infinite size.

When a process puts an item on the queue for the first time, a background thread is started which is responsible for writing items to the underlying pipe.  This does mean that, on exit, the process should wait for the background thread to flush all the data to the pipe.  This happens automatically unless you specifically prevent it by calling cancel_join_thread() method.

If you stick to those methods supported by standard queue objects, then things should work correctly.  

(Maybe cancel_join_thread() would be better named allow_exit_without_flush().)
msg192188 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013-07-02 12:02
New changeset f35401dba89f by Richard Oudkerk in branch '2.7':
Issue #14206: Clarify docs for Queue.join_cancel_thread().
http://hg.python.org/cpython/rev/f35401dba89f

New changeset 9746f217a270 by Richard Oudkerk in branch '3.3':
Issue #14206: Clarify docs for Queue.join_cancel_thread().
http://hg.python.org/cpython/rev/9746f217a270

New changeset d0b48efac9de by Richard Oudkerk in branch 'default':
Issue #14206: Clarify docs for Queue.join_cancel_thread().
http://hg.python.org/cpython/rev/d0b48efac9de
History
Date User Action Args
2013-07-02 12:04:18sbtsetstatus: open -> closed
2013-07-02 12:04:00sbtsetresolution: fixed
stage: resolved
2013-07-02 12:02:31python-devsetnosy: + python-dev
messages: + msg192188
2012-03-06 12:28:37sbtsetnosy: + sbt
messages: + msg155011
2012-03-06 00:32:41Garrett.Moorecreate