classification
Title: multiprocessing.Queue does not order objects
Type: behavior Stage:
Components: Extension Modules Versions: Python 2.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: jnoller Nosy List: Stefan.P, asksol, georg.brandl, ggenellina, jnoller, ndfred, pitrou, rhettinger, sbt
Priority: normal Keywords: patch

Created on 2009-01-19 12:57 by ndfred, last changed 2014-05-14 11:45 by exarkun.

Files
File name Uploaded Description Edit
testthreads.py ndfred, 2009-01-19 12:57 Test script
nick.py jnoller, 2009-01-23 01:59
warn_fifo.patch skrah, 2010-02-21 12:58 Document that FIFO behavior is not guaranteed. review
issue-4999.diff asksol, 2010-10-24 07:47
Messages (13)
msg80162 - (view) Author: Frédéric Sagnes (ndfred) Date: 2009-01-19 12:57
Objects contained in a multiprocessing.Queue object are not comming out
of the queue in the same order as they went in. For instance, if I put
in object1, object2 and object3 in this very time sequence from multiple
processes, they can end up comming out of the queue as object2, object1
then object3 instead of the original order.

When using the threading module instead of multiprocessing everything is
fine.

The provided test script adds strings to the queue with timestamps.
These messages are not ordered by timestamp when they are printed. This
is an output of the test script with format "[pid@time] message":

[2120@00406] Got lock
[2120@02424] Released lock
[2121@02426] Got lock
[2121@04439] Released lock
[...]
[2121@16459] Released lock
[2120@16461] Got lock
[2121@18464] Got lock
[2120@18462] Released lock
[2121@20466] Released lock

Using print to print the message immediatly prints the messages in the
right order.

See this mailing-list thread for details:
http://groups.google.com/group/comp.lang.python/browse_thread/thread/11a5c4ce4ff4382d/033dcd3607eacbf9
msg80164 - (view) Author: Frédéric Sagnes (ndfred) Date: 2009-01-19 13:01
Tested using Python 2.6.1 on Mac OS 10.5 and Linux 2.6.26
msg80386 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-01-22 22:16
Rather than providing strict ordering of Queue input/output, perhaps a
separate or derived class (PriorityQueue?) should be provided just for
that matter (heapq-based?).
msg80387 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2009-01-22 22:46
The Queue module design should be followed as closely as possible.  It
provides FIFO order as the default, with options for LIFO and priority
order.
msg80389 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2009-01-22 23:10
I agree, at very least it should provide the same default behavior that 
Queue does, however, it is not obvious to me how to resolve this in 
mp.Queue
msg80390 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-01-22 23:20
That's why I was suggesting to leave Queue as it is and provide a
separate PriorityQueue. With a PriorityQueue the user is free to put
(timestamp, object) tuples in it if he wants a strict chronologic
ordering. He can also use sequence numbers instead of timestamps, etc.
msg80391 - (view) Author: Jean-Paul Calderone (exarkun) * (Python committer) Date: 2009-01-22 23:23
Jesse, can you explain the cause of the bug?  Maybe that will inspire
someone to come up with an idea for a fix.
msg80394 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2009-01-23 01:59
As a note to myself: adding a simple print to the multiprocessing.Queue 
put method, I can see the calls occurring from the children in order, 
for example:

obj:  0.025193 [proc1] Got lock
obj:  0.227725 [proc1] Released lock
obj:  0.228401 [proc2] Got lock
obj:  0.430501 [proc2] Released lock
obj:  0.431082 [proc0] Got lock
obj:  0.633048 [proc0] Released lock
obj:  0.633723 [proc4] Got lock
obj:  0.835692 [proc4] Released lock
obj:  0.836262 [proc3] Got lock
obj:  1.038197 [proc3] Released lock
obj:  1.038753 [proc1] Got lock
obj:  1.239288 [proc1] Released lock
obj:  1.240079 [proc2] Got lock
obj:  1.440773 [proc2] Released lock
obj:  1.441577 [proc0] Got lock
obj:  1.642142 [proc0] Released lock
obj:  1.642959 [proc4] Got lock
obj:  1.843331 [proc4] Released lock
obj:  1.844178 [proc3] Got lock
obj:  2.044549 [proc3] Released lock

Also attaching the process-based script
msg80396 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2009-01-23 02:48
On a put to the queue, the object is appended onto a deque (self._buffer) 
- this buffer is managed by a thread (_start_thread) which is handed the 
_feed method as the target. I suspect the bug is actually in the _feed 
method.
msg99652 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2010-02-21 12:58
I think it would be nice to update the documentation if this isn't
resolved yet. The patch adds a warning that FIFO behavior is not
guaranteed.
msg101579 - (view) Author: Stefan Praszalowicz (Stefan.P) Date: 2010-03-23 12:52
I just got surprised by this, and I agree that updating the doc would be nice, because as of now, it states quite explicitly that "the Queue and JoinableQueue types are multi-producer, multi-consumer FIFO queues".
msg119494 - (view) Author: Ask Solem (asksol) (Python committer) Date: 2010-10-24 07:47
Updated doc patch
msg119497 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2010-10-24 08:22
Are you sure this needs to be a warning?  We try to use them very sparingly.

(Also note the 3-space indent for reStructuredText markup.)
History
Date User Action Args
2014-05-14 11:45:03exarkunsetnosy: - exarkun
2014-05-13 22:23:48ned.deilysetnosy: + sbt
2014-05-13 21:46:36skrahsetnosy: - skrah
2010-10-24 08:22:57georg.brandlsetnosy: + georg.brandl
messages: + msg119497
2010-10-24 07:47:46asksolsetfiles: + issue-4999.diff
nosy: + asksol
messages: + msg119494

2010-03-23 12:52:00Stefan.Psettype: behavior

messages: + msg101579
nosy: + Stefan.P
2010-02-21 12:58:09skrahsetfiles: + warn_fifo.patch

nosy: + skrah
messages: + msg99652

keywords: + patch
2009-01-24 02:10:50ggenellinasetnosy: + ggenellina
2009-01-23 02:48:01jnollersetmessages: + msg80396
2009-01-23 01:59:16jnollersetfiles: + nick.py
messages: + msg80394
2009-01-22 23:23:16exarkunsetnosy: + exarkun
messages: + msg80391
2009-01-22 23:20:40pitrousetmessages: + msg80390
2009-01-22 23:10:43jnollersetmessages: + msg80389
2009-01-22 22:46:07rhettingersetnosy: + rhettinger
messages: + msg80387
2009-01-22 22:16:10pitrousetnosy: + pitrou
messages: + msg80386
2009-01-22 22:08:42jnollersetpriority: normal
2009-01-19 13:58:17jnollersetassignee: jnoller
nosy: + jnoller
2009-01-19 13:01:45ndfredsetmessages: + msg80164
2009-01-19 12:57:24ndfredcreate