This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: multiprocessing.Queue does not order objects
Type: behavior Stage: resolved
Components: Extension Modules Versions: Python 2.6
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: jnoller Nosy List: Stefan.P, asksol, georg.brandl, ggenellina, iritkatriel, jnoller, ndfred, pitrou, rhettinger, sbt
Priority: normal Keywords: patch

Created on 2009-01-19 12:57 by ndfred, last changed 2022-04-11 14:56 by admin. This issue is now closed.

File name Uploaded Description Edit ndfred, 2009-01-19 12:57 Test script jnoller, 2009-01-23 01:59
warn_fifo.patch skrah, 2010-02-21 12:58 Document that FIFO behavior is not guaranteed. review
issue-4999.diff asksol, 2010-10-24 07:47
Messages (14)
msg80162 - (view) Author: Frédéric Sagnes (ndfred) Date: 2009-01-19 12:57
Objects contained in a multiprocessing.Queue object are not comming out
of the queue in the same order as they went in. For instance, if I put
in object1, object2 and object3 in this very time sequence from multiple
processes, they can end up comming out of the queue as object2, object1
then object3 instead of the original order.

When using the threading module instead of multiprocessing everything is

The provided test script adds strings to the queue with timestamps.
These messages are not ordered by timestamp when they are printed. This
is an output of the test script with format "[pid@time] message":

[2120@00406] Got lock
[2120@02424] Released lock
[2121@02426] Got lock
[2121@04439] Released lock
[2121@16459] Released lock
[2120@16461] Got lock
[2121@18464] Got lock
[2120@18462] Released lock
[2121@20466] Released lock

Using print to print the message immediatly prints the messages in the
right order.

See this mailing-list thread for details:
msg80164 - (view) Author: Frédéric Sagnes (ndfred) Date: 2009-01-19 13:01
Tested using Python 2.6.1 on Mac OS 10.5 and Linux 2.6.26
msg80386 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-01-22 22:16
Rather than providing strict ordering of Queue input/output, perhaps a
separate or derived class (PriorityQueue?) should be provided just for
that matter (heapq-based?).
msg80387 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2009-01-22 22:46
The Queue module design should be followed as closely as possible.  It
provides FIFO order as the default, with options for LIFO and priority
msg80389 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2009-01-22 23:10
I agree, at very least it should provide the same default behavior that 
Queue does, however, it is not obvious to me how to resolve this in 
msg80390 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-01-22 23:20
That's why I was suggesting to leave Queue as it is and provide a
separate PriorityQueue. With a PriorityQueue the user is free to put
(timestamp, object) tuples in it if he wants a strict chronologic
ordering. He can also use sequence numbers instead of timestamps, etc.
msg80391 - (view) Author: Jean-Paul Calderone (exarkun) * (Python committer) Date: 2009-01-22 23:23
Jesse, can you explain the cause of the bug?  Maybe that will inspire
someone to come up with an idea for a fix.
msg80394 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2009-01-23 01:59
As a note to myself: adding a simple print to the multiprocessing.Queue 
put method, I can see the calls occurring from the children in order, 
for example:

obj:  0.025193 [proc1] Got lock
obj:  0.227725 [proc1] Released lock
obj:  0.228401 [proc2] Got lock
obj:  0.430501 [proc2] Released lock
obj:  0.431082 [proc0] Got lock
obj:  0.633048 [proc0] Released lock
obj:  0.633723 [proc4] Got lock
obj:  0.835692 [proc4] Released lock
obj:  0.836262 [proc3] Got lock
obj:  1.038197 [proc3] Released lock
obj:  1.038753 [proc1] Got lock
obj:  1.239288 [proc1] Released lock
obj:  1.240079 [proc2] Got lock
obj:  1.440773 [proc2] Released lock
obj:  1.441577 [proc0] Got lock
obj:  1.642142 [proc0] Released lock
obj:  1.642959 [proc4] Got lock
obj:  1.843331 [proc4] Released lock
obj:  1.844178 [proc3] Got lock
obj:  2.044549 [proc3] Released lock

Also attaching the process-based script
msg80396 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2009-01-23 02:48
On a put to the queue, the object is appended onto a deque (self._buffer) 
- this buffer is managed by a thread (_start_thread) which is handed the 
_feed method as the target. I suspect the bug is actually in the _feed 
msg99652 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2010-02-21 12:58
I think it would be nice to update the documentation if this isn't
resolved yet. The patch adds a warning that FIFO behavior is not
msg101579 - (view) Author: Stefan Praszalowicz (Stefan.P) Date: 2010-03-23 12:52
I just got surprised by this, and I agree that updating the doc would be nice, because as of now, it states quite explicitly that "the Queue and JoinableQueue types are multi-producer, multi-consumer FIFO queues".
msg119494 - (view) Author: Ask Solem (asksol) (Python committer) Date: 2010-10-24 07:47
Updated doc patch
msg119497 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2010-10-24 08:22
Are you sure this needs to be a warning?  We try to use them very sparingly.

(Also note the 3-space indent for reStructuredText markup.)
msg380309 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2020-11-04 02:41
I see the documentation mentions it now (in the second gray box here: )

and it also suggests to use a shared queue if this is an issue. 

Any objections to closing this issue?
Date User Action Args
2022-04-11 14:56:44adminsetgithub: 49249
2020-11-20 11:41:49iritkatrielsetstatus: pending -> closed
resolution: out of date
stage: resolved
2020-11-04 02:41:12iritkatrielsetstatus: open -> pending
nosy: + iritkatriel
messages: + msg380309

2014-05-14 11:45:03exarkunsetnosy: - exarkun
2014-05-13 22:23:48ned.deilysetnosy: + sbt
2014-05-13 21:46:36skrahsetnosy: - skrah
2010-10-24 08:22:57georg.brandlsetnosy: + georg.brandl
messages: + msg119497
2010-10-24 07:47:46asksolsetfiles: + issue-4999.diff
nosy: + asksol
messages: + msg119494

2010-03-23 12:52:00Stefan.Psettype: behavior

messages: + msg101579
nosy: + Stefan.P
2010-02-21 12:58:09skrahsetfiles: + warn_fifo.patch

nosy: + skrah
messages: + msg99652

keywords: + patch
2009-01-24 02:10:50ggenellinasetnosy: + ggenellina
2009-01-23 02:48:01jnollersetmessages: + msg80396
2009-01-23 01:59:16jnollersetfiles: +
messages: + msg80394
2009-01-22 23:23:16exarkunsetnosy: + exarkun
messages: + msg80391
2009-01-22 23:20:40pitrousetmessages: + msg80390
2009-01-22 23:10:43jnollersetmessages: + msg80389
2009-01-22 22:46:07rhettingersetnosy: + rhettinger
messages: + msg80387
2009-01-22 22:16:10pitrousetnosy: + pitrou
messages: + msg80386
2009-01-22 22:08:42jnollersetpriority: normal
2009-01-19 13:58:17jnollersetassignee: jnoller
nosy: + jnoller
2009-01-19 13:01:45ndfredsetmessages: + msg80164
2009-01-19 12:57:24ndfredcreate