Title: Race condition in multiprocessing Queue
Type: behavior Stage:
Components: Versions: Python 3.6
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: TwistedSim, highvelcty, tim.peters
Priority: normal Keywords:

Created on 2018-02-20 01:55 by TwistedSim, last changed 2019-11-10 22:05 by highvelcty.

File name Uploaded Description Edit TwistedSim, 2018-02-20 01:55 Code that reproduce the behavior
Messages (4)
msg312391 - (view) Author: Simon Bouchard (TwistedSim) * Date: 2018-02-20 01:55
The clear list function call in made after the put(data) on the queue. But the data is sometime clear in the queue (randomly). Since both function are call within the same process, a race condition is not expected.
msg312392 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2018-02-20 02:15
The docs could be clearer about this:  the argument to .put() is _not_ pickled at the time .put() is called.  The object is remembered (by reference, not by value), and a feeder thread pickles the value and puts the pickle on the queue when the feeder thread gets around to that.  So if the object is mutated in any way in between, it's not defined whether the pre- or post-mutation state is put on the queue (or, in some cases, even some partially mutated value).

To make it wholly deterministic you could, e.g., change the .put() to this:


Or you could use a Manager().Queue() instead, which doesn't use a feeder thread.
msg354134 - (view) Author: Eric Meyer (highvelcty) Date: 2019-10-07 19:50
multiprocessing.SimpleQueue is another workaround to this issue.

I agree the docs should be clearer about this. Additionally, it would be helpful if there was a way to optionally put an item on a multiprocessing.Queue and block until the item has been written to the pipe (or block until it is safe to mutate,garbage collect,etc the item in the calling thread).
msg356340 - (view) Author: Eric Meyer (highvelcty) Date: 2019-11-10 22:05
The problem I was seeing ended up being improper GIL management in a c++ extension. It seems putting items that were created without the GIL on the queue causes a seg fault.
Date User Action Args
2019-11-10 22:05:56highvelctysetmessages: + msg356340
2019-10-07 19:50:44highvelctysetnosy: + highvelcty
messages: + msg354134
2018-02-20 02:15:10tim.peterssetnosy: + tim.peters
messages: + msg312392
2018-02-20 01:55:01TwistedSimcreate