classification
Title: Implement multiprocessing.Barrier
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.3
process
Status: closed Resolution: fixed
Dependencies: 14087 Superseder:
Assigned To: Nosy List: anacrolix, neologix, pitrou, python-dev, sbt
Priority: normal Keywords: patch

Created on 2012-02-20 00:52 by anacrolix, last changed 2012-06-15 20:35 by sbt. This issue is now closed.

Files
File name Uploaded Description Edit
barrier.py sbt, 2012-02-20 17:16
barrier_tests.py sbt, 2012-02-20 17:20
mp_barrier.patch sbt, 2012-02-23 17:19
mp_barrier.patch sbt, 2012-06-05 20:30 review
Messages (13)
msg153744 - (view) Author: Matt Joiner (anacrolix) Date: 2012-02-20 00:52
There is no Barrier in multiprocessing corresponding to threading.Barrier.
msg153785 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-02-20 17:16
Here is an initial implementation.  Differences from threading.Barrier:

- I have not implemented reset().

- wait() returns 0 or -1.  One thread returns 0, the remainder return -1.
  This is different to threading.Barrier where each of the N threads 
  returns a unique index in range(N).

- I added an "action_args" parameter to the constructor.  This is a tuple 
  which provides argument to the "action" callback.  (This is because closures
  are not picklable, making no-argument callbacks rather limiting for 
  multiprocessing.)
msg153786 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-02-20 17:20
barrier_tests.py contains minor modifications of the unit tests for
threading.Barrier.  (The two tests using reset() are commented out.)

The implementation passes for me on Linux and Windows.
msg153857 - (view) Author: Charles-Fran├žois Natali (neologix) * (Python committer) Date: 2012-02-21 07:50
> Here is an initial implementation.

Wouldn't it be simpler with a mp.Condition?

Otherwise, this should be added to Lib/multiprocesing.synchronize.py, and the tests to test_multiprocessing.
msg153958 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-02-22 15:28
> Wouldn't it be simpler with a mp.Condition?

Well, it is a fair bit shorter than the implementation in threading.py.  But that is not a fair comparison because it does implement reset().

I was trying to avoid using shared memory/ctypes since multiprocessing.synchronize does not currently use them.  However, I think it would be better (and much simpler) to just subclass threading.Barrier, making self._state and self._counter properties which delegate to RawValue objects.  That gets rid of the differences in behaviour.  I have this working, although I had to monkey patch multiprocessing.Condition to add a wait_for() method.  See Issue 14087.

> Otherwise, this should be added to Lib/multiprocesing.synchronize.py,
> and the tests to test_multiprocessing.

I will provide a patch later.
msg154073 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-02-23 17:19
Patch which subclasses threading.Barrier.
msg154078 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-02-23 19:06
Forgot to mention, mp_barrier.patch needs to be applied on top of cond_wait_for.patch for Issue14087.
msg159335 - (view) Author: Charles-Fran├žois Natali (neologix) * (Python committer) Date: 2012-04-25 20:27
The patch looks good.
However, I've had two failures while testing it:
- a BrokenBarrierError on test_default_timeout: I see you've already increased the timeout from the original threading code, but you can probably double it (we have some slow buildbots, and creating processes is expensive on Windows)
- TestNumberOfObjects is failing too:
"""
test_number_of_objects (test.test_multiprocessing.WithManagerTestZZZNumberOfObjects) ...   b6cb776c:       refcount=1
    <threading.Barrier object at 0xb6cb776c>
  b6d72dfc:       refcount=2
    <multiprocessing.pool.Pool object at 0xb6d72dfc>
  b6cb776c:       refcount=1
    <threading.Barrier object at 0xb6cb776c>
  b6d72dfc:       refcount=2
    <multiprocessing.pool.Pool object at 0xb6d72dfc>
FAIL
"""
msg159455 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-04-27 14:09
RawValue uses ctypes, right? That's problematic for platforms which don't support ctypes.
msg161671 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-05-26 17:42
> RawValue uses ctypes, right? That's problematic for platforms which don't 
> support ctypes.

Are there many posix systems (we care about) where ctypes doesn't work?

It would be fairly easy to use memoryview instead of ctypes.  (In fact Value/RawValue could be implemented that way when using a typecode instead of a ctypes type.)
msg161673 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-05-26 17:56
> > RawValue uses ctypes, right? That's problematic for platforms which don't 
> > support ctypes.
> 
> Are there many posix systems (we care about) where ctypes doesn't
> work?

It depends what you call "caring about" :-)
But proprietary Unix C compilers generally don't mix very well with
ctypes.

> It would be fairly easy to use memoryview instead of ctypes.  (In fact
> Value/RawValue could be implemented that way when using a typecode
> instead of a ctypes type.)

That would be a good idea then. Also, not using ctypes could make code
theoretically more robust.
msg162371 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-06-05 20:30
The attached patch uses memoryview instead of ctypes.

If the patch for Issue #14953 (reimplementing RawArray/RawValue in terms of memoryview) is applied, then it could be simplified a bit.
msg162923 - (view) Author: Roundup Robot (python-dev) Date: 2012-06-15 17:33
New changeset 2d2f206d040e by Richard Oudkerk in branch 'default':
Issue #14059: Implement multiprocessing.Barrier
http://hg.python.org/cpython/rev/2d2f206d040e
History
Date User Action Args
2012-06-15 20:35:12sbtsetstatus: open -> closed
resolution: fixed
stage: resolved
2012-06-15 17:33:07python-devsetnosy: + python-dev
messages: + msg162923
2012-06-05 20:30:04sbtsetfiles: + mp_barrier.patch

messages: + msg162371
2012-05-26 17:56:20pitrousetmessages: + msg161673
2012-05-26 17:42:15sbtsetmessages: + msg161671
2012-04-27 14:09:39pitrousetnosy: + pitrou
messages: + msg159455
2012-04-25 20:27:16neologixsetmessages: + msg159335
2012-03-31 23:42:54pitrousetdependencies: + multiprocessing.Condition.wait_for missing
2012-02-23 19:06:50sbtsetmessages: + msg154078
2012-02-23 17:19:41sbtsetfiles: + mp_barrier.patch
keywords: + patch
messages: + msg154073
2012-02-22 15:28:21sbtsetmessages: + msg153958
2012-02-21 07:50:49neologixsetnosy: + neologix
messages: + msg153857
2012-02-20 17:20:03sbtsetfiles: + barrier_tests.py

messages: + msg153786
2012-02-20 17:16:39sbtsetfiles: + barrier.py
nosy: + sbt
messages: + msg153785

2012-02-20 00:52:18anacrolixcreate