As of revision 65077 of the trunk, I'm getting errors in 
test_multiprocessing that seem to point to memory corruption in object 
allocation/deallocation.  The failures are intermittent, and of a 
similar nature to the errors I was seeing previously, outlined in issue 
3088.

The platform is OS X 10.5.4 (not a fresh install---it was an upgrade 
from OS X 10.4, in case this makes any difference), running on a MacBook 
Pro.  I'm running a freshly checked out debug build of the trunk.

Here's what I did:

(1) make a fresh svn+ssh checkout of the trunk
(2) ./configure --with-pydebug && make
(3) ./python.exe Lib/test/test_multiprocessing.py
(4) repeat step (3) until something nasty happens.

The results vary from run to run, and 80-90% of the runs of 
test_multiprocessing pass.  Here are 3 of the failures I've seen, 
occurring on three separate runs of test_multiprocessing.

Failure 1:

test_notify_all (__main__.WithManagerTestCondition) ... Assertion 
failed: (pool->ref.count > 0), function PyObject_Free, file 
Objects/obmalloc.c, line 1100.

Failure 2:

test_imap_unordered (__main__.WithManagerTestPool) ...
python.exe(32381,0xb0513000) malloc: *** error for object 0xdbdbdbdb:
pointer being reallocated was not allocated
*** set a breakpoint in malloc_error_break to debug
python.exe(32381,0xb0513000) malloc: *** error for object 0xdbdbdbdb:
Non-aligned pointer being freed
*** set a breakpoint in malloc_error_break to debug
Fatal Python error: UNREF invalid object
ERROR

Failure 3:

test_imap_unordered (__main__.WithManagerTestPool) ... Fatal Python
error: UNREF invalid object
ERROR


I have very little (i.e. no) experience of debugging this kind of 
failure, and little understanding of how the multiprocessing module 
works.  But I can and will follow instructions and suggestions about how 
to debug this.

Stupid question:  it appears from reading the comments in that file that 
obmalloc.c is (intentionally) not thread-safe.  Could this have anything 
to do with the failures above?

And one more:

Failure 4:

test_make_pool (__main__.WithManagerTestPool) ... Assertion failed: (bp != 
NULL), function PyObject_Malloc, file Objects/obmalloc.c, line 746.

And another:

Failure 5:

test_notify (__main__.WithManagerTestCondition) ... Assertion failed: 
(usable_arenas->freepools == NULL), function PyObject_Malloc, file 
Objects/obmalloc.c, line 809.
ERROR

On Jul 17, 2008, at 6:22 PM, Mark Dickinson <report@bugs.python.org>  
wrote:

>
> New submission from Mark Dickinson <dickinsm@gmail.com>:
>
> As of revision 65077 of the trunk, I'm getting errors in
> test_multiprocessing that seem to point to memory corruption in object
> allocation/deallocation.  The failures are intermittent, and of a
> similar nature to the errors I was seeing previously, outlined in  
> issue
> 3088.
>
> The platform is OS X 10.5.4 (not a fresh install---it was an upgrade
> from OS X 10.4, in case this makes any difference), running on a  
> MacBook
> Pro.  I'm running a freshly checked out debug build of the trunk.
>
> Here's what I did:
>
> (1) make a fresh svn+ssh checkout of the trunk
> (2) ./configure --with-pydebug && make
> (3) ./python.exe Lib/test/test_multiprocessing.py
> (4) repeat step (3) until something nasty happens.
>
> The results vary from run to run, and 80-90% of the runs of
> test_multiprocessing pass.  Here are 3 of the failures I've seen,
> occurring on three separate runs of test_multiprocessing.
>

I am/was going to help you with this when you emailed me your last  
email - I'm disturbed none of my machines or the buildbots for that  
matter are seeing this. Can you post the output from:

Echo $LD_LIBRARY_PATH
which gcc
gcc -v

LD_LIBRARY_PATH isn't set.  gcc is the system gcc from Apple:

Macintosh-3:trunk dickinsm$ echo $LD_LIBRARY_PATH

Macintosh-3:trunk dickinsm$ which gcc
/usr/bin/gcc
Macintosh-3:trunk dickinsm$ gcc -v
Using built-in specs.
Target: i686-apple-darwin9
Configured with: /var/tmp/gcc/gcc-5484~1/src/configure --disable-
checking -enable-werror --prefix=/usr --mandir=/share/man --enable-
languages=c,objc,c++,obj-c++ --program-transform-name=/^[cg][^.-
]*$/s/$/-4.0/ --with-gxx-include-dir=/include/c++/4.0.0 --with-
slibdir=/usr/lib --build=i686-apple-darwin9 --with-arch=apple --with-
tune=generic --host=i686-apple-darwin9 --target=i686-apple-darwin9
Thread model: posix
gcc version 4.0.1 (Apple Inc. build 5484)

Can you try removing the --with-pydebug flag from configure and  
running that way?

Okay:  I just tried the following:

(1) clean svn checkout
(2) ./configure && make
(3) 100 runs of test_multiprocessing, via the shell command:
for ((i=0;i<100;i+=1)); do ./python.exe 
Lib/test/test_multiprocessing.py; sleep 1; done

I got 4 failed runs out of those 100 runs (details below);  2 hangs in 
test_notify_all, a KeyError in test_remote, and a failure of 
test_number_of_objects.

Failed run 1
------------
test_notify_all (__main__.WithManagerTestCondition) ... Process Process-
48:
Traceback (most recent call last):
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/process.py", 
line 232, in _bootstrap
    self.run()
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/process.py", 
line 88, in run
    self._target(*self._args, **self._kwargs)
  File "Lib/test/test_multiprocessing.py", line 600, in f
    cond.acquire()
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/managers.py", 
line 946, in acquire
    return self._callmethod('acquire', (blocking,))
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/managers.py", 
line 718, in _callmethod
    self._connect()
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/managers.py", 
line 705, in _connect
    conn = self._Client(self._token.address, authkey=self._authkey)
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/connection.py", 
line 133, in Client
    c = SocketClient(address)
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/connection.py", 
line 254, in SocketClient
    s.connect(address)
  File "<string>", line 1, in connect
error: [Errno 61] Connection refused
^CProcess PoolWorker-5:4:
Traceback (most recent call last):
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/process.py", 
line 232, in _bootstrap
Process PoolWorker-5:3:
Traceback (most recent call last):
    self.run()
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/process.py", 
line 88, in run
    self._target(*self._args, **self._kwargs)
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/pool.py", line 
57, in worker
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/process.py", 
line 232, in _bootstrap
    task = get()
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/queues.py", 
line 337, in get
    self.run()
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/process.py", 
line 88, in run
    self._target(*self._args, **self._kwargs)
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/pool.py", line 
57, in worker
    racquire()
KeyboardInterrupt
    task = get()
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/queues.py", 
line 337, in get
    racquire()
KeyboardInterrupt
Process PoolWorker-5:1:
Traceback (most recent call last):
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/process.py", 
line 232, in _bootstrap
Process Process-50:
Process Process-49:
Traceback (most recent call last):
    self.run()
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/process.py", 
line 88, in run
    self._target(*self._args, **self._kwargs)
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/process.py", 
line 232, in _bootstrap
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/pool.py", line 
57, in worker
    task = get()
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/queues.py", 
line 339, in get
    self.run()
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/process.py", 
line 88, in run
    self._target(*self._args, **self._kwargs)
  File "Lib/test/test_multiprocessing.py", line 602, in f
    return recv()
KeyboardInterrupt
    cond.wait(timeout)
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/managers.py", 
line 959, in wait
    return self._callmethod('wait', (timeout,))
Traceback (most recent call last):
  File "Lib/test/test_multiprocessing.py", line 1786, in <module>
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/managers.py", 
line 722, in _callmethod
    kind, result = conn.recv()
KeyboardInterrupt
    main()
  File "Lib/test/test_multiprocessing.py", line 1783, in main
    test_main(unittest.TextTestRunner(verbosity=2).run)
  File "Lib/test/test_multiprocessing.py", line 1773, in test_main
    run(suite)
  File "/Users/dickinsm/python_source/trunk/Lib/unittest.py", line 750, 
in run
Process PoolWorker-5:2:
    test(result)
  File "/Users/dickinsm/python_source/trunk/Lib/unittest.py", line 461, 
in __call__
    return self.run(*args, **kwds)
  File "/Users/dickinsm/python_source/trunk/Lib/unittest.py", line 457, 
in run
    test(result)
  File "/Users/dickinsm/python_source/trunk/Lib/unittest.py", line 461, 
in __call__
    return self.run(*args, **kwds)
  File "/Users/dickinsm/python_source/trunk/Lib/unittest.py", line 457, 
in run
    test(result)
  File "/Users/dickinsm/python_source/trunk/Lib/unittest.py", line 300, 
in __call__
    return self.run(*args, **kwds)
  File "/Users/dickinsm/python_source/trunk/Lib/unittest.py", line 279, 
in run
    testMethod()
  File "Lib/test/test_multiprocessing.py", line 701, in test_notify_all
    sleeping.acquire()
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/managers.py", 
line 946, in acquire
    return self._callmethod('acquire', (blocking,))
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/managers.py", 
line 722, in _callmethod
    kind, result = conn.recv()
KeyboardInterrupt
Traceback (most recent call last):
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/process.py", 
line 232, in _bootstrap
    self.run()
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/process.py", 
line 88, in run
    self._target(*self._args, **self._kwargs)
  File "Lib/test/test_multiprocessing.py", line 602, in f
    cond.wait(timeout)
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/managers.py", 
line 959, in wait
    return self._callmethod('wait', (timeout,))
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/managers.py", 
line 722, in _callmethod
    kind, result = conn.recv()
KeyboardInterrupt
Traceback (most recent call last):


Failed run 2
------------
test_task_done (__main__.WithManagerTestQueue) ... ok
test_remote (__main__.WithManagerTestRemoteManager) ... ERROR
test_bounded_semaphore (__main__.WithManagerTestSemaphore) ... ok
test_semaphore (__main__.WithManagerTestSemaphore) ... ok
test_timeout (__main__.WithManagerTestSemaphore) ... ok
test_getobj_getlock (__main__.WithManagerTestValue) ... ok
test_rawvalue (__main__.WithManagerTestValue) ... ok
test_value (__main__.WithManagerTestValue) ... ok
test_number_of_objects (__main__.WithManagerTestZZZNumberOfObjects) ... 
ok

======================================================================
ERROR: test_remote (__main__.WithManagerTestRemoteManager)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Lib/test/test_multiprocessing.py", line 1157, in test_remote
    queue = manager2.get_queue()
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/managers.py", 
line 635, in temp
    authkey=self._authkey, exposed=exp
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/managers.py", 
line 887, in AutoProxy
    incref=incref)
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/managers.py", 
line 696, in __init__
    self._incref()
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/managers.py", 
line 743, in _incref
    dispatch(conn, None, 'incref', (self._id,))
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/managers.py", 
line 79, in dispatch
    raise convert_to_error(kind, result)
RemoteError: 
------------------------------------------------------------------------
---
Traceback (most recent call last):
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/managers.py", 
line 181, in handle_request
    result = func(c, *args, **kwds)
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/managers.py", 
line 397, in incref
    self.id_to_refcount[ident] += 1
KeyError: '5bf968'
------------------------------------------------------------------------
---

----------------------------------------------------------------------
Ran 121 tests in 9.230s

FAILED (errors=1)

Failed run 3
------------
test_number_of_objects (__main__.WithManagerTestZZZNumberOfObjects) ...   
680490:       refcount=1
    <threading._Semaphore object at 0x680490>
  680bd0:       refcount=1
    <multiprocessing.pool.Pool object at 0x680bd0>
FAIL

======================================================================
FAIL: test_number_of_objects 
(__main__.WithManagerTestZZZNumberOfObjects)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Lib/test/test_multiprocessing.py", line 1042, in 
test_number_of_objects
    self.assertEqual(refs, EXPECTED_NUMBER)
AssertionError: 2 != 1

----------------------------------------------------------------------
Ran 121 tests in 9.228s

FAILED (failures=1)

Failed run 4
------------
test_notify_all (__main__.WithManagerTestCondition) ... Process Process-
50:
Traceback (most recent call last):
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/process.py", 
line 232, in _bootstrap
    self.run()
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/process.py", 
line 88, in run
    self._target(*self._args, **self._kwargs)
  File "Lib/test/test_multiprocessing.py", line 600, in f
    cond.acquire()
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/managers.py", 
line 946, in acquire
    return self._callmethod('acquire', (blocking,))
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/managers.py", 
line 718, in _callmethod
    self._connect()
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/managers.py", 
line 705, in _connect
    conn = self._Client(self._token.address, authkey=self._authkey)
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/connection.py", 
line 133, in Client
    c = SocketClient(address)
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/connection.py", 
line 254, in SocketClient
    s.connect(address)
  File "<string>", line 1, in connect
error: [Errno 61] Connection refused
^CProcess PoolWorker-5:4:
Traceback (most recent call last):
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/process.py", 
line 232, in _bootstrap
Process PoolWorker-5:3:
Traceback (most recent call last):
    self.run()
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/process.py", 
line 88, in run
    self._target(*self._args, **self._kwargs)
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/pool.py", line 
57, in worker
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/process.py", 
line 232, in _bootstrap
    task = get()
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/queues.py", 
line 337, in get
    racquire()
    self.run()
KeyboardInterrupt
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/process.py", 
line 88, in run
    self._target(*self._args, **self._kwargs)
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/pool.py", line 
57, in worker
    task = get()
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/queues.py", 
line 337, in get
    racquire()
KeyboardInterrupt
Process PoolWorker-5:1:
Traceback (most recent call last):
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/process.py", 
line 232, in _bootstrap
Process Process-48:
    self.run()
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/process.py", 
line 88, in run
    self._target(*self._args, **self._kwargs)
Traceback (most recent call last):
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/pool.py", line 
57, in worker
    task = get()
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/queues.py", 
line 339, in get
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/process.py", 
line 232, in _bootstrap
    return recv()
KeyboardInterrupt
    self.run()
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/process.py", 
line 88, in run
    self._target(*self._args, **self._kwargs)
  File "Lib/test/test_multiprocessing.py", line 602, in f
    cond.wait(timeout)
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/managers.py", 
line 959, in wait
Traceback (most recent call last):
  File "Lib/test/test_multiprocessing.py", line 1786, in <module>
    return self._callmethod('wait', (timeout,))
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/managers.py", 
line 722, in _callmethod
    kind, result = conn.recv()
KeyboardInterrupt
    main()
  File "Lib/test/test_multiprocessing.py", line 1783, in main
    test_main(unittest.TextTestRunner(verbosity=2).run)
  File "Lib/test/test_multiprocessing.py", line 1773, in test_main
Process PoolWorker-5:2:
    run(suite)
  File "/Users/dickinsm/python_source/trunk/Lib/unittest.py", line 750, 
in run
Traceback (most recent call last):
    test(result)
  File "/Users/dickinsm/python_source/trunk/Lib/unittest.py", line 461, 
in __call__
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/process.py", 
line 232, in _bootstrap
    return self.run(*args, **kwds)
  File "/Users/dickinsm/python_source/trunk/Lib/unittest.py", line 457, 
in run
    test(result)
  File "/Users/dickinsm/python_source/trunk/Lib/unittest.py", line 461, 
in __call__
    self.run()
    return self.run(*args, **kwds)
  File "/Users/dickinsm/python_source/trunk/Lib/unittest.py", line 457, 
in run
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/process.py", 
line 88, in run
    self._target(*self._args, **self._kwargs)
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/pool.py", line 
57, in worker
    test(result)
  File "/Users/dickinsm/python_source/trunk/Lib/unittest.py", line 300, 
in __call__
    return self.run(*args, **kwds)
  File "/Users/dickinsm/python_source/trunk/Lib/unittest.py", line 279, 
in run
    task = get()
    testMethod()
  File "Lib/test/test_multiprocessing.py", line 701, in test_notify_all
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/queues.py", 
line 337, in get
    sleeping.acquire()
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/managers.py", 
line 946, in acquire
    racquire()
KeyboardInterrupt
    return self._callmethod('acquire', (blocking,))
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/managers.py", 
line 722, in _callmethod
    kind, result = conn.recv()
KeyboardInterrupt
Process Process-49:
Traceback (most recent call last):
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/process.py", 
line 232, in _bootstrap
    self.run()
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/process.py", 
line 88, in run
    self._target(*self._args, **self._kwargs)
  File "Lib/test/test_multiprocessing.py", line 602, in f
    cond.wait(timeout)
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/managers.py", 
line 959, in wait
    return self._callmethod('wait', (timeout,))
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/managers.py", 
line 722, in _callmethod
    kind, result = conn.recv()
KeyboardInterrupt

I should add to the previous message that this was revision 65090, and
that it was a non-debug build.

It looks like this isn't just me.  See the buildbot output at:

http://www.python.org/dev/buildbot/all/x86%20osx.5%20trunk/builds/33/ste
p-test/0

which shows:

test_multiprocessing
Assertion failed: (bp != NULL), function PyObject_Malloc, file 
Objects/obmalloc.c, line 746.
test test_multiprocessing failed -- errors occurred; run in verbose mode 
for details

Ok, so for the moment, let's set aside the connection refused messages: 
that may be a case of not cleaning up a socket correctly (which is still 
bad, but not memory corruption).

Of note from the buildbot failure:
Assertion failed: (bp != NULL), function PyObject_Malloc, file 
Objects/obmalloc.c, line 746.
test test_multiprocessing failed -- errors occurred; run in verbose mode 
for details

I don't know enough about obmalloc.c to state if this is a problem with 
it not being multithreaded

Here's another failure (from my own buildbot to boot):

test_multiprocessing
/Users/buildbot/buildarea/trunk.noller-
osx86/build/Lib/multiprocessing/__init__.py:82: ImportWarning: Not 
importing directory '/Users/buildbot/buildarea/trunk.noller-
osx86/build/Modules/_multiprocessing': missing __init__.py
  import _multiprocessing
Fatal Python error: Objects/tupleobject.c:169 object at 0x539d538 has 
negative ref count -606348326
make: *** [buildbottest] Abort trap
program finished with exit code 2

I finally found some more time to look at this.  I cut down the test-suite 
to try to find a minimal failing example.  I can fairly reliably make a 
debug build of the trunk crash using the following nine lines

import multiprocessing.managers
def sqr(x): return x*x
manager = multiprocessing.managers.SyncManager()
manager.start()
pool = manager.Pool(4)
it = pool.imap_unordered(sqr, range(10000))
assert sorted(it) == map(sqr, range(10000))
pool.terminate()
manager.shutdown()

Typical output is:

Fatal Python error: UNREF invalid object
(followed by traceback)

or:

Assertion failed: (bp != NULL), function PyObject_Malloc, file 
Objects/obmalloc.c, line 755.

or:

Debug memory block at address p=0x247778:
    26 bytes originally requested
    The 4 pad bytes at p-4 are not all FORBIDDENBYTE (0xfb):
        at p-4: 0xdb *** OUCH
        at p-3: 0xdb *** OUCH
        at p-2: 0xdb *** OUCH
        at p-1: 0xdb *** OUCH
    Because memory is corrupted at the start, the count of bytes requested
       may be bogus, and checking the trailing pad bytes may segfault.
    The 4 pad bytes at tail=0x247792 are not all FORBIDDENBYTE (0xfb):
        at tail+0: 0x35 *** OUCH
        at tail+1: 0x00 *** OUCH
        at tail+2: 0xfb
        at tail+3: 0xfb
    The block was made by call #4227530756 to debug malloc/realloc.
    Data at p: 00 00 00 00 00 00 00 00 ... 00 00 08 00 00 00 b0 72
Fatal Python error: bad leading pad byte

> Assertion failed: (bp != NULL), function PyObject_Malloc, file 
> Objects/obmalloc.c, line 755.

This one gives one probable cause of the problem:

- in Modules/_multiprocessing/connection.h, connection_send_obj()
releases the GIL around a call to conn_send_string().
- in Modules/_multiprocessing/socket_connection.c, conn_send_string()
uses PyMem_Malloc()

This is wrong (the GIL must be held when using the PyMem_* and
PyObject_* functions), and is probably the cause of the failed assertion.

> This is wrong (the GIL must be held when using the PyMem_* and
> PyObject_* functions), and is probably the cause of the failed 
assertion.

This sounds quite likely.

I just managed (using the low-tech method of setting a static variable 
on entry and clearing it on exit) to confirm that PyObject_Malloc in 
obmalloc.c is being accessed simultaneously by multiple threads when
test_multiprocessing is run.

Here's a patch that fixes the problem for me.  It releases the GIL around
the calls to _conn_sendall within conn_send_string, instead of releasing 
the GIL for the whole call to conn_send_string.

To be complete, the patch should also deal with conn_recv_string() which
has the same problem.
And please do not forget the win32 implementation, in pipe_connection.c.

Thanks, Amaury!  How's this?

I have no access to a Windows machine, so this patch is untested on 
Windows.

Mark,
There are 3 semicolons missing in your patch, in pipe_connection.c, just
after the calls to WriteFile and ReadFile.
After this, it compiles and runs correctly. Tests pass.

Note that on Windows, your "nine lines" cannot work as is, because the
processes are spawned, not forked: the sqr() function is not copied.
And if you save the lines in a script file, it will be imported by every
subprocess, and every subprocess will start its own manager... and
memory explodes.

import multiprocessing.managers
def sqr(x): return x*x
if __name__ == '__main__':
    manager = multiprocessing.managers.SyncManager()
    manager.start()
    pool = manager.Pool(4)
    it = pool.imap_unordered(sqr, range(1000))
    assert sorted(it) == [sqr(x) for x in range(1000)]
    pool.terminate()
    manager.shutdown()

I added the semicolons Amaury, and have it teed up in my local repo for 
submit. Can you review this diff just to confirm?

I've committed this as-is based off my last patch. I will watch the 
buildbots for failures. 

Mark/Amaury - if I see you guys at pycon, I owe you a drink.