For me, test_multiprocessing hangs consistently on OS X 10.5.3.  It
passes just fine on Ubuntu 8.04.

I can confirm this on Leopard too.

It passes for me on Leopard. Can you post the test running in verbose
mode so we can see where it hangs?

On python-3000 trunk, _multiprocessing doesn't even compile:

/Users/jesse/open_source/subversion/python-
3000/Modules/_multiprocessing/semaphore.c: In function ‘semlock_iszero’:
/Users/jesse/open_source/subversion/python-
3000/Modules/_multiprocessing/semaphore.c:515: warning: unused variable 
‘sval’

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Jun 12, 2008, at 9:02 AM, Benjamin Peterson wrote:

> It passes for me on Leopard. Can you post the test running in verbose
> mode so we can see where it hangs?

It never hangs when run standalone, though it crashes about half the  
time.  Running it under gdb doesn't help; it always gives me an  
interrupted syste call in os.waidpid() in forking.py.

The hang occurs during 'make test', and it's always the second run  
that hangs.  Perhaps some lock isn't getting cleaned up properly the  
first time?

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSFEicXEjvBPtnXfVAQLGNwP/S6f2IrO7c7SET0Gx8FXqdPmot3jcmopx
TFxDA5csh/pVaDQCVW6DiLMXsu2TkQGPPbbo8Bx9iPmV/iIHFqy4nDtETqqjKdRp
BvVtBmvSrP6wmymlKFlFC5qdfbbvguZq/hO60XulQk+WU4F8N7oHQck0tA2JhdDh
lS5SAFAIovA=
=xzs6
-----END PGP SIGNATURE-----

I did a make clean && ./configure && make and it started compiling for me 
again. Sorry for the noise.

If it's only failing during the second run of "make test", typically
there's some implicit dependency on something that is disturbed by
running a test that's later in the suite of tests.  This could be either
the fault of that other test (not restoring some global setting or
environment var) or the fault of the test that fails (making unwarranted
assumptions or not initializing some needed settings before starting).

If it works for some folks and not for others, on the same platform,
compare the set of extension modules that are not built, reported by
"make" in a message starting with "Failed to find the necessary bits to
build these modules:". Likely, Barry has the most complete set, while
Jesse has a few more extensions missing.

Finding this is usually a painful process of bisecting the set of tests
run.  Randomizing the tests with regrtest.py -r might also be helpful.

FWIW, when I tried (on Leopard) "make test
TESTOPTS=test_multiprocessing" it hung on the first set.  When I ran
"./python Lib/test/test_multiprocess.py -v" it reported 122 tests
passed.  But when I ran "./python Lib/test/regrtest.py -v
test_multiprocessing" one test failed:

======================================================================
ERROR: test_remote (test.test_multiprocessing.WithManagerTestRemoteManager)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/guido/p/Lib/test/test_multiprocessing.py", line 1167, in
test_remote
    queue = manager2.get_queue()
  File "/Users/guido/p/Lib/multiprocessing/managers.py", line 650, in temp
    authkey=self._authkey, exposed=exp
  File "/Users/guido/p/Lib/multiprocessing/managers.py", line 902, in
AutoProxy
    incref=incref)
  File "/Users/guido/p/Lib/multiprocessing/managers.py", line 711, in
__init__
    self._incref()
  File "/Users/guido/p/Lib/multiprocessing/managers.py", line 758, in
_incref
    dispatch(conn, None, 'incref', (self._id,))
  File "/Users/guido/p/Lib/multiprocessing/managers.py", line 94, in
dispatch
    raise convert_to_error(kind, result)
RemoteError: 
---------------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/guido/p/Lib/multiprocessing/managers.py", line 196, in
handle_request
    result = func(c, *args, **kwds)
  File "/Users/guido/p/Lib/multiprocessing/managers.py", line 412, in incref
    self.id_to_refcount[ident] += 1
KeyError: '5f2828'
---------------------------------------------------------------------------

I should add this was in the trunk (2.6).

I think I'm having a similar lockup on fedora core 4 (smp machine). This
is with the py3k branch, freshly svn updated. When running "make test
TESTOPTS=test_multiprocessing" the first of the two test runs always
succeeds in something like 10-15 seconds, while the second one
occasionally hangs. I tried running with -v as extra argument but can't
get it to hang in that case (to figure out which test is the culprit).

After a few more runs with -v and redirecting output to a file it seems
the lockup I get is in 
test_notify_all (test.test_multiprocessing.WithManagerTestCondition)

It's taking me longer to get to this than I planned, any help is 
appreciated.

I can get an intermittent (1 every 15 or so runs) lock in:
 test_get (__main__.WithProcessesTestQueue) ... 

Executed like this:
./python Lib/test/test_multiprocessing.py

When I control-c it the stack looks like this:
...snip
  File "/root/py/python-3000/Lib/multiprocessing/pool.py", line 57, in 
worker
    task = get()
  File "/root/py/python-3000/Lib/multiprocessing/queues.py", line 337, 
in get
    task = get()
  File "/root/py/python-3000/Lib/multiprocessing/queues.py", line 339, 
in get
    racquire()
KeyboardInterrupt
    task = get()
  File "/root/py/python-3000/Lib/multiprocessing/queues.py", line 337, 
in get
    task = get()
  File "/root/py/python-3000/Lib/multiprocessing/queues.py", line 337, 
in get
    return recv()
  File "/root/py/python-3000/Lib/pickle.py", line 1327, in loads
    racquire()
KeyboardInterrupt
    racquire()
KeyboardInterrupt
    def loads(s, *, encoding="ASCII", errors="strict"):
KeyboardInterrupt


I'm not seeing frequent locks/failures when run with regrtest, but I am 
seeing them with "make test TESTOPTS=test_multiprocessing"

I've attached full output. Still trying to figure it out

I made a copy of test_multiprocessing.py (to test_mp.py) and basically
removed all test classes, except _TestCondition. In it, I commented all
test methods except test_notify_all. When run with make test
TESTOPTS="-v test_mp" there's the lockups every few runs, but I also got
(only three times in 40 or so runs so far):


9:57|paul@tabu:~/c/py3k-svn> make test TESTOPTS="-v test_mp"

Failed to find the necessary bits to build these modules:
_gestalt
To find the necessary bits, look in setup.py in detect_modules() for the
module's name.

find ./Lib -name '*.py[co]' -print | xargs rm -f
./python -E -bb ./Lib/test/regrtest.py -v test_mp
test_mp
test_notify_all (test.test_mp.WithProcessesTestCondition) ... ok
test_notify_all (test.test_mp.WithThreadsTestCondition) ... ok
test_notify_all (test.test_mp.WithManagerTestCondition) ... ok

----------------------------------------------------------------------
Ran 3 tests in 1.087s

OK
1 test OK.
CAUTION:  stdout isn't compared in verbose mode:
a test that passes in verbose mode may fail without it.
./python -E -bb ./Lib/test/regrtest.py -v test_mp
test_mp
test_notify_all (test.test_mp.WithProcessesTestCondition) ... ok
test_notify_all (test.test_mp.WithThreadsTestCondition) ... ok
test_notify_all (test.test_mp.WithManagerTestCondition) ... Exception in
thread Thread-28:
Traceback (most recent call last):
  File "/home/paul/c/py3k-svn/Lib/threading.py", line 492, in
_bootstrap_inner
    self.run()
  File "/home/paul/c/py3k-svn/Lib/threading.py", line 447, in run
    self._target(*self._args, **self._kwargs)
  File "/home/paul/c/py3k-svn/Lib/test/test_mp.py", line 208, in f
    cond.wait(timeout)
  File "/home/paul/c/py3k-svn/Lib/multiprocessing/managers.py", line
973, in wait
    return self._callmethod('wait', (timeout,))
  File "/home/paul/c/py3k-svn/Lib/multiprocessing/managers.py", line
748, in _callmethod
    raise convert_to_error(kind, result)
RuntimeError: cannot wait on un-aquired lock

FWIW: In order to boost the logging level within the test(s) do the 
following:

Search for LOG_LEVEL, set it to:
LOG_LEVEL=util.SUBDEBUG

And then in the main() replace:
    multiprocessing.get_logger().setLevel(LOG_LEVEL)
With:
multiprocessing.log_to_stderr(level=LOG_LEVEL)

I also isolated the test(s) like Paul did, and it looks like a semi-
consistent lock up in:
  File "/root/py/python-3000/Lib/multiprocessing/queues.py", line 337, in 
get
    racquire()

This is running only the test_event test. The racquire traces back to 
synchronize.SemLock which calls into _multiprocessing.SemLock

Seems to work fine for me now with latest py3k branch.

I suspect the problems with WithManagerTestCondition.notify_all() may
have to do with the thread safety of the proxies.  If you replace
Thread(...) by self.Process(...) in that test then the problem may go away.

After talking with Richard, I think the best way to attack this issue 
(and the other ones around suite unreliability) is to remove the 
unreliable test cases for the first beta, and then refactor the suite 
post beta with an eye towards reliability and clarity. Personally, I 
would like to break the suites up in the the test_multiprocessing.py 
script to be more in the vein of other tests in Lib/test/...

I removed the more unreliable test cases while keeping the core ones and 
wrote a quick bash script to do a burn-in of the "make tests" command, 
for 100 loops, I was unable to get the tests to hang with the reduced 
suite.

I ran the same thing on trunk and py3k just to make sure I could not get 
it to hang/crash.

Here is the loop I ran the tests with:

#!/bin/sh

for (( i=1;i<=100;i+=1 )); do
    make test TESTOPTS="-v test_multiprocessing"        
done

I don't have commit rights, so I can't apply the test_multiprocessing_reduced.diff myself. Anyone willing? I think this 
should help the buildbots.

I'm going to knock this one down to critical since it's working for me
now on OS X and buildbot looks green.  We can address any additional
patches after the beta release.

Still hangs for me on the 2.6 trunk on Ubuntu 8.04

Where exactly does it hang Miki?

Jesse,

I just run "make test", it runs until test_multiprocessing and then
hangs there

test_multiprocessing is also still hanging for me, perhaps 30% of the 
times I run the test suite.

When running the test by itself it seems to pass much more often, but 
not always.  I just got the following output (on OS X 10.5.3/Intel).
There was a hang at test_get;  after around half-an-hour I hit Ctrl-C.

Macintosh-3:trunk dickinsm$ ./python.exe 
Lib/test/test_multiprocessing.py
test_array (__main__.WithProcessesTestArray) ... ok
test_getobj_getlock_obj (__main__.WithProcessesTestArray) ... ok
test_rawarray (__main__.WithProcessesTestArray) ... ok
test_notify (__main__.WithProcessesTestCondition) ... ok
test_notify_all (__main__.WithProcessesTestCondition) ... ok
test_timeout (__main__.WithProcessesTestCondition) ... ok
test_connection (__main__.WithProcessesTestConnection) ... ok
test_duplex_false (__main__.WithProcessesTestConnection) ... ok
test_sendbytes (__main__.WithProcessesTestConnection) ... ok
test_spawn_close (__main__.WithProcessesTestConnection) ... ok
test_event (__main__.WithProcessesTestEvent) ... ok
test_finalize (__main__.WithProcessesTestFinalize) ... ok
test_heap (__main__.WithProcessesTestHeap) ... ok
test_import (__main__.WithProcessesTestImportStar) ... ok
test_lock (__main__.WithProcessesTestLock) ... ok
test_rlock (__main__.WithProcessesTestLock) ... ok
test_enable_logging (__main__.WithProcessesTestLogging) ... ok
test_level (__main__.WithProcessesTestLogging) ... ok
test_active_children (__main__.WithProcessesTestProcess) ... ok
test_cpu_count (__main__.WithProcessesTestProcess) ... ok
test_current (__main__.WithProcessesTestProcess) ... ok
test_process (__main__.WithProcessesTestProcess) ... ok
test_recursion (__main__.WithProcessesTestProcess) ... ok
test_terminate (__main__.WithProcessesTestProcess) ... ok
test_fork (__main__.WithProcessesTestQueue) ... ok
test_get (__main__.WithProcessesTestQueue) ... ^CTraceback (most recent 
call last):
  File "Lib/test/test_multiprocessing.py", line 1799, in <module>
main()
  File "Lib/test/test_multiprocessing.py", line 1796, in main
test_main(unittest.TextTestRunner(verbosity=2).run)
  File "Lib/test/test_multiprocessing.py", line 1786, in test_main
run(suite)
  File "/Users/dickinsm/python_source/trunk/Lib/unittest.py", line 743, 
in run
test(result)
  File "/Users/dickinsm/python_source/trunk/Lib/unittest.py", line 454, 
in __call__
return self.run(*args, **kwds)
  File "/Users/dickinsm/python_source/trunk/Lib/unittest.py", line 450, 
in run
test(result)
  File "/Users/dickinsm/python_source/trunk/Lib/unittest.py", line 454, 
in __call__
return self.run(*args, **kwds)
  File "/Users/dickinsm/python_source/trunk/Lib/unittest.py", line 450, 
in run
test(result)
  File "/Users/dickinsm/python_source/trunk/Lib/unittest.py", line 293, 
in __call__
return self.run(*args, **kwds)
  File "/Users/dickinsm/python_source/trunk/Lib/unittest.py", line 272, 
in run
testMethod()
  File "Lib/test/test_multiprocessing.py", line 415, in test_get
parent_can_continue.wait()
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/synchronize.py"
, line 292, in wait
self._cond.wait(timeout)
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/synchronize.py"
, line 201, in wait
self._wait_semaphore.acquire(True, timeout)
KeyboardInterrupt
[50284 refs]
Macintosh-3:trunk dickinsm$

I think I narrowed the problem to a race condition in *subclasses* of
threading.local:
In threadmodule.c::local_getattro, there is a chance that self->dict is
changed before PyObject_GenericGetAttr is called.

After 4 hours spent with the debugger I think I have an explanation and
a correction of one of the problems.
Note that this only affects the "Manager" tests.

- The threading.local type overrides getattr and setattr: on every call,
it fetches a dictionary stored in the current ThreadState, and installs
it in its self->dict member, allowing the regular
PyObject_Generic[Get|Set]Attr functions to actually work on thread-local
data.
- Of course, for this to work the GIL must not be released in any way
between the moment self->dict is set, until object's dictionary is
retrieved inside PyObject_Generic[Get|Set]Attr. This condition makes the
code a bit fragile.
With current 2.6b1, there is no path in PyObject_Generic[Get|Set]Attr
that releases the GIL before the dict is retrieved - no decref, no jump
into python code (at least when the attribute name is a string). It's
almost a miracle, indeed.
- The problem is in threadmodule.c, the paragraph starting at line 285:
the first time the local object is accessed, it calls the __init__
method of the threading.local derived class. And this calls python code,
and can release the GIL.
- The consequences are dramatic: this let the possibility for another
thread to start, use the same local object, install its thread-local
dict in self->dict, and stop here; the first thread will wake up with
the wrong dict...
- in the multiprocessing case, a remote call to a synchronization
function grabs the wrong channel, and get bogus results... hence the
"cannot wait on un-aquired (sic) lock" message.

The actual correction is very short: only one "else" to remove... Here
is a patch with a test.

Committed my changes to threading.local as r64601.

I think we could enable the "Manager" tests again.

Alright - I'll do that asap

On Jun 30, 2008, at 7:06 PM, Amaury Forgeot d'Arc <report@bugs.python.org 
 > wrote:

>
> Amaury Forgeot d'Arc <amauryfa@gmail.com> added the comment:
>
> Committed my changes to threading.local as r64601.
>
> I think we could enable the "Manager" tests again.
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue3088>
> _______________________________________

Tests are back on as of r64663 on trunk.

Not sure that all problems reported here are resolved.
For example, "test_get (__main__.WithProcessesTestQueue)" does not seem
to use threading.local at all.

Still hangs for me on revision 64665

Miki where does it hang? I've run it in a tight loop for over 500 
iterations without a single hang.

The g4 buildbot still isn't happy, either.  See:

http://www.python.org/dev/buildbot/trunk.stable/g4%20osx.4%20trunk/builds/
3645/step-test/0

(This was a run of revision 64663, I think.)

The G4 buildbot is green right now.

Looking through the buildbot's history (OSX G4) I can see it hung *prior* 
and after the change to turn the manager tests back on, so it's unrelated 
to those tests. I'll reopen this bug.

> The G4 buildbot is green right now.

Sure.  But it looks like test_multiprocessing hung on build 3645, which 
as far as I can tell included all the recent fixes.

My machine's behaving the same way:  when doing 'make test', sometimes 
test_multiprocessing passes, sometimes (about half the time, maybe a 
little less) it hangs.  Unfortunately, when running test_multiprocessing 
by itself it hardly ever hangs;  and when running it under regrtest.py 
with verbose output it doesn't hang either, so it's a little difficult 
to figure out where things are going wrong.

What platform mark?

Here's an interesting snippet from the my most recent attempt to run 
'make test'.  I got a failed assertion:

[...]
test_multibytecodec_support
test_multifile
test_multiprocessing
Assertion failed: (bp != NULL), function PyObject_Malloc, file 
Objects/obmalloc.c, line 746.
make: *** [test] Abort trap
./python.exe -E -tt ./Lib/test/regrtest.py -l 
test_grammar
test_opcodes
[...]

This is again on OS X 10.5.3;  I built with:

LDFLAGS=-L/opt/local/lib CPPFLAGS=-I/opt/local/include ./configure --
with-pydebug && make

(the /opt/local stuff is there because that's where my readline library 
is...)

Python 2.6b1+ (trunk:64671, Jul  2 2008, 21:51:37) 
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin

> What platform mark?

OS X 10.5.4/Intel Core 2 Duo.  (It's a 2007 Macbook Pro.)

Thanks. I'm a been mystified why it hanging on you - I'm on a 2008 Macbook 
Pro running latest 10.5.4. I'm still trying to get it to hang though.

Mark, can you try commenting out _TestCondition and seeing if you can 
still get it to hang?;

> Mark, can you try commenting out _TestCondition and seeing if you can 
> still get it to hang?;

I removed the _TestCondition class entirely from test_multiprocessing, 
and did make test again.  It didn't hang! :-)  It crashed instead.  :-(
 
The output's below, in all its gory glory.  I can't make head or tail or 
it, but maybe someone else can.  The last two runs before this gave me a 
Segmentation fault.

test_multiprocessing
Debug memory block at address p=0x2ecd928:
Debug memory block at address p=0x2ecd928:
    49 bytes originally requested
    49 bytes originally requested
    The 4 pad bytes at p-4 are     The 4 pad bytes at p-4 are 
FORBIDDENBYTE, as expected.
FORBIDDENBYTE, as expected.
    The 4 pad bytes at tail=0x2ecd959 are     The 4 pad bytes at 
tail=0x2ecd959 are not all FORBIDDENBYTE (0xfb):
not all FORBIDDENBYTE (0xfb):
        at tail+0: 0x7d        at tail+0: 0x7d *** OUCH *** OUCH

        at tail+1: 0x74        at tail+1: 0x74 *** OUCH *** OUCH

        at tail+2: 0x71        at tail+2: 0x71 *** OUCH *** OUCH

        at tail+3: 0x02        at tail+3: 0x02 *** OUCH *** OUCH

    The block was made by call #782281511 to debug malloc/realloc.
    The block was made by call #782281511 to debug malloc/realloc.
    Data at p:    Data at p: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 ... ... 87 87 71 71 02 02 2e 2e 00 00 1c 1c 02 02 85 85

Fatal Python error: bad trailing pad byte
Fatal Python error: bad trailing pad byte
make: *** [test] Abort trap

That output exceeds my knowledge of python internals by several light 
years. :(

On Wed, Jul 2, 2008 at 3:44 PM, Mark Dickinson <report@bugs.python.org> wrote:
>
> Mark Dickinson <dickinsm@gmail.com> added the comment:
>
>> Mark, can you try commenting out _TestCondition and seeing if you can
>> still get it to hang?;
>
> I removed the _TestCondition class entirely from test_multiprocessing,
> and did make test again.  It didn't hang! :-)  It crashed instead.  :-(

Try running "ulimit -c unlimited" in the shell before running the test
(from the same shell).  After it aborts it should dump a core file,
which you can then inspect using "gdb ./python core", to which "bt"
will give you a stack trace ("backtrace").

On a minor note, I'd suggest running "./python -m test.regrtest"
explicitly, rather than "make test".  The latter runs the test suite
twice, deleting all .pyc files before the first run, to detect
problems in their creation.

And please add the "-v" option at the end, so we can see which function
fails

Okay.  I just got about 5 perfect runs of the test suite, followed by:

Macintosh-3:trunk dickinsm$ ./python.exe -m test.regrtest
[...]
test_multiprocessing
Assertion failed: (bp != NULL), function PyObject_Malloc, file 
Objects/obmalloc.c, line 746.
Abort trap (core dumped)

I then did:

gdb -c /cores/core.16235

I've attached the traceback as traceback.txt

On Wed, Jul 2, 2008 at 5:08 PM, Mark Dickinson <report@bugs.python.org> wrote:
>
> Mark Dickinson <dickinsm@gmail.com> added the comment:
>
> Okay.  I just got about 5 perfect runs of the test suite, followed by:
>
> Macintosh-3:trunk dickinsm$ ./python.exe -m test.regrtest
> [...]
> test_multiprocessing
> Assertion failed: (bp != NULL), function PyObject_Malloc, file
> Objects/obmalloc.c, line 746.
> Abort trap (core dumped)
>
> I then did:
>
> gdb -c /cores/core.16235
>
> I've attached the traceback as traceback.txt

Are you sure that's right?  That traceback has no mention of
PyObject_Malloc or obmalloc.c.  Try checking the date.  Also, if  you
use "gdb ./python.exe <corefile>" to start gdb it should print a
warning if the program doesn't match the core.

I just run "make test" and it never moves past test_multiprocessing.

Maybe it's my machine which is dual cpu quad core (total of 8 cores)?

Doubtful Miki - I do the work on the module on an 8 Core Gentoo, 8 Core 
Mac Pro and Dual Core Macbook Pro - it's not a # of cores issue, unless 
it's simply a >1 issue.

> Are you sure that's right?

Not at all.  :-)

> That traceback has no mention of
> PyObject_Malloc or obmalloc.c.  Try checking the date.  Also, if  you
> use "gdb ./python.exe <corefile>" to start gdb it should print a
> warning if the program doesn't match the core.

The date and time on the core file look right (Jul 2, 23:52 GMT+1), and 
gdb ./python.exe ... doesn't give any warning.  So I'm not sure what I 
did wrong.  I'll try again and see if I get the same thing.

Here's a new traceback (a different error again, this time: a negative 
refcount in Objects/tupleobject.c.)

That looks better.  It crashed while deleting an exception, who's args
tuple has a bogus refcount.  Could be a refcount issue of the
exception or the args, or of something that that references them, or a
dangling pointer, or a buffer overrun, etc.

Things to try:
1) Run "pystack" in gdb, from Misc/gdbinit
2) Print the exception type.  Use "up" until you reach
BaseException_clear, then do "print self->ob_type->tp_name".  Also do
"print *self" and make sure the ob_refcnt is at 0 and the other fields
look sane.
3) Compile using --without-pymalloc and throw it at a real memory
debugger.  I'd suggest starting with your libc's own debugging
options, as they tend to be less invasive:
http://developer.apple.com/documentation/Performance/Conceptual/ManagingMemory/Articles/MallocDebug.html
.  If that doesn't work, look at Electric Fence, Valgrind, or your
tool of choice.

Also, make sure you do a "make clean" since you last updated the tree or
touched any file or ran configure.  The automatic dependency checking
isn't 100% reliable.

Barring the segfaults Mark is seeing, I went through and removed all of 
the tests, and I have been incrementally adding them back one by one. _TestQueue seems to be the one (at least, the first) which is hanging 
intermittently in a racquire(). If anyone else who is having hangs minds, 
please try removing _TestQueue and see if you can still get it to hang.

The two tracebacks provided by Mark seem to correspond to the following
python stack (innermost last):

Lib/test/test_multiprocessing.py, line 1005, in _test_map_unordered
    self.assertEqual(sorted(it), map(sqr, range(1000)))
Lib/multiprocessing/pool.py, line 500, in IMapIterator.next()
    self._cond.acquire()
Lib/threading.py, line 123, in _RLock.acquire():
    rc = self.__block.acquire(blocking)

The test hanged for me at first try but worked fine on the second test,
weird.

On a Linux system (FC4) with r64686 of the Py3k branch I also still get
occassional hangs (with ./python -E -bb ./Lib/test/regrtest.py -v
test_multiprocessing). Mostly this seems to occur with the very first
test executed, i.e. before any of the "test_... " lines have been generated.

The following may or may not be related. Some time ago I decided to give
valgrind a try to see if it could detect anything strange going on with
the multiprocessing tests, specifically using the 'helgrind'
thread-debugging tool that comes with it. 

Valgrind reports as its first error:

==9719== Thread #1: Bug in libpthread: sem_wait succeeded on semaphore
without prior sem_post
==9719==    at 0x4007FFF: sem_wait_WRK (hg_intercepts.c:1057)
==9719==    by 0x4008094: sem_wait@* (hg_intercepts.c:1073)
==9719==    by 0x46A0087: semlock_acquire (semaphore.c:310)
==9719==    by 0x808C121: PyEval_EvalFrameEx (ceval.c:3371)
==9719==    by 0x808D0FE: PyEval_EvalCodeEx (ceval.c:2808)
==9719==    by 0x808B9D0: PyEval_EvalFrameEx (ceval.c:3469)
==9719==    by 0x808D0FE: PyEval_EvalCodeEx (ceval.c:2808)
==9719==    by 0x80F4B65: function_call (funcobject.c:628)
==9719==    by 0x80D1207: PyObject_Call (abstract.c:2178)
==9719==    by 0x80890EC: PyEval_EvalFrameEx (ceval.c:3672)
==9719==    by 0x808C1A9: PyEval_EvalFrameEx (ceval.c:3459)
==9719==    by 0x808C1A9: PyEval_EvalFrameEx (ceval.c:3459)
==9716== Thread #1 is the program's root thread

I've been hesitant to report this as the claim that libpthread is broken
is pretty bold. I contacted the valgrind devs about this, see [1]. 
More recently, someone on the valgrind list reported problems that do
seem to indicate there are broken libpthreads out there (see [2]), as
this individual reports a semaphore wait not blocking where it should.

Could it be that the multiprocessing tests are exposing one or more bugs
in libpthread?

[1] http://thread.gmane.org/gmane.comp.debugging.valgrind/8345
[2] http://thread.gmane.org/gmane.comp.debugging.valgrind/8384

> Are you sure that's right?  That traceback has no mention of
> PyObject_Malloc or obmalloc.c.

So now I think that the traceback was right.  There was no mention of 
PyObject_Malloc or obmalloc.c because the traceback only showed 1 of the 9 
threads, and the failed assert occurred in a different thread.

I've attached another traceback, showing all the threads, and applying 'tb 
full' to the relevant thread.  (This was from a different run, but with 
the same failed assertion at line 746 of Objects/obmalloc.c.)

I managed to hang on Ubuntu, here is the backtrace that I got with CTRL-C:

Process PoolWorker-5:1:
Traceback (most recent call last):
  File "/home/cartman/Sources/py3k/Lib/multiprocessing/process.py", line
232, in _bootstrap
    test_bsddb test_bsddb3 test_cProfile test_kqueue test_lib2to3
2 skips unexpected on linux2:
    test_bsddb3 test_bsddb
Process PoolWorker-5:3:
Traceback (most recent call last):
  File "/home/cartman/Sources/py3k/Lib/multiprocessing/process.py", line
232, in _bootstrap
    self.run()
  File "/home/cartman/Sources/py3k/Lib/multiprocessing/process.py", line
88, in run
    self._target(*self._args, **self._kwargs)
  File "/home/cartman/Sources/py3k/Lib/multiprocessing/pool.py", line
57, in worker
    self.run()
  File "/home/cartman/Sources/py3k/Lib/multiprocessing/process.py", line
88, in run
    self._target(*self._args, **self._kwargs)
  File "/home/cartman/Sources/py3k/Lib/multiprocessing/pool.py", line
57, in worker
    task = get()
  File "/home/cartman/Sources/py3k/Lib/multiprocessing/queues.py", line
339, in get
    task = get()
  File "/home/cartman/Sources/py3k/Lib/multiprocessing/queues.py", line
337, in get
    return recv()
  File "/home/cartman/Sources/py3k/Lib/pickle.py", line 1327, in loads
    racquire()
KeyboardInterrupt
Process PoolWorker-5:2:
Traceback (most recent call last):
  File "/home/cartman/Sources/py3k/Lib/multiprocessing/process.py", line
232, in _bootstrap
    self.run()
  File "/home/cartman/Sources/py3k/Lib/multiprocessing/process.py", line
88, in run
    self._target(*self._args, **self._kwargs)
  File "/home/cartman/Sources/py3k/Lib/multiprocessing/pool.py", line
57, in worker
    def loads(s, *, encoding="ASCII", errors="strict"):
KeyboardInterrupt
Process PoolWorker-5:4:
Traceback (most recent call last):
  File "/home/cartman/Sources/py3k/Lib/multiprocessing/process.py", line
232, in _bootstrap
    self.run()
  File "/home/cartman/Sources/py3k/Lib/multiprocessing/process.py", line
88, in run
    self._target(*self._args, **self._kwargs)
  File "/home/cartman/Sources/py3k/Lib/multiprocessing/pool.py", line
57, in worker
    task = get()
  File "/home/cartman/Sources/py3k/Lib/multiprocessing/queues.py", line
337, in get
    racquire()
KeyboardInterrupt
    task = get()
  File "/home/cartman/Sources/py3k/Lib/multiprocessing/queues.py", line
337, in get
    racquire()
KeyboardInterrupt
^CError in atexit._run_exitfuncs:
make: *** [testall] Segmentation fault

I found that on my Debian64, running test_multiprocessing under gdb
hangs even before the first test is started - somewhere in the
installation of the Manager.

And it appears that the problem is described in issue874900: "threading
module can deadlock after fork".
I don't know if it's a good idea to mix fork and threads, but the patch
I attached to issue874900 seems to correct the hang.

Thanks Amaury - I've been working through the tests and identifying
the "problem children" - I'll finish that up and then try re-running
them with the 874900 patch.

I'm still seeing intermittent lockups on Ubuntu 7.10 - traceback on
ctrl-C is similar to that posted by Ismail above.

Since Jesse seems to be on top of this, I'll stick to using -x
test_multiprocessing for the moment.

Bumping back to release blocker for beta 2 (Barry may choose to defer it
again, but it should at least be on his radar).

Updated issue title to more accurately reflect scope of the problem.

I forgot to mention that I am seeing the intermittent hangs on the trunk
(2.6). I haven't been testing it on Py3k.

Sadly _multiprocessing apparently doesn't even build on my Ubuntu 8.04
machine and it still hangs on my 10.5 machine.

On Jul 15, 2008, at 8:38 PM, "Barry A. Warsaw"  
<report@bugs.python.org> wrote:

>
> Barry A. Warsaw <barry@python.org> added the comment:
>
> Sadly _multiprocessing apparently doesn't even build on my Ubuntu 8.04
> machine and it still hangs on my 10.5 machine.
>

There is no reason it shouldn't compile on ubuntu - without the patch  
for the bug I added as a dependency, we will keep seeing hangs

Barry - can you email the compile errors?

Something's very strange.  The first make after configure fails to build
_multiprocessing, but a subsequent make succeeds.  I'll see if I can
capture the compilation error message.

Here's the 'make' output.  What's strange is that moving
_multiprocessing{_failed,}.so, the import works just fine.

building '_multiprocessing' extension
creating
build/temp.linux-i686-3.0/home/barry/projects/python/python30/Modules/_multiprocessing
gcc -pthread -fPIC -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall
-Wstrict-prototypes -DHAVE_SEM_OPEN=1 -DHAVE_FD_TRANSFER=1
-DHAVE_SEM_TIMEDWAIT=1 -IModules/_multiprocessing -I.
-I/home/barry/projects/python/python30/./Include -I. -IInclude
-I./Include -I/usr/local/include
-I/home/barry/projects/python/python30/Include
-I/home/barry/projects/python/python30 -c
/home/barry/projects/python/python30/Modules/_multiprocessing/multiprocessing.c
-o
build/temp.linux-i686-3.0/home/barry/projects/python/python30/Modules/_multiprocessing/multiprocessing.o
gcc -pthread -fPIC -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall
-Wstrict-prototypes -DHAVE_SEM_OPEN=1 -DHAVE_FD_TRANSFER=1
-DHAVE_SEM_TIMEDWAIT=1 -IModules/_multiprocessing -I.
-I/home/barry/projects/python/python30/./Include -I. -IInclude
-I./Include -I/usr/local/include
-I/home/barry/projects/python/python30/Include
-I/home/barry/projects/python/python30 -c
/home/barry/projects/python/python30/Modules/_multiprocessing/socket_connection.c
-o
build/temp.linux-i686-3.0/home/barry/projects/python/python30/Modules/_multiprocessing/socket_connection.o
gcc -pthread -fPIC -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall
-Wstrict-prototypes -DHAVE_SEM_OPEN=1 -DHAVE_FD_TRANSFER=1
-DHAVE_SEM_TIMEDWAIT=1 -IModules/_multiprocessing -I.
-I/home/barry/projects/python/python30/./Include -I. -IInclude
-I./Include -I/usr/local/include
-I/home/barry/projects/python/python30/Include
-I/home/barry/projects/python/python30 -c
/home/barry/projects/python/python30/Modules/_multiprocessing/semaphore.c -o
build/temp.linux-i686-3.0/home/barry/projects/python/python30/Modules/_multiprocessing/semaphore.o
gcc -pthread -shared
build/temp.linux-i686-3.0/home/barry/projects/python/python30/Modules/_multiprocessing/multiprocessing.o
build/temp.linux-i686-3.0/home/barry/projects/python/python30/Modules/_multiprocessing/socket_connection.o
build/temp.linux-i686-3.0/home/barry/projects/python/python30/Modules/_multiprocessing/semaphore.o
-L/usr/local/lib -o build/lib.linux-i686-3.0/_multiprocessing.so
*** WARNING: renaming "_multiprocessing" since importing it failed: No
module named _multiprocessing

Issue 874900's patch seems to have resolve the hangs. I am closing this 
issue as fixed.

I confirm this is solved for me in beta 2