classification
Title: test_multiprocessing hangs intermittently on POSIX platforms
Type: Stage:
Components: Library (Lib) Versions: Python 3.0, Python 2.6
process
Status: closed Resolution: fixed
Dependencies: 874900 Superseder:
Assigned To: jnoller Nosy List: Rhamphoryncus, amaury.forgeotdarc, barry, benjamin.peterson, cartman, gvanrossum, jnoller, mark.dickinson, ncoghlan, paulmelis, roudkerk, tebeka
Priority: release blocker Keywords: patch

Created on 2008-06-12 04:48 by barry, last changed 2008-07-18 14:43 by tebeka. This issue is now closed.

Files
File name Uploaded Description Edit
test_get.output jnoller, 2008-06-12 19:56
test_multiprocessing_reduced.diff jnoller, 2008-06-15 15:20
threadlocal.patch amaury.forgeotdarc, 2008-06-28 23:41
traceback.txt mark.dickinson, 2008-07-02 23:08 Traceback from test_multiprocessing crash
traceback2.txt mark.dickinson, 2008-07-03 00:44 A second traceback
multithread_traceback.txt mark.dickinson, 2008-07-03 10:06 Traceback showing all threads
Messages (73)
msg68053 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2008-06-12 04:48
For me, test_multiprocessing hangs consistently on OS X 10.5.3.  It
passes just fine on Ubuntu 8.04.
msg68055 - (view) Author: Ismail Donmez (cartman) Date: 2008-06-12 05:23
I can confirm this on Leopard too.
msg68059 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2008-06-12 13:02
It passes for me on Leopard. Can you post the test running in verbose
mode so we can see where it hangs?
msg68062 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-06-12 13:18
On python-3000 trunk, _multiprocessing doesn't even compile:

/Users/jesse/open_source/subversion/python-
3000/Modules/_multiprocessing/semaphore.c: In function ‘semlock_iszero’:
/Users/jesse/open_source/subversion/python-
3000/Modules/_multiprocessing/semaphore.c:515: warning: unused variable 
‘sval’
msg68063 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2008-06-12 13:19
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Jun 12, 2008, at 9:02 AM, Benjamin Peterson wrote:

> It passes for me on Leopard. Can you post the test running in verbose
> mode so we can see where it hangs?

It never hangs when run standalone, though it crashes about half the  
time.  Running it under gdb doesn't help; it always gives me an  
interrupted syste call in os.waidpid() in forking.py.

The hang occurs during 'make test', and it's always the second run  
that hangs.  Perhaps some lock isn't getting cleaned up properly the  
first time?

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSFEicXEjvBPtnXfVAQLGNwP/S6f2IrO7c7SET0Gx8FXqdPmot3jcmopx
TFxDA5csh/pVaDQCVW6DiLMXsu2TkQGPPbbo8Bx9iPmV/iIHFqy4nDtETqqjKdRp
BvVtBmvSrP6wmymlKFlFC5qdfbbvguZq/hO60XulQk+WU4F8N7oHQck0tA2JhdDh
lS5SAFAIovA=
=xzs6
-----END PGP SIGNATURE-----
msg68064 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-06-12 13:25
I did a make clean && ./configure && make and it started compiling for me 
again. Sorry for the noise.
msg68065 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2008-06-12 14:07
If it's only failing during the second run of "make test", typically
there's some implicit dependency on something that is disturbed by
running a test that's later in the suite of tests.  This could be either
the fault of that other test (not restoring some global setting or
environment var) or the fault of the test that fails (making unwarranted
assumptions or not initializing some needed settings before starting).

If it works for some folks and not for others, on the same platform,
compare the set of extension modules that are not built, reported by
"make" in a message starting with "Failed to find the necessary bits to
build these modules:". Likely, Barry has the most complete set, while
Jesse has a few more extensions missing.

Finding this is usually a painful process of bisecting the set of tests
run.  Randomizing the tests with regrtest.py -r might also be helpful.

FWIW, when I tried (on Leopard) "make test
TESTOPTS=test_multiprocessing" it hung on the first set.  When I ran
"./python Lib/test/test_multiprocess.py -v" it reported 122 tests
passed.  But when I ran "./python Lib/test/regrtest.py -v
test_multiprocessing" one test failed:

======================================================================
ERROR: test_remote (test.test_multiprocessing.WithManagerTestRemoteManager)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/guido/p/Lib/test/test_multiprocessing.py", line 1167, in
test_remote
    queue = manager2.get_queue()
  File "/Users/guido/p/Lib/multiprocessing/managers.py", line 650, in temp
    authkey=self._authkey, exposed=exp
  File "/Users/guido/p/Lib/multiprocessing/managers.py", line 902, in
AutoProxy
    incref=incref)
  File "/Users/guido/p/Lib/multiprocessing/managers.py", line 711, in
__init__
    self._incref()
  File "/Users/guido/p/Lib/multiprocessing/managers.py", line 758, in
_incref
    dispatch(conn, None, 'incref', (self._id,))
  File "/Users/guido/p/Lib/multiprocessing/managers.py", line 94, in
dispatch
    raise convert_to_error(kind, result)
RemoteError: 
---------------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/guido/p/Lib/multiprocessing/managers.py", line 196, in
handle_request
    result = func(c, *args, **kwds)
  File "/Users/guido/p/Lib/multiprocessing/managers.py", line 412, in incref
    self.id_to_refcount[ident] += 1
KeyError: '5f2828'
---------------------------------------------------------------------------
msg68066 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2008-06-12 14:08
I should add this was in the trunk (2.6).
msg68067 - (view) Author: Paul Melis (paulmelis) Date: 2008-06-12 15:16
I think I'm having a similar lockup on fedora core 4 (smp machine). This
is with the py3k branch, freshly svn updated. When running "make test
TESTOPTS=test_multiprocessing" the first of the two test runs always
succeeds in something like 10-15 seconds, while the second one
occasionally hangs. I tried running with -v as extra argument but can't
get it to hang in that case (to figure out which test is the culprit).
msg68068 - (view) Author: Paul Melis (paulmelis) Date: 2008-06-12 15:25
After a few more runs with -v and redirecting output to a file it seems
the lockup I get is in 
test_notify_all (test.test_multiprocessing.WithManagerTestCondition)
msg68072 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-06-12 18:40
It's taking me longer to get to this than I planned, any help is 
appreciated.
msg68077 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-06-12 19:56
I can get an intermittent (1 every 15 or so runs) lock in:
 test_get (__main__.WithProcessesTestQueue) ... 

Executed like this:
./python Lib/test/test_multiprocessing.py

When I control-c it the stack looks like this:
...snip
  File "/root/py/python-3000/Lib/multiprocessing/pool.py", line 57, in 
worker
    task = get()
  File "/root/py/python-3000/Lib/multiprocessing/queues.py", line 337, 
in get
    task = get()
  File "/root/py/python-3000/Lib/multiprocessing/queues.py", line 339, 
in get
    racquire()
KeyboardInterrupt
    task = get()
  File "/root/py/python-3000/Lib/multiprocessing/queues.py", line 337, 
in get
    task = get()
  File "/root/py/python-3000/Lib/multiprocessing/queues.py", line 337, 
in get
    return recv()
  File "/root/py/python-3000/Lib/pickle.py", line 1327, in loads
    racquire()
KeyboardInterrupt
    racquire()
KeyboardInterrupt
    def loads(s, *, encoding="ASCII", errors="strict"):
KeyboardInterrupt


I'm not seeing frequent locks/failures when run with regrtest, but I am 
seeing them with "make test TESTOPTS=test_multiprocessing"

I've attached full output. Still trying to figure it out
msg68136 - (view) Author: Paul Melis (paulmelis) Date: 2008-06-13 07:59
I made a copy of test_multiprocessing.py (to test_mp.py) and basically
removed all test classes, except _TestCondition. In it, I commented all
test methods except test_notify_all. When run with make test
TESTOPTS="-v test_mp" there's the lockups every few runs, but I also got
(only three times in 40 or so runs so far):


9:57|paul@tabu:~/c/py3k-svn> make test TESTOPTS="-v test_mp"

Failed to find the necessary bits to build these modules:
_gestalt
To find the necessary bits, look in setup.py in detect_modules() for the
module's name.

find ./Lib -name '*.py[co]' -print | xargs rm -f
./python -E -bb ./Lib/test/regrtest.py -v test_mp
test_mp
test_notify_all (test.test_mp.WithProcessesTestCondition) ... ok
test_notify_all (test.test_mp.WithThreadsTestCondition) ... ok
test_notify_all (test.test_mp.WithManagerTestCondition) ... ok

----------------------------------------------------------------------
Ran 3 tests in 1.087s

OK
1 test OK.
CAUTION:  stdout isn't compared in verbose mode:
a test that passes in verbose mode may fail without it.
./python -E -bb ./Lib/test/regrtest.py -v test_mp
test_mp
test_notify_all (test.test_mp.WithProcessesTestCondition) ... ok
test_notify_all (test.test_mp.WithThreadsTestCondition) ... ok
test_notify_all (test.test_mp.WithManagerTestCondition) ... Exception in
thread Thread-28:
Traceback (most recent call last):
  File "/home/paul/c/py3k-svn/Lib/threading.py", line 492, in
_bootstrap_inner
    self.run()
  File "/home/paul/c/py3k-svn/Lib/threading.py", line 447, in run
    self._target(*self._args, **self._kwargs)
  File "/home/paul/c/py3k-svn/Lib/test/test_mp.py", line 208, in f
    cond.wait(timeout)
  File "/home/paul/c/py3k-svn/Lib/multiprocessing/managers.py", line
973, in wait
    return self._callmethod('wait', (timeout,))
  File "/home/paul/c/py3k-svn/Lib/multiprocessing/managers.py", line
748, in _callmethod
    raise convert_to_error(kind, result)
RuntimeError: cannot wait on un-aquired lock
msg68150 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-06-13 12:38
FWIW: In order to boost the logging level within the test(s) do the 
following:

Search for LOG_LEVEL, set it to:
LOG_LEVEL=util.SUBDEBUG

And then in the main() replace:
    multiprocessing.get_logger().setLevel(LOG_LEVEL)
With:
multiprocessing.log_to_stderr(level=LOG_LEVEL)
msg68153 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-06-13 13:17
I also isolated the test(s) like Paul did, and it looks like a semi-
consistent lock up in:
  File "/root/py/python-3000/Lib/multiprocessing/queues.py", line 337, in 
get
    racquire()

This is running only the test_event test. The racquire traces back to 
synchronize.SemLock which calls into _multiprocessing.SemLock
msg68198 - (view) Author: Ismail Donmez (cartman) Date: 2008-06-14 06:43
Seems to work fine for me now with latest py3k branch.
msg68209 - (view) Author: roudkerk (roudkerk) Date: 2008-06-14 17:35
I suspect the problems with WithManagerTestCondition.notify_all() may
have to do with the thread safety of the proxies.  If you replace
Thread(...) by self.Process(...) in that test then the problem may go away.
msg68237 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-06-15 15:20
After talking with Richard, I think the best way to attack this issue 
(and the other ones around suite unreliability) is to remove the 
unreliable test cases for the first beta, and then refactor the suite 
post beta with an eye towards reliability and clarity. Personally, I 
would like to break the suites up in the the test_multiprocessing.py 
script to be more in the vein of other tests in Lib/test/...

I removed the more unreliable test cases while keeping the core ones and 
wrote a quick bash script to do a burn-in of the "make tests" command, 
for 100 loops, I was unable to get the tests to hang with the reduced 
suite.

I ran the same thing on trunk and py3k just to make sure I could not get 
it to hang/crash.
msg68240 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-06-15 15:26
Here is the loop I ran the tests with:

#!/bin/sh

for (( i=1;i<=100;i+=1 )); do
    make test TESTOPTS="-v test_multiprocessing"        
done
msg68295 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-06-16 20:37
I don't have commit rights, so I can't apply the test_multiprocessing_reduced.diff myself. Anyone willing? I think this 
should help the buildbots.
msg68384 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2008-06-19 00:21
I'm going to knock this one down to critical since it's working for me
now on OS X and buildbot looks green.  We can address any additional
patches after the beta release.
msg68466 - (view) Author: Miki Tebeka (tebeka) * Date: 2008-06-20 15:41
Still hangs for me on the 2.6 trunk on Ubuntu 8.04
msg68467 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-06-20 15:46
Where exactly does it hang Miki?
msg68485 - (view) Author: Miki Tebeka (tebeka) * Date: 2008-06-20 23:25
Jesse,

I just run "make test", it runs until test_multiprocessing and then
hangs there
msg68856 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2008-06-27 21:04
test_multiprocessing is also still hanging for me, perhaps 30% of the 
times I run the test suite.

When running the test by itself it seems to pass much more often, but 
not always.  I just got the following output (on OS X 10.5.3/Intel).
There was a hang at test_get;  after around half-an-hour I hit Ctrl-C.

Macintosh-3:trunk dickinsm$ ./python.exe 
Lib/test/test_multiprocessing.py
test_array (__main__.WithProcessesTestArray) ... ok
test_getobj_getlock_obj (__main__.WithProcessesTestArray) ... ok
test_rawarray (__main__.WithProcessesTestArray) ... ok
test_notify (__main__.WithProcessesTestCondition) ... ok
test_notify_all (__main__.WithProcessesTestCondition) ... ok
test_timeout (__main__.WithProcessesTestCondition) ... ok
test_connection (__main__.WithProcessesTestConnection) ... ok
test_duplex_false (__main__.WithProcessesTestConnection) ... ok
test_sendbytes (__main__.WithProcessesTestConnection) ... ok
test_spawn_close (__main__.WithProcessesTestConnection) ... ok
test_event (__main__.WithProcessesTestEvent) ... ok
test_finalize (__main__.WithProcessesTestFinalize) ... ok
test_heap (__main__.WithProcessesTestHeap) ... ok
test_import (__main__.WithProcessesTestImportStar) ... ok
test_lock (__main__.WithProcessesTestLock) ... ok
test_rlock (__main__.WithProcessesTestLock) ... ok
test_enable_logging (__main__.WithProcessesTestLogging) ... ok
test_level (__main__.WithProcessesTestLogging) ... ok
test_active_children (__main__.WithProcessesTestProcess) ... ok
test_cpu_count (__main__.WithProcessesTestProcess) ... ok
test_current (__main__.WithProcessesTestProcess) ... ok
test_process (__main__.WithProcessesTestProcess) ... ok
test_recursion (__main__.WithProcessesTestProcess) ... ok
test_terminate (__main__.WithProcessesTestProcess) ... ok
test_fork (__main__.WithProcessesTestQueue) ... ok
test_get (__main__.WithProcessesTestQueue) ... ^CTraceback (most recent 
call last):
  File "Lib/test/test_multiprocessing.py", line 1799, in <module>
main()
  File "Lib/test/test_multiprocessing.py", line 1796, in main
test_main(unittest.TextTestRunner(verbosity=2).run)
  File "Lib/test/test_multiprocessing.py", line 1786, in test_main
run(suite)
  File "/Users/dickinsm/python_source/trunk/Lib/unittest.py", line 743, 
in run
test(result)
  File "/Users/dickinsm/python_source/trunk/Lib/unittest.py", line 454, 
in __call__
return self.run(*args, **kwds)
  File "/Users/dickinsm/python_source/trunk/Lib/unittest.py", line 450, 
in run
test(result)
  File "/Users/dickinsm/python_source/trunk/Lib/unittest.py", line 454, 
in __call__
return self.run(*args, **kwds)
  File "/Users/dickinsm/python_source/trunk/Lib/unittest.py", line 450, 
in run
test(result)
  File "/Users/dickinsm/python_source/trunk/Lib/unittest.py", line 293, 
in __call__
return self.run(*args, **kwds)
  File "/Users/dickinsm/python_source/trunk/Lib/unittest.py", line 272, 
in run
testMethod()
  File "Lib/test/test_multiprocessing.py", line 415, in test_get
parent_can_continue.wait()
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/synchronize.py"
, line 292, in wait
self._cond.wait(timeout)
  File 
"/Users/dickinsm/python_source/trunk/Lib/multiprocessing/synchronize.py"
, line 201, in wait
self._wait_semaphore.acquire(True, timeout)
KeyboardInterrupt
[50284 refs]
Macintosh-3:trunk dickinsm$
msg68877 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2008-06-28 15:16
I think I narrowed the problem to a race condition in *subclasses* of
threading.local:
In threadmodule.c::local_getattro, there is a chance that self->dict is
changed before PyObject_GenericGetAttr is called.
msg68929 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2008-06-28 23:41
After 4 hours spent with the debugger I think I have an explanation and
a correction of one of the problems.
Note that this only affects the "Manager" tests.

- The threading.local type overrides getattr and setattr: on every call,
it fetches a dictionary stored in the current ThreadState, and installs
it in its self->dict member, allowing the regular
PyObject_Generic[Get|Set]Attr functions to actually work on thread-local
data.
- Of course, for this to work the GIL must not be released in any way
between the moment self->dict is set, until object's dictionary is
retrieved inside PyObject_Generic[Get|Set]Attr. This condition makes the
code a bit fragile.
With current 2.6b1, there is no path in PyObject_Generic[Get|Set]Attr
that releases the GIL before the dict is retrieved - no decref, no jump
into python code (at least when the attribute name is a string). It's
almost a miracle, indeed.
- The problem is in threadmodule.c, the paragraph starting at line 285:
the first time the local object is accessed, it calls the __init__
method of the threading.local derived class. And this calls python code,
and can release the GIL.
- The consequences are dramatic: this let the possibility for another
thread to start, use the same local object, install its thread-local
dict in self->dict, and stop here; the first thread will wake up with
the wrong dict...
- in the multiprocessing case, a remote call to a synchronization
function grabs the wrong channel, and get bogus results... hence the
"cannot wait on un-aquired (sic) lock" message.

The actual correction is very short: only one "else" to remove... Here
is a patch with a test.
msg69025 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2008-06-30 23:06
Committed my changes to threading.local as r64601.

I think we could enable the "Manager" tests again.
msg69026 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-06-30 23:11
Alright - I'll do that asap

On Jun 30, 2008, at 7:06 PM, Amaury Forgeot d'Arc <report@bugs.python.org 
 > wrote:

>
> Amaury Forgeot d'Arc <amauryfa@gmail.com> added the comment:
>
> Committed my changes to threading.local as r64601.
>
> I think we could enable the "Manager" tests again.
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue3088>
> _______________________________________
msg69106 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-07-02 16:48
Tests are back on as of r64663 on trunk.
msg69108 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2008-07-02 16:53
Not sure that all problems reported here are resolved.
For example, "test_get (__main__.WithProcessesTestQueue)" does not seem
to use threading.local at all.
msg69109 - (view) Author: Miki Tebeka (tebeka) * Date: 2008-07-02 17:54
Still hangs for me on revision 64665
msg69110 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-07-02 18:01
Miki where does it hang? I've run it in a tight loop for over 500 
iterations without a single hang.
msg69119 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2008-07-02 20:48
The g4 buildbot still isn't happy, either.  See:

http://www.python.org/dev/buildbot/trunk.stable/g4%20osx.4%20trunk/builds/
3645/step-test/0

(This was a run of revision 64663, I think.)
msg69120 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-07-02 20:51
The G4 buildbot is green right now.
msg69121 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-07-02 20:57
Looking through the buildbot's history (OSX G4) I can see it hung *prior* 
and after the change to turn the manager tests back on, so it's unrelated 
to those tests. I'll reopen this bug.
msg69122 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2008-07-02 21:00
> The G4 buildbot is green right now.

Sure.  But it looks like test_multiprocessing hung on build 3645, which 
as far as I can tell included all the recent fixes.

My machine's behaving the same way:  when doing 'make test', sometimes 
test_multiprocessing passes, sometimes (about half the time, maybe a 
little less) it hangs.  Unfortunately, when running test_multiprocessing 
by itself it hardly ever hangs;  and when running it under regrtest.py 
with verbose output it doesn't hang either, so it's a little difficult 
to figure out where things are going wrong.
msg69123 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-07-02 21:02
What platform mark?
msg69124 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2008-07-02 21:05
Here's an interesting snippet from the my most recent attempt to run 
'make test'.  I got a failed assertion:

[...]
test_multibytecodec_support
test_multifile
test_multiprocessing
Assertion failed: (bp != NULL), function PyObject_Malloc, file 
Objects/obmalloc.c, line 746.
make: *** [test] Abort trap
./python.exe -E -tt ./Lib/test/regrtest.py -l 
test_grammar
test_opcodes
[...]

This is again on OS X 10.5.3;  I built with:

LDFLAGS=-L/opt/local/lib CPPFLAGS=-I/opt/local/include ./configure --
with-pydebug && make

(the /opt/local stuff is there because that's where my readline library 
is...)

Python 2.6b1+ (trunk:64671, Jul  2 2008, 21:51:37) 
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
msg69125 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2008-07-02 21:07
> What platform mark?

OS X 10.5.4/Intel Core 2 Duo.  (It's a 2007 Macbook Pro.)
msg69127 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-07-02 21:11
Thanks. I'm a been mystified why it hanging on you - I'm on a 2008 Macbook 
Pro running latest 10.5.4. I'm still trying to get it to hang though.
msg69128 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-07-02 21:18
Mark, can you try commenting out _TestCondition and seeing if you can 
still get it to hang?;
msg69130 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2008-07-02 21:44
> Mark, can you try commenting out _TestCondition and seeing if you can 
> still get it to hang?;

I removed the _TestCondition class entirely from test_multiprocessing, 
and did make test again.  It didn't hang! :-)  It crashed instead.  :-(
 
The output's below, in all its gory glory.  I can't make head or tail or 
it, but maybe someone else can.  The last two runs before this gave me a 
Segmentation fault.

test_multiprocessing
Debug memory block at address p=0x2ecd928:
Debug memory block at address p=0x2ecd928:
    49 bytes originally requested
    49 bytes originally requested
    The 4 pad bytes at p-4 are     The 4 pad bytes at p-4 are 
FORBIDDENBYTE, as expected.
FORBIDDENBYTE, as expected.
    The 4 pad bytes at tail=0x2ecd959 are     The 4 pad bytes at 
tail=0x2ecd959 are not all FORBIDDENBYTE (0xfb):
not all FORBIDDENBYTE (0xfb):
        at tail+0: 0x7d        at tail+0: 0x7d *** OUCH *** OUCH

        at tail+1: 0x74        at tail+1: 0x74 *** OUCH *** OUCH

        at tail+2: 0x71        at tail+2: 0x71 *** OUCH *** OUCH

        at tail+3: 0x02        at tail+3: 0x02 *** OUCH *** OUCH

    The block was made by call #782281511 to debug malloc/realloc.
    The block was made by call #782281511 to debug malloc/realloc.
    Data at p:    Data at p: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 ... ... 87 87 71 71 02 02 2e 2e 00 00 1c 1c 02 02 85 85

Fatal Python error: bad trailing pad byte
Fatal Python error: bad trailing pad byte
make: *** [test] Abort trap
msg69131 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-07-02 21:50
That output exceeds my knowledge of python internals by several light 
years. :(
msg69133 - (view) Author: Adam Olsen (Rhamphoryncus) Date: 2008-07-02 22:00
On Wed, Jul 2, 2008 at 3:44 PM, Mark Dickinson <report@bugs.python.org> wrote:
>
> Mark Dickinson <dickinsm@gmail.com> added the comment:
>
>> Mark, can you try commenting out _TestCondition and seeing if you can
>> still get it to hang?;
>
> I removed the _TestCondition class entirely from test_multiprocessing,
> and did make test again.  It didn't hang! :-)  It crashed instead.  :-(

Try running "ulimit -c unlimited" in the shell before running the test
(from the same shell).  After it aborts it should dump a core file,
which you can then inspect using "gdb ./python core", to which "bt"
will give you a stack trace ("backtrace").

On a minor note, I'd suggest running "./python -m test.regrtest"
explicitly, rather than "make test".  The latter runs the test suite
twice, deleting all .pyc files before the first run, to detect
problems in their creation.
msg69138 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2008-07-02 22:33
And please add the "-v" option at the end, so we can see which function
fails
msg69142 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2008-07-02 23:08
Okay.  I just got about 5 perfect runs of the test suite, followed by:

Macintosh-3:trunk dickinsm$ ./python.exe -m test.regrtest
[...]
test_multiprocessing
Assertion failed: (bp != NULL), function PyObject_Malloc, file 
Objects/obmalloc.c, line 746.
Abort trap (core dumped)

I then did:

gdb -c /cores/core.16235

I've attached the traceback as traceback.txt
msg69149 - (view) Author: Adam Olsen (Rhamphoryncus) Date: 2008-07-02 23:55
On Wed, Jul 2, 2008 at 5:08 PM, Mark Dickinson <report@bugs.python.org> wrote:
>
> Mark Dickinson <dickinsm@gmail.com> added the comment:
>
> Okay.  I just got about 5 perfect runs of the test suite, followed by:
>
> Macintosh-3:trunk dickinsm$ ./python.exe -m test.regrtest
> [...]
> test_multiprocessing
> Assertion failed: (bp != NULL), function PyObject_Malloc, file
> Objects/obmalloc.c, line 746.
> Abort trap (core dumped)
>
> I then did:
>
> gdb -c /cores/core.16235
>
> I've attached the traceback as traceback.txt

Are you sure that's right?  That traceback has no mention of
PyObject_Malloc or obmalloc.c.  Try checking the date.  Also, if  you
use "gdb ./python.exe <corefile>" to start gdb it should print a
warning if the program doesn't match the core.
msg69151 - (view) Author: Miki Tebeka (tebeka) * Date: 2008-07-02 23:58
I just run "make test" and it never moves past test_multiprocessing.

Maybe it's my machine which is dual cpu quad core (total of 8 cores)?
msg69154 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-07-03 00:25
Doubtful Miki - I do the work on the module on an 8 Core Gentoo, 8 Core 
Mac Pro and Dual Core Macbook Pro - it's not a # of cores issue, unless 
it's simply a >1 issue.
msg69155 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2008-07-03 00:30
> Are you sure that's right?

Not at all.  :-)

> That traceback has no mention of
> PyObject_Malloc or obmalloc.c.  Try checking the date.  Also, if  you
> use "gdb ./python.exe <corefile>" to start gdb it should print a
> warning if the program doesn't match the core.

The date and time on the core file look right (Jul 2, 23:52 GMT+1), and 
gdb ./python.exe ... doesn't give any warning.  So I'm not sure what I 
did wrong.  I'll try again and see if I get the same thing.
msg69156 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2008-07-03 00:44
Here's a new traceback (a different error again, this time: a negative 
refcount in Objects/tupleobject.c.)
msg69158 - (view) Author: Adam Olsen (Rhamphoryncus) Date: 2008-07-03 01:19
That looks better.  It crashed while deleting an exception, who's args
tuple has a bogus refcount.  Could be a refcount issue of the
exception or the args, or of something that that references them, or a
dangling pointer, or a buffer overrun, etc.

Things to try:
1) Run "pystack" in gdb, from Misc/gdbinit
2) Print the exception type.  Use "up" until you reach
BaseException_clear, then do "print self->ob_type->tp_name".  Also do
"print *self" and make sure the ob_refcnt is at 0 and the other fields
look sane.
3) Compile using --without-pymalloc and throw it at a real memory
debugger.  I'd suggest starting with your libc's own debugging
options, as they tend to be less invasive:
http://developer.apple.com/documentation/Performance/Conceptual/ManagingMemory/Articles/MallocDebug.html
.  If that doesn't work, look at Electric Fence, Valgrind, or your
tool of choice.
msg69159 - (view) Author: Adam Olsen (Rhamphoryncus) Date: 2008-07-03 01:28
Also, make sure you do a "make clean" since you last updated the tree or
touched any file or ran configure.  The automatic dependency checking
isn't 100% reliable.
msg69160 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-07-03 02:07
Barring the segfaults Mark is seeing, I went through and removed all of 
the tests, and I have been incrementally adding them back one by one. _TestQueue seems to be the one (at least, the first) which is hanging 
intermittently in a racquire(). If anyone else who is having hangs minds, 
please try removing _TestQueue and see if you can still get it to hang.
msg69181 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2008-07-03 07:43
The two tracebacks provided by Mark seem to correspond to the following
python stack (innermost last):

Lib/test/test_multiprocessing.py, line 1005, in _test_map_unordered
    self.assertEqual(sorted(it), map(sqr, range(1000)))
Lib/multiprocessing/pool.py, line 500, in IMapIterator.next()
    self._cond.acquire()
Lib/threading.py, line 123, in _RLock.acquire():
    rc = self.__block.acquire(blocking)
msg69182 - (view) Author: Ismail Donmez (cartman) Date: 2008-07-03 07:51
The test hanged for me at first try but worked fine on the second test,
weird.
msg69185 - (view) Author: Paul Melis (paulmelis) Date: 2008-07-03 08:14
On a Linux system (FC4) with r64686 of the Py3k branch I also still get
occassional hangs (with ./python -E -bb ./Lib/test/regrtest.py -v
test_multiprocessing). Mostly this seems to occur with the very first
test executed, i.e. before any of the "test_... " lines have been generated.

The following may or may not be related. Some time ago I decided to give
valgrind a try to see if it could detect anything strange going on with
the multiprocessing tests, specifically using the 'helgrind'
thread-debugging tool that comes with it. 

Valgrind reports as its first error:

==9719== Thread #1: Bug in libpthread: sem_wait succeeded on semaphore
without prior sem_post
==9719==    at 0x4007FFF: sem_wait_WRK (hg_intercepts.c:1057)
==9719==    by 0x4008094: sem_wait@* (hg_intercepts.c:1073)
==9719==    by 0x46A0087: semlock_acquire (semaphore.c:310)
==9719==    by 0x808C121: PyEval_EvalFrameEx (ceval.c:3371)
==9719==    by 0x808D0FE: PyEval_EvalCodeEx (ceval.c:2808)
==9719==    by 0x808B9D0: PyEval_EvalFrameEx (ceval.c:3469)
==9719==    by 0x808D0FE: PyEval_EvalCodeEx (ceval.c:2808)
==9719==    by 0x80F4B65: function_call (funcobject.c:628)
==9719==    by 0x80D1207: PyObject_Call (abstract.c:2178)
==9719==    by 0x80890EC: PyEval_EvalFrameEx (ceval.c:3672)
==9719==    by 0x808C1A9: PyEval_EvalFrameEx (ceval.c:3459)
==9719==    by 0x808C1A9: PyEval_EvalFrameEx (ceval.c:3459)
==9716== Thread #1 is the program's root thread

I've been hesitant to report this as the claim that libpthread is broken
is pretty bold. I contacted the valgrind devs about this, see [1]. 
More recently, someone on the valgrind list reported problems that do
seem to indicate there are broken libpthreads out there (see [2]), as
this individual reports a semaphore wait not blocking where it should.

Could it be that the multiprocessing tests are exposing one or more bugs
in libpthread?

[1] http://thread.gmane.org/gmane.comp.debugging.valgrind/8345
[2] http://thread.gmane.org/gmane.comp.debugging.valgrind/8384
msg69188 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2008-07-03 10:06
> Are you sure that's right?  That traceback has no mention of
> PyObject_Malloc or obmalloc.c.

So now I think that the traceback was right.  There was no mention of 
PyObject_Malloc or obmalloc.c because the traceback only showed 1 of the 9 
threads, and the failed assert occurred in a different thread.

I've attached another traceback, showing all the threads, and applying 'tb 
full' to the relevant thread.  (This was from a different run, but with 
the same failed assertion at line 746 of Objects/obmalloc.c.)
msg69414 - (view) Author: Ismail Donmez (cartman) Date: 2008-07-08 05:17
I managed to hang on Ubuntu, here is the backtrace that I got with CTRL-C:

Process PoolWorker-5:1:
Traceback (most recent call last):
  File "/home/cartman/Sources/py3k/Lib/multiprocessing/process.py", line
232, in _bootstrap
    test_bsddb test_bsddb3 test_cProfile test_kqueue test_lib2to3
2 skips unexpected on linux2:
    test_bsddb3 test_bsddb
Process PoolWorker-5:3:
Traceback (most recent call last):
  File "/home/cartman/Sources/py3k/Lib/multiprocessing/process.py", line
232, in _bootstrap
    self.run()
  File "/home/cartman/Sources/py3k/Lib/multiprocessing/process.py", line
88, in run
    self._target(*self._args, **self._kwargs)
  File "/home/cartman/Sources/py3k/Lib/multiprocessing/pool.py", line
57, in worker
    self.run()
  File "/home/cartman/Sources/py3k/Lib/multiprocessing/process.py", line
88, in run
    self._target(*self._args, **self._kwargs)
  File "/home/cartman/Sources/py3k/Lib/multiprocessing/pool.py", line
57, in worker
    task = get()
  File "/home/cartman/Sources/py3k/Lib/multiprocessing/queues.py", line
339, in get
    task = get()
  File "/home/cartman/Sources/py3k/Lib/multiprocessing/queues.py", line
337, in get
    return recv()
  File "/home/cartman/Sources/py3k/Lib/pickle.py", line 1327, in loads
    racquire()
KeyboardInterrupt
Process PoolWorker-5:2:
Traceback (most recent call last):
  File "/home/cartman/Sources/py3k/Lib/multiprocessing/process.py", line
232, in _bootstrap
    self.run()
  File "/home/cartman/Sources/py3k/Lib/multiprocessing/process.py", line
88, in run
    self._target(*self._args, **self._kwargs)
  File "/home/cartman/Sources/py3k/Lib/multiprocessing/pool.py", line
57, in worker
    def loads(s, *, encoding="ASCII", errors="strict"):
KeyboardInterrupt
Process PoolWorker-5:4:
Traceback (most recent call last):
  File "/home/cartman/Sources/py3k/Lib/multiprocessing/process.py", line
232, in _bootstrap
    self.run()
  File "/home/cartman/Sources/py3k/Lib/multiprocessing/process.py", line
88, in run
    self._target(*self._args, **self._kwargs)
  File "/home/cartman/Sources/py3k/Lib/multiprocessing/pool.py", line
57, in worker
    task = get()
  File "/home/cartman/Sources/py3k/Lib/multiprocessing/queues.py", line
337, in get
    racquire()
KeyboardInterrupt
    task = get()
  File "/home/cartman/Sources/py3k/Lib/multiprocessing/queues.py", line
337, in get
    racquire()
KeyboardInterrupt
^CError in atexit._run_exitfuncs:
make: *** [testall] Segmentation fault
msg69424 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2008-07-08 12:26
I found that on my Debian64, running test_multiprocessing under gdb
hangs even before the first test is started - somewhere in the
installation of the Manager.

And it appears that the problem is described in issue874900: "threading
module can deadlock after fork".
I don't know if it's a good idea to mix fork and threads, but the patch
I attached to issue874900 seems to correct the hang.
msg69425 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-07-08 12:51
Thanks Amaury - I've been working through the tests and identifying
the "problem children" - I'll finish that up and then try re-running
them with the 874900 patch.
msg69431 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2008-07-08 13:43
I'm still seeing intermittent lockups on Ubuntu 7.10 - traceback on
ctrl-C is similar to that posted by Ismail above.

Since Jesse seems to be on top of this, I'll stick to using -x
test_multiprocessing for the moment.
msg69433 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2008-07-08 13:46
Bumping back to release blocker for beta 2 (Barry may choose to defer it
again, but it should at least be on his radar).
msg69434 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2008-07-08 13:47
Updated issue title to more accurately reflect scope of the problem.
msg69435 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2008-07-08 13:50
I forgot to mention that I am seeing the intermittent hangs on the trunk
(2.6). I haven't been testing it on Py3k.
msg69737 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2008-07-16 00:38
Sadly _multiprocessing apparently doesn't even build on my Ubuntu 8.04
machine and it still hangs on my 10.5 machine.
msg69738 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-07-16 00:42
On Jul 15, 2008, at 8:38 PM, "Barry A. Warsaw"  
<report@bugs.python.org> wrote:

>
> Barry A. Warsaw <barry@python.org> added the comment:
>
> Sadly _multiprocessing apparently doesn't even build on my Ubuntu 8.04
> machine and it still hangs on my 10.5 machine.
>

There is no reason it shouldn't compile on ubuntu - without the patch  
for the bug I added as a dependency, we will keep seeing hangs
msg69740 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-07-16 01:31
Barry - can you email the compile errors?
msg69747 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2008-07-16 02:29
Something's very strange.  The first make after configure fails to build
_multiprocessing, but a subsequent make succeeds.  I'll see if I can
capture the compilation error message.
msg69754 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2008-07-16 03:06
Here's the 'make' output.  What's strange is that moving
_multiprocessing{_failed,}.so, the import works just fine.

building '_multiprocessing' extension
creating
build/temp.linux-i686-3.0/home/barry/projects/python/python30/Modules/_multiprocessing
gcc -pthread -fPIC -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall
-Wstrict-prototypes -DHAVE_SEM_OPEN=1 -DHAVE_FD_TRANSFER=1
-DHAVE_SEM_TIMEDWAIT=1 -IModules/_multiprocessing -I.
-I/home/barry/projects/python/python30/./Include -I. -IInclude
-I./Include -I/usr/local/include
-I/home/barry/projects/python/python30/Include
-I/home/barry/projects/python/python30 -c
/home/barry/projects/python/python30/Modules/_multiprocessing/multiprocessing.c
-o
build/temp.linux-i686-3.0/home/barry/projects/python/python30/Modules/_multiprocessing/multiprocessing.o
gcc -pthread -fPIC -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall
-Wstrict-prototypes -DHAVE_SEM_OPEN=1 -DHAVE_FD_TRANSFER=1
-DHAVE_SEM_TIMEDWAIT=1 -IModules/_multiprocessing -I.
-I/home/barry/projects/python/python30/./Include -I. -IInclude
-I./Include -I/usr/local/include
-I/home/barry/projects/python/python30/Include
-I/home/barry/projects/python/python30 -c
/home/barry/projects/python/python30/Modules/_multiprocessing/socket_connection.c
-o
build/temp.linux-i686-3.0/home/barry/projects/python/python30/Modules/_multiprocessing/socket_connection.o
gcc -pthread -fPIC -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall
-Wstrict-prototypes -DHAVE_SEM_OPEN=1 -DHAVE_FD_TRANSFER=1
-DHAVE_SEM_TIMEDWAIT=1 -IModules/_multiprocessing -I.
-I/home/barry/projects/python/python30/./Include -I. -IInclude
-I./Include -I/usr/local/include
-I/home/barry/projects/python/python30/Include
-I/home/barry/projects/python/python30 -c
/home/barry/projects/python/python30/Modules/_multiprocessing/semaphore.c -o
build/temp.linux-i686-3.0/home/barry/projects/python/python30/Modules/_multiprocessing/semaphore.o
gcc -pthread -shared
build/temp.linux-i686-3.0/home/barry/projects/python/python30/Modules/_multiprocessing/multiprocessing.o
build/temp.linux-i686-3.0/home/barry/projects/python/python30/Modules/_multiprocessing/socket_connection.o
build/temp.linux-i686-3.0/home/barry/projects/python/python30/Modules/_multiprocessing/semaphore.o
-L/usr/local/lib -o build/lib.linux-i686-3.0/_multiprocessing.so
*** WARNING: renaming "_multiprocessing" since importing it failed: No
module named _multiprocessing
msg69895 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-07-17 17:41
Issue 874900's patch seems to have resolve the hangs. I am closing this 
issue as fixed.
msg69955 - (view) Author: Miki Tebeka (tebeka) * Date: 2008-07-18 14:42
I confirm this is solved for me in beta 2
History
Date User Action Args
2008-07-18 14:43:00tebekasetmessages: + msg69955
2008-07-17 17:41:47jnollersetstatus: open -> closed
resolution: fixed
messages: + msg69895
2008-07-16 12:58:01barrylinkissue3375 dependencies
2008-07-16 03:06:08barrysetmessages: + msg69754
2008-07-16 02:29:07barrysetmessages: + msg69747
2008-07-16 01:31:01jnollersetmessages: + msg69740
2008-07-16 00:42:56jnollersetmessages: + msg69738
2008-07-16 00:38:04barrysetmessages: + msg69737
2008-07-15 17:16:51jnollersetdependencies: + threading module can deadlock after fork
2008-07-08 13:50:33ncoghlansetmessages: + msg69435
versions: + Python 2.6
2008-07-08 13:47:47ncoghlansetmessages: + msg69434
title: test_multiprocessing hangs on OS X 10.5.3 -> test_multiprocessing hangs intermittently on POSIX platforms
2008-07-08 13:46:35ncoghlansetpriority: critical -> release blocker
resolution: accepted -> (no value)
messages: + msg69433
2008-07-08 13:43:51ncoghlansetnosy: + ncoghlan
messages: + msg69431
2008-07-08 12:51:28jnollersetmessages: + msg69425
2008-07-08 12:26:47amaury.forgeotdarcsetmessages: + msg69424
2008-07-08 05:17:25cartmansetmessages: + msg69414
2008-07-03 10:06:13mark.dickinsonsetfiles: + multithread_traceback.txt
messages: + msg69188
2008-07-03 08:14:38paulmelissetmessages: + msg69185
2008-07-03 07:51:14cartmansetmessages: + msg69182
2008-07-03 07:43:35amaury.forgeotdarcsetmessages: + msg69181
2008-07-03 02:07:49jnollersetmessages: + msg69160
2008-07-03 01:28:57Rhamphoryncussetmessages: + msg69159
2008-07-03 01:19:42Rhamphoryncussetmessages: + msg69158
2008-07-03 00:44:29mark.dickinsonsetfiles: + traceback2.txt
messages: + msg69156
2008-07-03 00:30:14mark.dickinsonsetmessages: + msg69155
2008-07-03 00:25:51jnollersetmessages: + msg69154
2008-07-02 23:58:35tebekasetmessages: + msg69151
2008-07-02 23:55:43Rhamphoryncussetmessages: + msg69149
2008-07-02 23:08:24mark.dickinsonsetfiles: + traceback.txt
messages: + msg69142
2008-07-02 22:33:49amaury.forgeotdarcsetmessages: + msg69138
2008-07-02 22:00:44Rhamphoryncussetmessages: + msg69133
2008-07-02 21:50:13jnollersetmessages: + msg69131
2008-07-02 21:44:08mark.dickinsonsetmessages: + msg69130
2008-07-02 21:18:14jnollersetmessages: + msg69128
2008-07-02 21:11:05jnollersetmessages: + msg69127
2008-07-02 21:07:11mark.dickinsonsetmessages: + msg69125
2008-07-02 21:05:28mark.dickinsonsetmessages: + msg69124
2008-07-02 21:02:01jnollersetmessages: + msg69123
2008-07-02 21:00:29mark.dickinsonsetmessages: + msg69122
2008-07-02 20:57:10jnollersetstatus: closed -> open
resolution: fixed -> accepted
messages: + msg69121
2008-07-02 20:51:31jnollersetmessages: + msg69120
2008-07-02 20:48:14mark.dickinsonsetmessages: + msg69119
2008-07-02 18:01:52jnollersetmessages: + msg69110
2008-07-02 17:54:44tebekasetmessages: + msg69109
2008-07-02 16:53:49amaury.forgeotdarcsetmessages: + msg69108
2008-07-02 16:48:51jnollersetstatus: open -> closed
resolution: fixed
messages: + msg69106
2008-06-30 23:11:44jnollersetmessages: + msg69026
2008-06-30 23:06:53amaury.forgeotdarcsetmessages: + msg69025
2008-06-28 23:41:27amaury.forgeotdarcsetfiles: + threadlocal.patch
messages: + msg68929
2008-06-28 15:16:13amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg68877
2008-06-27 21:04:03mark.dickinsonsetnosy: + mark.dickinson
messages: + msg68856
2008-06-20 23:25:56tebekasetmessages: + msg68485
2008-06-20 15:46:01jnollersetmessages: + msg68467
2008-06-20 15:41:10tebekasetnosy: + tebeka
messages: + msg68466
2008-06-20 00:19:09benjamin.petersonlinkissue3148 superseder
2008-06-19 13:35:24jnollersetassignee: jnoller
2008-06-19 00:22:00barrysetpriority: release blocker -> critical
messages: + msg68384
2008-06-16 20:37:09jnollersetmessages: + msg68295
2008-06-15 15:26:58jnollersetmessages: + msg68240
2008-06-15 15:20:21jnollersetfiles: + test_multiprocessing_reduced.diff
keywords: + patch
messages: + msg68237
2008-06-14 18:14:05Rhamphoryncussetnosy: + Rhamphoryncus
2008-06-14 17:35:42roudkerksetmessages: + msg68209
2008-06-14 06:43:16cartmansetmessages: + msg68198
2008-06-13 13:17:56jnollersetmessages: + msg68153
2008-06-13 12:38:12jnollersetmessages: + msg68150
2008-06-13 07:59:10paulmelissetmessages: + msg68136
2008-06-12 19:59:25jnollersetnosy: + roudkerk
2008-06-12 19:56:56jnollersetfiles: + test_get.output
messages: + msg68077
2008-06-12 18:40:33jnollersetmessages: + msg68072
2008-06-12 15:25:56paulmelissetmessages: + msg68068
2008-06-12 15:16:40paulmelissetnosy: + paulmelis
messages: + msg68067
2008-06-12 14:08:30gvanrossumsetmessages: + msg68066
2008-06-12 14:07:49gvanrossumsetnosy: + gvanrossum
messages: + msg68065
2008-06-12 13:25:46jnollersetmessages: + msg68064
2008-06-12 13:19:49barrysetmessages: + msg68063
2008-06-12 13:18:17jnollersetnosy: + jnoller
messages: + msg68062
2008-06-12 13:02:14benjamin.petersonsetnosy: + benjamin.peterson
messages: + msg68059
2008-06-12 05:23:57cartmansetnosy: + cartman
messages: + msg68055
2008-06-12 04:48:42barrycreate