Issue3088
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2008-06-12 04:48 by barry, last changed 2022-04-11 14:56 by admin. This issue is now closed.
Files | ||||
---|---|---|---|---|
File name | Uploaded | Description | Edit | |
test_get.output | jnoller, 2008-06-12 19:56 | |||
test_multiprocessing_reduced.diff | jnoller, 2008-06-15 15:20 | |||
threadlocal.patch | amaury.forgeotdarc, 2008-06-28 23:41 | |||
traceback.txt | mark.dickinson, 2008-07-02 23:08 | Traceback from test_multiprocessing crash | ||
traceback2.txt | mark.dickinson, 2008-07-03 00:44 | A second traceback | ||
multithread_traceback.txt | mark.dickinson, 2008-07-03 10:06 | Traceback showing all threads |
Messages (73) | |||
---|---|---|---|
msg68053 - (view) | Author: Barry A. Warsaw (barry) * ![]() |
Date: 2008-06-12 04:48 | |
For me, test_multiprocessing hangs consistently on OS X 10.5.3. It passes just fine on Ubuntu 8.04. |
|||
msg68055 - (view) | Author: Ismail Donmez (donmez) * | Date: 2008-06-12 05:23 | |
I can confirm this on Leopard too. |
|||
msg68059 - (view) | Author: Benjamin Peterson (benjamin.peterson) * ![]() |
Date: 2008-06-12 13:02 | |
It passes for me on Leopard. Can you post the test running in verbose mode so we can see where it hangs? |
|||
msg68062 - (view) | Author: Jesse Noller (jnoller) * ![]() |
Date: 2008-06-12 13:18 | |
On python-3000 trunk, _multiprocessing doesn't even compile: /Users/jesse/open_source/subversion/python- 3000/Modules/_multiprocessing/semaphore.c: In function ‘semlock_iszero’: /Users/jesse/open_source/subversion/python- 3000/Modules/_multiprocessing/semaphore.c:515: warning: unused variable ‘sval’ |
|||
msg68063 - (view) | Author: Barry A. Warsaw (barry) * ![]() |
Date: 2008-06-12 13:19 | |
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Jun 12, 2008, at 9:02 AM, Benjamin Peterson wrote: > It passes for me on Leopard. Can you post the test running in verbose > mode so we can see where it hangs? It never hangs when run standalone, though it crashes about half the time. Running it under gdb doesn't help; it always gives me an interrupted syste call in os.waidpid() in forking.py. The hang occurs during 'make test', and it's always the second run that hangs. Perhaps some lock isn't getting cleaned up properly the first time? - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) iQCVAwUBSFEicXEjvBPtnXfVAQLGNwP/S6f2IrO7c7SET0Gx8FXqdPmot3jcmopx TFxDA5csh/pVaDQCVW6DiLMXsu2TkQGPPbbo8Bx9iPmV/iIHFqy4nDtETqqjKdRp BvVtBmvSrP6wmymlKFlFC5qdfbbvguZq/hO60XulQk+WU4F8N7oHQck0tA2JhdDh lS5SAFAIovA= =xzs6 -----END PGP SIGNATURE----- |
|||
msg68064 - (view) | Author: Jesse Noller (jnoller) * ![]() |
Date: 2008-06-12 13:25 | |
I did a make clean && ./configure && make and it started compiling for me again. Sorry for the noise. |
|||
msg68065 - (view) | Author: Guido van Rossum (gvanrossum) * ![]() |
Date: 2008-06-12 14:07 | |
If it's only failing during the second run of "make test", typically there's some implicit dependency on something that is disturbed by running a test that's later in the suite of tests. This could be either the fault of that other test (not restoring some global setting or environment var) or the fault of the test that fails (making unwarranted assumptions or not initializing some needed settings before starting). If it works for some folks and not for others, on the same platform, compare the set of extension modules that are not built, reported by "make" in a message starting with "Failed to find the necessary bits to build these modules:". Likely, Barry has the most complete set, while Jesse has a few more extensions missing. Finding this is usually a painful process of bisecting the set of tests run. Randomizing the tests with regrtest.py -r might also be helpful. FWIW, when I tried (on Leopard) "make test TESTOPTS=test_multiprocessing" it hung on the first set. When I ran "./python Lib/test/test_multiprocess.py -v" it reported 122 tests passed. But when I ran "./python Lib/test/regrtest.py -v test_multiprocessing" one test failed: ====================================================================== ERROR: test_remote (test.test_multiprocessing.WithManagerTestRemoteManager) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/guido/p/Lib/test/test_multiprocessing.py", line 1167, in test_remote queue = manager2.get_queue() File "/Users/guido/p/Lib/multiprocessing/managers.py", line 650, in temp authkey=self._authkey, exposed=exp File "/Users/guido/p/Lib/multiprocessing/managers.py", line 902, in AutoProxy incref=incref) File "/Users/guido/p/Lib/multiprocessing/managers.py", line 711, in __init__ self._incref() File "/Users/guido/p/Lib/multiprocessing/managers.py", line 758, in _incref dispatch(conn, None, 'incref', (self._id,)) File "/Users/guido/p/Lib/multiprocessing/managers.py", line 94, in dispatch raise convert_to_error(kind, result) RemoteError: --------------------------------------------------------------------------- Traceback (most recent call last): File "/Users/guido/p/Lib/multiprocessing/managers.py", line 196, in handle_request result = func(c, *args, **kwds) File "/Users/guido/p/Lib/multiprocessing/managers.py", line 412, in incref self.id_to_refcount[ident] += 1 KeyError: '5f2828' --------------------------------------------------------------------------- |
|||
msg68066 - (view) | Author: Guido van Rossum (gvanrossum) * ![]() |
Date: 2008-06-12 14:08 | |
I should add this was in the trunk (2.6). |
|||
msg68067 - (view) | Author: Paul Melis (paulmelis) | Date: 2008-06-12 15:16 | |
I think I'm having a similar lockup on fedora core 4 (smp machine). This is with the py3k branch, freshly svn updated. When running "make test TESTOPTS=test_multiprocessing" the first of the two test runs always succeeds in something like 10-15 seconds, while the second one occasionally hangs. I tried running with -v as extra argument but can't get it to hang in that case (to figure out which test is the culprit). |
|||
msg68068 - (view) | Author: Paul Melis (paulmelis) | Date: 2008-06-12 15:25 | |
After a few more runs with -v and redirecting output to a file it seems the lockup I get is in test_notify_all (test.test_multiprocessing.WithManagerTestCondition) |
|||
msg68072 - (view) | Author: Jesse Noller (jnoller) * ![]() |
Date: 2008-06-12 18:40 | |
It's taking me longer to get to this than I planned, any help is appreciated. |
|||
msg68077 - (view) | Author: Jesse Noller (jnoller) * ![]() |
Date: 2008-06-12 19:56 | |
I can get an intermittent (1 every 15 or so runs) lock in: test_get (__main__.WithProcessesTestQueue) ... Executed like this: ./python Lib/test/test_multiprocessing.py When I control-c it the stack looks like this: ...snip File "/root/py/python-3000/Lib/multiprocessing/pool.py", line 57, in worker task = get() File "/root/py/python-3000/Lib/multiprocessing/queues.py", line 337, in get task = get() File "/root/py/python-3000/Lib/multiprocessing/queues.py", line 339, in get racquire() KeyboardInterrupt task = get() File "/root/py/python-3000/Lib/multiprocessing/queues.py", line 337, in get task = get() File "/root/py/python-3000/Lib/multiprocessing/queues.py", line 337, in get return recv() File "/root/py/python-3000/Lib/pickle.py", line 1327, in loads racquire() KeyboardInterrupt racquire() KeyboardInterrupt def loads(s, *, encoding="ASCII", errors="strict"): KeyboardInterrupt I'm not seeing frequent locks/failures when run with regrtest, but I am seeing them with "make test TESTOPTS=test_multiprocessing" I've attached full output. Still trying to figure it out |
|||
msg68136 - (view) | Author: Paul Melis (paulmelis) | Date: 2008-06-13 07:59 | |
I made a copy of test_multiprocessing.py (to test_mp.py) and basically removed all test classes, except _TestCondition. In it, I commented all test methods except test_notify_all. When run with make test TESTOPTS="-v test_mp" there's the lockups every few runs, but I also got (only three times in 40 or so runs so far): 9:57|paul@tabu:~/c/py3k-svn> make test TESTOPTS="-v test_mp" Failed to find the necessary bits to build these modules: _gestalt To find the necessary bits, look in setup.py in detect_modules() for the module's name. find ./Lib -name '*.py[co]' -print | xargs rm -f ./python -E -bb ./Lib/test/regrtest.py -v test_mp test_mp test_notify_all (test.test_mp.WithProcessesTestCondition) ... ok test_notify_all (test.test_mp.WithThreadsTestCondition) ... ok test_notify_all (test.test_mp.WithManagerTestCondition) ... ok ---------------------------------------------------------------------- Ran 3 tests in 1.087s OK 1 test OK. CAUTION: stdout isn't compared in verbose mode: a test that passes in verbose mode may fail without it. ./python -E -bb ./Lib/test/regrtest.py -v test_mp test_mp test_notify_all (test.test_mp.WithProcessesTestCondition) ... ok test_notify_all (test.test_mp.WithThreadsTestCondition) ... ok test_notify_all (test.test_mp.WithManagerTestCondition) ... Exception in thread Thread-28: Traceback (most recent call last): File "/home/paul/c/py3k-svn/Lib/threading.py", line 492, in _bootstrap_inner self.run() File "/home/paul/c/py3k-svn/Lib/threading.py", line 447, in run self._target(*self._args, **self._kwargs) File "/home/paul/c/py3k-svn/Lib/test/test_mp.py", line 208, in f cond.wait(timeout) File "/home/paul/c/py3k-svn/Lib/multiprocessing/managers.py", line 973, in wait return self._callmethod('wait', (timeout,)) File "/home/paul/c/py3k-svn/Lib/multiprocessing/managers.py", line 748, in _callmethod raise convert_to_error(kind, result) RuntimeError: cannot wait on un-aquired lock |
|||
msg68150 - (view) | Author: Jesse Noller (jnoller) * ![]() |
Date: 2008-06-13 12:38 | |
FWIW: In order to boost the logging level within the test(s) do the following: Search for LOG_LEVEL, set it to: LOG_LEVEL=util.SUBDEBUG And then in the main() replace: multiprocessing.get_logger().setLevel(LOG_LEVEL) With: multiprocessing.log_to_stderr(level=LOG_LEVEL) |
|||
msg68153 - (view) | Author: Jesse Noller (jnoller) * ![]() |
Date: 2008-06-13 13:17 | |
I also isolated the test(s) like Paul did, and it looks like a semi- consistent lock up in: File "/root/py/python-3000/Lib/multiprocessing/queues.py", line 337, in get racquire() This is running only the test_event test. The racquire traces back to synchronize.SemLock which calls into _multiprocessing.SemLock |
|||
msg68198 - (view) | Author: Ismail Donmez (donmez) * | Date: 2008-06-14 06:43 | |
Seems to work fine for me now with latest py3k branch. |
|||
msg68209 - (view) | Author: roudkerk (roudkerk) | Date: 2008-06-14 17:35 | |
I suspect the problems with WithManagerTestCondition.notify_all() may have to do with the thread safety of the proxies. If you replace Thread(...) by self.Process(...) in that test then the problem may go away. |
|||
msg68237 - (view) | Author: Jesse Noller (jnoller) * ![]() |
Date: 2008-06-15 15:20 | |
After talking with Richard, I think the best way to attack this issue (and the other ones around suite unreliability) is to remove the unreliable test cases for the first beta, and then refactor the suite post beta with an eye towards reliability and clarity. Personally, I would like to break the suites up in the the test_multiprocessing.py script to be more in the vein of other tests in Lib/test/... I removed the more unreliable test cases while keeping the core ones and wrote a quick bash script to do a burn-in of the "make tests" command, for 100 loops, I was unable to get the tests to hang with the reduced suite. I ran the same thing on trunk and py3k just to make sure I could not get it to hang/crash. |
|||
msg68240 - (view) | Author: Jesse Noller (jnoller) * ![]() |
Date: 2008-06-15 15:26 | |
Here is the loop I ran the tests with: #!/bin/sh for (( i=1;i<=100;i+=1 )); do make test TESTOPTS="-v test_multiprocessing" done |
|||
msg68295 - (view) | Author: Jesse Noller (jnoller) * ![]() |
Date: 2008-06-16 20:37 | |
I don't have commit rights, so I can't apply the test_multiprocessing_reduced.diff myself. Anyone willing? I think this should help the buildbots. |
|||
msg68384 - (view) | Author: Barry A. Warsaw (barry) * ![]() |
Date: 2008-06-19 00:21 | |
I'm going to knock this one down to critical since it's working for me now on OS X and buildbot looks green. We can address any additional patches after the beta release. |
|||
msg68466 - (view) | Author: Miki Tebeka (tebeka) * | Date: 2008-06-20 15:41 | |
Still hangs for me on the 2.6 trunk on Ubuntu 8.04 |
|||
msg68467 - (view) | Author: Jesse Noller (jnoller) * ![]() |
Date: 2008-06-20 15:46 | |
Where exactly does it hang Miki? |
|||
msg68485 - (view) | Author: Miki Tebeka (tebeka) * | Date: 2008-06-20 23:25 | |
Jesse, I just run "make test", it runs until test_multiprocessing and then hangs there |
|||
msg68856 - (view) | Author: Mark Dickinson (mark.dickinson) * ![]() |
Date: 2008-06-27 21:04 | |
test_multiprocessing is also still hanging for me, perhaps 30% of the times I run the test suite. When running the test by itself it seems to pass much more often, but not always. I just got the following output (on OS X 10.5.3/Intel). There was a hang at test_get; after around half-an-hour I hit Ctrl-C. Macintosh-3:trunk dickinsm$ ./python.exe Lib/test/test_multiprocessing.py test_array (__main__.WithProcessesTestArray) ... ok test_getobj_getlock_obj (__main__.WithProcessesTestArray) ... ok test_rawarray (__main__.WithProcessesTestArray) ... ok test_notify (__main__.WithProcessesTestCondition) ... ok test_notify_all (__main__.WithProcessesTestCondition) ... ok test_timeout (__main__.WithProcessesTestCondition) ... ok test_connection (__main__.WithProcessesTestConnection) ... ok test_duplex_false (__main__.WithProcessesTestConnection) ... ok test_sendbytes (__main__.WithProcessesTestConnection) ... ok test_spawn_close (__main__.WithProcessesTestConnection) ... ok test_event (__main__.WithProcessesTestEvent) ... ok test_finalize (__main__.WithProcessesTestFinalize) ... ok test_heap (__main__.WithProcessesTestHeap) ... ok test_import (__main__.WithProcessesTestImportStar) ... ok test_lock (__main__.WithProcessesTestLock) ... ok test_rlock (__main__.WithProcessesTestLock) ... ok test_enable_logging (__main__.WithProcessesTestLogging) ... ok test_level (__main__.WithProcessesTestLogging) ... ok test_active_children (__main__.WithProcessesTestProcess) ... ok test_cpu_count (__main__.WithProcessesTestProcess) ... ok test_current (__main__.WithProcessesTestProcess) ... ok test_process (__main__.WithProcessesTestProcess) ... ok test_recursion (__main__.WithProcessesTestProcess) ... ok test_terminate (__main__.WithProcessesTestProcess) ... ok test_fork (__main__.WithProcessesTestQueue) ... ok test_get (__main__.WithProcessesTestQueue) ... ^CTraceback (most recent call last): File "Lib/test/test_multiprocessing.py", line 1799, in <module> main() File "Lib/test/test_multiprocessing.py", line 1796, in main test_main(unittest.TextTestRunner(verbosity=2).run) File "Lib/test/test_multiprocessing.py", line 1786, in test_main run(suite) File "/Users/dickinsm/python_source/trunk/Lib/unittest.py", line 743, in run test(result) File "/Users/dickinsm/python_source/trunk/Lib/unittest.py", line 454, in __call__ return self.run(*args, **kwds) File "/Users/dickinsm/python_source/trunk/Lib/unittest.py", line 450, in run test(result) File "/Users/dickinsm/python_source/trunk/Lib/unittest.py", line 454, in __call__ return self.run(*args, **kwds) File "/Users/dickinsm/python_source/trunk/Lib/unittest.py", line 450, in run test(result) File "/Users/dickinsm/python_source/trunk/Lib/unittest.py", line 293, in __call__ return self.run(*args, **kwds) File "/Users/dickinsm/python_source/trunk/Lib/unittest.py", line 272, in run testMethod() File "Lib/test/test_multiprocessing.py", line 415, in test_get parent_can_continue.wait() File "/Users/dickinsm/python_source/trunk/Lib/multiprocessing/synchronize.py" , line 292, in wait self._cond.wait(timeout) File "/Users/dickinsm/python_source/trunk/Lib/multiprocessing/synchronize.py" , line 201, in wait self._wait_semaphore.acquire(True, timeout) KeyboardInterrupt [50284 refs] Macintosh-3:trunk dickinsm$ |
|||
msg68877 - (view) | Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * ![]() |
Date: 2008-06-28 15:16 | |
I think I narrowed the problem to a race condition in *subclasses* of threading.local: In threadmodule.c::local_getattro, there is a chance that self->dict is changed before PyObject_GenericGetAttr is called. |
|||
msg68929 - (view) | Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * ![]() |
Date: 2008-06-28 23:41 | |
After 4 hours spent with the debugger I think I have an explanation and a correction of one of the problems. Note that this only affects the "Manager" tests. - The threading.local type overrides getattr and setattr: on every call, it fetches a dictionary stored in the current ThreadState, and installs it in its self->dict member, allowing the regular PyObject_Generic[Get|Set]Attr functions to actually work on thread-local data. - Of course, for this to work the GIL must not be released in any way between the moment self->dict is set, until object's dictionary is retrieved inside PyObject_Generic[Get|Set]Attr. This condition makes the code a bit fragile. With current 2.6b1, there is no path in PyObject_Generic[Get|Set]Attr that releases the GIL before the dict is retrieved - no decref, no jump into python code (at least when the attribute name is a string). It's almost a miracle, indeed. - The problem is in threadmodule.c, the paragraph starting at line 285: the first time the local object is accessed, it calls the __init__ method of the threading.local derived class. And this calls python code, and can release the GIL. - The consequences are dramatic: this let the possibility for another thread to start, use the same local object, install its thread-local dict in self->dict, and stop here; the first thread will wake up with the wrong dict... - in the multiprocessing case, a remote call to a synchronization function grabs the wrong channel, and get bogus results... hence the "cannot wait on un-aquired (sic) lock" message. The actual correction is very short: only one "else" to remove... Here is a patch with a test. |
|||
msg69025 - (view) | Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * ![]() |
Date: 2008-06-30 23:06 | |
Committed my changes to threading.local as r64601. I think we could enable the "Manager" tests again. |
|||
msg69026 - (view) | Author: Jesse Noller (jnoller) * ![]() |
Date: 2008-06-30 23:11 | |
Alright - I'll do that asap On Jun 30, 2008, at 7:06 PM, Amaury Forgeot d'Arc <report@bugs.python.org > wrote: > > Amaury Forgeot d'Arc <amauryfa@gmail.com> added the comment: > > Committed my changes to threading.local as r64601. > > I think we could enable the "Manager" tests again. > > _______________________________________ > Python tracker <report@bugs.python.org> > <http://bugs.python.org/issue3088> > _______________________________________ |
|||
msg69106 - (view) | Author: Jesse Noller (jnoller) * ![]() |
Date: 2008-07-02 16:48 | |
Tests are back on as of r64663 on trunk. |
|||
msg69108 - (view) | Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * ![]() |
Date: 2008-07-02 16:53 | |
Not sure that all problems reported here are resolved. For example, "test_get (__main__.WithProcessesTestQueue)" does not seem to use threading.local at all. |
|||
msg69109 - (view) | Author: Miki Tebeka (tebeka) * | Date: 2008-07-02 17:54 | |
Still hangs for me on revision 64665 |
|||
msg69110 - (view) | Author: Jesse Noller (jnoller) * ![]() |
Date: 2008-07-02 18:01 | |
Miki where does it hang? I've run it in a tight loop for over 500 iterations without a single hang. |
|||
msg69119 - (view) | Author: Mark Dickinson (mark.dickinson) * ![]() |
Date: 2008-07-02 20:48 | |
The g4 buildbot still isn't happy, either. See: http://www.python.org/dev/buildbot/trunk.stable/g4%20osx.4%20trunk/builds/ 3645/step-test/0 (This was a run of revision 64663, I think.) |
|||
msg69120 - (view) | Author: Jesse Noller (jnoller) * ![]() |
Date: 2008-07-02 20:51 | |
The G4 buildbot is green right now. |
|||
msg69121 - (view) | Author: Jesse Noller (jnoller) * ![]() |
Date: 2008-07-02 20:57 | |
Looking through the buildbot's history (OSX G4) I can see it hung *prior* and after the change to turn the manager tests back on, so it's unrelated to those tests. I'll reopen this bug. |
|||
msg69122 - (view) | Author: Mark Dickinson (mark.dickinson) * ![]() |
Date: 2008-07-02 21:00 | |
> The G4 buildbot is green right now. Sure. But it looks like test_multiprocessing hung on build 3645, which as far as I can tell included all the recent fixes. My machine's behaving the same way: when doing 'make test', sometimes test_multiprocessing passes, sometimes (about half the time, maybe a little less) it hangs. Unfortunately, when running test_multiprocessing by itself it hardly ever hangs; and when running it under regrtest.py with verbose output it doesn't hang either, so it's a little difficult to figure out where things are going wrong. |
|||
msg69123 - (view) | Author: Jesse Noller (jnoller) * ![]() |
Date: 2008-07-02 21:02 | |
What platform mark? |
|||
msg69124 - (view) | Author: Mark Dickinson (mark.dickinson) * ![]() |
Date: 2008-07-02 21:05 | |
Here's an interesting snippet from the my most recent attempt to run 'make test'. I got a failed assertion: [...] test_multibytecodec_support test_multifile test_multiprocessing Assertion failed: (bp != NULL), function PyObject_Malloc, file Objects/obmalloc.c, line 746. make: *** [test] Abort trap ./python.exe -E -tt ./Lib/test/regrtest.py -l test_grammar test_opcodes [...] This is again on OS X 10.5.3; I built with: LDFLAGS=-L/opt/local/lib CPPFLAGS=-I/opt/local/include ./configure -- with-pydebug && make (the /opt/local stuff is there because that's where my readline library is...) Python 2.6b1+ (trunk:64671, Jul 2 2008, 21:51:37) [GCC 4.0.1 (Apple Inc. build 5465)] on darwin |
|||
msg69125 - (view) | Author: Mark Dickinson (mark.dickinson) * ![]() |
Date: 2008-07-02 21:07 | |
> What platform mark? OS X 10.5.4/Intel Core 2 Duo. (It's a 2007 Macbook Pro.) |
|||
msg69127 - (view) | Author: Jesse Noller (jnoller) * ![]() |
Date: 2008-07-02 21:11 | |
Thanks. I'm a been mystified why it hanging on you - I'm on a 2008 Macbook Pro running latest 10.5.4. I'm still trying to get it to hang though. |
|||
msg69128 - (view) | Author: Jesse Noller (jnoller) * ![]() |
Date: 2008-07-02 21:18 | |
Mark, can you try commenting out _TestCondition and seeing if you can still get it to hang?; |
|||
msg69130 - (view) | Author: Mark Dickinson (mark.dickinson) * ![]() |
Date: 2008-07-02 21:44 | |
> Mark, can you try commenting out _TestCondition and seeing if you can > still get it to hang?; I removed the _TestCondition class entirely from test_multiprocessing, and did make test again. It didn't hang! :-) It crashed instead. :-( The output's below, in all its gory glory. I can't make head or tail or it, but maybe someone else can. The last two runs before this gave me a Segmentation fault. test_multiprocessing Debug memory block at address p=0x2ecd928: Debug memory block at address p=0x2ecd928: 49 bytes originally requested 49 bytes originally requested The 4 pad bytes at p-4 are The 4 pad bytes at p-4 are FORBIDDENBYTE, as expected. FORBIDDENBYTE, as expected. The 4 pad bytes at tail=0x2ecd959 are The 4 pad bytes at tail=0x2ecd959 are not all FORBIDDENBYTE (0xfb): not all FORBIDDENBYTE (0xfb): at tail+0: 0x7d at tail+0: 0x7d *** OUCH *** OUCH at tail+1: 0x74 at tail+1: 0x74 *** OUCH *** OUCH at tail+2: 0x71 at tail+2: 0x71 *** OUCH *** OUCH at tail+3: 0x02 at tail+3: 0x02 *** OUCH *** OUCH The block was made by call #782281511 to debug malloc/realloc. The block was made by call #782281511 to debug malloc/realloc. Data at p: Data at p: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ... ... 87 87 71 71 02 02 2e 2e 00 00 1c 1c 02 02 85 85 Fatal Python error: bad trailing pad byte Fatal Python error: bad trailing pad byte make: *** [test] Abort trap |
|||
msg69131 - (view) | Author: Jesse Noller (jnoller) * ![]() |
Date: 2008-07-02 21:50 | |
That output exceeds my knowledge of python internals by several light years. :( |
|||
msg69133 - (view) | Author: Adam Olsen (Rhamphoryncus) | Date: 2008-07-02 22:00 | |
On Wed, Jul 2, 2008 at 3:44 PM, Mark Dickinson <report@bugs.python.org> wrote: > > Mark Dickinson <dickinsm@gmail.com> added the comment: > >> Mark, can you try commenting out _TestCondition and seeing if you can >> still get it to hang?; > > I removed the _TestCondition class entirely from test_multiprocessing, > and did make test again. It didn't hang! :-) It crashed instead. :-( Try running "ulimit -c unlimited" in the shell before running the test (from the same shell). After it aborts it should dump a core file, which you can then inspect using "gdb ./python core", to which "bt" will give you a stack trace ("backtrace"). On a minor note, I'd suggest running "./python -m test.regrtest" explicitly, rather than "make test". The latter runs the test suite twice, deleting all .pyc files before the first run, to detect problems in their creation. |
|||
msg69138 - (view) | Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * ![]() |
Date: 2008-07-02 22:33 | |
And please add the "-v" option at the end, so we can see which function fails |
|||
msg69142 - (view) | Author: Mark Dickinson (mark.dickinson) * ![]() |
Date: 2008-07-02 23:08 | |
Okay. I just got about 5 perfect runs of the test suite, followed by: Macintosh-3:trunk dickinsm$ ./python.exe -m test.regrtest [...] test_multiprocessing Assertion failed: (bp != NULL), function PyObject_Malloc, file Objects/obmalloc.c, line 746. Abort trap (core dumped) I then did: gdb -c /cores/core.16235 I've attached the traceback as traceback.txt |
|||
msg69149 - (view) | Author: Adam Olsen (Rhamphoryncus) | Date: 2008-07-02 23:55 | |
On Wed, Jul 2, 2008 at 5:08 PM, Mark Dickinson <report@bugs.python.org> wrote: > > Mark Dickinson <dickinsm@gmail.com> added the comment: > > Okay. I just got about 5 perfect runs of the test suite, followed by: > > Macintosh-3:trunk dickinsm$ ./python.exe -m test.regrtest > [...] > test_multiprocessing > Assertion failed: (bp != NULL), function PyObject_Malloc, file > Objects/obmalloc.c, line 746. > Abort trap (core dumped) > > I then did: > > gdb -c /cores/core.16235 > > I've attached the traceback as traceback.txt Are you sure that's right? That traceback has no mention of PyObject_Malloc or obmalloc.c. Try checking the date. Also, if you use "gdb ./python.exe <corefile>" to start gdb it should print a warning if the program doesn't match the core. |
|||
msg69151 - (view) | Author: Miki Tebeka (tebeka) * | Date: 2008-07-02 23:58 | |
I just run "make test" and it never moves past test_multiprocessing. Maybe it's my machine which is dual cpu quad core (total of 8 cores)? |
|||
msg69154 - (view) | Author: Jesse Noller (jnoller) * ![]() |
Date: 2008-07-03 00:25 | |
Doubtful Miki - I do the work on the module on an 8 Core Gentoo, 8 Core Mac Pro and Dual Core Macbook Pro - it's not a # of cores issue, unless it's simply a >1 issue. |
|||
msg69155 - (view) | Author: Mark Dickinson (mark.dickinson) * ![]() |
Date: 2008-07-03 00:30 | |
> Are you sure that's right? Not at all. :-) > That traceback has no mention of > PyObject_Malloc or obmalloc.c. Try checking the date. Also, if you > use "gdb ./python.exe <corefile>" to start gdb it should print a > warning if the program doesn't match the core. The date and time on the core file look right (Jul 2, 23:52 GMT+1), and gdb ./python.exe ... doesn't give any warning. So I'm not sure what I did wrong. I'll try again and see if I get the same thing. |
|||
msg69156 - (view) | Author: Mark Dickinson (mark.dickinson) * ![]() |
Date: 2008-07-03 00:44 | |
Here's a new traceback (a different error again, this time: a negative refcount in Objects/tupleobject.c.) |
|||
msg69158 - (view) | Author: Adam Olsen (Rhamphoryncus) | Date: 2008-07-03 01:19 | |
That looks better. It crashed while deleting an exception, who's args tuple has a bogus refcount. Could be a refcount issue of the exception or the args, or of something that that references them, or a dangling pointer, or a buffer overrun, etc. Things to try: 1) Run "pystack" in gdb, from Misc/gdbinit 2) Print the exception type. Use "up" until you reach BaseException_clear, then do "print self->ob_type->tp_name". Also do "print *self" and make sure the ob_refcnt is at 0 and the other fields look sane. 3) Compile using --without-pymalloc and throw it at a real memory debugger. I'd suggest starting with your libc's own debugging options, as they tend to be less invasive: http://developer.apple.com/documentation/Performance/Conceptual/ManagingMemory/Articles/MallocDebug.html . If that doesn't work, look at Electric Fence, Valgrind, or your tool of choice. |
|||
msg69159 - (view) | Author: Adam Olsen (Rhamphoryncus) | Date: 2008-07-03 01:28 | |
Also, make sure you do a "make clean" since you last updated the tree or touched any file or ran configure. The automatic dependency checking isn't 100% reliable. |
|||
msg69160 - (view) | Author: Jesse Noller (jnoller) * ![]() |
Date: 2008-07-03 02:07 | |
Barring the segfaults Mark is seeing, I went through and removed all of the tests, and I have been incrementally adding them back one by one. _TestQueue seems to be the one (at least, the first) which is hanging intermittently in a racquire(). If anyone else who is having hangs minds, please try removing _TestQueue and see if you can still get it to hang. |
|||
msg69181 - (view) | Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * ![]() |
Date: 2008-07-03 07:43 | |
The two tracebacks provided by Mark seem to correspond to the following python stack (innermost last): Lib/test/test_multiprocessing.py, line 1005, in _test_map_unordered self.assertEqual(sorted(it), map(sqr, range(1000))) Lib/multiprocessing/pool.py, line 500, in IMapIterator.next() self._cond.acquire() Lib/threading.py, line 123, in _RLock.acquire(): rc = self.__block.acquire(blocking) |
|||
msg69182 - (view) | Author: Ismail Donmez (donmez) * | Date: 2008-07-03 07:51 | |
The test hanged for me at first try but worked fine on the second test, weird. |
|||
msg69185 - (view) | Author: Paul Melis (paulmelis) | Date: 2008-07-03 08:14 | |
On a Linux system (FC4) with r64686 of the Py3k branch I also still get occassional hangs (with ./python -E -bb ./Lib/test/regrtest.py -v test_multiprocessing). Mostly this seems to occur with the very first test executed, i.e. before any of the "test_... " lines have been generated. The following may or may not be related. Some time ago I decided to give valgrind a try to see if it could detect anything strange going on with the multiprocessing tests, specifically using the 'helgrind' thread-debugging tool that comes with it. Valgrind reports as its first error: ==9719== Thread #1: Bug in libpthread: sem_wait succeeded on semaphore without prior sem_post ==9719== at 0x4007FFF: sem_wait_WRK (hg_intercepts.c:1057) ==9719== by 0x4008094: sem_wait@* (hg_intercepts.c:1073) ==9719== by 0x46A0087: semlock_acquire (semaphore.c:310) ==9719== by 0x808C121: PyEval_EvalFrameEx (ceval.c:3371) ==9719== by 0x808D0FE: PyEval_EvalCodeEx (ceval.c:2808) ==9719== by 0x808B9D0: PyEval_EvalFrameEx (ceval.c:3469) ==9719== by 0x808D0FE: PyEval_EvalCodeEx (ceval.c:2808) ==9719== by 0x80F4B65: function_call (funcobject.c:628) ==9719== by 0x80D1207: PyObject_Call (abstract.c:2178) ==9719== by 0x80890EC: PyEval_EvalFrameEx (ceval.c:3672) ==9719== by 0x808C1A9: PyEval_EvalFrameEx (ceval.c:3459) ==9719== by 0x808C1A9: PyEval_EvalFrameEx (ceval.c:3459) ==9716== Thread #1 is the program's root thread I've been hesitant to report this as the claim that libpthread is broken is pretty bold. I contacted the valgrind devs about this, see [1]. More recently, someone on the valgrind list reported problems that do seem to indicate there are broken libpthreads out there (see [2]), as this individual reports a semaphore wait not blocking where it should. Could it be that the multiprocessing tests are exposing one or more bugs in libpthread? [1] http://thread.gmane.org/gmane.comp.debugging.valgrind/8345 [2] http://thread.gmane.org/gmane.comp.debugging.valgrind/8384 |
|||
msg69188 - (view) | Author: Mark Dickinson (mark.dickinson) * ![]() |
Date: 2008-07-03 10:06 | |
> Are you sure that's right? That traceback has no mention of > PyObject_Malloc or obmalloc.c. So now I think that the traceback was right. There was no mention of PyObject_Malloc or obmalloc.c because the traceback only showed 1 of the 9 threads, and the failed assert occurred in a different thread. I've attached another traceback, showing all the threads, and applying 'tb full' to the relevant thread. (This was from a different run, but with the same failed assertion at line 746 of Objects/obmalloc.c.) |
|||
msg69414 - (view) | Author: Ismail Donmez (donmez) * | Date: 2008-07-08 05:17 | |
I managed to hang on Ubuntu, here is the backtrace that I got with CTRL-C: Process PoolWorker-5:1: Traceback (most recent call last): File "/home/cartman/Sources/py3k/Lib/multiprocessing/process.py", line 232, in _bootstrap test_bsddb test_bsddb3 test_cProfile test_kqueue test_lib2to3 2 skips unexpected on linux2: test_bsddb3 test_bsddb Process PoolWorker-5:3: Traceback (most recent call last): File "/home/cartman/Sources/py3k/Lib/multiprocessing/process.py", line 232, in _bootstrap self.run() File "/home/cartman/Sources/py3k/Lib/multiprocessing/process.py", line 88, in run self._target(*self._args, **self._kwargs) File "/home/cartman/Sources/py3k/Lib/multiprocessing/pool.py", line 57, in worker self.run() File "/home/cartman/Sources/py3k/Lib/multiprocessing/process.py", line 88, in run self._target(*self._args, **self._kwargs) File "/home/cartman/Sources/py3k/Lib/multiprocessing/pool.py", line 57, in worker task = get() File "/home/cartman/Sources/py3k/Lib/multiprocessing/queues.py", line 339, in get task = get() File "/home/cartman/Sources/py3k/Lib/multiprocessing/queues.py", line 337, in get return recv() File "/home/cartman/Sources/py3k/Lib/pickle.py", line 1327, in loads racquire() KeyboardInterrupt Process PoolWorker-5:2: Traceback (most recent call last): File "/home/cartman/Sources/py3k/Lib/multiprocessing/process.py", line 232, in _bootstrap self.run() File "/home/cartman/Sources/py3k/Lib/multiprocessing/process.py", line 88, in run self._target(*self._args, **self._kwargs) File "/home/cartman/Sources/py3k/Lib/multiprocessing/pool.py", line 57, in worker def loads(s, *, encoding="ASCII", errors="strict"): KeyboardInterrupt Process PoolWorker-5:4: Traceback (most recent call last): File "/home/cartman/Sources/py3k/Lib/multiprocessing/process.py", line 232, in _bootstrap self.run() File "/home/cartman/Sources/py3k/Lib/multiprocessing/process.py", line 88, in run self._target(*self._args, **self._kwargs) File "/home/cartman/Sources/py3k/Lib/multiprocessing/pool.py", line 57, in worker task = get() File "/home/cartman/Sources/py3k/Lib/multiprocessing/queues.py", line 337, in get racquire() KeyboardInterrupt task = get() File "/home/cartman/Sources/py3k/Lib/multiprocessing/queues.py", line 337, in get racquire() KeyboardInterrupt ^CError in atexit._run_exitfuncs: make: *** [testall] Segmentation fault |
|||
msg69424 - (view) | Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * ![]() |
Date: 2008-07-08 12:26 | |
I found that on my Debian64, running test_multiprocessing under gdb hangs even before the first test is started - somewhere in the installation of the Manager. And it appears that the problem is described in issue874900: "threading module can deadlock after fork". I don't know if it's a good idea to mix fork and threads, but the patch I attached to issue874900 seems to correct the hang. |
|||
msg69425 - (view) | Author: Jesse Noller (jnoller) * ![]() |
Date: 2008-07-08 12:51 | |
Thanks Amaury - I've been working through the tests and identifying the "problem children" - I'll finish that up and then try re-running them with the 874900 patch. |
|||
msg69431 - (view) | Author: Alyssa Coghlan (ncoghlan) * ![]() |
Date: 2008-07-08 13:43 | |
I'm still seeing intermittent lockups on Ubuntu 7.10 - traceback on ctrl-C is similar to that posted by Ismail above. Since Jesse seems to be on top of this, I'll stick to using -x test_multiprocessing for the moment. |
|||
msg69433 - (view) | Author: Alyssa Coghlan (ncoghlan) * ![]() |
Date: 2008-07-08 13:46 | |
Bumping back to release blocker for beta 2 (Barry may choose to defer it again, but it should at least be on his radar). |
|||
msg69434 - (view) | Author: Alyssa Coghlan (ncoghlan) * ![]() |
Date: 2008-07-08 13:47 | |
Updated issue title to more accurately reflect scope of the problem. |
|||
msg69435 - (view) | Author: Alyssa Coghlan (ncoghlan) * ![]() |
Date: 2008-07-08 13:50 | |
I forgot to mention that I am seeing the intermittent hangs on the trunk (2.6). I haven't been testing it on Py3k. |
|||
msg69737 - (view) | Author: Barry A. Warsaw (barry) * ![]() |
Date: 2008-07-16 00:38 | |
Sadly _multiprocessing apparently doesn't even build on my Ubuntu 8.04 machine and it still hangs on my 10.5 machine. |
|||
msg69738 - (view) | Author: Jesse Noller (jnoller) * ![]() |
Date: 2008-07-16 00:42 | |
On Jul 15, 2008, at 8:38 PM, "Barry A. Warsaw" <report@bugs.python.org> wrote: > > Barry A. Warsaw <barry@python.org> added the comment: > > Sadly _multiprocessing apparently doesn't even build on my Ubuntu 8.04 > machine and it still hangs on my 10.5 machine. > There is no reason it shouldn't compile on ubuntu - without the patch for the bug I added as a dependency, we will keep seeing hangs |
|||
msg69740 - (view) | Author: Jesse Noller (jnoller) * ![]() |
Date: 2008-07-16 01:31 | |
Barry - can you email the compile errors? |
|||
msg69747 - (view) | Author: Barry A. Warsaw (barry) * ![]() |
Date: 2008-07-16 02:29 | |
Something's very strange. The first make after configure fails to build _multiprocessing, but a subsequent make succeeds. I'll see if I can capture the compilation error message. |
|||
msg69754 - (view) | Author: Barry A. Warsaw (barry) * ![]() |
Date: 2008-07-16 03:06 | |
Here's the 'make' output. What's strange is that moving _multiprocessing{_failed,}.so, the import works just fine. building '_multiprocessing' extension creating build/temp.linux-i686-3.0/home/barry/projects/python/python30/Modules/_multiprocessing gcc -pthread -fPIC -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DHAVE_SEM_OPEN=1 -DHAVE_FD_TRANSFER=1 -DHAVE_SEM_TIMEDWAIT=1 -IModules/_multiprocessing -I. -I/home/barry/projects/python/python30/./Include -I. -IInclude -I./Include -I/usr/local/include -I/home/barry/projects/python/python30/Include -I/home/barry/projects/python/python30 -c /home/barry/projects/python/python30/Modules/_multiprocessing/multiprocessing.c -o build/temp.linux-i686-3.0/home/barry/projects/python/python30/Modules/_multiprocessing/multiprocessing.o gcc -pthread -fPIC -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DHAVE_SEM_OPEN=1 -DHAVE_FD_TRANSFER=1 -DHAVE_SEM_TIMEDWAIT=1 -IModules/_multiprocessing -I. -I/home/barry/projects/python/python30/./Include -I. -IInclude -I./Include -I/usr/local/include -I/home/barry/projects/python/python30/Include -I/home/barry/projects/python/python30 -c /home/barry/projects/python/python30/Modules/_multiprocessing/socket_connection.c -o build/temp.linux-i686-3.0/home/barry/projects/python/python30/Modules/_multiprocessing/socket_connection.o gcc -pthread -fPIC -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DHAVE_SEM_OPEN=1 -DHAVE_FD_TRANSFER=1 -DHAVE_SEM_TIMEDWAIT=1 -IModules/_multiprocessing -I. -I/home/barry/projects/python/python30/./Include -I. -IInclude -I./Include -I/usr/local/include -I/home/barry/projects/python/python30/Include -I/home/barry/projects/python/python30 -c /home/barry/projects/python/python30/Modules/_multiprocessing/semaphore.c -o build/temp.linux-i686-3.0/home/barry/projects/python/python30/Modules/_multiprocessing/semaphore.o gcc -pthread -shared build/temp.linux-i686-3.0/home/barry/projects/python/python30/Modules/_multiprocessing/multiprocessing.o build/temp.linux-i686-3.0/home/barry/projects/python/python30/Modules/_multiprocessing/socket_connection.o build/temp.linux-i686-3.0/home/barry/projects/python/python30/Modules/_multiprocessing/semaphore.o -L/usr/local/lib -o build/lib.linux-i686-3.0/_multiprocessing.so *** WARNING: renaming "_multiprocessing" since importing it failed: No module named _multiprocessing |
|||
msg69895 - (view) | Author: Jesse Noller (jnoller) * ![]() |
Date: 2008-07-17 17:41 | |
Issue 874900's patch seems to have resolve the hangs. I am closing this issue as fixed. |
|||
msg69955 - (view) | Author: Miki Tebeka (tebeka) * | Date: 2008-07-18 14:42 | |
I confirm this is solved for me in beta 2 |
History | |||
---|---|---|---|
Date | User | Action | Args |
2022-04-11 14:56:35 | admin | set | github: 47338 |
2008-07-18 14:43:00 | tebeka | set | messages: + msg69955 |
2008-07-17 17:41:47 | jnoller | set | status: open -> closed resolution: fixed messages: + msg69895 |
2008-07-16 12:58:01 | barry | link | issue3375 dependencies |
2008-07-16 03:06:08 | barry | set | messages: + msg69754 |
2008-07-16 02:29:07 | barry | set | messages: + msg69747 |
2008-07-16 01:31:01 | jnoller | set | messages: + msg69740 |
2008-07-16 00:42:56 | jnoller | set | messages: + msg69738 |
2008-07-16 00:38:04 | barry | set | messages: + msg69737 |
2008-07-15 17:16:51 | jnoller | set | dependencies: + threading module can deadlock after fork |
2008-07-08 13:50:33 | ncoghlan | set | messages:
+ msg69435 versions: + Python 2.6 |
2008-07-08 13:47:47 | ncoghlan | set | messages:
+ msg69434 title: test_multiprocessing hangs on OS X 10.5.3 -> test_multiprocessing hangs intermittently on POSIX platforms |
2008-07-08 13:46:35 | ncoghlan | set | priority: critical -> release blocker resolution: accepted -> (no value) messages: + msg69433 |
2008-07-08 13:43:51 | ncoghlan | set | nosy:
+ ncoghlan messages: + msg69431 |
2008-07-08 12:51:28 | jnoller | set | messages: + msg69425 |
2008-07-08 12:26:47 | amaury.forgeotdarc | set | messages: + msg69424 |
2008-07-08 05:17:25 | donmez | set | messages: + msg69414 |
2008-07-03 10:06:13 | mark.dickinson | set | files:
+ multithread_traceback.txt messages: + msg69188 |
2008-07-03 08:14:38 | paulmelis | set | messages: + msg69185 |
2008-07-03 07:51:14 | donmez | set | messages: + msg69182 |
2008-07-03 07:43:35 | amaury.forgeotdarc | set | messages: + msg69181 |
2008-07-03 02:07:49 | jnoller | set | messages: + msg69160 |
2008-07-03 01:28:57 | Rhamphoryncus | set | messages: + msg69159 |
2008-07-03 01:19:42 | Rhamphoryncus | set | messages: + msg69158 |
2008-07-03 00:44:29 | mark.dickinson | set | files:
+ traceback2.txt messages: + msg69156 |
2008-07-03 00:30:14 | mark.dickinson | set | messages: + msg69155 |
2008-07-03 00:25:51 | jnoller | set | messages: + msg69154 |
2008-07-02 23:58:35 | tebeka | set | messages: + msg69151 |
2008-07-02 23:55:43 | Rhamphoryncus | set | messages: + msg69149 |
2008-07-02 23:08:24 | mark.dickinson | set | files:
+ traceback.txt messages: + msg69142 |
2008-07-02 22:33:49 | amaury.forgeotdarc | set | messages: + msg69138 |
2008-07-02 22:00:44 | Rhamphoryncus | set | messages: + msg69133 |
2008-07-02 21:50:13 | jnoller | set | messages: + msg69131 |
2008-07-02 21:44:08 | mark.dickinson | set | messages: + msg69130 |
2008-07-02 21:18:14 | jnoller | set | messages: + msg69128 |
2008-07-02 21:11:05 | jnoller | set | messages: + msg69127 |
2008-07-02 21:07:11 | mark.dickinson | set | messages: + msg69125 |
2008-07-02 21:05:28 | mark.dickinson | set | messages: + msg69124 |
2008-07-02 21:02:01 | jnoller | set | messages: + msg69123 |
2008-07-02 21:00:29 | mark.dickinson | set | messages: + msg69122 |
2008-07-02 20:57:10 | jnoller | set | status: closed -> open resolution: fixed -> accepted messages: + msg69121 |
2008-07-02 20:51:31 | jnoller | set | messages: + msg69120 |
2008-07-02 20:48:14 | mark.dickinson | set | messages: + msg69119 |
2008-07-02 18:01:52 | jnoller | set | messages: + msg69110 |
2008-07-02 17:54:44 | tebeka | set | messages: + msg69109 |
2008-07-02 16:53:49 | amaury.forgeotdarc | set | messages: + msg69108 |
2008-07-02 16:48:51 | jnoller | set | status: open -> closed resolution: fixed messages: + msg69106 |
2008-06-30 23:11:44 | jnoller | set | messages: + msg69026 |
2008-06-30 23:06:53 | amaury.forgeotdarc | set | messages: + msg69025 |
2008-06-28 23:41:27 | amaury.forgeotdarc | set | files:
+ threadlocal.patch messages: + msg68929 |
2008-06-28 15:16:13 | amaury.forgeotdarc | set | nosy:
+ amaury.forgeotdarc messages: + msg68877 |
2008-06-27 21:04:03 | mark.dickinson | set | nosy:
+ mark.dickinson messages: + msg68856 |
2008-06-20 23:25:56 | tebeka | set | messages: + msg68485 |
2008-06-20 15:46:01 | jnoller | set | messages: + msg68467 |
2008-06-20 15:41:10 | tebeka | set | nosy:
+ tebeka messages: + msg68466 |
2008-06-20 00:19:09 | benjamin.peterson | link | issue3148 superseder |
2008-06-19 13:35:24 | jnoller | set | assignee: jnoller |
2008-06-19 00:22:00 | barry | set | priority: release blocker -> critical messages: + msg68384 |
2008-06-16 20:37:09 | jnoller | set | messages: + msg68295 |
2008-06-15 15:26:58 | jnoller | set | messages: + msg68240 |
2008-06-15 15:20:21 | jnoller | set | files:
+ test_multiprocessing_reduced.diff keywords: + patch messages: + msg68237 |
2008-06-14 18:14:05 | Rhamphoryncus | set | nosy: + Rhamphoryncus |
2008-06-14 17:35:42 | roudkerk | set | messages: + msg68209 |
2008-06-14 06:43:16 | donmez | set | messages: + msg68198 |
2008-06-13 13:17:56 | jnoller | set | messages: + msg68153 |
2008-06-13 12:38:12 | jnoller | set | messages: + msg68150 |
2008-06-13 07:59:10 | paulmelis | set | messages: + msg68136 |
2008-06-12 19:59:25 | jnoller | set | nosy: + roudkerk |
2008-06-12 19:56:56 | jnoller | set | files:
+ test_get.output messages: + msg68077 |
2008-06-12 18:40:33 | jnoller | set | messages: + msg68072 |
2008-06-12 15:25:56 | paulmelis | set | messages: + msg68068 |
2008-06-12 15:16:40 | paulmelis | set | nosy:
+ paulmelis messages: + msg68067 |
2008-06-12 14:08:30 | gvanrossum | set | messages: + msg68066 |
2008-06-12 14:07:49 | gvanrossum | set | nosy:
+ gvanrossum messages: + msg68065 |
2008-06-12 13:25:46 | jnoller | set | messages: + msg68064 |
2008-06-12 13:19:49 | barry | set | messages: + msg68063 |
2008-06-12 13:18:17 | jnoller | set | nosy:
+ jnoller messages: + msg68062 |
2008-06-12 13:02:14 | benjamin.peterson | set | nosy:
+ benjamin.peterson messages: + msg68059 |
2008-06-12 05:23:57 | donmez | set | nosy:
+ donmez messages: + msg68055 |
2008-06-12 04:48:42 | barry | create |