classification
Title: test_gdb broken PPC64 Linux
Type: behavior Stage: resolved
Components: Tests Versions: Python 3.4, Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: dmalcolm Nosy List: David.Edelsohn, dmalcolm, python-dev, siddhesh
Priority: normal Keywords: patch

Created on 2013-04-24 18:35 by David.Edelsohn, last changed 2013-05-07 06:35 by ezio.melotti. This issue is now closed.

Files
File name Uploaded Description Edit
find-more-information.patch dmalcolm, 2013-04-24 19:25 review
test_gdb.out David.Edelsohn, 2013-04-24 19:33 Output of test_gdb with patch
possible-fix.patch dmalcolm, 2013-04-24 19:47 review
Messages (10)
msg187720 - (view) Author: David Edelsohn (David.Edelsohn) * Date: 2013-04-24 18:35
Verify that "py-bt" indicates threads that are waiting for the GIL ... FAIL

FAIL: test_threads (test.test_gdb.PyBtTests)
Verify that "py-bt" indicates threads that are waiting for the GIL
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/shager/cpython-buildarea/3.x.edelsohn-powerlinux-ppc64/build/Lib/test/test_gdb.py", line 717, in test_threads
    self.assertIn('Waiting for the GIL', gdb_output)
AssertionError: 'Waiting for the GIL' not found in 'Breakpoint 1 at 0x100ee8b0: file Python/bltinmodule.c, line 967.\n[Thread debugging using libthread_db enabled]\nUsing host libthread_db library "/lib64/libthread_db.so.1".\n[New Thread 0x3fffb14af200 (LWP 1699)]\n[New Thread 0x3fffb0caf200 (LWP 1703)]\n[New Thread 0x3fffabfff200 (LWP 1711)]\n[New Thread 0x3fffab7ff200 (LWP 1730)]\n\nBreakpoint 1, builtin_id (self=<module at remote 0x3fffb17dc058>, v=42) at Python/bltinmodule.c:967\n967\t    return PyLong_FromVoidPtr(v);\n\nThread 5 (Thread 0x3fffab7ff200 (LWP 1730)):\nTraceback (most recent call first):\n  File "<string>", line 10, in run\n  File "/home/shager/cpython-buildarea/3.x.edelsohn-powerlinux-ppc64/build/Lib/threading.py", line 642, in _bootstrap_inner\n    self.run()\n  File "/home/shager/cpython-buildarea/3.x.edelsohn-powerlinux-ppc64/build/Lib/threading.py", line 619, in _bootstrap\n    self._bootstrap_inner()\n\nThread 4 (Thread 0x3fffabfff200 (LWP 1711)):\nTraceback (most recent call first):\n  File "<string>", line 10, in run\n  File "/home/shager/cpython-buildarea/3.x.edelsohn-powerlinux-ppc64/build/Lib/threading.py", line 642, in _bootstrap_inner\n    self.run()\n  File "/home/shager/cpython-buildarea/3.x.edelsohn-powerlinux-ppc64/build/Lib/threading.py", line 619, in _bootstrap\n    self._bootstrap_inner()\n\nThread 3 (Thread 0x3fffb0caf200 (LWP 1703)):\nTraceback (most recent call first):\n  File "<string>", line 10, in run\n  File "/home/shager/cpython-buildarea/3.x.edelsohn-powerlinux-ppc64/build/Lib/threading.py", line 642, in _bootstrap_inner\n    self.run()\n  File "/home/shager/cpython-buildarea/3.x.edelsohn-powerlinux-ppc64/build/Lib/threading.py", line 619, in _bootstrap\n    self._bootstrap_inner()\n\nThread 2 (Thread 0x3fffb14af200 (LWP 1699)):\nTraceback (most recent call first):\n  File "<string>", line 10, in run\n  File "/home/shager/cpython-buildarea/3.x.edelsohn-powerlinux-ppc64/build/Lib/threading.py", line 642, in _bootstrap_inner\n    self.run()\n  File "/home/shager/cpython-buildarea/3.x.edelsohn-powerlinux-ppc64/build/Lib/threading.py", line 619, in _bootstrap\n    self._bootstrap_inner()\n\nThread 1 (Thread 0x8074e46670 (LWP 1616)):\nTraceback (most recent call first):\n  File "<string>", line 18, in <module>\n'
msg187722 - (view) Author: Dave Malcolm (dmalcolm) (Python committer) Date: 2013-04-24 18:43
Is this an optimized or a debug build?  (aka --with-pydebug)

What are the optimization flags passed to the C compiler?
msg187724 - (view) Author: David Edelsohn (David.Edelsohn) * Date: 2013-04-24 18:58
Unoptimized debug build (configured using --with-pydebug). Buildbot test failure running on gcc110 in GCC Compile Farm (running Fedora).

gcc -pthread -c -Wno-unused-result -g -O0 -Wall -Wstrict-prototypes -m64
msg187730 - (view) Author: Dave Malcolm (dmalcolm) (Python committer) Date: 2013-04-24 19:25
Thanks.  I can't see from that output what's going wrong.

Can you apply the following patch, which (I hope) will print more detailed info where the failure happens.
msg187733 - (view) Author: Dave Malcolm (dmalcolm) (Python committer) Date: 2013-04-24 19:40
Looking at test_gdb.out, the issue is that the threads waiting for the GIL have e.g. this at the top of their backtrace:

Thread 2 (Thread 0x3fffb14af200 (LWP 37119)):
#0  0x0000008075181ea8 in __pthread_cond_timedwait (cond=0x103ea140 <gil_cond>, mutex=0x103ea170 <gil_mutex>, abstime=0x3fffb14abf68) at pthread_cond_timedwait.c:167
#1  0x00000000100f59f8 in PyCOND_TIMEDWAIT (cond=0x103ea140 <gil_cond>, mut=0x103ea170 <gil_mutex>, us=5000) at /home/dje/src/cpython/Python/condvar.h:103
#2  0x00000000100f5ffc in take_gil (tstate=0x1043b020) at /home/dje/src/cpython/Python/ceval_gil.h:224

i.e. the topmost frame has name "__pthread_cond_timedwait" (two leading underscores.

However, the gdb hooks (in Tools/gdb/libpython.py) have:

    def is_waiting_for_gil(self):
        '''Is this frame waiting on the GIL?'''
        # This assumes the _POSIX_THREADS version of Python/ceval_gil.h:
        name = self._gdbframe.name()
        if name:
            return name.startswith('pthread_cond_timedwait')

i.e. no underscores, hence that name.startswith conditional doesn't match.
msg187734 - (view) Author: Dave Malcolm (dmalcolm) (Python committer) Date: 2013-04-24 19:47
Can you try this candidate fix?
msg187735 - (view) Author: David Edelsohn (David.Edelsohn) * Date: 2013-04-24 19:54
__pthread_cond_timedwait with underscore may be a general GLibc change.

The patch fixes the failure.
msg187781 - (view) Author: Siddhesh Poyarekar (siddhesh) Date: 2013-04-25 13:19
It's not a change in glibc.  __pthread_cond_timedwait is the internal function name while pthread_cond_timedwait is the exported alias.  You're seeing __pthread_cond_timedwait here because either your glibc installation has debug symbols or you have debug info packages installed, which allows gdb to resolve the function name to the internal name.  Without glibc debug info you should see just pthread_cond_timedwait@@... or just pthread_cond_timedwait.

In any case, I guess you'd be better off just using .find("pthread_cond_timedwait") instead of startswith() since I've also seen this on ppc64, which might break your test again:

Thread 2 (Thread 0x3fffb7d1f200 (LWP 5746)):
#0  0x00003fffb7f21ec8 in .pthread_cond_timedwait () from /lib64/libpthread.so.0

I'm not entirely sure what the preceding dot means, but it seems to indicate a function call outside the binary in ppc64.
msg187782 - (view) Author: David Edelsohn (David.Edelsohn) * Date: 2013-04-25 13:24
Thanks for explaining the issue with GLibc symbols.  The Python test definitely should not fail if debugging symbols are installed.

The "dot" symbols should no longer occur with modern PowerLinux installations.  They were a carry-over from the AIX ABI on which the PPC64 Linux ABI is based.
msg188568 - (view) Author: Roundup Robot (python-dev) Date: 2013-05-06 18:51
New changeset f4a6b731905a by David Malcolm in branch '3.3':
#17833: fix test_gdb failures seen on PPC64 Linux in test_threads (test.test_gdb.PyBtTests)
http://hg.python.org/cpython/rev/f4a6b731905a

New changeset 6d971b172389 by David Malcolm in branch 'default':
#17833: merge with 3.3
http://hg.python.org/cpython/rev/6d971b172389
History
Date User Action Args
2013-05-07 06:35:48ezio.melottisetmessages: - msg188621
2013-05-07 06:35:01python-devsetmessages: + msg188621
2013-05-06 18:56:15dmalcolmsetstatus: open -> closed
resolution: fixed
stage: commit review -> resolved
2013-05-06 18:51:41python-devsetnosy: + python-dev
messages: + msg188568
2013-05-01 21:30:20pitrousetassignee: dmalcolm
stage: commit review
type: behavior
versions: + Python 3.3
2013-04-25 13:24:25David.Edelsohnsetmessages: + msg187782
2013-04-25 13:19:46siddheshsetnosy: + siddhesh
messages: + msg187781
2013-04-24 19:54:58David.Edelsohnsetmessages: + msg187735
2013-04-24 19:47:04dmalcolmsetfiles: + possible-fix.patch

messages: + msg187734
2013-04-24 19:40:29dmalcolmsetmessages: + msg187733
2013-04-24 19:33:40David.Edelsohnsetfiles: + test_gdb.out
2013-04-24 19:25:12dmalcolmsetfiles: + find-more-information.patch
keywords: + patch
messages: + msg187730
2013-04-24 18:58:26David.Edelsohnsetmessages: + msg187724
2013-04-24 18:43:03dmalcolmsetnosy: + dmalcolm
messages: + msg187722
2013-04-24 18:35:30David.Edelsohncreate