Verify that "py-bt" indicates threads that are waiting for the GIL ... FAIL

FAIL: test_threads (test.test_gdb.PyBtTests)
Verify that "py-bt" indicates threads that are waiting for the GIL
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/shager/cpython-buildarea/3.x.edelsohn-powerlinux-ppc64/build/Lib/test/test_gdb.py", line 717, in test_threads
    self.assertIn('Waiting for the GIL', gdb_output)
AssertionError: 'Waiting for the GIL' not found in 'Breakpoint 1 at 0x100ee8b0: file Python/bltinmodule.c, line 967.\n[Thread debugging using libthread_db enabled]\nUsing host libthread_db library "/lib64/libthread_db.so.1".\n[New Thread 0x3fffb14af200 (LWP 1699)]\n[New Thread 0x3fffb0caf200 (LWP 1703)]\n[New Thread 0x3fffabfff200 (LWP 1711)]\n[New Thread 0x3fffab7ff200 (LWP 1730)]\n\nBreakpoint 1, builtin_id (self=<module at remote 0x3fffb17dc058>, v=42) at Python/bltinmodule.c:967\n967\t    return PyLong_FromVoidPtr(v);\n\nThread 5 (Thread 0x3fffab7ff200 (LWP 1730)):\nTraceback (most recent call first):\n  File "<string>", line 10, in run\n  File "/home/shager/cpython-buildarea/3.x.edelsohn-powerlinux-ppc64/build/Lib/threading.py", line 642, in _bootstrap_inner\n    self.run()\n  File "/home/shager/cpython-buildarea/3.x.edelsohn-powerlinux-ppc64/build/Lib/threading.py", line 619, in _bootstrap\n    self._bootstrap_inner()\n\nThread 4 (Thread 0x3fffabfff200 (LWP 1711)):\nTraceback (most recent call first):\n  File "<string>", line 10, in run\n  File "/home/shager/cpython-buildarea/3.x.edelsohn-powerlinux-ppc64/build/Lib/threading.py", line 642, in _bootstrap_inner\n    self.run()\n  File "/home/shager/cpython-buildarea/3.x.edelsohn-powerlinux-ppc64/build/Lib/threading.py", line 619, in _bootstrap\n    self._bootstrap_inner()\n\nThread 3 (Thread 0x3fffb0caf200 (LWP 1703)):\nTraceback (most recent call first):\n  File "<string>", line 10, in run\n  File "/home/shager/cpython-buildarea/3.x.edelsohn-powerlinux-ppc64/build/Lib/threading.py", line 642, in _bootstrap_inner\n    self.run()\n  File "/home/shager/cpython-buildarea/3.x.edelsohn-powerlinux-ppc64/build/Lib/threading.py", line 619, in _bootstrap\n    self._bootstrap_inner()\n\nThread 2 (Thread 0x3fffb14af200 (LWP 1699)):\nTraceback (most recent call first):\n  File "<string>", line 10, in run\n  File "/home/shager/cpython-buildarea/3.x.edelsohn-powerlinux-ppc64/build/Lib/threading.py", line 642, in _bootstrap_inner\n    self.run()\n  File "/home/shager/cpython-buildarea/3.x.edelsohn-powerlinux-ppc64/build/Lib/threading.py", line 619, in _bootstrap\n    self._bootstrap_inner()\n\nThread 1 (Thread 0x8074e46670 (LWP 1616)):\nTraceback (most recent call first):\n  File "<string>", line 18, in <module>\n'

Is this an optimized or a debug build?  (aka --with-pydebug)

What are the optimization flags passed to the C compiler?

Unoptimized debug build (configured using --with-pydebug). Buildbot test failure running on gcc110 in GCC Compile Farm (running Fedora).

gcc -pthread -c -Wno-unused-result -g -O0 -Wall -Wstrict-prototypes -m64

Thanks.  I can't see from that output what's going wrong.

Can you apply the following patch, which (I hope) will print more detailed info where the failure happens.

Looking at test_gdb.out, the issue is that the threads waiting for the GIL have e.g. this at the top of their backtrace:

Thread 2 (Thread 0x3fffb14af200 (LWP 37119)):
#0  0x0000008075181ea8 in __pthread_cond_timedwait (cond=0x103ea140 <gil_cond>, mutex=0x103ea170 <gil_mutex>, abstime=0x3fffb14abf68) at pthread_cond_timedwait.c:167
#1  0x00000000100f59f8 in PyCOND_TIMEDWAIT (cond=0x103ea140 <gil_cond>, mut=0x103ea170 <gil_mutex>, us=5000) at /home/dje/src/cpython/Python/condvar.h:103
#2  0x00000000100f5ffc in take_gil (tstate=0x1043b020) at /home/dje/src/cpython/Python/ceval_gil.h:224

i.e. the topmost frame has name "__pthread_cond_timedwait" (two leading underscores.

However, the gdb hooks (in Tools/gdb/libpython.py) have:

    def is_waiting_for_gil(self):
        '''Is this frame waiting on the GIL?'''
        # This assumes the _POSIX_THREADS version of Python/ceval_gil.h:
        name = self._gdbframe.name()
        if name:
            return name.startswith('pthread_cond_timedwait')

i.e. no underscores, hence that name.startswith conditional doesn't match.

Can you try this candidate fix?

__pthread_cond_timedwait with underscore may be a general GLibc change.

The patch fixes the failure.

It's not a change in glibc.  __pthread_cond_timedwait is the internal function name while pthread_cond_timedwait is the exported alias.  You're seeing __pthread_cond_timedwait here because either your glibc installation has debug symbols or you have debug info packages installed, which allows gdb to resolve the function name to the internal name.  Without glibc debug info you should see just pthread_cond_timedwait@@... or just pthread_cond_timedwait.

In any case, I guess you'd be better off just using .find("pthread_cond_timedwait") instead of startswith() since I've also seen this on ppc64, which might break your test again:

Thread 2 (Thread 0x3fffb7d1f200 (LWP 5746)):
#0  0x00003fffb7f21ec8 in .pthread_cond_timedwait () from /lib64/libpthread.so.0

I'm not entirely sure what the preceding dot means, but it seems to indicate a function call outside the binary in ppc64.

Thanks for explaining the issue with GLibc symbols.  The Python test definitely should not fail if debugging symbols are installed.

The "dot" symbols should no longer occur with modern PowerLinux installations.  They were a carry-over from the AIX ABI on which the PPC64 Linux ABI is based.

New changeset f4a6b731905a by David Malcolm in branch '3.3':
#17833: fix test_gdb failures seen on PPC64 Linux in test_threads (test.test_gdb.PyBtTests)
http://hg.python.org/cpython/rev/f4a6b731905a

New changeset 6d971b172389 by David Malcolm in branch 'default':
#17833: merge with 3.3
http://hg.python.org/cpython/rev/6d971b172389