test_gdb fails in s390x SLES buildbots #78188
Comments
|
test_gdb is currently failing on the following buildbots: http://buildbot.python.org/all/#/builders/54/builds/465 Sample error lines: ====================================================================== Traceback (most recent call last):
File "/home/dje/cpython-buildarea/3.7.edelsohn-sles-z/build/Lib/test/test_gdb.py", line 776, in test_threads
cmds_after_breakpoint=['thread apply all py-bt'])
File "/home/dje/cpython-buildarea/3.7.edelsohn-sles-z/build/Lib/test/test_gdb.py", line 213, in get_stack_trace
self.assertEqual(unexpected_errlines, [])
AssertionError: Lists differ: ["Python Exception <class 'gdb.error'> PC [58 chars]ved'] != []
First list contains 2 additional elements.
First extra element 0:
"Python Exception <class 'gdb.error'> PC not saved: "
+ []
- ["Python Exception <class 'gdb.error'> PC not saved: ",
- 'Error occurred in Python command: PC not saved']Ran 46 tests in 19.888s |
|
In gdb source code, frame_unwind_pc(), I see: else if (this_frame->prev_pc.status == CC_NOT_SAVED) |
|
All the builds fail consistently after http://buildbot.python.org/all/#/builders/54/builds/465 and things were passing before 465. Maybe is this something to do with the commit that happened there that we could revert and test it in the machine by doing a git-bisect? But I couldn't see anything on GDB related code being changed in Python on #465. My initial hunch was that it had something to do with GDB 8.0+ but Ubuntu 18.04 LTS also runs on 8.0+ fine. Tests pass : 00e0524 ➜ cpython git:(master) git log 00e0524~1..d6a283b commit d6a283b commit a730279 commit 00e0524 |
|
New failure on s390x SLES 3.7: |
|
This failure seems related to the issue: https://buildbot.python.org/all/#/builders/139/builds/74 ===================================================================== Traceback (most recent call last):
File "/srv/buildbot/buildarea/2.7.bolen-ubuntu/build/Lib/test/test_gdb.py", line 808, in test_threads
self.assertIn('Waiting for the GIL', gdb_output)
test test_gdb failed -- Traceback (most recent call last):
File "/srv/buildbot/buildarea/2.7.bolen-ubuntu/build/Lib/test/test_gdb.py", line 808, in test_threads
self.assertIn('Waiting for the GIL', gdb_output)
AssertionError: 'Waiting for the GIL' not found in 'Breakpoint 1 (PyObject_Print) pending.\n..... |
|
I requested a SLES VM in https://linuxone20.cloud.marist.edu as indicated by @david Edelsohn (the builtbot maintainer) and I can reproduce the issue on master, 3.7, 3.6 and 3.5. So it seems that a change in gdb itself caused this. Interestingly, in Python 3.5 the error is different: ====================================================================== Traceback (most recent call last):
File "/home/linux1/cpython/Lib/test/test_gdb.py", line 771, in test_threads
self.assertIn('Waiting for the GIL', gdb_output)
AssertionError: 'Waiting for the GIL' not found in 'Breakpoint 1 at 0x11751d8: file Python/bltinmodule.c, line 1093.\n[Thread debugging using libthread_db enabled]\nUsing host libthread_db library "/lib64/libthread_db.so.1".\n[New Thread 0x3fffd2ff910 (LWP 33928)]\n[New Thread 0x3fffcaff910 (LWP 33929)]\n'Ran 45 tests in 19.522s |
|
I think I understand what is happening. The SLES buildbots do not have debugging symbols for glibc: (gdb) set verbose on And when requesting the stacktrace of a thread it cannot go into libc because the program counter is not available: (gdb) thread apply 2 where #19 0x0000000001324f02 in method_call (method=<method at remote 0x3fffdeee258>, args=(), kwargs=0x0) at Objects/classobject.c:306 This get triggered when printing the python stack trace with py-bt and the exception is getting printed on stderr and this fails the test. |
|
I need to do some more experimenting because those symbols are not available on other systems with the test passing. Something is corrupting/not saving the program counter but I do not know what. |
|
Walking the stack up one by one only triggers the "PC not saved" when the stack goes into libc: (gdb) break builtin_id Thread 1 "python" hit Breakpoint 1, builtin_id (self=0x3fffdf5c0c8, v=0x14f7690 <small_ints+2256>) at Python/bltinmodule.c:1182 |
|
Interestingly, everything points to something special in the glibc version of the buildbot. Usually the end of the stack for threads is: (gdb) but in this buildbot we get: #24 0x000003fffde88b22 in start_thread () from /lib64/libpthread.so.0 I think that the combination of glibc + gdb is producing this problem probably because the optimization level of glibc. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: