New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
test_gdb fails on AMD64 Fedora Rawhide 3.x [gdb 9.2 bug] #85645
Comments
https://buildbot.python.org/all/#builders/1/builds/1194 0:07:05 load avg: 16.74 [421/423/1] test_gdb failed (2 min 29 sec) -- running: test_capi (1 min 28 sec), test_lib2to3 (1 min 22 sec) Traceback (most recent call last):
File "/home/buildbot/buildarea/3.x.cstratak-fedora-rawhide-x86_64/build/Lib/test/test_gdb.py", line 332, in test_dicts
self.assertGdbRepr({'foo': 'bar'}, "{'foo': 'bar'}")
File "/home/buildbot/buildarea/3.x.cstratak-fedora-rawhide-x86_64/build/Lib/test/test_gdb.py", line 308, in assertGdbRepr
gdb_repr, gdb_output = self.get_gdb_repr('id(' + ascii(val) + ')')
File "/home/buildbot/buildarea/3.x.cstratak-fedora-rawhide-x86_64/build/Lib/test/test_gdb.py", line 284, in get_gdb_repr
self.fail('Unexpected gdb output: %r\n%s' % (gdb_output, gdb_output))
AssertionError: Unexpected gdb output: 'Breakpoint 1 at 0x5fbc79: file Python/bltinmodule.c, line 1173.\n[Thread debugging using libthread_db enabled]\nUsing host libthread_db library "/lib64/libthread_db.so.1".\n'
Breakpoint 1 at 0x5fbc79: file Python/bltinmodule.c, line 1173.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1". Stderr: ====================================================================== Traceback (most recent call last):
File "/home/buildbot/buildarea/3.x.cstratak-fedora-rawhide-x86_64/build/Lib/test/test_gdb.py", line 912, in test_pycfunction
self.assertIn(f'<built-in method {func_name}', gdb_output)
AssertionError: '<built-in method meth_noargs' not found in 'Breakpoint 1 (meth_noargs) pending.\n[Thread debugging using libthread_db enabled]\nUsing host libthread_db library "/lib64/libthread_db.so.1".\nUnable to locate python frame\n' Stderr: 0:07:19 load avg: 15.24 Re-running test_gdb in verbose mode |
pythoninfo: sys.version: 3.10.0a0 (heads/master:da4e09fff6, Aug 4 2020, 03:30:43) [GCC 10.2.1 20200723 (Red Hat 10.2.1-1)] gdb_version: GNU gdb (GDB) Fedora 9.2-2.fc33 |
I failed to reproduce the issue on an up-to-date Fedora Rawhide with gdb-9.2-2.fc33.x86_64 ("GNU gdb (GDB) Fedora 9.2-2.fc33"). |
Other failures on a different architecture, aarch64 Fedora Rawhide Clang 3.9: https://buildbot.python.org/all/#/builders/687/builds/152 "gdb_version: GNU gdb (GDB) Fedora 9.2-2.fc33" "Stderr: https://buildbot.python.org/all/#/builders/687/builds/160 "gdb_version: GNU gdb (GDB) Fedora 9.2-2.fc33" "Stderr: See also bpo-40746: "test_gdb failing on 32-bit armv7l when built with GCC -Og (fail on Raspbian on 3.9, regression from 3.8)". |
I looked at the aarch64 failure on Fedora Rawhide using GCC and the master branch. When I run "./python -m test -v test_gdb -m test_pycfunction" multiple times, I get a different error at each run, and sometimes the test pass, whereas nothing changes between the runs.
When gdb fails to rebuild the stack file, it logs the message: "opening file=(...)/_testcapi.cpython-310d-aarch64-linux-gnu.so [0]; direct_opencount=1". It seems to be an issue in gdb, not in Python. I modified test_gdb.py to get subprocess arguments and output (out/err). == Logs of a success == args=('gdb', '--batch', '-nx', '-iex', 'add-auto-load-safe-path /home/vstinner/python/master/python-gdb.py', '--eval-command=set breakpoint pending yes', '--eval-command=break meth_fastcall', '--eval-command=set print address off', '--eva
l-command=run', '--eval-command=set print entry-values no', '--eval-command=bt', '--eval-command=py-bt', '--args', '/home/vstinner/python/master/python', '-S', '-c', '\nimport _testcapi\ndef foo():\n _testcapi.meth_fastcall()\ndef bar(
):\n foo()\nbar()\n')
--out--
Breakpoint 1 (meth_fastcall) pending.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Breakpoint 1, meth_fastcall (self=<module at remote 0xffffea684e90>, args=, nargs=0) at /home/vstinner/python/master/Modules/_testcapimodule.c:5255
5255 {
#0 meth_fastcall (self=<module at remote 0xffffea684e90>, args=, nargs=0) at /home/vstinner/python/master/Modules/_testcapimodule.c:5255
#1 cfunction_vectorcall_FASTCALL (func=<built-in method meth_fastcall of module object at remote 0xffffea684e90>, args=, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/methodobject.c:424
#2 _PyObject_VectorcallTstate (tstate=, callable=<built-in method meth_fastcall of module object at remote 0xffffea684e90>, args=, nargsf=9223372036854775808, kwnames=0x0) at ./Include/cpython/abstract.h:114
#3 PyObject_Vectorcall (kwnames=0x0, nargsf=9223372036854775808, args=, callable=<built-in method meth_fastcall of module object at remote 0xffffea684e90>) at ./Include/cpython/abstract.h:123
#4 call_function (tstate=, pp_stack=, oparg=0, kwnames=0x0) at Python/ceval.c:5121
#5 _PyEval_EvalFrameDefault (tstate=, f=Frame 0x883070, for file <string>, line 4, in foo (), throwflag=<optimized out>) at Python/ceval.c:3516
#6 _PyEval_EvalFrame (throwflag=0, f=Frame 0x883070, for file <string>, line 4, in foo (), tstate=) at ./Include/internal/pycore_ceval.h:40
#7 function_code_fastcall (tstate=, co=<optimized out>, args=, nargs=0, globals=<optimized out>) at Objects/call.c:329
#8 _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:366
#9 _PyObject_VectorcallTstate (tstate=, callable=<function at remote 0xffffea711a50>, args=, nargsf=9223372036854775808, kwnames=0x0) at ./Include/cpython/abstract.h:114
#10 PyObject_Vectorcall (kwnames=0x0, nargsf=9223372036854775808, args=, callable=<function at remote 0xffffea711a50>) at ./Include/cpython/abstract.h:123
#11 call_function (tstate=, pp_stack=, oparg=0, kwnames=0x0) at Python/ceval.c:5121
#12 _PyEval_EvalFrameDefault (tstate=, f=Frame 0xffffea67f050, for file <string>, line 6, in bar (), throwflag=<optimized out>) at Python/ceval.c:3547
#13 _PyEval_EvalFrame (throwflag=0, f=Frame 0xffffea67f050, for file <string>, line 6, in bar (), tstate=) at ./Include/internal/pycore_ceval.h:40
#14 function_code_fastcall (tstate=, co=<optimized out>, args=, nargs=0, globals=<optimized out>) at Objects/call.c:329
#15 _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:366
#16 _PyObject_VectorcallTstate (tstate=, callable=<function at remote 0xffffea682410>, args=, nargsf=9223372036854775808, kwnames=0x0) at ./Include/cpython/abstract.h:114
#17 PyObject_Vectorcall (kwnames=0x0, nargsf=9223372036854775808, args=, callable=<function at remote 0xffffea682410>) at ./Include/cpython/abstract.h:123
#18 call_function (tstate=, pp_stack=, oparg=0, kwnames=0x0) at Python/ceval.c:5121
#19 _PyEval_EvalFrameDefault (tstate=, f=Frame 0xffffea6c15c0, for file <string>, line 7, in <module> (), throwflag=<optimized out>) at Python/ceval.c:3547
(...)
--out--done
--err--
Function "meth_fastcall" not defined.
--err--done == Logs of a failure == args=('gdb', '--batch', '-nx', '-iex', 'add-auto-load-safe-path /home/vstinner/python/master/python-gdb.py', '--eval-command=set breakpoint pending yes', '--eval-command=break meth_fastcall', '--eval-command=set print address off', '--eva
l-command=run', '--eval-command=set print entry-values no', '--eval-command=bt', '--eval-command=py-bt', '--args', '/home/vstinner/python/master/python', '-S', '-c', '\nimport _testcapi\ndef foo():\n _testcapi.meth_fastcall()\ndef bar(
):\n foo()\nbar()\n')
--out--
Breakpoint 1 (meth_fastcall) pending.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1". Program received signal SIGTRAP, Trace/breakpoint trap. --out--done --err--done |
More logs on the AArch64 issue. It seems like *sometimes*, gdb fails to parse debug symbols of _testcapi.cpython-310d-aarch64-linux-gnu.so, whereas most of the time, "bt" shows symbols as expected. stdout+stderr when the bug happens. ####################################################### Program received signal SIGTRAP, Trace/breakpoint trap. Extract of a successful run: Breakpoint 1, meth_fastcall (self=<module at remote 0xffffea683e90>, args=, nargs=0) at /home/vstinner/python/master/Modules/_testcapimodule.c:5255 I can reproduce the issue with command "./script.sh" using the following files. script.sh: $ cat script.sh
OUT=$(mktemp)
while true; do
gdb --batch -x cmds -args ./python -S x.py < /dev/null 2>&1 | tee $OUT
grep -q -F '<built-in method meth_fastcall' $OUT || break
echo
echo "===================================================="
echo
done echo "#######################################################" rm -f $OUT x.py: import _testcapi
def foo():
# _testcapi.meth_varargs()
_testcapi.meth_fastcall()
def bar():
foo()
bar() cmds: |
I reported https://bugzilla.redhat.com/show_bug.cgi?id=1866884 to gdb: "Arch64: sometimes, gdb fails to load symbols of a dynamic library with a pending breakpoint". |
Since I identified a gdb regression in gdb 9.2, I wrote PR 21768 to skip test_gdb on gdb >= 9.2 until the bug is fixed: |
https://bugzilla.redhat.com/show_bug.cgi?id=1866884 "AArch64: sometimes, gdb fails to load symbols of a dynamic library with a pending breakpoint": gdb bug has been confirmed. |
test_gdb pass again on AMD64 Fedora Rawhide 3.x with "GNU gdb (GDB) Fedora 10.1-2.fc34": https://buildbot.python.org/all/#/builders/285/builds/348 I close the issue. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: