https://buildbot.python.org/all/#builders/1/builds/1194

0:07:05 load avg: 16.74 [421/423/1] test_gdb failed (2 min 29 sec) -- running: test_capi (1 min 28 sec), test_lib2to3 (1 min 22 sec)
(...)
======================================================================
FAIL: test_dicts (test.test_gdb.PrettyPrintTests)
Verify the pretty-printing of dictionaries
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-rawhide-x86_64/build/Lib/test/test_gdb.py", line 332, in test_dicts
    self.assertGdbRepr({'foo': 'bar'}, "{'foo': 'bar'}")
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-rawhide-x86_64/build/Lib/test/test_gdb.py", line 308, in assertGdbRepr
    gdb_repr, gdb_output = self.get_gdb_repr('id(' + ascii(val) + ')')
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-rawhide-x86_64/build/Lib/test/test_gdb.py", line 284, in get_gdb_repr
    self.fail('Unexpected gdb output: %r\n%s' % (gdb_output, gdb_output))
AssertionError: Unexpected gdb output: 'Breakpoint 1 at 0x5fbc79: file Python/bltinmodule.c, line 1173.\n[Thread debugging using libthread_db enabled]\nUsing host libthread_db library "/lib64/libthread_db.so.1".\n'
Breakpoint 1 at 0x5fbc79: file Python/bltinmodule.c, line 1173.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".


Stderr:
Couldn't get registers: No such process.
Selected thread is running.

======================================================================
FAIL: test_pycfunction (test.test_gdb.PyBtTests) [_testcapi.meth_noargs]
Verify that "py-bt" displays invocations of PyCFunction instances
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-rawhide-x86_64/build/Lib/test/test_gdb.py", line 912, in test_pycfunction
    self.assertIn(f'<built-in method {func_name}', gdb_output)
AssertionError: '<built-in method meth_noargs' not found in 'Breakpoint 1 (meth_noargs) pending.\n[Thread debugging using libthread_db enabled]\nUsing host libthread_db library "/lib64/libthread_db.so.1".\nUnable to locate python frame\n'

Stderr:
Function "meth_varargs" not defined.
Function "meth_varargs" not defined.
(...)
Function "meth_varargs_keywords" not defined.
Function "meth_varargs_keywords" not defined.
(...)
Function "meth_o" not defined.
Function "meth_o" not defined.
(...)
Couldn't get registers: No such process.
Selected thread is running.
Function "meth_noargs" not defined.
(...)



0:07:19 load avg: 15.24 Re-running test_gdb in verbose mode
test_NULL_ob_type (test.test_gdb.PrettyPrintTests)
Ensure that a PyObject* with NULL ob_type is handled gracefully ... ok
test_NULL_ptr (test.test_gdb.PrettyPrintTests)
Ensure that a NULL PyObject* is handled gracefully ... ok
test_builtin_method (test.test_gdb.PrettyPrintTests) ... ok
test_builtins_help (test.test_gdb.PrettyPrintTests)
Ensure that the new-style class _Helper in site.py can be handled ... ok
test_bytes (test.test_gdb.PrettyPrintTests)
Verify the pretty-printing of bytes ... ok
test_corrupt_ob_type (test.test_gdb.PrettyPrintTests)
Ensure that a PyObject* with a corrupt ob_type is handled gracefully ... ok
test_corrupt_tp_flags (test.test_gdb.PrettyPrintTests)
Ensure that a PyObject* with a type with corrupt tp_flags is handled ... ok
test_corrupt_tp_name (test.test_gdb.PrettyPrintTests)
Ensure that a PyObject* with a type with corrupt tp_name is handled ... ok
test_dicts (test.test_gdb.PrettyPrintTests)
Verify the pretty-printing of dictionaries ... ok
test_exceptions (test.test_gdb.PrettyPrintTests) ... ok
test_frames (test.test_gdb.PrettyPrintTests) ... ok
test_frozensets (test.test_gdb.PrettyPrintTests)
Verify the pretty-printing of frozensets ... ok
test_getting_backtrace (test.test_gdb.PrettyPrintTests) ... ok
test_int (test.test_gdb.PrettyPrintTests)
Verify the pretty-printing of various int values ... ok
test_lists (test.test_gdb.PrettyPrintTests)
Verify the pretty-printing of lists ... ok
test_modern_class (test.test_gdb.PrettyPrintTests)
Verify the pretty-printing of new-style class instances ... ok
test_selfreferential_dict (test.test_gdb.PrettyPrintTests)
Ensure that a reference loop involving a dict doesn't lead proxyval ... ok
test_selfreferential_list (test.test_gdb.PrettyPrintTests)
Ensure that a reference loop involving a list doesn't lead proxyval ... ok
test_selfreferential_new_style_instance (test.test_gdb.PrettyPrintTests) ... ok
test_selfreferential_old_style_instance (test.test_gdb.PrettyPrintTests) ... ok
test_sets (test.test_gdb.PrettyPrintTests)
Verify the pretty-printing of sets ... ok
test_singletons (test.test_gdb.PrettyPrintTests)
Verify the pretty-printing of True, False and None ... ok
test_strings (test.test_gdb.PrettyPrintTests)
Verify the pretty-printing of unicode strings ... ok
test_subclassing_list (test.test_gdb.PrettyPrintTests)
Verify the pretty-printing of an instance of a list subclass ... ok
test_subclassing_tuple (test.test_gdb.PrettyPrintTests)
Verify the pretty-printing of an instance of a tuple subclass ... ok
test_truncation (test.test_gdb.PrettyPrintTests)
Verify that very long output is truncated ... ok
test_tuples (test.test_gdb.PrettyPrintTests)
Verify the pretty-printing of tuples ... ok
test_bt (test.test_gdb.PyBtTests)
Verify that the "py-bt" command works ... ok
test_bt_full (test.test_gdb.PyBtTests)
Verify that the "py-bt-full" command works ... ok
test_gc (test.test_gdb.PyBtTests)
Verify that "py-bt" indicates if a thread is garbage-collecting ... ok
test_pycfunction (test.test_gdb.PyBtTests)
Timeout (0:15:00)!
Thread 0x00007f195dda7740 (most recent call first):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-rawhide-x86_64/build/Lib/selectors.py", line 416 in select
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-rawhide-x86_64/build/Lib/subprocess.py", line 1977 in _communicate
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-rawhide-x86_64/build/Lib/subprocess.py", line 1130 in communicate
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-rawhide-x86_64/build/Lib/test/test_gdb.py", line 112 in run_gdb
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-rawhide-x86_64/build/Lib/test/test_gdb.py", line 225 in get_stack_trace
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-rawhide-x86_64/build/Lib/test/test_gdb.py", line 907 in test_pycfunction
...

> https://buildbot.python.org/all/#builders/1/builds/1194

pythoninfo:

sys.version: 3.10.0a0 (heads/master:da4e09fff6, Aug  4 2020, 03:30:43)  [GCC 10.2.1 20200723 (Red Hat 10.2.1-1)]

gdb_version: GNU gdb (GDB) Fedora 9.2-2.fc33

I failed to reproduce the issue on an up-to-date Fedora Rawhide with gdb-9.2-2.fc33.x86_64 ("GNU gdb (GDB) Fedora 9.2-2.fc33").

Other failures on a different architecture, aarch64 Fedora Rawhide Clang 3.9:

https://buildbot.python.org/all/#/builders/687/builds/152

"gdb_version: GNU gdb (GDB) Fedora 9.2-2.fc33"

"Stderr:
Unable to fetch general registers.: No such process.
Selected thread is running."

https://buildbot.python.org/all/#/builders/687/builds/160

"gdb_version: GNU gdb (GDB) Fedora 9.2-2.fc33"

"Stderr:
Unable to fetch general registers.: No such process.
Selected thread is running."

See also bpo-40746: "test_gdb failing on 32-bit armv7l when built with GCC -Og (fail on Raspbian on 3.9, regression from 3.8)".

I looked at the aarch64 failure on Fedora Rawhide using GCC and the master branch.

When I run "./python -m test -v test_gdb  -m test_pycfunction" multiple times, I get a different error at each run, and sometimes the test pass, whereas nothing changes between the runs.

* gcc (GCC) 10.2.1 20200723 (Red Hat 10.2.1-1) [gcc-10.2.1-1.fc33.aarch64]
* GNU gdb (GDB) Fedora 9.2-2.fc33 [gdb-9.2-2.fc33.aarch64]

When gdb fails to rebuild the stack file, it logs the message: "opening file=(...)/_testcapi.cpython-310d-aarch64-linux-gnu.so [0]; direct_opencount=1".

It seems to be an issue in gdb, not in Python.

I modified test_gdb.py to get subprocess arguments and output (out/err).


== Logs of a success ==

args=('gdb', '--batch', '-nx', '-iex', 'add-auto-load-safe-path /home/vstinner/python/master/python-gdb.py', '--eval-command=set breakpoint pending yes', '--eval-command=break meth_fastcall', '--eval-command=set print address off', '--eva
l-command=run', '--eval-command=set print entry-values no', '--eval-command=bt', '--eval-command=py-bt', '--args', '/home/vstinner/python/master/python', '-S', '-c', '\nimport _testcapi\ndef foo():\n    _testcapi.meth_fastcall()\ndef bar(
):\n    foo()\nbar()\n')                                                                                                                                                                                                                      
--out--                                                                                                                                                                                                                                       
Breakpoint 1 (meth_fastcall) pending.                                                                                                                                                                                                         
[Thread debugging using libthread_db enabled]                                                                                                                                                                                                 
Using host libthread_db library "/lib64/libthread_db.so.1".                                                                                                                                                                                   
                                                                                                                                                                                                                                              
Breakpoint 1, meth_fastcall (self=<module at remote 0xffffea684e90>, args=, nargs=0) at /home/vstinner/python/master/Modules/_testcapimodule.c:5255                                                                                           
5255    {                                                                                                                                                                                                                                     
#0  meth_fastcall (self=<module at remote 0xffffea684e90>, args=, nargs=0) at /home/vstinner/python/master/Modules/_testcapimodule.c:5255                                                                                                     
#1  cfunction_vectorcall_FASTCALL (func=<built-in method meth_fastcall of module object at remote 0xffffea684e90>, args=, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/methodobject.c:424                                      
#2  _PyObject_VectorcallTstate (tstate=, callable=<built-in method meth_fastcall of module object at remote 0xffffea684e90>, args=, nargsf=9223372036854775808, kwnames=0x0) at ./Include/cpython/abstract.h:114                              
#3  PyObject_Vectorcall (kwnames=0x0, nargsf=9223372036854775808, args=, callable=<built-in method meth_fastcall of module object at remote 0xffffea684e90>) at ./Include/cpython/abstract.h:123                                              
#4  call_function (tstate=, pp_stack=, oparg=0, kwnames=0x0) at Python/ceval.c:5121                                                                                                                                                           
#5  _PyEval_EvalFrameDefault (tstate=, f=Frame 0x883070, for file <string>, line 4, in foo (), throwflag=<optimized out>) at Python/ceval.c:3516                                                                                              
#6  _PyEval_EvalFrame (throwflag=0, f=Frame 0x883070, for file <string>, line 4, in foo (), tstate=) at ./Include/internal/pycore_ceval.h:40                                                                                                  
#7  function_code_fastcall (tstate=, co=<optimized out>, args=, nargs=0, globals=<optimized out>) at Objects/call.c:329                                                                                                                       
#8  _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:366                                                                                               
#9  _PyObject_VectorcallTstate (tstate=, callable=<function at remote 0xffffea711a50>, args=, nargsf=9223372036854775808, kwnames=0x0) at ./Include/cpython/abstract.h:114                                                                    
#10 PyObject_Vectorcall (kwnames=0x0, nargsf=9223372036854775808, args=, callable=<function at remote 0xffffea711a50>) at ./Include/cpython/abstract.h:123                                                                                    
#11 call_function (tstate=, pp_stack=, oparg=0, kwnames=0x0) at Python/ceval.c:5121                                                                                                                                                           
#12 _PyEval_EvalFrameDefault (tstate=, f=Frame 0xffffea67f050, for file <string>, line 6, in bar (), throwflag=<optimized out>) at Python/ceval.c:3547                                                                                        
#13 _PyEval_EvalFrame (throwflag=0, f=Frame 0xffffea67f050, for file <string>, line 6, in bar (), tstate=) at ./Include/internal/pycore_ceval.h:40                                                                                            
#14 function_code_fastcall (tstate=, co=<optimized out>, args=, nargs=0, globals=<optimized out>) at Objects/call.c:329                                                                                                                       
#15 _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:366                                                                                               
#16 _PyObject_VectorcallTstate (tstate=, callable=<function at remote 0xffffea682410>, args=, nargsf=9223372036854775808, kwnames=0x0) at ./Include/cpython/abstract.h:114                                                                    
#17 PyObject_Vectorcall (kwnames=0x0, nargsf=9223372036854775808, args=, callable=<function at remote 0xffffea682410>) at ./Include/cpython/abstract.h:123                                                                                    
#18 call_function (tstate=, pp_stack=, oparg=0, kwnames=0x0) at Python/ceval.c:5121                                                                                                                                                           
#19 _PyEval_EvalFrameDefault (tstate=, f=Frame 0xffffea6c15c0, for file <string>, line 7, in <module> (), throwflag=<optimized out>) at Python/ceval.c:3547                                
(...)
                                                                                                                                                                                        
--out--done                                                                                                                                                                                                                                   
--err--                                                                                                                                                                                                                                       
Function "meth_fastcall" not defined.                                                                                                                                                                                                         
                                                                                                                                                                                                                                              
--err--done                                         


== Logs of a failure ==

args=('gdb', '--batch', '-nx', '-iex', 'add-auto-load-safe-path /home/vstinner/python/master/python-gdb.py', '--eval-command=set breakpoint pending yes', '--eval-command=break meth_fastcall', '--eval-command=set print address off', '--eva
l-command=run', '--eval-command=set print entry-values no', '--eval-command=bt', '--eval-command=py-bt', '--args', '/home/vstinner/python/master/python', '-S', '-c', '\nimport _testcapi\ndef foo():\n    _testcapi.meth_fastcall()\ndef bar(
):\n    foo()\nbar()\n')
--out--              
Breakpoint 1 (meth_fastcall) pending.                    
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1". 

Program received signal SIGTRAP, Trace/breakpoint trap.
?? () from /home/vstinner/python/master/build/lib.linux-aarch64-3.10-pydebug/_testcapi.cpython-310d-aarch64-linux-gnu.so
#0  ?? () from /home/vstinner/python/master/build/lib.linux-aarch64-3.10-pydebug/_testcapi.cpython-310d-aarch64-linux-gnu.so
#1  ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
Unable to locate python frame

--out--done
--err--
Function "meth_fastcall" not defined.
   1056556:     opening file=/home/vstinner/python/master/build/lib.linux-aarch64-3.10-pydebug/_testcapi.cpython-310d-aarch64-linux-gnu.so [0]; direct_opencount=1
   1056556:

--err--done

More logs on the AArch64 issue.

It seems like *sometimes*, gdb fails to parse debug symbols of _testcapi.cpython-310d-aarch64-linux-gnu.so, whereas most of the time, "bt" shows symbols as expected.

stdout+stderr when the bug happens.

#######################################################
Function "meth_fastcall" not defined.
Breakpoint 1 (meth_fastcall) pending.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
   1069559:	opening file=/home/vstinner/python/master/build/lib.linux-aarch64-3.10-pydebug/_testcapi.cpython-310d-aarch64-linux-gnu.so [0]; direct_opencount=1
   1069559:	

Program received signal SIGTRAP, Trace/breakpoint trap.
?? () from /home/vstinner/python/master/build/lib.linux-aarch64-3.10-pydebug/_testcapi.cpython-310d-aarch64-linux-gnu.so
#0  ?? () from /home/vstinner/python/master/build/lib.linux-aarch64-3.10-pydebug/_testcapi.cpython-310d-aarch64-linux-gnu.so
#1  ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
Unable to locate python frame
#######################################################

Extract of a successful run:
#######################################################
Function "meth_fastcall" not defined.
Breakpoint 1 (meth_fastcall) pending.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

Breakpoint 1, meth_fastcall (self=<module at remote 0xffffea683e90>, args=, nargs=0) at /home/vstinner/python/master/Modules/_testcapimodule.c:5255
5255	{
#0  meth_fastcall (self=<module at remote 0xffffea683e90>, args=, nargs=0) at /home/vstinner/python/master/Modules/_testcapimodule.c:5255
#1  cfunction_vectorcall_FASTCALL (func=<built-in method meth_fastcall of module object at remote 0xffffea683e90>, args=, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/methodobject.c:424
#2  _PyObject_VectorcallTstate (tstate=, callable=<built-in method meth_fastcall of module object at remote 0xffffea683e90>, args=, nargsf=9223372036854775808, kwnames=0x0) at ./Include/cpython/abstract.h:114
#######################################################


I can reproduce the issue with command "./script.sh" using the following files.


script.sh:
---
$ cat script.sh 
OUT=$(mktemp)
while true; do
  gdb --batch -x cmds -args ./python -S x.py < /dev/null 2>&1 | tee $OUT 
  grep -q -F '<built-in method meth_fastcall' $OUT || break
  echo
  echo "===================================================="
  echo
done

echo "#######################################################"
echo "BUG!"
echo "#######################################################"
cat $OUT
echo "#######################################################"

rm -f $OUT
---


x.py:
---
import _testcapi
def foo():
#    _testcapi.meth_varargs()
    _testcapi.meth_fastcall()
def bar():
    foo()
bar()
---


cmds:
---
set breakpoint pending on
add-auto-load-safe-path /home/vstinner/python/master/python-gdb.py
set breakpoint pending yes
set print address off
break meth_fastcall
run
set print entry-values no
bt
py-bt
---

I reported https://bugzilla.redhat.com/show_bug.cgi?id=1866884 to gdb: "Arch64: sometimes, gdb fails to load symbols of a dynamic library with a pending breakpoint".

Since I identified a gdb regression in gdb 9.2, I wrote PR 21768 to skip test_gdb on gdb >= 9.2 until the bug is fixed:
https://bugzilla.redhat.com/show_bug.cgi?id=1866884

New changeset e27a51c11e10d5df79b3e48dc3e7bfedfad5a794 by Victor Stinner in branch 'master':
bpo-41473: Skip test_gdb with gdb 9.2 to work around gdb bug (GH-21768)
https://github.com/python/cpython/commit/e27a51c11e10d5df79b3e48dc3e7bfedfad5a794

New changeset 87bc22051fcfcf181160d06a57ac2b16ee071f8f by Miss Islington (bot) in branch '3.8':
bpo-41473: Skip test_gdb with gdb 9.2 to work around gdb bug (GH-21768)
https://github.com/python/cpython/commit/87bc22051fcfcf181160d06a57ac2b16ee071f8f

New changeset 5e12a5b82230cfa34a9c32f58467770e2076313c by Miss Islington (bot) in branch '3.9':
bpo-41473: Skip test_gdb with gdb 9.2 to work around gdb bug (GH-21768)
https://github.com/python/cpython/commit/5e12a5b82230cfa34a9c32f58467770e2076313c

https://bugzilla.redhat.com/show_bug.cgi?id=1866884 "AArch64: sometimes, gdb fails to load symbols of a dynamic library with a pending breakpoint": gdb bug has been confirmed.

New changeset 066394018a8463643cc63d933493f0afa99d72cc by Victor Stinner in branch 'master':
bpo-41473: Reenable test_gdb on gdb 9.2 and newer (GH-23637)
https://github.com/python/cpython/commit/066394018a8463643cc63d933493f0afa99d72cc

New changeset 8e8f82dd9459b9f62c21480528d737cffd6146bc by Miss Islington (bot) in branch '3.8':
bpo-41473: Reenable test_gdb on gdb 9.2 and newer (GH-23637)
https://github.com/python/cpython/commit/8e8f82dd9459b9f62c21480528d737cffd6146bc

New changeset c7cf66d2fe1b85cc02153be6422dfc5e34811638 by Miss Islington (bot) in branch '3.9':
bpo-41473: Reenable test_gdb on gdb 9.2 and newer (GH-23637)
https://github.com/python/cpython/commit/c7cf66d2fe1b85cc02153be6422dfc5e34811638

test_gdb pass again on AMD64 Fedora Rawhide 3.x with "GNU gdb (GDB) Fedora 10.1-2.fc34":

https://buildbot.python.org/all/#/builders/285/builds/348
0:02:52 load avg: 11.34 [242/426] test_gdb passed (1 min 28 sec) (...)

I close the issue.