https://buildbot.python.org/all/#/builders/727

test_tuples (test.test_gdb.PrettyPrintTests)
Verify the pretty-printing of tuples ... ok
test_bt (test.test_gdb.PyBtTests)
Verify that the "py-bt" command works ... FAIL
Stderr:
Python Exception <class 'gdb.MemoryError'> Cannot access memory at address 0xfffffedc: 
Error occurred in Python command: Cannot access memory at address 0xfffffedc

and a pile of similar errors after test_bt

Marking release blocker as this isn't present in 3.8 or earlier, we've got a regression here.

i haven't tried to debug yet, but that pointer value smells like something did a negative offset from NULL...

Some data from a recent build:
https://buildbot.python.org/all/#/builders/645/builds/480

That's 32-bit ARM, armv7l ABI with GCC 8.3 and GDB 8.2.

Python is built with -Og optimization level. In my experience, gdb fails to read debug symbols when Python is optimized at -Og level. Last time I asked a gdb developer, he told me that he always use -O0, so do I.


test.pythoninfo:

platform.architecture: 32bit ELF
platform.libc_ver: glibc 2.28
platform.platform: Linux-4.19.97-v7l+-armv7l-with-glibc2.28
platform.python_implementation: CPython

CC.version: gcc (Raspbian 8.3.0-6+rpi1) 8.3.0
sysconfig[PY_CFLAGS]: -Wno-unused-result -Wsign-compare -g -Og -Wall
sysconfig[PY_CFLAGS_NODIST]: -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden -I./Include/internal
sysconfig[PY_STDMODULE_CFLAGS]: -Wno-unused-result -Wsign-compare -g -Og -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden -I./Include/internal -I. -I./Include

gdb_version: GNU gdb (Raspbian 8.2.1-2) 8.2.1

> Marking release blocker as this isn't present in 3.8 or earlier, we've got a regression here.

FYI we have issue with test_gdb on Fedora for at least 5 years. Most issues come from -Og optimization level of test_gdb.

Recently, I modified test_gdb to skip tests when we detect that gdb fails to read debug symbols, which happens often when using -Og: bpo-40019.

commit 7bf069b6110278102c8f4719975a5eb5a5af25f9
Author: Victor Stinner <vstinner@python.org>
Date:   Fri Mar 20 08:23:26 2020 +0100

    bpo-40019: Skip test_gdb if Python was optimized (GH-19081)
    
    test_gdb now skips tests if it detects that gdb failed to read debug
    information because the Python binary is optimized.

I closed bpo-17737 "test_gdb fails on armv7hl" as out of date.

This will miss 3.9.0b4.

Note: 3.9.0b5, the last beta before 3.9.0, is next week.

gdb doesn't work fully when Python is built with -Og. I don't think that there is much that we can do. I suggest to skip test_gdb on 32-bit ARM when Python is built with -Og.

I don't think that this regular requires the "release blocker" priority. It would be nice to fix but it should not hold the 3.9.0 final release.

Downgrading since this missed rc1 anyway.

test_gdb now pass on ARM Raspbian 3.x. I'm not sure why test_gdb fails on 3.10 but pass on 3.x.

For me, it looks more like a gcc or gdb issue, rather than a Python bug.

ARM Raspbian 3.10:
https://buildbot.python.org/all/#/builders/685/builds/76

test.pythoninfo:

CC.version: gcc (Raspbian 10.2.1-6+rpi1) 10.2.1 20210110
gdb_version: GNU gdb (Raspbian 10.1-1.7) 10.1.90.20210103-git

sysconfig[CCSHARED]: -fPIC
sysconfig[CC]: gcc -pthread
sysconfig[CFLAGS]: -Wno-unused-result -Wsign-compare -g -Og -Wall
sysconfig[CONFIG_ARGS]: '--prefix' '/var/lib/buildbot/workers/3.10.gps-raspbian/build/target' '--with-pydebug'
sysconfig[HOST_GNU_TYPE]: armv7l-unknown-linux-gnueabihf
sysconfig[MACHDEP]: linux
sysconfig[MULTIARCH]: arm-linux-gnueabihf
sysconfig[OPT]: -g -Og -Wall
sysconfig[PY_CFLAGS]: -Wno-unused-result -Wsign-compare -g -Og -Wall
sysconfig[PY_CFLAGS_NODIST]: -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden -I./Include/internal
sysconfig[PY_STDMODULE_CFLAGS]: -Wno-unused-result -Wsign-compare -g -Og -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden -I./Include/internal -I. -I./Include
sysconfig[Py_DEBUG]: 1

Logs:

======================================================================
FAIL: test_bt (test.test_gdb.PyBtTests)
Verify that the "py-bt" command works
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/lib/buildbot/workers/3.10.gps-raspbian/build/Lib/test/test_gdb.py", line 776, in test_bt
    self.assertMultilineMatches(bt,
  File "/var/lib/buildbot/workers/3.10.gps-raspbian/build/Lib/test/test_gdb.py", line 295, in assertMultilineMatches
    self.fail(msg='%r did not match %r' % (actual, pattern))
AssertionError: 'Breakpoint 1 at 0x258ba4: file Python/bltinmodule.c, line 1194.\n[Thread debugging using libthread_db enabled]\nUsing host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".\n\nBreakpoint 1, builtin_id (self=, v=42) at Python/bltinmodule.c:1194\n1194\t{\nTraceback (most recent call first):\n  <built-in method id of module object at remote 0xb68b8360>\n' did not match '^.*\nTraceback \\(most recent call first\\):\n  <built-in method id of module object .*>\n  File ".*gdb_sample.py", line 10, in baz\n    id\\(42\\)\n  File ".*gdb_sample.py", line 7, in bar\n    baz\\(a, b, c\\)\n  File ".*gdb_sample.py", line 4, in foo\n    bar\\(a, b, c\\)\n  File ".*gdb_sample.py", line 12, in <module>\n    foo\\(1, 2, 3\\)\n'

Stderr:
Python Exception <class 'gdb.MemoryError'> Cannot access memory at address 0xfdfdfd83: 
Error occurred in Python: Cannot access memory at address 0xfdfdfd83

fwiw I updated my arm raspbian buildbot from raspbian Buster to raspbian Bullseye in the last few days.  That could also explain the difference.  More recent toolchain versions.

although it looks like the 3.10 failure you linked to ran after that.  so... the upgrade may not explain things.

I manually tested this on 3.11 main and it appears to be working.