classification
Title: test_gdb failing on 32-bit armv7l when built with GCC -Og: Cannot access memory at address 0xfffffedc
Type: Stage:
Components: Tests Versions: Python 3.10, Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: gregory.p.smith, lukasz.langa, vstinner
Priority: high Keywords:

Created on 2020-05-23 18:30 by gregory.p.smith, last changed 2021-09-15 16:24 by gregory.p.smith.

Messages (11)
msg369736 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2020-05-23 18:30
https://buildbot.python.org/all/#/builders/727

test_tuples (test.test_gdb.PrettyPrintTests)
Verify the pretty-printing of tuples ... ok
test_bt (test.test_gdb.PyBtTests)
Verify that the "py-bt" command works ... FAIL
Stderr:
Python Exception <class 'gdb.MemoryError'> Cannot access memory at address 0xfffffedc: 
Error occurred in Python command: Cannot access memory at address 0xfffffedc

and a pile of similar errors after test_bt

Marking release blocker as this isn't present in 3.8 or earlier, we've got a regression here.

i haven't tried to debug yet, but that pointer value smells like something did a negative offset from NULL...
msg371269 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-11 13:07
Some data from a recent build:
https://buildbot.python.org/all/#/builders/645/builds/480

That's 32-bit ARM, armv7l ABI with GCC 8.3 and GDB 8.2.

Python is built with -Og optimization level. In my experience, gdb fails to read debug symbols when Python is optimized at -Og level. Last time I asked a gdb developer, he told me that he always use -O0, so do I.


test.pythoninfo:

platform.architecture: 32bit ELF
platform.libc_ver: glibc 2.28
platform.platform: Linux-4.19.97-v7l+-armv7l-with-glibc2.28
platform.python_implementation: CPython

CC.version: gcc (Raspbian 8.3.0-6+rpi1) 8.3.0
sysconfig[PY_CFLAGS]: -Wno-unused-result -Wsign-compare -g -Og -Wall
sysconfig[PY_CFLAGS_NODIST]: -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden -I./Include/internal
sysconfig[PY_STDMODULE_CFLAGS]: -Wno-unused-result -Wsign-compare -g -Og -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden -I./Include/internal -I. -I./Include

gdb_version: GNU gdb (Raspbian 8.2.1-2) 8.2.1
msg371272 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-11 13:20
> Marking release blocker as this isn't present in 3.8 or earlier, we've got a regression here.

FYI we have issue with test_gdb on Fedora for at least 5 years. Most issues come from -Og optimization level of test_gdb.

Recently, I modified test_gdb to skip tests when we detect that gdb fails to read debug symbols, which happens often when using -Og: bpo-40019.

commit 7bf069b6110278102c8f4719975a5eb5a5af25f9
Author: Victor Stinner <vstinner@python.org>
Date:   Fri Mar 20 08:23:26 2020 +0100

    bpo-40019: Skip test_gdb if Python was optimized (GH-19081)
    
    test_gdb now skips tests if it detects that gdb failed to read debug
    information because the Python binary is optimized.
msg371274 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-11 13:21
I closed bpo-17737 "test_gdb fails on armv7hl" as out of date.
msg372633 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2020-06-29 21:02
This will miss 3.9.0b4.
msg373593 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2020-07-13 11:34
Note: 3.9.0b5, the last beta before 3.9.0, is next week.
msg376719 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-09-11 10:55
gdb doesn't work fully when Python is built with -Og. I don't think that there is much that we can do. I suggest to skip test_gdb on 32-bit ARM when Python is built with -Og.

I don't think that this regular requires the "release blocker" priority. It would be nice to fix but it should not hold the 3.9.0 final release.
msg376887 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2020-09-14 17:51
Downgrading since this missed rc1 anyway.
msg401818 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-09-15 09:18
test_gdb now pass on ARM Raspbian 3.x. I'm not sure why test_gdb fails on 3.10 but pass on 3.x.

For me, it looks more like a gcc or gdb issue, rather than a Python bug.

ARM Raspbian 3.10:
https://buildbot.python.org/all/#/builders/685/builds/76

test.pythoninfo:

CC.version: gcc (Raspbian 10.2.1-6+rpi1) 10.2.1 20210110
gdb_version: GNU gdb (Raspbian 10.1-1.7) 10.1.90.20210103-git

sysconfig[CCSHARED]: -fPIC
sysconfig[CC]: gcc -pthread
sysconfig[CFLAGS]: -Wno-unused-result -Wsign-compare -g -Og -Wall
sysconfig[CONFIG_ARGS]: '--prefix' '/var/lib/buildbot/workers/3.10.gps-raspbian/build/target' '--with-pydebug'
sysconfig[HOST_GNU_TYPE]: armv7l-unknown-linux-gnueabihf
sysconfig[MACHDEP]: linux
sysconfig[MULTIARCH]: arm-linux-gnueabihf
sysconfig[OPT]: -g -Og -Wall
sysconfig[PY_CFLAGS]: -Wno-unused-result -Wsign-compare -g -Og -Wall
sysconfig[PY_CFLAGS_NODIST]: -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden -I./Include/internal
sysconfig[PY_STDMODULE_CFLAGS]: -Wno-unused-result -Wsign-compare -g -Og -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden -I./Include/internal -I. -I./Include
sysconfig[Py_DEBUG]: 1

Logs:

======================================================================
FAIL: test_bt (test.test_gdb.PyBtTests)
Verify that the "py-bt" command works
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/lib/buildbot/workers/3.10.gps-raspbian/build/Lib/test/test_gdb.py", line 776, in test_bt
    self.assertMultilineMatches(bt,
  File "/var/lib/buildbot/workers/3.10.gps-raspbian/build/Lib/test/test_gdb.py", line 295, in assertMultilineMatches
    self.fail(msg='%r did not match %r' % (actual, pattern))
AssertionError: 'Breakpoint 1 at 0x258ba4: file Python/bltinmodule.c, line 1194.\n[Thread debugging using libthread_db enabled]\nUsing host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".\n\nBreakpoint 1, builtin_id (self=, v=42) at Python/bltinmodule.c:1194\n1194\t{\nTraceback (most recent call first):\n  <built-in method id of module object at remote 0xb68b8360>\n' did not match '^.*\nTraceback \\(most recent call first\\):\n  <built-in method id of module object .*>\n  File ".*gdb_sample.py", line 10, in baz\n    id\\(42\\)\n  File ".*gdb_sample.py", line 7, in bar\n    baz\\(a, b, c\\)\n  File ".*gdb_sample.py", line 4, in foo\n    bar\\(a, b, c\\)\n  File ".*gdb_sample.py", line 12, in <module>\n    foo\\(1, 2, 3\\)\n'

Stderr:
Python Exception <class 'gdb.MemoryError'> Cannot access memory at address 0xfdfdfd83: 
Error occurred in Python: Cannot access memory at address 0xfdfdfd83
msg401850 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2021-09-15 16:23
fwiw I updated my arm raspbian buildbot from raspbian Buster to raspbian Bullseye in the last few days.  That could also explain the difference.  More recent toolchain versions.
msg401851 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2021-09-15 16:24
although it looks like the 3.10 failure you linked to ran after that.  so... the upgrade may not explain things.
History
Date User Action Args
2021-09-15 16:24:41gregory.p.smithsetmessages: + msg401851
2021-09-15 16:23:15gregory.p.smithsetmessages: + msg401850
2021-09-15 09:19:26vstinnersettitle: test_gdb failing on 32-bit armv7l when built with GCC -Og (fail on Raspbian on 3.9, regression from 3.8) -> test_gdb failing on 32-bit armv7l when built with GCC -Og: <class 'gdb.MemoryError'> Cannot access memory at address 0xfffffedc
2021-09-15 09:18:59vstinnersetmessages: + msg401818
2020-09-14 17:51:43lukasz.langasetpriority: release blocker -> high

messages: + msg376887
2020-09-11 10:55:39vstinnersetmessages: + msg376719
2020-07-13 11:34:51lukasz.langasetmessages: + msg373593
2020-06-29 21:02:11lukasz.langasetnosy: + lukasz.langa
messages: + msg372633
2020-06-11 13:21:32vstinnersetmessages: + msg371274
2020-06-11 13:20:17vstinnersetmessages: + msg371272
2020-06-11 13:18:54vstinnersettitle: test_gdb failing on Raspbian on 3.9, regression from 3.8 -> test_gdb failing on 32-bit armv7l when built with GCC -Og (fail on Raspbian on 3.9, regression from 3.8)
2020-06-11 13:07:35vstinnersetnosy: + vstinner
messages: + msg371269
2020-05-23 18:30:25gregory.p.smithcreate