classification
Title: test_gdb fails on AMD64 Fedora Stable LTO 3.8 and AMD64 RHEL8 LTO 3.x: Unexpected gdb output
Type: Stage: resolved
Components: Tests Versions: Python 3.9, Python 3.8, Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: cstratak, miss-islington, steve.dower, vstinner
Priority: normal Keywords: patch

Created on 2019-09-20 21:32 by vstinner, last changed 2019-09-27 12:53 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 16422 merged vstinner, 2019-09-26 13:49
PR 16424 merged miss-islington, 2019-09-26 14:54
PR 16426 merged vstinner, 2019-09-26 15:07
Messages (16)
msg352896 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-20 21:32
AMD64 RHEL8 LTO 3.x:
https://buildbot.python.org/all/#/builders/312/builds/4

FAIL: test_NULL_ob_type (test.test_gdb.PrettyPrintTests)
FAIL: test_NULL_ptr (test.test_gdb.PrettyPrintTests)
FAIL: test_builtin_method (test.test_gdb.PrettyPrintTests)
FAIL: test_builtins_help (test.test_gdb.PrettyPrintTests)
FAIL: test_bytes (test.test_gdb.PrettyPrintTests)
FAIL: test_corrupt_ob_type (test.test_gdb.PrettyPrintTests)
FAIL: test_corrupt_tp_flags (test.test_gdb.PrettyPrintTests)
FAIL: test_corrupt_tp_name (test.test_gdb.PrettyPrintTests)
FAIL: test_dicts (test.test_gdb.PrettyPrintTests)
FAIL: test_exceptions (test.test_gdb.PrettyPrintTests)
FAIL: test_frozensets (test.test_gdb.PrettyPrintTests)
FAIL: test_int (test.test_gdb.PrettyPrintTests)
FAIL: test_lists (test.test_gdb.PrettyPrintTests)
FAIL: test_modern_class (test.test_gdb.PrettyPrintTests)
FAIL: test_selfreferential_dict (test.test_gdb.PrettyPrintTests)
FAIL: test_selfreferential_list (test.test_gdb.PrettyPrintTests)
FAIL: test_selfreferential_new_style_instance (test.test_gdb.PrettyPrintTests)
FAIL: test_selfreferential_old_style_instance (test.test_gdb.PrettyPrintTests)
FAIL: test_sets (test.test_gdb.PrettyPrintTests)
FAIL: test_singletons (test.test_gdb.PrettyPrintTests)
FAIL: test_strings (test.test_gdb.PrettyPrintTests)
FAIL: test_subclassing_list (test.test_gdb.PrettyPrintTests)
FAIL: test_subclassing_tuple (test.test_gdb.PrettyPrintTests)
FAIL: test_truncation (test.test_gdb.PrettyPrintTests)
FAIL: test_tuples (test.test_gdb.PrettyPrintTests)

Example of failure:

======================================================================
FAIL: test_NULL_ob_type (test.test_gdb.PrettyPrintTests)
Ensure that a PyObject* with NULL ob_type is handled gracefully
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto/build/Lib/test/test_gdb.py", line 515, in test_NULL_ob_type
    self.assertSane('id(42)',
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto/build/Lib/test/test_gdb.py", line 486, in assertSane
    self.get_gdb_repr(source,
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto/build/Lib/test/test_gdb.py", line 261, in get_gdb_repr
    self.fail('Unexpected gdb output: %r\n%s' % (gdb_output, gdb_output))
AssertionError: Unexpected gdb output: 'Breakpoint 1 at 0x565b60: file Objects/longobject.c, line 1113.\n[Thread debugging using libthread_db enabled]\nUsing host libthread_db library "/lib64/libthread_db.so.1".\n\nBreakpoint 1, builtin_id (self=, v=42) at Objects/longobject.c:1113\n1113\tPyLong_FromVoidPtr(void *p)\n#0  builtin_id (self=, v=<unknown at remote 0x94e000>) at Objects/longobject.c:1113\n#1  cfunction_vectorcall_O (func=<built-in method id of module object at remote 0x7ffff7f6eea0>, args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/methodobject.c:442\n#2  _PyObject_Vectorcall (kwnames=0x0, nargsf=9223372036854775809, args=<optimized out>, callable=<built-in method id of module object at remote 0x7ffff7f6eea0>) at ./Include/cpython/abstract.h:96\n#3  call_function (tstate=, pp_stack=, oparg=<optimized out>, kwnames=0x0) at Python/ceval.c:4984\n#4  _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3496\n#5  _PyEval_EvalCodeWithName (_co=<code at remote 0x7ffff7f13920>, globals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, locals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, args=, argcount=0, kwnames=, kwargs=, kwcount=0, kwstep=2, defs=, defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) at Python/ceval.c:4296\n#6  PyEval_EvalCodeEx (closure=0x0, kwdefs=0x0, defcount=0, defs=, kwcount=0, kws=, argcount=0, args=, locals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, globals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, _co=<code at remote 0x7ffff7f13920>) at Python/ceval.c:712\n#7  PyEval_EvalCode (locals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, globals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, co=<code at remote 0x7ffff7f13920>) at Python/ceval.c:714\n#8  run_eval_code_obj (locals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, globals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, co=) at Python/pythonrun.c:1117\n#9  run_mod (mod=<optimized out>, filename=\'<string>\', globals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, locals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, flags=, arena=) at Python/pythonrun.c:1139\n#10 PyRun_StringFlags (flags=, locals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, globals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, start=257, str="id(42)\\n") at Python/pythonrun.c:1008\n#11 PyRun_SimpleStringFlags (command="id(42)\\n", flags=) at Python/pythonrun.c:460\n#12 pymain_run_command (cf=, command=<optimized out>) at Modules/main.c:260\n#13 pymain_run_python (exitcode=) at Modules/main.c:558\n#14 Py_RunMain () at Modules/main.c:646\n#15 pymain_main (args=) at Modules/main.c:676\n#16 Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:700\n#17 __libc_start_main () from /lib64/libc.so.6\n#18 _start () at Python/ceval.c:5444\n'
Breakpoint 1 at 0x565b60: file Objects/longobject.c, line 1113.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

Breakpoint 1, builtin_id (self=, v=42) at Objects/longobject.c:1113
1113	PyLong_FromVoidPtr(void *p)
#0  builtin_id (self=, v=<unknown at remote 0x94e000>) at Objects/longobject.c:1113
#1  cfunction_vectorcall_O (func=<built-in method id of module object at remote 0x7ffff7f6eea0>, args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/methodobject.c:442
#2  _PyObject_Vectorcall (kwnames=0x0, nargsf=9223372036854775809, args=<optimized out>, callable=<built-in method id of module object at remote 0x7ffff7f6eea0>) at ./Include/cpython/abstract.h:96
#3  call_function (tstate=, pp_stack=, oparg=<optimized out>, kwnames=0x0) at Python/ceval.c:4984
#4  _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3496
#5  _PyEval_EvalCodeWithName (_co=<code at remote 0x7ffff7f13920>, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, args=, argcount=0, kwnames=, kwargs=, kwcount=0, kwstep=2, defs=, defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) at Python/ceval.c:4296
#6  PyEval_EvalCodeEx (closure=0x0, kwdefs=0x0, defcount=0, defs=, kwcount=0, kws=, argcount=0, args=, locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, _co=<code at remote 0x7ffff7f13920>) at Python/ceval.c:712
#7  PyEval_EvalCode (locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, co=<code at remote 0x7ffff7f13920>) at Python/ceval.c:714
#8  run_eval_code_obj (locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, co=) at Python/pythonrun.c:1117
#9  run_mod (mod=<optimized out>, filename='<string>', globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, flags=, arena=) at Python/pythonrun.c:1139
#10 PyRun_StringFlags (flags=, locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, start=257, str="id(42)\n") at Python/pythonrun.c:1008
#11 PyRun_SimpleStringFlags (command="id(42)\n", flags=) at Python/pythonrun.c:460
#12 pymain_run_command (cf=, command=<optimized out>) at Modules/main.c:260
#13 pymain_run_python (exitcode=) at Modules/main.c:558
#14 Py_RunMain () at Modules/main.c:646
#15 pymain_main (args=) at Modules/main.c:676
#16 Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:700
#17 __libc_start_main () from /lib64/libc.so.6
#18 _start () at Python/ceval.c:5444
msg352897 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-20 21:35
Similar errors on:

AMD64 RHEL8 LTO + PGO 3.7:
https://buildbot.python.org/all/#/builders/255/builds/13

AMD64 RHEL8 LTO 3.7:
https://buildbot.python.org/all/#/builders/293/builds/13

AMD64 RHEL8 LTO 3.8:
https://buildbot.python.org/all/#/builders/265/builds/12
msg352910 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-20 23:41
Similar issue on AMD64 Fedora Stable LTO 3.8:
https://buildbot.python.org/all/#/builders/235/builds/2
msg352949 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-09-21 18:31
Shouldn't this test be suppressed on LTO builds?

#0  builtin_id (self=, v=<unknown at remote 0x94e000>) at Objects/longobject.c:1113

builtid_id isn't in longobject.c, so this is incorrect debug information. The test expects to find it in bltinmodule.c:

re.match(r'.*#0\s+builtin_id\s+\(self\=.*,\s+v=\s*(.*?)?\)\s+at\s+\S*Python/bltinmodule.c.*' ...
msg353012 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-23 12:25
AMD64 Fedora Stable LTO + PGO 3.7 is also affected:
https://buildbot.python.org/all/#/builders/252/builds/2

And AMD64 Fedora Stable LTO 3.7:
https://buildbot.python.org/all/#/builders/270/builds/2
msg353020 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-23 14:34
> builtid_id isn't in longobject.c, so this is incorrect debug information. The test expects to find it in bltinmodule.c:

gdb may be confused by LTO. I'm not sure if how LTO works to debug symbols.

> Shouldn't this test be suppressed on LTO builds?

Maybe. I don't know.
msg353023 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-09-23 14:48
Greg seems to have done LTO related things in the past, maybe he knows? (All my PGO contributions only apply to Windows :) )
msg353121 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2019-09-24 22:11
The test probably has a fragile regex.  I don't spend time with gdb things much myself.
msg353122 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-09-24 22:38
Um, okay then.

The test already includes this:

if ((sysconfig.get_config_var('PGO_PROF_USE_FLAG') or 'xxx') in
    (sysconfig.get_config_var('PY_CORE_CFLAGS') or '')):
    raise unittest.SkipTest("test_gdb is not reliable on PGO builds")

Apparently this no longer works. Who understands these variables enough to fix the issue?
msg353127 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-25 00:41
> Apparently this no longer works. Who understands these variables enough to fix the issue?

Some failures are LTO + PGO builds, but test_gdb also fails on LTO without PGO builds.

The PGO_PROF_USE_FLAG check only checks for PGO.
msg353313 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-26 14:54
New changeset 64b4a3a2deabcd4103fac2759a311fe94159b4d1 by Victor Stinner in branch 'master':
bpo-38239: Fix test_gdb for Link Time Optimization (LTO) (GH-16422)
https://github.com/python/cpython/commit/64b4a3a2deabcd4103fac2759a311fe94159b4d1
msg353317 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-26 15:08
Python 2.7 doesn't seem to be affected: I failed to reproduce the test_gdb failure on Python 2.7 with LTO.
msg353318 - (view) Author: miss-islington (miss-islington) Date: 2019-09-26 15:13
New changeset c9893400652f38804aed0be8d8f70feda9465c47 by Miss Islington (bot) in branch '3.8':
bpo-38239: Fix test_gdb for Link Time Optimization (LTO) (GH-16422)
https://github.com/python/cpython/commit/c9893400652f38804aed0be8d8f70feda9465c47
msg353321 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-26 15:30
New changeset e6b5ed1fbdbb7b44c7ab2a353fa0bc726b073740 by Victor Stinner in branch '3.7':
bpo-38239: Fix test_gdb for Link Time Optimization (LTO) (GH-16422) (GH-16426)
https://github.com/python/cpython/commit/e6b5ed1fbdbb7b44c7ab2a353fa0bc726b073740
msg353325 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-26 16:17
AMD64 RHEL8 LTO 3.x is green again. I will keep the issue open until all mentionned buildbots are back to green (when test_gdb pass on these workers).
msg353357 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-27 12:53
> AMD64 RHEL8 LTO 3.x is green again. I will keep the issue open until all mentionned buildbots are back to green (when test_gdb pass on these workers).

I checked an all buildbots mentioned in this issues are back to green (success). I close the issue.


> AMD64 RHEL8 LTO 3.x:
> https://buildbot.python.org/all/#/builders/312/builds/4

Green

> AMD64 RHEL8 LTO + PGO 3.7:
> https://buildbot.python.org/all/#/builders/255/builds/13

Green

> AMD64 RHEL8 LTO 3.7:
> https://buildbot.python.org/all/#/builders/293/builds/13

Green

> AMD64 RHEL8 LTO 3.8:
> https://buildbot.python.org/all/#/builders/265/builds/12

Green

> Similar issue on AMD64 Fedora Stable LTO 3.8:
> https://buildbot.python.org/all/#/builders/235/builds/2

Green

> AMD64 Fedora Stable LTO + PGO 3.7 is also affected:
> https://buildbot.python.org/all/#/builders/252/builds/2

Green

> And AMD64 Fedora Stable LTO 3.7:
> https://buildbot.python.org/all/#/builders/270/builds/2

Green
History
Date User Action Args
2019-09-27 12:53:10vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg353357

stage: patch review -> resolved
2019-09-26 16:17:23vstinnersetmessages: + msg353325
2019-09-26 15:30:19vstinnersetmessages: + msg353321
2019-09-26 15:13:42miss-islingtonsetnosy: + miss-islington
messages: + msg353318
2019-09-26 15:08:28vstinnersetmessages: + msg353317
versions: + Python 3.7, Python 3.8
2019-09-26 15:07:47vstinnersetpull_requests: + pull_request16005
2019-09-26 14:54:29miss-islingtonsetpull_requests: + pull_request16003
2019-09-26 14:54:28vstinnersetmessages: + msg353313
2019-09-26 13:49:12vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request16001
2019-09-25 00:41:18vstinnersetmessages: + msg353127
2019-09-24 22:38:20steve.dowersetmessages: + msg353122
2019-09-24 22:12:24gregory.p.smithsetnosy: - gregory.p.smith
2019-09-24 22:11:52gregory.p.smithsetnosy: gregory.p.smith, vstinner, steve.dower, cstratak
messages: + msg353121
2019-09-24 12:14:50cstrataksetnosy: + cstratak
2019-09-23 14:48:10steve.dowersetnosy: + gregory.p.smith
messages: + msg353023
2019-09-23 14:34:21vstinnersetmessages: + msg353020
2019-09-23 12:25:34vstinnersetmessages: + msg353012
2019-09-21 18:31:47steve.dowersetnosy: + steve.dower
messages: + msg352949
2019-09-20 23:41:48vstinnersetmessages: + msg352910
title: test_gdb fails on AMD64 RHEL8 LTO 3.x: Unexpected gdb output -> test_gdb fails on AMD64 Fedora Stable LTO 3.8 and AMD64 RHEL8 LTO 3.x: Unexpected gdb output
2019-09-20 21:35:30vstinnersetmessages: + msg352897
2019-09-20 21:32:40vstinnercreate