AMD64 RHEL8 LTO 3.x:
https://buildbot.python.org/all/#/builders/312/builds/4

FAIL: test_NULL_ob_type (test.test_gdb.PrettyPrintTests)
FAIL: test_NULL_ptr (test.test_gdb.PrettyPrintTests)
FAIL: test_builtin_method (test.test_gdb.PrettyPrintTests)
FAIL: test_builtins_help (test.test_gdb.PrettyPrintTests)
FAIL: test_bytes (test.test_gdb.PrettyPrintTests)
FAIL: test_corrupt_ob_type (test.test_gdb.PrettyPrintTests)
FAIL: test_corrupt_tp_flags (test.test_gdb.PrettyPrintTests)
FAIL: test_corrupt_tp_name (test.test_gdb.PrettyPrintTests)
FAIL: test_dicts (test.test_gdb.PrettyPrintTests)
FAIL: test_exceptions (test.test_gdb.PrettyPrintTests)
FAIL: test_frozensets (test.test_gdb.PrettyPrintTests)
FAIL: test_int (test.test_gdb.PrettyPrintTests)
FAIL: test_lists (test.test_gdb.PrettyPrintTests)
FAIL: test_modern_class (test.test_gdb.PrettyPrintTests)
FAIL: test_selfreferential_dict (test.test_gdb.PrettyPrintTests)
FAIL: test_selfreferential_list (test.test_gdb.PrettyPrintTests)
FAIL: test_selfreferential_new_style_instance (test.test_gdb.PrettyPrintTests)
FAIL: test_selfreferential_old_style_instance (test.test_gdb.PrettyPrintTests)
FAIL: test_sets (test.test_gdb.PrettyPrintTests)
FAIL: test_singletons (test.test_gdb.PrettyPrintTests)
FAIL: test_strings (test.test_gdb.PrettyPrintTests)
FAIL: test_subclassing_list (test.test_gdb.PrettyPrintTests)
FAIL: test_subclassing_tuple (test.test_gdb.PrettyPrintTests)
FAIL: test_truncation (test.test_gdb.PrettyPrintTests)
FAIL: test_tuples (test.test_gdb.PrettyPrintTests)

Example of failure:

======================================================================
FAIL: test_NULL_ob_type (test.test_gdb.PrettyPrintTests)
Ensure that a PyObject* with NULL ob_type is handled gracefully
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto/build/Lib/test/test_gdb.py", line 515, in test_NULL_ob_type
    self.assertSane('id(42)',
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto/build/Lib/test/test_gdb.py", line 486, in assertSane
    self.get_gdb_repr(source,
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.lto/build/Lib/test/test_gdb.py", line 261, in get_gdb_repr
    self.fail('Unexpected gdb output: %r\n%s' % (gdb_output, gdb_output))
AssertionError: Unexpected gdb output: 'Breakpoint 1 at 0x565b60: file Objects/longobject.c, line 1113.\n[Thread debugging using libthread_db enabled]\nUsing host libthread_db library "/lib64/libthread_db.so.1".\n\nBreakpoint 1, builtin_id (self=, v=42) at Objects/longobject.c:1113\n1113\tPyLong_FromVoidPtr(void *p)\n#0  builtin_id (self=, v=<unknown at remote 0x94e000>) at Objects/longobject.c:1113\n#1  cfunction_vectorcall_O (func=<built-in method id of module object at remote 0x7ffff7f6eea0>, args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/methodobject.c:442\n#2  _PyObject_Vectorcall (kwnames=0x0, nargsf=9223372036854775809, args=<optimized out>, callable=<built-in method id of module object at remote 0x7ffff7f6eea0>) at ./Include/cpython/abstract.h:96\n#3  call_function (tstate=, pp_stack=, oparg=<optimized out>, kwnames=0x0) at Python/ceval.c:4984\n#4  _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3496\n#5  _PyEval_EvalCodeWithName (_co=<code at remote 0x7ffff7f13920>, globals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, locals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, args=, argcount=0, kwnames=, kwargs=, kwcount=0, kwstep=2, defs=, defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) at Python/ceval.c:4296\n#6  PyEval_EvalCodeEx (closure=0x0, kwdefs=0x0, defcount=0, defs=, kwcount=0, kws=, argcount=0, args=, locals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, globals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, _co=<code at remote 0x7ffff7f13920>) at Python/ceval.c:712\n#7  PyEval_EvalCode (locals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, globals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, co=<code at remote 0x7ffff7f13920>) at Python/ceval.c:714\n#8  run_eval_code_obj (locals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, globals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, co=) at Python/pythonrun.c:1117\n#9  run_mod (mod=<optimized out>, filename=\'<string>\', globals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, locals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, flags=, arena=) at Python/pythonrun.c:1139\n#10 PyRun_StringFlags (flags=, locals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, globals={\'__name__\': \'__main__\', \'__doc__\': None, \'__package__\': None, \'__loader__\': <type at remote 0x9669a0>, \'__spec__\': None, \'__annotations__\': {}, \'__builtins__\': <module at remote 0x7ffff7f6eea0>}, start=257, str="id(42)\\n") at Python/pythonrun.c:1008\n#11 PyRun_SimpleStringFlags (command="id(42)\\n", flags=) at Python/pythonrun.c:460\n#12 pymain_run_command (cf=, command=<optimized out>) at Modules/main.c:260\n#13 pymain_run_python (exitcode=) at Modules/main.c:558\n#14 Py_RunMain () at Modules/main.c:646\n#15 pymain_main (args=) at Modules/main.c:676\n#16 Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:700\n#17 __libc_start_main () from /lib64/libc.so.6\n#18 _start () at Python/ceval.c:5444\n'
Breakpoint 1 at 0x565b60: file Objects/longobject.c, line 1113.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

Breakpoint 1, builtin_id (self=, v=42) at Objects/longobject.c:1113
1113	PyLong_FromVoidPtr(void *p)
#0  builtin_id (self=, v=<unknown at remote 0x94e000>) at Objects/longobject.c:1113
#1  cfunction_vectorcall_O (func=<built-in method id of module object at remote 0x7ffff7f6eea0>, args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/methodobject.c:442
#2  _PyObject_Vectorcall (kwnames=0x0, nargsf=9223372036854775809, args=<optimized out>, callable=<built-in method id of module object at remote 0x7ffff7f6eea0>) at ./Include/cpython/abstract.h:96
#3  call_function (tstate=, pp_stack=, oparg=<optimized out>, kwnames=0x0) at Python/ceval.c:4984
#4  _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3496
#5  _PyEval_EvalCodeWithName (_co=<code at remote 0x7ffff7f13920>, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, args=, argcount=0, kwnames=, kwargs=, kwcount=0, kwstep=2, defs=, defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) at Python/ceval.c:4296
#6  PyEval_EvalCodeEx (closure=0x0, kwdefs=0x0, defcount=0, defs=, kwcount=0, kws=, argcount=0, args=, locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, _co=<code at remote 0x7ffff7f13920>) at Python/ceval.c:712
#7  PyEval_EvalCode (locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, co=<code at remote 0x7ffff7f13920>) at Python/ceval.c:714
#8  run_eval_code_obj (locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, co=) at Python/pythonrun.c:1117
#9  run_mod (mod=<optimized out>, filename='<string>', globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, flags=, arena=) at Python/pythonrun.c:1139
#10 PyRun_StringFlags (flags=, locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x9669a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7f6eea0>}, start=257, str="id(42)\n") at Python/pythonrun.c:1008
#11 PyRun_SimpleStringFlags (command="id(42)\n", flags=) at Python/pythonrun.c:460
#12 pymain_run_command (cf=, command=<optimized out>) at Modules/main.c:260
#13 pymain_run_python (exitcode=) at Modules/main.c:558
#14 Py_RunMain () at Modules/main.c:646
#15 pymain_main (args=) at Modules/main.c:676
#16 Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:700
#17 __libc_start_main () from /lib64/libc.so.6
#18 _start () at Python/ceval.c:5444

Similar errors on:

AMD64 RHEL8 LTO + PGO 3.7:
https://buildbot.python.org/all/#/builders/255/builds/13

AMD64 RHEL8 LTO 3.7:
https://buildbot.python.org/all/#/builders/293/builds/13

AMD64 RHEL8 LTO 3.8:
https://buildbot.python.org/all/#/builders/265/builds/12

Similar issue on AMD64 Fedora Stable LTO 3.8:
https://buildbot.python.org/all/#/builders/235/builds/2

Shouldn't this test be suppressed on LTO builds?

#0  builtin_id (self=, v=<unknown at remote 0x94e000>) at Objects/longobject.c:1113

builtid_id isn't in longobject.c, so this is incorrect debug information. The test expects to find it in bltinmodule.c:

re.match(r'.*#0\s+builtin_id\s+\(self\=.*,\s+v=\s*(.*?)?\)\s+at\s+\S*Python/bltinmodule.c.*' ...

AMD64 Fedora Stable LTO + PGO 3.7 is also affected:
https://buildbot.python.org/all/#/builders/252/builds/2

And AMD64 Fedora Stable LTO 3.7:
https://buildbot.python.org/all/#/builders/270/builds/2

> builtid_id isn't in longobject.c, so this is incorrect debug information. The test expects to find it in bltinmodule.c:

gdb may be confused by LTO. I'm not sure if how LTO works to debug symbols.

> Shouldn't this test be suppressed on LTO builds?

Maybe. I don't know.

Greg seems to have done LTO related things in the past, maybe he knows? (All my PGO contributions only apply to Windows :) )

The test probably has a fragile regex.  I don't spend time with gdb things much myself.

Um, okay then.

The test already includes this:

if ((sysconfig.get_config_var('PGO_PROF_USE_FLAG') or 'xxx') in
    (sysconfig.get_config_var('PY_CORE_CFLAGS') or '')):
    raise unittest.SkipTest("test_gdb is not reliable on PGO builds")

Apparently this no longer works. Who understands these variables enough to fix the issue?

> Apparently this no longer works. Who understands these variables enough to fix the issue?

Some failures are LTO + PGO builds, but test_gdb also fails on LTO without PGO builds.

The PGO_PROF_USE_FLAG check only checks for PGO.

New changeset 64b4a3a2deabcd4103fac2759a311fe94159b4d1 by Victor Stinner in branch 'master':
bpo-38239: Fix test_gdb for Link Time Optimization (LTO) (GH-16422)
https://github.com/python/cpython/commit/64b4a3a2deabcd4103fac2759a311fe94159b4d1

Python 2.7 doesn't seem to be affected: I failed to reproduce the test_gdb failure on Python 2.7 with LTO.

New changeset c9893400652f38804aed0be8d8f70feda9465c47 by Miss Islington (bot) in branch '3.8':
bpo-38239: Fix test_gdb for Link Time Optimization (LTO) (GH-16422)
https://github.com/python/cpython/commit/c9893400652f38804aed0be8d8f70feda9465c47

New changeset e6b5ed1fbdbb7b44c7ab2a353fa0bc726b073740 by Victor Stinner in branch '3.7':
bpo-38239: Fix test_gdb for Link Time Optimization (LTO) (GH-16422) (GH-16426)
https://github.com/python/cpython/commit/e6b5ed1fbdbb7b44c7ab2a353fa0bc726b073740

AMD64 RHEL8 LTO 3.x is green again. I will keep the issue open until all mentionned buildbots are back to green (when test_gdb pass on these workers).

> AMD64 RHEL8 LTO 3.x is green again. I will keep the issue open until all mentionned buildbots are back to green (when test_gdb pass on these workers).

I checked an all buildbots mentioned in this issues are back to green (success). I close the issue.


> AMD64 RHEL8 LTO 3.x:
> https://buildbot.python.org/all/#/builders/312/builds/4

Green

> AMD64 RHEL8 LTO + PGO 3.7:
> https://buildbot.python.org/all/#/builders/255/builds/13

Green

> AMD64 RHEL8 LTO 3.7:
> https://buildbot.python.org/all/#/builders/293/builds/13

Green

> AMD64 RHEL8 LTO 3.8:
> https://buildbot.python.org/all/#/builders/265/builds/12

Green

> Similar issue on AMD64 Fedora Stable LTO 3.8:
> https://buildbot.python.org/all/#/builders/235/builds/2

Green

> AMD64 Fedora Stable LTO + PGO 3.7 is also affected:
> https://buildbot.python.org/all/#/builders/252/builds/2

Green

> And AMD64 Fedora Stable LTO 3.7:
> https://buildbot.python.org/all/#/builders/270/builds/2

Green