classification
Title: test_gdb fails on Python 3.6 when built with LTO+PGO
Type: Stage:
Components: Build, Tests Versions: Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: cstratak, haypo, jkloth, koobs, mi
Priority: normal Keywords:

Created on 2017-05-11 16:16 by haypo, last changed 2017-06-16 05:19 by koobs.

Files
File name Uploaded Description Edit
build.log cstratak, 2017-05-11 16:26
root.log cstratak, 2017-05-11 16:27
pgo-lto-gdb-errors-build cstratak, 2017-05-11 20:35
Pull Requests
URL Status Linked Edit
PR 1549 merged haypo, 2017-05-11 16:27
Messages (11)
msg293503 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-05-11 16:16
cstratak reported the following test failure on Fedora 24 when building Python 3.6 with LTO + PGO:

======================================================================
FAIL: test_threads (test.test_gdb.PyBtTests)
Verify that "py-bt" indicates threads that are waiting for the GIL
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/Python-3.6.1/Lib/test/test_gdb.py", line 783, in test_threads
    cmds_after_breakpoint=['thread apply all py-bt'])
  File "/builddir/build/BUILD/Python-3.6.1/Lib/test/test_gdb.py", line 218, in get_stack_trace
    self.assertEqual(unexpected_errlines, [])
AssertionError: Lists differ: ["Python Exception <class 'ValueError'> Va[95 chars]nd."] != []
First list contains 2 additional elements.
First extra element 0:
"Python Exception <class 'ValueError'> Variable 'func_obj' not found.: "
+ []
- ["Python Exception <class 'ValueError'> Variable 'func_obj' not found.: ",
-  "Error occurred in Python command: Variable 'func_obj' not found."]
----------------------------------------------------------------------
msg293505 - (view) Author: Charalampos Stratakis (cstratak) * Date: 2017-05-11 16:26
Full build log
msg293506 - (view) Author: Charalampos Stratakis (cstratak) * Date: 2017-05-11 16:27
All the dependencies dragged.

gdb is of version 7.11. The failures do not happen with gdb 7.12 (which exists in later Fedora releases).
msg293507 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-05-11 16:30
I created https://github.com/python/cpython/pull/1549 to update test_gdb.py and python-gdb.py in Python 3.6. I don't know if it will fix the issus, but it shouldn't hurt :-)
msg293508 - (view) Author: Charalampos Stratakis (cstratak) * Date: 2017-05-11 16:33
Note: test_gdb is skipped on later Fedora's actually (possibly due to gdb package no being dragged at the minimal buildroot) so the issue might still be there, so the gdb version might have no effect on that. Will investigate further.
msg293516 - (view) Author: Charalampos Stratakis (cstratak) * Date: 2017-05-11 20:35
So the issue wasn't restricted to a specific gdb version or distro release, as due to some issues dependency issues the gdb binary wasn't pulled in the buildroot which makes test_gdb to get skipped.

So I was able to reproduce it on my system by installing gdb (version 7.12.1), compiling python 3.6 from sources with
--enable-optimizations and --with-lto flags enabled and running 'make test'

Also applied the relevant PR however it didn't fix the issue.

Attaching the full log of 'make test'
msg293582 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-05-12 22:21
New changeset d05f7fdf6cf77724bd3064fb5a0846ef5cfe0c88 by Victor Stinner in branch '3.6':
[3.6] bpo-30345: Update test_gdb.py and python-gdb.py from master (#1549)
https://github.com/python/cpython/commit/d05f7fdf6cf77724bd3064fb5a0846ef5cfe0c88
msg295299 - (view) Author: Mikhail (mi) Date: 2017-06-06 20:19
I rebuilt my python-3.6.1 with the path (https://github.com/python/cpython/commit/d05f7fdf6cf77724bd3064fb5a0846ef5cfe0c88) and tried debugging again -- same problem:

% gdb7121 /opt/bin/python3.6
...
(gdb) r SCRIPT.py
Thread 1 received signal SIGSEGV, Segmentation fault.
...
(gdb) py-bt
Python Exception <type 'exceptions.ValueError'> Variable 'func_obj' not found.: 
Error occurred in Python command: Variable 'func_obj' not found.
msg295339 - (view) Author: Mikhail (mi) Date: 2017-06-07 14:11
The actual stack, which I'm trying to debug, begins like this:

#0  0xbfbfd34e in ?? ()
#1  0x2a9ec81e in ?? () from /opt/lib/qt5/libQt5WebKit.so.5
#2  0x2acf0efe in ?? () from /opt/lib/qt5/libQt5WebKit.so.5
#3  0x2acd8b74 in ?? () from /opt/lib/qt5/libQt5WebKit.so.5
#4  0x2acd5d60 in ?? () from /opt/lib/qt5/libQt5WebKit.so.5
#5  0x2acd87ae in ?? () from /opt/lib/qt5/libQt5WebKit.so.5
#6  0x2a9fe2e3 in QWebFrameAdapter::load(QNetworkRequest const&, QNetworkAccessManager::Operation, QByteArray const&) () from /opt/lib/qt5/libQt5WebKit.so.5
#7  0x2d7a18dd in QWebFrame::setUrl(QUrl const&) () from /opt/lib/qt5/libQt5WebKitWidgets.so.5
#8  0x2d7ad5eb in QWebView::setUrl(QUrl const&) () from /opt/lib/qt5/libQt5WebKitWidgets.so.5
#9  0x2d75efd4 in meth_QWebView_setUrl(_object*, _object*) ()
   from /opt/lib/python3.6/site-packages/PyQt5/QtWebKitWidgets.so
#10 0x28125151 in _PyCFunction_FastCallDict () from /opt/lib/libpython3.6m.so.1.0
#11 0x28125326 in _PyCFunction_FastCallKeywords () from /opt/lib/libpython3.6m.so.1.0
#12 0x2819a458 in ?? () from /opt/lib/libpython3.6m.so.1.0
#13 0x28193ab2 in _PyEval_EvalFrameDefault () from /opt/lib/libpython3.6m.so.1.0
#14 0x2819b790 in ?? () from /opt/lib/libpython3.6m.so.1.0
#15 0x2819a425 in ?? () from /opt/lib/libpython3.6m.so.1.0
#16 0x28193ab2 in _PyEval_EvalFrameDefault () from /opt/lib/libpython3.6m.so.1.0
[...]

Maybe, it is "too deep" into the native (not Python) code for the feature to work?
msg295341 - (view) Author: Jeremy Kloth (jkloth) * Date: 2017-06-07 14:53
It seems that commit (https://github.com/python/cpython/commit/c52572319cbd50adff85050a54122c25239a516d) changed the parameter name in the definition of _PyCFunction_FastCallDict().  I believe that changing 'func_obj' to just 'func' should fix it (in Tools/gdb/libpython.py).
msg295348 - (view) Author: Mikhail (mi) Date: 2017-06-07 15:41
So, I tried the modified patch (see http://aldan.algebra.com/~mi/tmp/patch-issue30345) -- and now I simply get a different variable name in the error-message:

(gdb) py-bt
Python Exception <type 'exceptions.ValueError'> Variable 'func' not found.: 
Error occurred in Python command: Variable 'func' not found.

However, the older version of the patch only referenced "func_obj" in test_gdb.py -- not in libpython.py -- so I may have misunderstood Jeremy's suggestion entirely...
History
Date User Action Args
2017-06-16 05:19:35koobssetnosy: + koobs
2017-06-07 15:41:32misetmessages: + msg295348
2017-06-07 14:53:32jklothsetnosy: + jkloth
messages: + msg295341
2017-06-07 14:11:55misetmessages: + msg295339
2017-06-06 20:19:47misetnosy: + mi
messages: + msg295299
2017-05-12 22:21:53hayposetmessages: + msg293582
2017-05-11 20:35:23cstrataksetfiles: + pgo-lto-gdb-errors-build

messages: + msg293516
2017-05-11 16:33:16cstrataksetmessages: + msg293508
2017-05-11 16:30:28hayposetmessages: + msg293507
2017-05-11 16:27:55hayposetpull_requests: + pull_request1647
2017-05-11 16:27:13cstrataksetfiles: + root.log

messages: + msg293506
2017-05-11 16:26:03cstrataksetfiles: + build.log
nosy: + cstratak
messages: + msg293505

2017-05-11 16:16:32haypocreate