classification
Title: python-gdb error: Python Exception Type does not have a target
Type: behavior Stage: resolved
Components: Demos and Tools Versions: Python 3.8, Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Dylan Cali, lisroach, vstinner
Priority: normal Keywords: patch

Created on 2018-11-01 10:40 by Dylan Cali, last changed 2019-03-13 00:16 by Dylan Cali. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 11848 merged lisroach, 2019-02-14 04:51
PR 12284 merged miss-islington, 2019-03-12 03:21
Messages (9)
msg329052 - (view) Author: Dylan Cali (Dylan Cali) Date: 2018-11-01 10:40
Python version: 3.6.6-debug
System: Kernel: 4.15.0-38-generic x86_64
        Distro: Linux Mint 18.2 Sonya
CPU:    Quad core Intel Xeon E3-1505M
Memory: 32018.6MB


Expected:

py-list and py-bt to print the current python frame and traceback when attaching to a hung python process with gdb, a debug build of python, and cpython/Tools/gdb/libpython.py loaded.


Actual:

py-list and py-bt fail with:

    Python Exception <class 'RuntimeError'> Type does not have a target.:
    Error occurred in Python command: Type does not have a target.

Invoking 'set python print-stack full' in gdb produces the more useful:

    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "~/git/cpython/Tools/gdb/libpython.py", line 916, in filename
      File "~/git/cpython/Tools/gdb/libpython.py", line 1158, in proxyval
    RuntimeError: Type does not have a target.

so it is failing at:

    fields = gdb.lookup_type('PyUnicodeObject').target().fields()

in libpython.py [1].


Reproduce:

I haven't been able to create a simple standalone program that triggers the failure.  I am working on a PR for Keras to try and fix the multiprocessing support in some of their utility classes.  Currently the tests are sporadically hanging and my intention was to use python's gdb integration to identify exactly where and why the hangs are occuring... but I can't get that information at the moment because of the error above when invoking py-list and py-bt.

So, unfortunately, the shortest path to reproduce is to checkout the PR branch, run the offending tests, connect with gdb, and invoke py-list/py-bt:

    * install a debug version of 3.6.6 if one isn't already available
    * git clone https://github.com/calid/keras.git -b fix-multiprocessing-hang
    * cd keras
    * pip install -e .[tests]
    * pip install tensorflow
    * py.test tests/keras/utils/data_utils_test.py
    * wait for hang
    * gdb -p <parent_pid_of_harness>
    * invoke py-list or py-bt


I am happy to poke around in libpython.py and try to fix/submit a PR myself, but I'm not at all familiar with the python internals so I would need some guidance.  And obviously let me know if this isn't actually a bug but rather environment related/user error.

Thank you!


[1] https://github.com/python/cpython/blob/v3.6.6/Tools/gdb/libpython.py#L1158
msg334196 - (view) Author: Lisa Roach (lisroach) * (Python committer) Date: 2019-01-22 00:40
I experienced this bug as well and have tried to dig into it a little. 

I experimented with a service I have that uses shared libraries. If I compile with high level of optimizations for the C++ code (-O3) I don't experience the issue, but compiling without any optimizations I do. 

I also tried with a script that does not use any shared libraries and I do not see the issue.


Digging into it a little, when running with the optimized version I found:

gdb.lookup_type('PyUnicodeObject')

returns a gdb.TYPE_CODE_TYPEDEF, which is the correct type. Running with the non-optimized version instead returns a gdb.TYPE_CODE_STRUCT.


The struct type errors because it has no target(), but I actually think target() might be superfluous here. 

Both versions if I only call `fields = gdb.lookup_type('PyUnicodeObject').fields()` returns a list of fields, one of which is named `data` for the purposes of the pep check, so target() doesn't seem vital.
msg335518 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-02-14 11:39
> Distro: Linux Mint 18.2 Sonya

Hum, Dylan: what is your gdb version?

On Fedora 29 with gdb 8.2-6.fc29, it seems like .target() is useless:

$ gdb ./python
GNU gdb (GDB) Fedora 8.2-6.fc29
...
(gdb) python print([field.name for field in gdb.lookup_type('PyUnicodeObject').target().fields()])
['_base', 'data']

(gdb) python print([field.name for field in gdb.lookup_type('PyUnicodeObject').fields()])
['_base', 'data']

I tested on a Python compiled manually from source, I ran my test in the directory of Python source  code.

I also tested on /usr/bin/python3 (system Python), same behavior.
msg335519 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-02-14 11:42
Python 3.6 no longer accept bugfixes, only security fixes:
http://devguide.python.org/#status-of-python-branches

You can try faulthandler has a workaround. For example, install faulthandler signal handler using faulthandler.register(signal.SIGUSR1) and then send a SIGUSR1 signal to the blocked process. You might want to write the output into a file if you don't have access the program stdout: use the 'file' parameter of faulthandler.register(). See also faulthandler.dump_traceback_later() to use a timeout.

https://docs.python.org/dev/library/faulthandler.html
msg335618 - (view) Author: Dylan Cali (Dylan Cali) Date: 2019-02-15 14:48
> Hum, Dylan: what is your gdb version?

$ gdb --version
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1

> On Fedora 29 with gdb 8.2-6.fc29, it seems like .target() is useless

I can confirm removing the .target() call in libpython.py resolved the issue for me with no ill effects

> You can try faulthandler has a workaround.

I'll give this a try, thank you
msg337713 - (view) Author: Lisa Roach (lisroach) * (Python committer) Date: 2019-03-12 03:21
New changeset 1ceb3a3d172dcf0ddff38d5d6b559443ad065b84 by Lisa Roach in branch 'master':
bpo-35132: Fixes missing target in gdb pep0393 check. (GH-11848)
https://github.com/python/cpython/commit/1ceb3a3d172dcf0ddff38d5d6b559443ad065b84
msg337729 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-03-12 09:17
New changeset 047f8f25b93e2649d234fa565a59383fceb40e16 by Victor Stinner (Miss Islington (bot)) in branch '3.7':
bpo-35132: Fixes missing target in gdb pep0393 check. (GH-11848) (GH-12284)
https://github.com/python/cpython/commit/047f8f25b93e2649d234fa565a59383fceb40e16
msg337730 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-03-12 09:22
Thanks Dylan Cali for the bug report. Lisa Roach fixed the bug in 3.7 and master branches. Until a new release is published, you can copy manually the file from:
https://github.com/python/cpython/blob/3.7/Tools/gdb/libpython.py
msg337824 - (view) Author: Dylan Cali (Dylan Cali) Date: 2019-03-13 00:16
Thank you!  And thank you Lisa Roach for the investigation and fix.
History
Date User Action Args
2019-03-13 00:16:18Dylan Calisetmessages: + msg337824
2019-03-12 09:22:05vstinnersetstatus: open -> closed
versions: + Python 3.7, Python 3.8, - Python 3.6
messages: + msg337730

resolution: fixed
stage: patch review -> resolved
2019-03-12 09:17:25vstinnersetmessages: + msg337729
2019-03-12 03:21:54miss-islingtonsetpull_requests: + pull_request12263
2019-03-12 03:21:31lisroachsetmessages: + msg337713
2019-02-15 14:48:23Dylan Calisetmessages: + msg335618
2019-02-14 11:42:40vstinnersetmessages: + msg335519
2019-02-14 11:39:48vstinnersetnosy: + vstinner
messages: + msg335518
2019-02-14 04:51:19lisroachsetkeywords: + patch
stage: patch review
pull_requests: + pull_request11879
2019-01-22 00:40:36lisroachsetnosy: + lisroach
messages: + msg334196
2018-11-01 10:40:19Dylan Calicreate