classification
Title: faulthandler should indicate if the fault happened in garbage collection
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.11, Python 3.10
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: maxballenger, pablogsal, vstinner
Priority: normal Keywords: patch

Created on 2021-06-20 20:24 by maxballenger, last changed 2021-06-21 18:46 by maxballenger. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 26823 merged vstinner, 2021-06-21 10:49
PR 26826 merged vstinner, 2021-06-21 11:23
Messages (7)
msg396196 - (view) Author: Maxwell Ballenger (maxballenger) Date: 2021-06-20 20:24
I have been working on debugging a segfault. When faulthandler catches the fault, it makes a printout like this:

Current thread 0x00007f4fa62b2700 (most recent call first):
  File "/usr/lib/python3.6/site-packages/tornado/ioloop.py", line 919, in call_at
  File "/usr/lib/python3.6/site-packages/tornado/ioloop.py", line 502, in add_timeout
  ...

However, when I run the same app with gdb, catch the segfault with gdb and and run py-bt, it makes a printout like this

(gdb) py-bt
Traceback (most recent call first):
  Garbage-collecting
  File "/usr/lib/python3.6/site-packages/tornado/ioloop.py", line 919, in call_at
    functools.partial(stack_context.wrap(callback), *args, **kwargs),
  File "/usr/lib/python3.6/site-packages/tornado/ioloop.py", line 502, in add_timeout
    return self.call_at(deadline, callback, *args, **kwargs)
  ...

The important distinction here for me is the "Garbage-collecting" line. When debugging this issue with faulthandler, I thought that the segfault was happening somewhere in the execution stack of this ioloop.py function. It wasn't until I ran under gdb that I realized it was actually happening in garbage collection and more or less has nothing to do with ioloop.py. It seems like faulthandler should be able to tell that the segfault was actually generated in garbage collection and this would make faulthandler much more helpful for cases like this.

Thank you for reading!
msg396202 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-06-20 22:59
faulthandler has no access to the C-stack so it cannot see the gc_collect_main() as gdb does.
msg396226 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-06-21 11:13
Maxwell: Oh, that's a good idea! I wrote PR 26823 to implement the feature. It's just 3 new lines in traceback.c :-)

        if (tstate == current_tstate && tstate->interp->gc.collecting) {
            PUTS(fd, "  Garbage-collecting\n");
        }
msg396228 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-06-21 11:15
New changeset d19163912bfc790283724f05328bd31e4e65003d by Victor Stinner in branch 'main':
bpo-44466: Faulthandler now detects the GC (GH-26823)
https://github.com/python/cpython/commit/d19163912bfc790283724f05328bd31e4e65003d
msg396237 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-06-21 12:23
New changeset 9b0bbb9143d15507aae0ff3afeb05969178b306c by Victor Stinner in branch '3.10':
bpo-44466: Faulthandler now detects the GC (GH-26823) (GH-26826)
https://github.com/python/cpython/commit/9b0bbb9143d15507aae0ff3afeb05969178b306c
msg396241 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-06-21 12:38
Maxwell: That was a nice idea and it's now added to Python 3.10 (will be part of the next Python 3.10 beta release)! I got an exception to land this feature in Python 3.10 even if we are past the feature freeze (see PR 26823 discussion).

As explained in PR 26823, Py_FatalError() also gets this _Py_DumpTracebackThreads() enhancement ;-) See an example at:
https://github.com/python/cpython/pull/26823#issuecomment-864950372
msg396281 - (view) Author: Maxwell Ballenger (maxballenger) Date: 2021-06-21 18:46
Thank you Victor, sounds great!
History
Date User Action Args
2021-06-21 18:46:37maxballengersetmessages: + msg396281
2021-06-21 12:38:14vstinnersetstatus: open -> closed
versions: + Python 3.10, Python 3.11, - Python 3.6
messages: + msg396241

resolution: fixed
stage: patch review -> resolved
2021-06-21 12:23:17vstinnersetmessages: + msg396237
2021-06-21 11:23:01vstinnersetpull_requests: + pull_request25407
2021-06-21 11:15:51vstinnersetmessages: + msg396228
2021-06-21 11:13:22vstinnersetmessages: + msg396226
2021-06-21 10:49:02vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request25404
2021-06-21 00:01:33xtreaksetnosy: + vstinner
2021-06-20 22:59:12pablogsalsetnosy: + pablogsal
messages: + msg396202
2021-06-20 20:24:28maxballengercreate