classification
Title: Allow traceback objects to be instantiated/mutated/annotated
Type: enhancement Stage: resolved
Components: Interpreter Core Versions: Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: brett.cannon, eric.snow, jcea, miss-islington, ncoghlan, njs, yselivanov
Priority: normal Keywords: patch

Created on 2017-06-06 03:41 by njs, last changed 2018-06-09 13:58 by jcea. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 4793 merged njs, 2017-12-11 10:10
PR 5653 merged ncoghlan, 2018-02-13 07:16
PR 5655 merged miss-islington, 2018-02-13 08:12
Messages (11)
msg295235 - (view) Author: Nathaniel Smith (njs) * (Python committer) Date: 2017-06-06 03:41
Currently, traceback objects don't expose any public constructor, are immutable, and don't have a __dict__ or allow subclassing, which makes it impossible to add extra annotations them.

It would be nice if these limitations were lifted, because there are rare but important cases where code needs to manipulate tracebacks directly, and currently these have to use awful stuff like ctypes. For example:

Jinja2:
https://github.com/pallets/jinja/blob/bec0065c4e7e89e7d893eda7840ba0219824b23c/jinja2/debug.py#L298-L372

Trio:
https://github.com/python-trio/trio/blob/496493afecc22d7d1a17175b6a2748a9c3510066/trio/_core/_multierror.py#L233-L318

(Notice that on PyPy there are no ctypes, but instead they have special extension that's supported for this case only to keep jinja2 working.)

For the above cases, what's needed is the ability to instantiate and assign to the fields of traceback objects.

In addition, in trio I'd like to be able to annotate traceback objects so that the traceback printing machinery can do things like hide "internal" tracebacks, or highlight places where exceptions jumped between tasks. This would be much easier if there were some way to attach data to tracebacks. Probably it doesn't make sense for traceback objects to have a __dict__ by default for speed/memory reasons, but we could add a dedicated metadata slot that is normally empty but can have arbitrary data assigned, or if they allowed subclassing then I could add a __dict__ in a subclass

I'm CC'ing the "import machinery" interest list, because my understanding is that with the new import system there's a similar desire to hide "internal" traceback frames, and while the features requested in this bug report won't solve that problem directly they might (are intended to) provide some useful machinery for it. Feel free to un-CC if I'm wrong :Jinja2:
https://github.com/pallets/jinja/blob/bec0065c4e7e89e7d893eda7840ba0219824b23c/jinja2/debug.py#L298-L372
-).

I think that this should be very straightforward to implement: it's just that no-one ever implemented setters etc. for tracebacks.
msg295236 - (view) Author: Nathaniel Smith (njs) * (Python committer) Date: 2017-06-06 03:42
Uh, please ignore the random second paste of the jinja2 URL in the middle of the second to last paragraph.
msg295237 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-06-06 04:00
Rather than allowing this on tracebacks directly, I'd prefer to build on the "TracebackException" work and allow traceback summaries in all the places where we currently require real tracebacks (including exception __traceback__ attributes): https://docs.python.org/3/library/traceback.html#tracebackexception-objects

The essential requirement here would be to abstract out a "traceback protocol" from the current concrete traceback objects, and adapt affected parts of the interpreter (such as the traceback display on shutdown) to use that abstract protocol when dealing with a non-native traceback object.
msg295238 - (view) Author: Nathaniel Smith (njs) * (Python committer) Date: 2017-06-06 04:24
My understanding is that the major difference between a real traceback object and a TracebackException object is that the latter is specialized for printing, so it can be lighter weight (no pinning of frame objects in memory), but loses some utility (can't do post-mortem debugging).

If that's right, then that's definitely not a solution, because trio and jinja2 and import errors all need to support post-mortem debugging.

I'm not against the idea of defining a traceback protocol, but it seems like a lot of work when the existing traceback objects are already perfectly good container objects that are just missing a few simple features.
msg295248 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-06-06 11:14
Aye, if you need full post-mortem debugging support, then updating the public API of the actual traceback objects based on the current ctypes-based workaround does seem like a better path forward.

TracebackException and friends would more come into play for the "make traceback printing easier to customise" part of your presentation.
msg308023 - (view) Author: Nathaniel Smith (njs) * (Python committer) Date: 2017-12-11 10:28
PR 4793 provides everything that Jinja2 and Trio actually need right now. It doesn't provide any way to annotate tracebacks with extra data, but I'm not entirely sure if that's necessary or what it would look like, so I figured I'd at least get this part in for 3.7.
msg309532 - (view) Author: Nathaniel Smith (njs) * (Python committer) Date: 2018-01-06 00:15
Ping -- anyone up for reviewing PR 4793?

https://github.com/python/cpython/pull/4793

It's pretty straightforward, and I figure better to ping now and beat the end-of-month rush :-)
msg309615 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2018-01-07 13:30
New changeset e46a8af450210ee5c7f0459ad6beddbc626ae60f by Nick Coghlan (Nathaniel J. Smith) in branch 'master':
bpo-30579: Allow TracebackType creation and tb_next mutation from Python (GH-4793)
https://github.com/python/cpython/commit/e46a8af450210ee5c7f0459ad6beddbc626ae60f
msg309616 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2018-01-07 13:31
Thanks for the patch!
msg312107 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2018-02-13 08:11
New changeset aec7532ed3ccbd29d3429a3f375e25f956c44003 by Nick Coghlan in branch 'master':
bpo-30579: Docs for dynamic traceback creation (GH-5653)
https://github.com/python/cpython/commit/aec7532ed3ccbd29d3429a3f375e25f956c44003
msg312109 - (view) Author: miss-islington (miss-islington) Date: 2018-02-13 08:53
New changeset 53374cc57f33f1afb22924da3a76ec6cf9e4afc1 by Miss Islington (bot) in branch '3.7':
bpo-30579: Docs for dynamic traceback creation (GH-5653)
https://github.com/python/cpython/commit/53374cc57f33f1afb22924da3a76ec6cf9e4afc1
History
Date User Action Args
2018-06-09 13:58:32jceasetnosy: + jcea
2018-02-13 08:53:45miss-islingtonsetnosy: + miss-islington
messages: + msg312109
2018-02-13 08:12:09miss-islingtonsetpull_requests: + pull_request5457
2018-02-13 08:11:01ncoghlansetmessages: + msg312107
2018-02-13 07:16:20ncoghlansetpull_requests: + pull_request5455
2018-01-07 13:31:58ncoghlansetstatus: open -> closed
resolution: fixed
messages: + msg309616

stage: patch review -> resolved
2018-01-07 13:30:20ncoghlansetmessages: + msg309615
2018-01-06 00:15:20njssetmessages: + msg309532
2017-12-11 12:58:10vstinnersetnosy: - vstinner
2017-12-11 10:28:46njssetmessages: + msg308023
2017-12-11 10:10:34giampaolo.rodolasetnosy: - giampaolo.rodola
2017-12-11 10:10:00njssetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request4692
2017-06-07 22:03:12Mariattasetstage: needs patch
2017-06-06 11:14:41ncoghlansetmessages: + msg295248
2017-06-06 04:24:23njssetmessages: + msg295238
2017-06-06 04:00:58ncoghlansetnosy: + vstinner, giampaolo.rodola, yselivanov
messages: + msg295237
2017-06-06 03:42:45njssetmessages: + msg295236
2017-06-06 03:41:53njscreate