classification
Title: format_exception() breaks on exception tuples from trace function
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.4, Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Aaron.Meurer, HJarausch, benjamin.peterson, bmac, inducer, ingrid, python-dev, r.david.murray
Priority: normal Keywords: easy, patch

Created on 2013-03-14 00:56 by inducer, last changed 2013-04-19 17:01 by r.david.murray. This issue is now closed.

Files
File name Uploaded Description Edit
exc-bug.py inducer, 2013-03-14 00:56
issue17413.patch ingrid, 2013-04-13 23:38 review
issue17413.patch ingrid, 2013-04-18 08:42 review
Messages (10)
msg184122 - (view) Author: Andreas Kloeckner (inducer) Date: 2013-03-14 00:56
traceback.format_exception() used to work properly with the 'arg' value passed to a tracing function set up via sys.settrace for an 'exception' event. In Python 3.x, this is no longer the case. Below you'll find what the attached test produces for a variety of interpreter versions.

If this is not intended to work, I would be much obliged for a hint on how to achieve the desired effect--i.e. get a formatted exception traceback for the exception tuple 'arg' passed to the trace function.

Thanks!

See also: https://github.com/inducer/pudb/issues/61

#####################################################
Python 2.7
#####################################################
ZZ call
ZZ line
ZZ line
ZZ exception
exception (<type 'exceptions.AttributeError'>, "'int' object has no attribute 'invalid'", <traceback object at 0x1951fc8>)
-------------------------------
Traceback (most recent call last):
  File "exc-bug.py", line 20, in f
    x.invalid
AttributeError: 'int' object has no attribute 'invalid'

-------------------------------
ZZ return
Traceback (most recent call last):
  File "exc-bug.py", line 22, in <module>
    f()
  File "exc-bug.py", line 20, in f
    x.invalid
AttributeError: 'int' object has no attribute 'invalid'
ZZ call
ZZ call
#####################################################

#####################################################
Python 3.2
#####################################################
ZZ call
ZZ line
ZZ line
ZZ exception
exception (<class 'AttributeError'>, "'int' object has no attribute 'invalid'", <traceback object at 0x7f4cf42e5f80>)
-------------------------------
Traceback (most recent call last):
  File "exc-bug.py", line 22, in <module>
    f()
  File "exc-bug.py", line 20, in f
    x.invalid
  File "exc-bug.py", line 9, in trace
    print("".join(format_exception(*arg)))
  File "/usr/lib/python3.2/traceback.py", line 180, in format_exception
    for value, tb in values:
  File "/usr/lib/python3.2/traceback.py", line 122, in _iter_chain
    cause = exc.__cause__
AttributeError: 'str' object has no attribute '__cause__'
#####################################################

#####################################################
Python 3.3
#####################################################
ZZ call
ZZ line
ZZ line
ZZ exception
exception (<class 'AttributeError'>, "'int' object has no attribute 'invalid'", <traceback object at 0x7f47383acb00>)
-------------------------------
Traceback (most recent call last):
  File "exc-bug.py", line 22, in <module>
    f()
  File "exc-bug.py", line 20, in f
    x.invalid
  File "exc-bug.py", line 9, in trace
    print("".join(format_exception(*arg)))
  File "/usr/lib/python3.3/traceback.py", line 181, in format_exception
    for value, tb in values:
  File "/usr/lib/python3.3/traceback.py", line 122, in _iter_chain
    context = exc.__context__
AttributeError: 'str' object has no attribute '__context__'
#####################################################
msg184125 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-03-14 01:26
It looks like a bug in the tracing machinery that has only been revealed by the changes to how tracebacks are interpreted in python3.  It should be a relatively simple fix, but I wonder if there is existing code that depends on the second argument getting turned into a string.

You can hack around the problem by creating a class to wrap around the arg[1] you get that has __cause__ and __context__ attributes, both set to none, and whose __str__ returns the original string.
msg184143 - (view) Author: Helmut Jarausch (HJarausch) Date: 2013-03-14 08:05
The problem is caused by the new format_exception in Python's traceback.py file. It reads

def format_exception(etype, value, tb, limit=None, chain=True): 
  list = []
  if chain:
     values = _iter_chain(value, tb)
  else:
     values = [(value, tb)]
  for value, tb in values:
      if isinstance(value, str):

and then

def _iter_chain(exc, custom_tb=None, seen=None):
    if seen is None:
        seen = set()
    seen.add(exc)
    its = []
    context = exc.__context__

As you can see, the new keyword parameter chain is True by default. Thus, iter_chain is called by default.

And there you have context= exc.__context__.
Now, if value is an object of type str Python tries to access the __context__ field of an object of type str.

And this raises an attribute error.

In an application (pudb) I've used the fixed

exc_info= sys.exc_info()
....
format_exception(*exc_info,chain=not isinstance(exc_info[1],str))

So, why is the keyword parameter 'chain' True by default.
This causes the problem.
msg184146 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-03-14 08:13
Because the second argument to format_traceback is supposed to be (is documented to be) an exception object.  The fact that it used to work anyway in Python2 if you passed a string was an accident of the implementation.  Likewise, settrace is documented to provide a value, not a string, so the fact that it provides a string is a bug.  That needs to be fixed.

(As to specifically why chain defaults to True, it defaults to having the same behavior as the normal exception machinery.)
msg184173 - (view) Author: Andreas Kloeckner (inducer) Date: 2013-03-14 16:59
Thanks for the suggestion. Since 3.2 and 3.3 will be with us for a while, I've implemented the workaround you've suggested. Works, too. :)
msg186867 - (view) Author: (ingrid) * Date: 2013-04-13 23:38
It seems that settrace works normally when an exception is raised in the python code with the raise keyword. If an exception is raised in the C code, settrace breaks as the C code passes all exceptions as strings. To fix this issue we just added a line to normalize the C exception within settrace. We included a regression test for this defect.

I paired on this with user bmac (http://bugs.python.org/user17692).
msg186910 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-04-14 11:09
Thanks Ingrid and Mark.  The patch looks good; I put a couple of FYI comments on the review.

I'm pretty sure this patch is correct, but I'd like someone with more experience modifying the ceval loop to confirm, so I'm nosying Benjamin.
msg187229 - (view) Author: (ingrid) * Date: 2013-04-18 08:42
Thank you, r.david.murray. I have updated the patch with your suggestions included.
msg187377 - (view) Author: Roundup Robot (python-dev) Date: 2013-04-19 16:58
New changeset d18df4c90515 by R David Murray in branch '3.3':
#17413: make sure settrace funcs get passed exception instances for 'value'.
http://hg.python.org/cpython/rev/d18df4c90515

New changeset 6297fcddf912 by R David Murray in branch 'default':
Merge #17413: make sure settrace funcs get passed exception instances for 'value'.
http://hg.python.org/cpython/rev/6297fcddf912
msg187378 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-04-19 17:01
Benjamin reviewed the patch and pointed out that the settrace state needed to be restored in the test, so I fixed that when I committed it.

Thanks Ingrid and Brendan.
History
Date User Action Args
2013-04-19 17:01:19r.david.murraysetstatus: open -> closed
resolution: fixed
messages: + msg187378

stage: commit review -> resolved
2013-04-19 16:58:26python-devsetnosy: + python-dev
messages: + msg187377
2013-04-18 08:42:28ingridsetfiles: + issue17413.patch

messages: + msg187229
2013-04-14 11:09:10r.david.murraysetversions: - Python 3.2
nosy: + benjamin.peterson

messages: + msg186910

stage: needs patch -> commit review
2013-04-13 23:43:20ingridsetnosy: + bmac
2013-04-13 23:38:32ingridsetfiles: + issue17413.patch

nosy: + ingrid
messages: + msg186867

keywords: + patch
2013-03-14 16:59:57inducersetmessages: + msg184173
2013-03-14 08:13:23r.david.murraysetmessages: + msg184146
2013-03-14 08:05:17HJarauschsetnosy: + HJarausch
messages: + msg184143
2013-03-14 01:30:20Aaron.Meurersetnosy: + Aaron.Meurer
2013-03-14 01:26:48r.david.murraysettitle: format_exception() breask on exception tuples from trace function -> format_exception() breaks on exception tuples from trace function

keywords: + easy
nosy: + r.david.murray
versions: + Python 3.2, Python 3.4
messages: + msg184125
stage: needs patch
2013-03-14 00:56:34inducercreate