Consider this:

$ python3 test_exc.py 
Traceback (most recent call last):
  File "test_exc.py", line 14, in <module>
    fail1()
  File "test_exc.py", line 11, in fail1
    fail2()
  File "test_exc.py", line 5, in fail2
    raise RuntimeError('Third') from None
RuntimeError: Third

$ cat test_exc.py 
def fail2():
    try:
        raise RuntimeError('Second')
    except RuntimeError:
        raise RuntimeError('Third') from None

def fail1():
    try:
        raise RuntimeError('First')
    except:
        fail2()
        raise

fail1()

Any exception raised in fail2() is the immediate consequence of the 'First' exception should thus be chained to the 'First' exception.

However, if somewhere in the call stack under fail2() an exception is caught and re-raised from None (to convert between exception types), this also results in a loss of the chain to the initial exception.

The correct stacktrace (in my opinion) would be:
Traceback (most recent call last):
  File "test_exc.py", line 9, in fail1
    raise RuntimeError('First')
RuntimeError: First

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test_exc.py", line 14, in <module>
    fail1()
  File "test_exc.py", line 11, in fail1
    fail2()
  File "test_exc.py", line 5, in fail2
    raise RuntimeError('Third')
RuntimeError: Third

The second paragraph should of course read "...consequence of the 'First' exception *and* should thus be chained..."

*ping*

No one any comments on this at all?

raise ... from None explictly silences the printing of exception context, so I don't see the problem.

Nothing is lost either way, it just isn't displayed by default.

While I can see the appeal of treating an "outer" context differently from an "inner" one, I don't see any way for the exception context suppression machinery to tell the difference.

The traceback *display* machinery might be able to figure it out by looking at the traceback details, but such a search could be quite expensive.

Benjamin: I think that in most cases the intention of a ".. from None" is to disable printing of a stack trace for a specific exception when it occurs in the try .. except block. (In the example, that would be suppressing the stacktrace for the "Second" exception, because it is explicity converted to "Third"). I do not think that this ought to effect the printing of exceptions that occured previously and higher-up in the call chain.

In other words, I think it is natural to expect that

def foo():
   try:
      do_stuff
   except Something:
      raise SomethingElse from None

disables printing the stack trace for Something exceptions when they occur in the try..except block -- but not to hide the exception context for exceptions that occured before foo() was even called.

I actually think Nikratio is right about the way this *should* work (intuitively).

I'm just not sure it's feasible to *implement* those semantics in CPython without significant changes to the way exception handling works - I don't believe the exception chaining code can currently tell the difference between the cases:

# No context on Exception3 is exactly what we want
try:
try:
raise Exception1
except Exception1:
raise Exception2
except Exception2 as exc
raise Exception3 from Exception2

# Not seeing Exception1 as the context for Exception3 is surprising!
try:
raise Exception1
except Exception1:
try:
raise Exception2
except Exception2 as exc
raise Exception3 from Exception2

In a certain sense, exceptions need to be able to have *multiple* contexts to handle this case properly without losing data. Frames would need to tag exceptions appropriately with the context details as an unhandled exception passed through a frame that was currently running an exception handler.

So even though it doesn't require new syntax, I think it *does* require a PEP if we're going to change this (and we still haven't fully dealt with the consequence of the last one - the display options for tracebacks are still a bit limited)

It may not immediately look like it, but I think bpo-17828 offers an example of a related problem. In that issue, I didn't want to *change* the exception raised, I wanted to annotate it to say "Hey, something I called raised an exception, here's some relevant local state to help you figure out what is going on" (in that case, the information to be added is the specific codec being invoked and whether it is an encoding or decoding operation).

Setting the context is somewhat similar - you don't just want to know which specific exception happened to be active when the eventually caught exception was first raised - you also want to know which already active exception handlers you passed through while unwinding the stack.

So really, what may be desirable here is almost an "annotated traceback", where the interpreter can decide to hang additional information off the frame currently being unwound in the exceptions traceback, while leaving the exception itself alone.

That's definitely PEP territory, but there are two distinct problems with the current exception chaining mechanism to help drive a prospective design for 3.5 (the status quo doesn't bother me enough for me to work on it myself, but if you're interested Nikolaus...)

Unrelated to my previous comment, I'm also wondering if this may actually represent a behavioural difference between contextlib.ExitStack and the interpreter's own exception handling machinery.

ExitStack uses a recursive->iterative transformation for its stack unwinding (see bpo-14963), and it needs to do a bit of fiddling to get the context right (see bpo-19092).

While the contextlib test suite goes to great lengths to try to ensure the semantics of normal stack unwinding are preserved, that definitely doesn't currently cover this case, and I'm thinking the way it works may actually be more like the behaviour Nikolaus expected in the original post (i.e. setting the context as the stack is unwound rather than when the replacement exception is raised).

Hi Nick,

I am interested in working on this, but I have never worked on the C parts of cpython before. Do you think this is a feasible project to start with? To me it looks a bit daunting, I'd certainly need some mentoring to even know where to start with this.

The first thing to do is to carefully specificy what the behavior should be.

2013/11/10 Nikolaus Rath <report@bugs.python.org>:

Nikolaus Rath added the comment:

Hi Nick,

I am interested in working on this, but I have never worked on the C parts of cpython before. Do you think this is a feasible project to start with? To me it looks a bit daunting, I'd certainly need some mentoring to even know where to start with this.

----------

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue18861\>

---

Yes, I suggest using ExitStack to figure out the behaviour we *want* first,
before diving into the messy practical details of how to make that a
reality in CPython. We somehow have to get the state of the exception
object and its traceback to represent an appropriate stack *tree*, rather
than the traditionally assumed linear stack.

It also occurred to me there's another potentially related issue: frame
hiding, where we want to avoid showing infrastructure code in end user
tracebacks. importlib currently has a very hacky version of that. The
Jinja2 template library uses a different approach.

The reason I bring these other problems up is because I think they
illustrate a theme around altering how a traceback is displayed that may be
amenable to a common solution (preferably one that is contextlib and
asyncio friendly).

So, I've been pondering the idea of traceback/frame annotations and exception trees a bit.

And what I'm wondering is if it may make sense to have the ability to annotate *frames* at runtime, and these annotations can be qualified by module names. So, for example, you might be able to write things like:

    sys.annotate_frame("codecs", "encoding", the_encoding)
    sys.annotate_frame("codecs", "decoding", the_encoding)
    sys.annotate_frame("traceback", "hide", True)
    sys.annotate_frame("traceback", "context", exc)

And then the traceback display machinery would be updated to do something useful with the annotations.

I'm not sure how ExitStack would cope with that (or other code that fakes tracebacks) but it's something to ponder.

Walter suggested annotating the exceptions directly might work: https://mail.python.org/pipermail/python-dev/2013-November/130155.html

However, that potentially runs into nesting problems (e.g. the idna codec invokes the ascii codec), although Walter suggested a simpler mechanism that just involved appending to a list.

Adjusting the target version, since it isn't feasible to move away from the current behaviour any earlier than Python 3.5.

Something else that this might make simpler: the unittest.TestCase.subTest API is currently kind of ugly. If the subtest details could be stored as a frame annotation instead...

Actually, this won't the subTest example because that actually *suppresses* the errors, and reports them later. Annotations only help when the exception is allowed to escape.

For the issue of which contexts need to be suppressed when __suppress_context__ is true, I think we don't need to keep the whole tree of exceptions. It might be enough to save the dynamic "nesting depth of the try-excepts". Here is a solution with two integers per exception, plus one on the thread state. Would it work?

We add an int field called except_depth to the thread state (incremented in ceval just before the try of a try-except and decremented after the last except: block).

Whenever an exception is caught, this value is copied to it (to a new int we put on BaseException, call it depth).

The __suppress_context__ field we change from bool to int (alternatively: add a new field and leave this one alone). Instead of setting it to True when there is __cause__, we set it to the except-depth of the raise...from that raised it.

Then the code displaying the traceback can use the depths information to determine which contexts to suppress (the ones up to and not including the first one with depth < the suppress_context value).

Returning to Nick's examples from 2013-11-08 14:34:

# No context on Exception3 is exactly what we want
try:
try:
raise Exception1
except Exception1:
raise Exception2
except Exception2 as exc
raise Exception3 from Exception2

Exception1 - depth = 2
Exception2 - depth = 1
Exception3 - depth = ? suppress_context = 1
(suppress both Exception1 and Exception2, they have depths >= 1)

# Not seeing Exception1 as the context for Exception3 is surprising!
try:
raise Exception1
except Exception1:
try:
raise Exception2
except Exception2 as exc
raise Exception3 from Exception2

Exception1 - depth = 1
Exception2 - depth = 2
Exception3 - depth = ? suppress_context = 2
(suppress only Exception2, Exception1 is too shallow)

To consider more elaborate nesting:

try:
try:
raise Exception0
except T: # Does not catch Exception0
pass
except Exception1:
try:
raise Exception2
except Exception2 as exc
raise Exception3 from Exception2

Whatever happens in the topmost try-block, if Exception0 is the context of anything raised inside the "except Exception1" block, then its depth is 1 because it was caught by this except. So it won't be suppressed by anything raised at a deeper level.

I think this problem is actually simpler than what we've been discussing.

First, note that by-and-large our current system works:

>>> try:
...   raise VE(1)
... except VE as e1:
...   try:
...     raise VE(2)
...   except VE as e2:
...     raise VE(3) from e2
... 
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
ValueError: 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 5, in <module>
ValueError: 2

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 7, in <module>
ValueError: 3

Here VE(2) is the cause of VE(3) and VE(1) is the context of VE(2), so VE(1) is not hidden by the fact that VE(3) has a cause.

The reason that Nick's example didn't work is because he is doing

    raise VE(3) from VE(2)

i.e., creating a new exception VE(2) that doesn't have VE(1) as a context. I'm not sure there is a use case for this, so I don't think we need to worry about it.

The case of None does need fixing. We use None to indicate that there was no cause (while suppressing context). But None can't have a context, so VE(1) gets lost. We could, instead of None, use a NoException(Exception) class. This exception would be chained with the current exc_info() as context so that it works like VE(1) as above, and the traceback printing logic will know that it needs to be omitted from the output.

This proposal involves no changes to the exception propagation mechanism of the interpreter. It would require these changes:

1. Define the new exception type
2. in do_raise, in the case of raise-from the None case changes
3. traceback display code needs to be updated to omit NoExceptions