Title: Interpreter aborts when chaining an infinite number of exceptions
Type: behavior Stage: patch review
Components: Interpreter Core Versions: Python 3.3, Python 3.2
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, amaury.forgeotdarc, benjamin.peterson, jcea, pconnell, pitrou, rhettinger, terry.reedy, yury
Priority: normal Keywords: patch

Created on 2009-05-15 08:29 by yury, last changed 2013-10-04 16:56 by jcea. This issue is now closed.

File name Uploaded Description Edit
stackoverflow.patch amaury.forgeotdarc, 2009-05-15 17:47 review
Messages (14)
msg87797 - (view) Author: Yury (yury) Date: 2009-05-15 08:29
def error_handle():


Fatal Python error: Cannot recover from stack overflow.

The interpreter should not crash. Perhaps a RuntimeError should be
thrown instead.
msg87804 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-05-15 11:22
This is normal behaviour, actually. The RuntimeError *is* raised, but
you catch it in the except clause and then recurse again ad infinitum.
The interpreter realizes that it "cannot recover from stack overflow",
as the message says, and then bails out.
msg87824 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2009-05-15 17:47
Hmm, the interpreter should not crash so easily with pyre Python code.
(The same code correctly raises RuntimeException wich python 2.x)
The issue should be corrected IMO.

The exact behavior seem to depend on where the recursion limit is detected:

- If it is detected at the beginning of the function (at the start of
PyEval_EvalFrameEx) or before the "try" statement, the exception is
correctly propagated and displayed. This case always happens with 2.x,
and you get the same on 3.0 if you add for example str([[[[[[[[]]]]]]]])
at the beginning of the function.

- The crash occurs when the recursion limit is detected while
PyEval_EvalFrameEx calls PyErr_NormalizeException() (in the "why ==
WHY_EXCEPTION" block). This does not change the control flow: the
original exception is simply replaced by the RuntimeError, and nested
calls continue 50 more frames.

Here is a tentative patch.
msg87836 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-05-15 20:40
Amaury, your patch might make some individual cases better, but it won't
prevent a FatalError from occurring in all cases.

Also, it makes things worse in the following case:

def recurse():

(the script goes into an uninterruptible infinite loop, rather than
raising an explicit Fatal error)

More useful, IMHO, would be to patch Py_FatalError() so that a traceback
is printed.
msg87856 - (view) Author: Yury (yury) Date: 2009-05-16 04:55
The code you posted causes an infinite loop in the 2.x branch as well.
Anyway, I do not see how crashing is a desired result. An infinite loop
means the programmer made a mistake somewhere. A crash means the
interpreter did.
msg87867 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-05-16 09:05
> The code you posted causes an infinite loop in the 2.x branch as well.
> Anyway, I do not see how crashing is a desired result.

I do not see what the "desired result" is in your example. The code is
obviously wrong. Did you get hit by that in production code or is it
just a proof-of-concept?

Also, this is not an actual crash (as in "segmentation fault"). It is
the interpreter /trying to protect itself from a crash/ which would be
caused by a stack overflow, and your code is trying to circumvent that

The fact that some errors cannot be recovered from is an unavoidable
fact of life. We may try to "fix" this particular case, but it will do
nothing for the more general case, so we might as well not "fix" it.
msg87890 - (view) Author: Yury (yury) Date: 2009-05-16 13:14
I knew that python handles infinite recursion and gracefully errors out,
and I knew that exception chaining was new to 3.0, so I wanted to see if
they would work together. Apparently, they do not. Yet, the code works
fine in the 2.x branch. So, the 3.0 branch introduces a bug. I am not
sure how much clearer I can make this.

The code is in no way trying to circumvent anything. If there is about
to be a stack overflow, an exception show be raised. It should then
float up the stack. This is the desired result. In fact, this is the
only sane result. In an operating system, a program should not be able
to crash the entire system. In X, a client should not be able to crash
the server. Same principle applies here.

I am sorry I simply do not see your point. This seems so obvious. There
is a correct way of handling this error. You can safely recover from it.
If you insist on a general case, I have outlined it above.

I am not terribly thrilled about your attitude of "we might as well
leave it broken." Why bother working on the project at all?
msg87891 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-05-16 13:35
While we seem to disagree on whether this a real bug (and I'll leave it
at that), I'll just stress once again that a "fatal error" is totally
different from an uncontrolled crash like a segmentation fault -- as I
explained and although you don't seem to understand, the "fatal error"
mechanism is there /precisely/ to avoid uncontrolled crashes. Also, your
analogy with processes crashing an X server is flawed. You should be in
control of all the code running in your own program; Python doesn't
offer any mechanism to protect good code from badly written code when
running in the same address space -- any such expectation is misguided.

(for various ways of producing fatal errors, please search for
"Py_FatalError()" in the source tree)
msg87898 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2009-05-16 14:45
> You should be in control of all the code running in your own program
This is not the case with applications that embed Python to provide 
users a way to script the application.
All the usages of Py_FatalError I've seen detect programming errors at 
the C level, or at interpreter startup.
I still take the rule that no python code should be able to crash the 
program unexpectedly.
msg87907 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2009-05-16 17:51
Well, perhaps something like #1195571 should be added.
msg150241 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-12-24 23:07
I believe #3555, #7338, and *13644 are basically duplicates of this issue. I have left this one open because it has a try at a patch. I think any patch should be tested with the other examples.

I agree with Antoine that an intentional exit is not a crash.
I also agree that the current procedure is not really a bug either. According to the language spec, the interpreter should recurse forever;-), just like "while True: pass" iterates 'forever'. Given that it does not due to finite limitations, the exact alternate behavior is undefined.

Now, if someone can find a way to better handle infinite recursion mixed with exceptions, without re-introducing real crashes or eliminating the benefits of the 3.0 changes, great.

Yury, I think Antoine's point is that gracefully handling all the different kinds of programming mistakes in a finite system is a delicate and difficult balancing act.
msg150243 - (view) Author: Yury (yury) Date: 2011-12-24 23:42
Rather than aborting with a stack overflow, I feel it is more natural to raise an exception. If it is not too difficult to implement, perhaps another type of exception should be raised. Since chained exceptions are new to 3.x, there should be a new exception to describe errors that happen in chaining. Perhaps stopping chaining at a certain depth and truncating all further exceptions with a blanket "ChainingException" exception. Or perhaps truncating the chain and wrapping it in another exception before returning it to user code.

While this is my proposed solution, I apologize I cannot volunteer to write the patch. The time investment on my part would involve learning most of the working of the interpreter. Perhaps someone who is already familiar with it would be interested.
msg150250 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2011-12-26 00:13
FWIW, I concur with Antoine on this one.
msg187835 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-04-26 07:08
Closing as won't fix. There is no sane way around the current behaviour.
Date User Action Args
2013-10-04 16:56:46jceasetnosy: + jcea
2013-10-04 03:56:36benjamin.petersonlinkissue18129 superseder
2013-04-26 07:08:08pitrousetstatus: open -> closed
resolution: wont fix
messages: + msg187835
2013-04-25 07:19:52pconnellsetnosy: + pconnell
2011-12-26 00:13:06rhettingersetnosy: + rhettinger
messages: + msg150250
2011-12-25 03:19:14Arfreversetnosy: + Arfrever
2011-12-24 23:42:43yurysetmessages: + msg150243
2011-12-24 23:07:50terry.reedysetversions: + Python 3.2, Python 3.3, - Python 3.0
type: crash -> behavior

nosy: + terry.reedy
title: Interpreter crashes when chaining an infinite number of exceptions -> Interpreter aborts when chaining an infinite number of exceptions
messages: + msg150241
stage: patch review
2011-12-24 22:47:36terry.reedylinkissue7338 superseder
2011-12-24 22:45:47terry.reedylinkissue13644 superseder
2009-05-16 17:51:32benjamin.petersonsetnosy: + benjamin.peterson
messages: + msg87907
2009-05-16 14:45:17amaury.forgeotdarcsetmessages: + msg87898
2009-05-16 13:35:56pitrousetmessages: + msg87891
2009-05-16 13:14:14yurysetmessages: + msg87890
2009-05-16 09:05:21pitrousetmessages: + msg87867
2009-05-16 04:55:19yurysetmessages: + msg87856
2009-05-15 20:40:04pitrousetmessages: + msg87836
2009-05-15 17:47:40amaury.forgeotdarcsetstatus: closed -> open
files: + stackoverflow.patch

keywords: + patch
nosy: + amaury.forgeotdarc
messages: + msg87824
resolution: not a bug -> (no value)
2009-05-15 11:22:17pitrousetstatus: open -> closed

nosy: + pitrou
messages: + msg87804

resolution: not a bug
2009-05-15 08:29:39yurycreate