classification
Title: Fatal Python error "XXX block stack overflow" when exception stacks >10
Type: crash Stage: patch review
Components: Interpreter Core Versions: Python 3.10, Python 3.9, Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Mark.Shannon, iritkatriel, myzhang1029, pablogsal, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2020-03-11 13:24 by myzhang1029, last changed 2020-10-07 23:24 by pablogsal.

Files
File name Uploaded Description Edit
exception_nest.py myzhang1029, 2020-03-11 13:24 Run exception_nest.py and run the generated python code
Pull Requests
URL Status Linked Edit
PR 22395 merged Mark.Shannon, 2020-09-24 11:22
Messages (12)
msg363914 - (view) Author: Maiyun Zhang (myzhang1029) Date: 2020-03-11 13:24
I apologize for describing this issue badly, but I'll try anyway.
The code to demonstrate the issue is attached, so it might be better to read that instead.
I noticed that when more than 10 exceptions are raised sequentially (i.e. one from another or one during the handling of another), the interpreter crashes saying "Fatal Python error: XXX block stack overflow".
This happens in python 3.7, 3.8 and development(git 39c3493) versions, but not in python2.7. Using ipython also fixes this issue.
I know this case is rare, but the maximum number of recursions is more than 2000, and the maximum number of statically nested blocks sepcified in frameobject.c is 20, so I'm pretty sure this isn't intended behavior.
msg377433 - (view) Author: Irit Katriel (iritkatriel) * (Python triager) Date: 2020-09-23 23:11
The error is coming from here: https://github.com/python/cpython/blob/cb9879b948a19c9434316f8ab6aba9c4601a8173/Objects/frameobject.c#L958

and CO_MAXBLOCKS is defined in Include/cpython/code.h
#define CO_MAXBLOCKS 20 /* Max static block nesting within a function */

This is not about recursion or about exception, it's about static nesting level. 

There is an example here showing an input that gives the same error with >20 nested while blocks: https://github.com/python/cpython/blob/c8f29ad986f8274fc5fbf889bdd2a211878856b9/Lib/test/test_syntax.py#L545
msg377434 - (view) Author: Irit Katriel (iritkatriel) * (Python triager) Date: 2020-09-23 23:12
In summary, I think this is not-a-bug.
msg377437 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-09-24 06:59
It is a bug. Compiler explicitly checks if the number of nested "try" blocks does not exceed the limit of CO_MAXBLOCKS, but it does not count implicit "try" blocks inserted when your assign an exception in the "except" clause.

    try:
        ...
    except Exception as e:
        ...

is actually translated to

    try:
        ...
    except Exception:
        try:
            e = ...
            ...
        finally:
            e = None
            del e

So we have double number of nested "try" blocks.
msg377439 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-09-24 07:34
Humm, my supposition was not absolutely correct. The cause is that the compiler and the interpreter use the stack of size CO_MAXBLOCKS for different things. The interpreter pushes a thing for the "except" clause, while the compiler does not do it.
msg377440 - (view) Author: Irit Katriel (iritkatriel) * (Python triager) Date: 2020-09-24 08:02
Unlike the 22 nested while,

if 1:
  if 2:
    ...
    if 22:
        pass

doesn't error. Is that correct?
msg377441 - (view) Author: Irit Katriel (iritkatriel) * (Python triager) Date: 2020-09-24 08:14
Another oddity:

This gives the error:
for x in '1':
 for x in '2':
  for x in '3':
   for x in '4':
    for x in '5':
     for x in '6':
      for x in '8':
       for x in '9':
        for x in '10':
         for x in '11':
          for x in '12':
           for x in '13':
            for x in '14':
             for x in '15':
              for x in '16':
               for x in '17':
                for x in '18':
                 for x in '19':
                  for x in '20':
                   for x in '21':
                    for x in '22':
                     pass

but this doesn't:

for x in '1':
 for x in '2':
  for x in '3':
   for x in '4':
    for x in '5':
     for x in '6':
      for x in '8':
       for x in '9':
        for x in '10':
         for x in '11':
          for x in '12':
           for x in '13':
            for x in '14':
             for x in '15':
              for x in '16':
               for x in '17':
                for x in '18':
                 for x in '19':
                  for x in '20':
                   for x in '21':
                    pass
                   else:
                    for x in '22':
                     pass
msg377444 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2020-09-24 10:47
iritkatriel

What error do you see and on what version?
msg377445 - (view) Author: Irit Katriel (iritkatriel) * (Python triager) Date: 2020-09-24 11:13
On windows 10, master:

> python.bat --version
Running Release|Win32 interpreter...
Python 3.10.0a0

>python.bat x.py
Running Release|Win32 interpreter...
SyntaxError: too many statically nested blocks

That's when x.py has the nested for loops without else. The error goes away if I add the else.
msg377490 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2020-09-25 13:04
New changeset 02d126aa09d96d03dcf9c5b51c858ce5ef386601 by Mark Shannon in branch 'master':
bpo-39934: Account for control blocks in 'except' in compiler. (GH-22395)
https://github.com/python/cpython/commit/02d126aa09d96d03dcf9c5b51c858ce5ef386601
msg378178 - (view) Author: Irit Katriel (iritkatriel) * (Python triager) Date: 2020-10-07 20:51
After studying Mark's patch and the rest of the code, I understand the for loop oddity.

The else block of the for loop comes after the 
compiler_pop_fblock(c, FOR_LOOP, start);

So you can do this all day and it won't complain:

for x in '1': pass
else:
 for x in '2': pass
 else:
  for x in '3': pass
  else:
   for x in '4': pass
   else:
    for x in '5': pass
    else:
     for x in '6': pass
     else:
      for x in '7': pass
      else:
       for x in '8': pass
       else:
        for x in '9': pass
        else:
         for x in '10': pass
         else:
          for x in '11': pass
          else:
           for x in '12': pass
           else:
            for x in '13': pass
            else:
             for x in '14': pass
             else:
              for x in '15': pass
              else:
               for x in '16': pass
               else:
                for x in '17': pass
                else:
                 for x in '18': pass
                 else:
                  for x in '19': pass
                  else:
                   for x in '20': pass
                   else:
                    for x in '21': pass
                    else:
                     for x in '22': pass
                     else:
                      for x in '23': pass
                      else:
                       for x in '24': pass
                       else:
                        for x in '25': pass
                        else:
                         for x in '26': pass
                         else:
                          for x in '27': pass
                          else:
                           for x in '28': pass
                           else:
                            for x in '29': pass



I guess the same goes for while loops, and the else of a try-except.


Since If blocks were deliberately left out of this game, I'm assuming that this "static nesting" is actually number of frames, rather than true static nesting. If that is the case then there is no issue here and we can close this ticket.

But if this is something to be fixed, then I am happy to make a patch along the lines of Mark's (which I partially already have).
msg378192 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-10-07 23:24
PR 22395 should be backported to 3.8 and 3.9
History
Date User Action Args
2020-10-07 23:24:40pablogsalsetnosy: + pablogsal
messages: + msg378192
2020-10-07 20:51:18iritkatrielsetmessages: + msg378178
2020-09-25 13:04:29Mark.Shannonsetmessages: + msg377490
2020-09-24 11:22:21Mark.Shannonsetkeywords: + patch
stage: patch review
pull_requests: + pull_request21435
2020-09-24 11:13:12iritkatrielsetmessages: + msg377445
2020-09-24 10:47:07Mark.Shannonsetmessages: + msg377444
2020-09-24 08:14:09iritkatrielsetmessages: + msg377441
2020-09-24 08:02:56iritkatrielsetmessages: + msg377440
2020-09-24 07:34:09serhiy.storchakasetmessages: + msg377439
2020-09-24 06:59:53serhiy.storchakasetversions: + Python 3.8, Python 3.9, Python 3.10, - Python 3.7
nosy: + Mark.Shannon, serhiy.storchaka

messages: + msg377437

type: behavior -> crash
2020-09-23 23:12:15iritkatrielsetmessages: + msg377434
2020-09-23 23:11:01iritkatrielsetnosy: + iritkatriel
messages: + msg377433
2020-03-11 13:24:52myzhang1029create