Fatal Python error "XXX block stack overflow" when exception stacks >10 #84115

myzhang1029 · 2020-03-11T13:24:52Z

BPO	39934
Nosy	@ambv, @markshannon, @serhiy-storchaka, @pablogsal, @myzhang1029, @iritkatriel
PRs	bpo-39934: Account for control blocks in 'except' in compiler. #22395 [3.9] bpo-39934: Account for control blocks in 'except' in compiler. (GH-22395) #23303
Files	exception_nest.py: Run exception_nest.py and run the generated python code

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-05-05.10:53:27.428>
created_at = <Date 2020-03-11.13:24:52.447>
labels = ['interpreter-core', '3.10', '3.8', '3.9', 'type-crash']
title = 'Fatal Python error "XXX block stack overflow" when exception stacks >10'
updated_at = <Date 2021-05-05.10:53:27.427>
user = 'https://github.com/myzhang1029'

bugs.python.org fields:

activity = <Date 2021-05-05.10:53:27.427>
actor = 'iritkatriel'
assignee = 'none'
closed = True
closed_date = <Date 2021-05-05.10:53:27.428>
closer = 'iritkatriel'
components = ['Interpreter Core']
creation = <Date 2020-03-11.13:24:52.447>
creator = 'myzhang1029'
dependencies = []
files = ['48969']
hgrepos = []
issue_num = 39934
keywords = ['patch']
message_count = 18.0
messages = ['363914', '377433', '377434', '377437', '377439', '377440', '377441', '377444', '377445', '377490', '378178', '378192', '381280', '385035', '385037', '385039', '385042', '393002']
nosy_count = 6.0
nosy_names = ['lukasz.langa', 'Mark.Shannon', 'serhiy.storchaka', 'pablogsal', 'myzhang1029', 'iritkatriel']
pr_nums = ['22395', '23303']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue39934'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

myzhang1029 · 2020-03-11T13:24:52Z

I apologize for describing this issue badly, but I'll try anyway.
The code to demonstrate the issue is attached, so it might be better to read that instead.
I noticed that when more than 10 exceptions are raised sequentially (i.e. one from another or one during the handling of another), the interpreter crashes saying "Fatal Python error: XXX block stack overflow".
This happens in python 3.7, 3.8 and development(git 39c3493) versions, but not in python2.7. Using ipython also fixes this issue.
I know this case is rare, but the maximum number of recursions is more than 2000, and the maximum number of statically nested blocks sepcified in frameobject.c is 20, so I'm pretty sure this isn't intended behavior.

iritkatriel · 2020-09-23T23:11:01Z

The error is coming from here:

cpython/Objects/frameobject.c

Line 958 in cb9879b

Py_FatalError("block stack overflow");

and CO_MAXBLOCKS is defined in Include/cpython/code.h
#define CO_MAXBLOCKS 20 /* Max static block nesting within a function */

This is not about recursion or about exception, it's about static nesting level.

There is an example here showing an input that gives the same error with >20 nested while blocks:

cpython/Lib/test/test_syntax.py

Line 545 in c8f29ad

iritkatriel · 2020-09-23T23:12:15Z

In summary, I think this is not-a-bug.

serhiy-storchaka · 2020-09-24T06:59:53Z

It is a bug. Compiler explicitly checks if the number of nested "try" blocks does not exceed the limit of CO_MAXBLOCKS, but it does not count implicit "try" blocks inserted when your assign an exception in the "except" clause.

try:
    ...
except Exception as e:
    ...

is actually translated to

try:
    ...
except Exception:
    try:
        e = ...
        ...
    finally:
        e = None
        del e

So we have double number of nested "try" blocks.

serhiy-storchaka · 2020-09-24T07:34:10Z

Humm, my supposition was not absolutely correct. The cause is that the compiler and the interpreter use the stack of size CO_MAXBLOCKS for different things. The interpreter pushes a thing for the "except" clause, while the compiler does not do it.

iritkatriel · 2020-09-24T08:02:56Z

Unlike the 22 nested while,

if 1:
  if 2:
    ...
    if 22:
        pass

doesn't error. Is that correct?

iritkatriel · 2020-09-24T08:14:10Z

Another oddity:

This gives the error:
for x in '1':
for x in '2':
for x in '3':
for x in '4':
for x in '5':
for x in '6':
for x in '8':
for x in '9':
for x in '10':
for x in '11':
for x in '12':
for x in '13':
for x in '14':
for x in '15':
for x in '16':
for x in '17':
for x in '18':
for x in '19':
for x in '20':
for x in '21':
for x in '22':
pass

but this doesn't:

for x in '1':
 for x in '2':
  for x in '3':
   for x in '4':
    for x in '5':
     for x in '6':
      for x in '8':
       for x in '9':
        for x in '10':
         for x in '11':
          for x in '12':
           for x in '13':
            for x in '14':
             for x in '15':
              for x in '16':
               for x in '17':
                for x in '18':
                 for x in '19':
                  for x in '20':
                   for x in '21':
                    pass
                   else:
                    for x in '22':
                     pass

markshannon · 2020-09-24T10:47:07Z

iritkatriel

What error do you see and on what version?

iritkatriel · 2020-09-24T11:13:13Z

On windows 10, master:

python.bat --version
Running Release|Win32 interpreter...
Python 3.10.0a0

python.bat x.py
Running Release|Win32 interpreter...
SyntaxError: too many statically nested blocks

That's when x.py has the nested for loops without else. The error goes away if I add the else.

markshannon · 2020-09-25T13:04:28Z

New changeset 02d126a by Mark Shannon in branch 'master':
bpo-39934: Account for control blocks in 'except' in compiler. (GH-22395)
02d126a

iritkatriel · 2020-10-07T20:51:18Z

After studying Mark's patch and the rest of the code, I understand the for loop oddity.

The else block of the for loop comes after the
compiler_pop_fblock(c, FOR_LOOP, start);

So you can do this all day and it won't complain:

for x in '1': pass
else:
 for x in '2': pass
 else:
  for x in '3': pass
  else:
   for x in '4': pass
   else:
    for x in '5': pass
    else:
     for x in '6': pass
     else:
      for x in '7': pass
      else:
       for x in '8': pass
       else:
        for x in '9': pass
        else:
         for x in '10': pass
         else:
          for x in '11': pass
          else:
           for x in '12': pass
           else:
            for x in '13': pass
            else:
             for x in '14': pass
             else:
              for x in '15': pass
              else:
               for x in '16': pass
               else:
                for x in '17': pass
                else:
                 for x in '18': pass
                 else:
                  for x in '19': pass
                  else:
                   for x in '20': pass
                   else:
                    for x in '21': pass
                    else:
                     for x in '22': pass
                     else:
                      for x in '23': pass
                      else:
                       for x in '24': pass
                       else:
                        for x in '25': pass
                        else:
                         for x in '26': pass
                         else:
                          for x in '27': pass
                          else:
                           for x in '28': pass
                           else:
                            for x in '29': pass

I guess the same goes for while loops, and the else of a try-except.

Since If blocks were deliberately left out of this game, I'm assuming that this "static nesting" is actually number of frames, rather than true static nesting. If that is the case then there is no issue here and we can close this ticket.

But if this is something to be fixed, then I am happy to make a patch along the lines of Mark's (which I partially already have).

pablogsal · 2020-10-07T23:24:40Z

PR 22395 should be backported to 3.8 and 3.9

markshannon · 2020-11-17T19:32:06Z

New changeset 48a9c0e by Irit Katriel in branch '3.9':
[3.9] bpo-39934: Account for control blocks in 'except' in compiler. (GH-22395) (GH-23303)
48a9c0e

markshannon · 2021-01-13T15:11:11Z

Does this need backporting to 3.8, or is 3.9 sufficient?

iritkatriel · 2021-01-13T15:14:36Z

There were additional merge conflicts when I tried to create a manual 3.8 backport, more significant than the 3.9 ones IIRC.

markshannon · 2021-01-13T15:41:57Z

Pablo, are you OK closing this without a 3.8 backport?

pablogsal · 2021-01-13T15:51:16Z

Pablo, are you OK closing this without a 3.8 backport?

I would be supportive, but we should check with Łukasz as he is the release manager of 3.8

iritkatriel · 2021-05-05T10:53:27Z

This was fixed in 3.9+ and 3.8 is in security fix now, so closing.

myzhang1029 mannequin added type-bug An unexpected behavior, bug, or error 3.7 (EOL) end of life interpreter-core (Objects, Python, Grammar, and Parser dirs) labels Mar 11, 2020

serhiy-storchaka added 3.8 only security fixes 3.9 only security fixes 3.10 only security fixes type-crash A hard crash of the interpreter, possibly with a core dump and removed 3.7 (EOL) end of life type-bug An unexpected behavior, bug, or error labels Sep 24, 2020

iritkatriel closed this as completed May 5, 2021

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fatal Python error "XXX block stack overflow" when exception stacks >10 #84115

Fatal Python error "XXX block stack overflow" when exception stacks >10 #84115

myzhang1029 mannequin commented Mar 11, 2020

myzhang1029 mannequin commented Mar 11, 2020

iritkatriel commented Sep 23, 2020

iritkatriel commented Sep 23, 2020

serhiy-storchaka commented Sep 24, 2020

serhiy-storchaka commented Sep 24, 2020

iritkatriel commented Sep 24, 2020

iritkatriel commented Sep 24, 2020

markshannon commented Sep 24, 2020

iritkatriel commented Sep 24, 2020

markshannon commented Sep 25, 2020

iritkatriel commented Oct 7, 2020

pablogsal commented Oct 7, 2020

markshannon commented Nov 17, 2020

markshannon commented Jan 13, 2021

iritkatriel commented Jan 13, 2021

markshannon commented Jan 13, 2021

pablogsal commented Jan 13, 2021

iritkatriel commented May 5, 2021

Fatal Python error "XXX block stack overflow" when exception stacks >10 #84115

Fatal Python error "XXX block stack overflow" when exception stacks >10 #84115

Comments

myzhang1029 mannequin commented Mar 11, 2020

myzhang1029 mannequin commented Mar 11, 2020

iritkatriel commented Sep 23, 2020

iritkatriel commented Sep 23, 2020

serhiy-storchaka commented Sep 24, 2020

serhiy-storchaka commented Sep 24, 2020

iritkatriel commented Sep 24, 2020

iritkatriel commented Sep 24, 2020

markshannon commented Sep 24, 2020

iritkatriel commented Sep 24, 2020

markshannon commented Sep 25, 2020

iritkatriel commented Oct 7, 2020

pablogsal commented Oct 7, 2020

markshannon commented Nov 17, 2020

markshannon commented Jan 13, 2021

iritkatriel commented Jan 13, 2021

markshannon commented Jan 13, 2021

pablogsal commented Jan 13, 2021

iritkatriel commented May 5, 2021