msg363914 - (view) |
Author: Maiyun Zhang (myzhang1029) * |
Date: 2020-03-11 13:24 |
I apologize for describing this issue badly, but I'll try anyway.
The code to demonstrate the issue is attached, so it might be better to read that instead.
I noticed that when more than 10 exceptions are raised sequentially (i.e. one from another or one during the handling of another), the interpreter crashes saying "Fatal Python error: XXX block stack overflow".
This happens in python 3.7, 3.8 and development(git 39c3493) versions, but not in python2.7. Using ipython also fixes this issue.
I know this case is rare, but the maximum number of recursions is more than 2000, and the maximum number of statically nested blocks sepcified in frameobject.c is 20, so I'm pretty sure this isn't intended behavior.
|
msg377433 - (view) |
Author: Irit Katriel (iritkatriel) *  |
Date: 2020-09-23 23:11 |
The error is coming from here: https://github.com/python/cpython/blob/cb9879b948a19c9434316f8ab6aba9c4601a8173/Objects/frameobject.c#L958
and CO_MAXBLOCKS is defined in Include/cpython/code.h
#define CO_MAXBLOCKS 20 /* Max static block nesting within a function */
This is not about recursion or about exception, it's about static nesting level.
There is an example here showing an input that gives the same error with >20 nested while blocks: https://github.com/python/cpython/blob/c8f29ad986f8274fc5fbf889bdd2a211878856b9/Lib/test/test_syntax.py#L545
|
msg377434 - (view) |
Author: Irit Katriel (iritkatriel) *  |
Date: 2020-09-23 23:12 |
In summary, I think this is not-a-bug.
|
msg377437 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) *  |
Date: 2020-09-24 06:59 |
It is a bug. Compiler explicitly checks if the number of nested "try" blocks does not exceed the limit of CO_MAXBLOCKS, but it does not count implicit "try" blocks inserted when your assign an exception in the "except" clause.
try:
...
except Exception as e:
...
is actually translated to
try:
...
except Exception:
try:
e = ...
...
finally:
e = None
del e
So we have double number of nested "try" blocks.
|
msg377439 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) *  |
Date: 2020-09-24 07:34 |
Humm, my supposition was not absolutely correct. The cause is that the compiler and the interpreter use the stack of size CO_MAXBLOCKS for different things. The interpreter pushes a thing for the "except" clause, while the compiler does not do it.
|
msg377440 - (view) |
Author: Irit Katriel (iritkatriel) *  |
Date: 2020-09-24 08:02 |
Unlike the 22 nested while,
if 1:
if 2:
...
if 22:
pass
doesn't error. Is that correct?
|
msg377441 - (view) |
Author: Irit Katriel (iritkatriel) *  |
Date: 2020-09-24 08:14 |
Another oddity:
This gives the error:
for x in '1':
for x in '2':
for x in '3':
for x in '4':
for x in '5':
for x in '6':
for x in '8':
for x in '9':
for x in '10':
for x in '11':
for x in '12':
for x in '13':
for x in '14':
for x in '15':
for x in '16':
for x in '17':
for x in '18':
for x in '19':
for x in '20':
for x in '21':
for x in '22':
pass
but this doesn't:
for x in '1':
for x in '2':
for x in '3':
for x in '4':
for x in '5':
for x in '6':
for x in '8':
for x in '9':
for x in '10':
for x in '11':
for x in '12':
for x in '13':
for x in '14':
for x in '15':
for x in '16':
for x in '17':
for x in '18':
for x in '19':
for x in '20':
for x in '21':
pass
else:
for x in '22':
pass
|
msg377444 - (view) |
Author: Mark Shannon (Mark.Shannon) *  |
Date: 2020-09-24 10:47 |
iritkatriel
What error do you see and on what version?
|
msg377445 - (view) |
Author: Irit Katriel (iritkatriel) *  |
Date: 2020-09-24 11:13 |
On windows 10, master:
> python.bat --version
Running Release|Win32 interpreter...
Python 3.10.0a0
>python.bat x.py
Running Release|Win32 interpreter...
SyntaxError: too many statically nested blocks
That's when x.py has the nested for loops without else. The error goes away if I add the else.
|
msg377490 - (view) |
Author: Mark Shannon (Mark.Shannon) *  |
Date: 2020-09-25 13:04 |
New changeset 02d126aa09d96d03dcf9c5b51c858ce5ef386601 by Mark Shannon in branch 'master':
bpo-39934: Account for control blocks in 'except' in compiler. (GH-22395)
https://github.com/python/cpython/commit/02d126aa09d96d03dcf9c5b51c858ce5ef386601
|
msg378178 - (view) |
Author: Irit Katriel (iritkatriel) *  |
Date: 2020-10-07 20:51 |
After studying Mark's patch and the rest of the code, I understand the for loop oddity.
The else block of the for loop comes after the
compiler_pop_fblock(c, FOR_LOOP, start);
So you can do this all day and it won't complain:
for x in '1': pass
else:
for x in '2': pass
else:
for x in '3': pass
else:
for x in '4': pass
else:
for x in '5': pass
else:
for x in '6': pass
else:
for x in '7': pass
else:
for x in '8': pass
else:
for x in '9': pass
else:
for x in '10': pass
else:
for x in '11': pass
else:
for x in '12': pass
else:
for x in '13': pass
else:
for x in '14': pass
else:
for x in '15': pass
else:
for x in '16': pass
else:
for x in '17': pass
else:
for x in '18': pass
else:
for x in '19': pass
else:
for x in '20': pass
else:
for x in '21': pass
else:
for x in '22': pass
else:
for x in '23': pass
else:
for x in '24': pass
else:
for x in '25': pass
else:
for x in '26': pass
else:
for x in '27': pass
else:
for x in '28': pass
else:
for x in '29': pass
I guess the same goes for while loops, and the else of a try-except.
Since If blocks were deliberately left out of this game, I'm assuming that this "static nesting" is actually number of frames, rather than true static nesting. If that is the case then there is no issue here and we can close this ticket.
But if this is something to be fixed, then I am happy to make a patch along the lines of Mark's (which I partially already have).
|
msg378192 - (view) |
Author: Pablo Galindo Salgado (pablogsal) *  |
Date: 2020-10-07 23:24 |
PR 22395 should be backported to 3.8 and 3.9
|
msg381280 - (view) |
Author: Mark Shannon (Mark.Shannon) *  |
Date: 2020-11-17 19:32 |
New changeset 48a9c0eb2a3304ea64d1b32fdf9db853d5d8c429 by Irit Katriel in branch '3.9':
[3.9] bpo-39934: Account for control blocks in 'except' in compiler. (GH-22395) (GH-23303)
https://github.com/python/cpython/commit/48a9c0eb2a3304ea64d1b32fdf9db853d5d8c429
|
msg385035 - (view) |
Author: Mark Shannon (Mark.Shannon) *  |
Date: 2021-01-13 15:11 |
Does this need backporting to 3.8, or is 3.9 sufficient?
|
msg385037 - (view) |
Author: Irit Katriel (iritkatriel) *  |
Date: 2021-01-13 15:14 |
There were additional merge conflicts when I tried to create a manual 3.8 backport, more significant than the 3.9 ones IIRC.
|
msg385039 - (view) |
Author: Mark Shannon (Mark.Shannon) *  |
Date: 2021-01-13 15:41 |
Pablo, are you OK closing this without a 3.8 backport?
|
msg385042 - (view) |
Author: Pablo Galindo Salgado (pablogsal) *  |
Date: 2021-01-13 15:51 |
> Pablo, are you OK closing this without a 3.8 backport?
I would be supportive, but we should check with Łukasz as he is the release manager of 3.8
|
msg393002 - (view) |
Author: Irit Katriel (iritkatriel) *  |
Date: 2021-05-05 10:53 |
This was fixed in 3.9+ and 3.8 is in security fix now, so closing.
|
|
Date |
User |
Action |
Args |
2022-04-11 14:59:28 | admin | set | github: 84115 |
2021-05-05 10:53:27 | iritkatriel | set | status: open -> closed resolution: fixed messages:
+ msg393002
stage: patch review -> resolved |
2021-01-13 16:21:31 | Mark.Shannon | set | nosy:
+ lukasz.langa
|
2021-01-13 15:51:16 | pablogsal | set | messages:
+ msg385042 |
2021-01-13 15:41:57 | Mark.Shannon | set | messages:
+ msg385039 |
2021-01-13 15:14:36 | iritkatriel | set | messages:
+ msg385037 |
2021-01-13 15:11:11 | Mark.Shannon | set | messages:
+ msg385035 |
2020-11-17 19:32:05 | Mark.Shannon | set | messages:
+ msg381280 |
2020-11-15 18:12:42 | iritkatriel | set | pull_requests:
+ pull_request22194 |
2020-10-07 23:24:40 | pablogsal | set | nosy:
+ pablogsal messages:
+ msg378192
|
2020-10-07 20:51:18 | iritkatriel | set | messages:
+ msg378178 |
2020-09-25 13:04:29 | Mark.Shannon | set | messages:
+ msg377490 |
2020-09-24 11:22:21 | Mark.Shannon | set | keywords:
+ patch stage: patch review pull_requests:
+ pull_request21435 |
2020-09-24 11:13:12 | iritkatriel | set | messages:
+ msg377445 |
2020-09-24 10:47:07 | Mark.Shannon | set | messages:
+ msg377444 |
2020-09-24 08:14:09 | iritkatriel | set | messages:
+ msg377441 |
2020-09-24 08:02:56 | iritkatriel | set | messages:
+ msg377440 |
2020-09-24 07:34:09 | serhiy.storchaka | set | messages:
+ msg377439 |
2020-09-24 06:59:53 | serhiy.storchaka | set | versions:
+ Python 3.8, Python 3.9, Python 3.10, - Python 3.7 nosy:
+ Mark.Shannon, serhiy.storchaka
messages:
+ msg377437
type: behavior -> crash |
2020-09-23 23:12:15 | iritkatriel | set | messages:
+ msg377434 |
2020-09-23 23:11:01 | iritkatriel | set | nosy:
+ iritkatriel messages:
+ msg377433
|
2020-03-11 13:24:52 | myzhang1029 | create | |