Title: RecursionError when parsing unwieldy expression (regression from 2.7 -> 3.x)
Type: compile error Stage:
Components: Interpreter Core Versions: Python 3.9
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: jack__d, mjo, terry.reedy
Priority: normal Keywords:

Created on 2020-12-15 17:55 by mjo, last changed 2021-08-13 18:53 by jack__d.

File name Uploaded Description Edit
DPT_defn mjo, 2020-12-15 17:55 jack__d, 2021-08-13 18:51
Messages (4)
msg383084 - (view) Author: Michael Orlitzky (mjo) Date: 2020-12-15 17:55
The attached file contains a huge, ugly expression that can be parsed in python-2.7 but not python-3.x (I've tested 3.7, 3.8, and 3.9). With the 3.x versions, I get

  $ python3 DPT_defn 
  RecursionError: maximum recursion depth exceeded during compilation

Python-2.7, on the other hand, is able to parse & compile much larger expressions -- this example was whittled down until it was roughly minimal.
msg383351 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2020-12-19 01:48
One can use sys.setrecursionlimit to increase allowed recursion depth.

But...  When I ran code from 3.10.0a3 IDLE editor, something hung.  Without touching recursion limit, when I split statement apart into

ex = compile("""<300 lines>""", '', 'eval')

I got 106994.5 in under a second.  With exec, ditto except that None is printed.  Same last result with 3.9.1.
msg399551 - (view) Author: Jack DeVries (jack__d) * Date: 2021-08-13 18:51
I spent some time experimenting with making the expression bigger and the recursion limit lower in python2. It seems like in python2, the depth that the compiler will recurse is unrelated to sys.recursionlimit.

Then, I lowered resource limits on stack and heap by ~1000x and increased the size of the expression by 10,000x, which caused a segmentation fault. Not only that, but the interpreter won't even respond to KeyboardInterrupt while it is doing the compiling. I believe that the recursion depth of the 2.x compiler is simply unbound, which seems like a bug to me.

As I tinkered with these things, I build out a reproduction script, which I've attached.

I think that documenting this change is more pragmatic than thinking about "fixing" it (not sure it can truly be fixed, as that would mean re-incorporating a bug). I don't know where such a note would belong, though. This document is the closed I can find, but it's too high-level for a detail like this to belong::
msg399552 - (view) Author: Jack DeVries (jack__d) * Date: 2021-08-13 18:53
edit; typo:

**This document is the **closest** I can find
Date User Action Args
2021-08-13 18:53:06jack__dsetmessages: + msg399552
2021-08-13 18:51:26jack__dsetfiles: +
nosy: + jack__d
messages: + msg399551

2020-12-19 01:48:32terry.reedysetnosy: + terry.reedy
messages: + msg383351
2020-12-15 17:56:14mjosettype: compile error
2020-12-15 17:55:57mjocreate