In bpo-30570, David Bolen noticed that "py -3.9 -m test test_pickle" consistently crashes on Windows (even though other methods of running that test do not crash, and the test succeeds when failed tests are retried).

Curiously, it seems that adding using support.infinite_recursion *introduced* the crash rather than preventing it.

I'm guessing this would have been fixed by GH-24501, but that fix was rolled back in GH-25179 for the sake of 3.9 ABI compatibility.

As of now, 3.9's pycore_ceval.c reads:

    static inline int _Py_RecursionLimitLowerWaterMark(int limit) {
        if (limit > 200) {
            return (limit - 50);
        }
        else {
            return (3 * (limit >> 2));
        }
    }

But support.infinite_recursion(max_depth=75) has a default 75, leaving a "low-water-mark" of 54, which the recursion apparently never recovers back to in the right way.


A couple of solutions could fix this:

(1) Remove the usage of support.infinite_recursion at the test.pickletester.AbstractPickleTests.test_bad_getattr call site.

(2) Use a different value max_depth. On my machine at least, it seems 75 was a "perfect storm" to cause this issue. Using infinite_recursion(60) or 61 or ... or 74 or 76 or 77 or 78 all pass, but infinite_recursion(75) in particular fails.

(3) Use fewer calls after the overflow by inlining something like assertRaises:

             with support.infinite_recursion():
-                self.assertRaises(RuntimeError, self.dumps, x, proto)
+                try:
+                    self.dumps(x, proto)
+                except RuntimeError:
+                    pass
+                else:
+                    self.fail("RuntimeError not raised")

(5) Re-visit an ABI-compliant version of GH-24501, such as GH-25160



The output I keep getting without any changes:

> py -3.9 -m test test_pickle -v
...
test_attribute_name_interning (test.test_pickle.CPicklerTests) ... ok
test_bad_getattr (test.test_pickle.CPicklerTests) ... Fatal Python error: _Py_CheckRecursiveCall: Cannot recover from stack overflow.Python runtime state: initialized

Current thread 0x000028b0 (most recent call first):
  File "C:\Users\sween\Source\Repos\cpython2\39\lib\test\pickletester.py", line 3300 in __getattr__
  File "C:\Users\sween\Source\Repos\cpython2\39\lib\test\pickletester.py", line 3300 in __getattr__
  File "C:\Users\sween\Source\Repos\cpython2\39\lib\test\pickletester.py", line 3300 in __getattr__
  ...
  File "C:\Users\sween\Source\Repos\cpython2\39\lib\test\pickletester.py", line 3300 in __getattr__
  File "C:\Users\sween\Source\Repos\cpython2\39\lib\test\pickletester.py", line 3300 in __getattr__
  File "C:\Users\sween\Source\Repos\cpython2\39\lib\test\pickletester.py", line 3300 in __getattr__
  File "C:\Users\sween\Source\Repos\cpython2\39\lib\copyreg.py", line 74 in _reduce_ex
  File "C:\Users\sween\Source\Repos\cpython2\39\lib\test\test_pickle.py", line 65 in dumps
  File "C:\Users\sween\Source\Repos\cpython2\39\lib\unittest\case.py", line 201 in handle
  File "C:\Users\sween\Source\Repos\cpython2\39\lib\unittest\case.py", line 739 in assertRaises
  File "C:\Users\sween\Source\Repos\cpython2\39\lib\test\pickletester.py", line 2381 in test_bad_getattr
  File "C:\Users\sween\Source\Repos\cpython2\39\lib\test\support\__init__.py", line 1770 in wrapper
  File "C:\Users\sween\Source\Repos\cpython2\39\lib\unittest\case.py", line 550 in _callTestMethod
  File "C:\Users\sween\Source\Repos\cpython2\39\lib\unittest\case.py", line 592 in run
  File "C:\Users\sween\Source\Repos\cpython2\39\lib\unittest\case.py", line 651 in __call__
  File "C:\Users\sween\Source\Repos\cpython2\39\lib\unittest\suite.py", line 122 in run
  File "C:\Users\sween\Source\Repos\cpython2\39\lib\unittest\suite.py", line 84 in __call__
  File "C:\Users\sween\Source\Repos\cpython2\39\lib\unittest\suite.py", line 122 in run
  File "C:\Users\sween\Source\Repos\cpython2\39\lib\unittest\suite.py", line 84 in __call__
  File "C:\Users\sween\Source\Repos\cpython2\39\lib\unittest\suite.py", line 122 in run

I'll note that it also fails on first run on the Windows 11 builder:

https://buildbot.python.org/all/#/builders/737/builds/65

New changeset 4296396db017d782d3aa16100b366748c9ea4a04 by Mark Shannon in branch '3.9':
[3.9] bpo-45806: Fix recovery from stack overflow for 3.9. Again. (GH-29640)
https://github.com/python/cpython/commit/4296396db017d782d3aa16100b366748c9ea4a04

Thanks! ✨ 🍰 ✨

Should this be backported to make the 3.8 Buildbots happy?