Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test_json.test_recursion.TestPyRecursion.test_endless_recursion stack overflow in AMD64 Windows8.1 Non-Debug 3.x #81154

Open
tirkarthi opened this issue May 20, 2019 · 6 comments
Open

test_json.test_recursion.TestPyRecursion.test_endless_recursion stack overflow in AMD64 Windows8.1 Non-Debug 3.x #81154

tirkarthi opened this issue May 20, 2019 · 6 comments
Labels
3.8 only security fixes tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error

Comments

@tirkarthi
Copy link
Member

BPO 36973
Nosy @vstinner, @serhiy-storchaka, @zooba, @tirkarthi

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-05-20.17:45:36.362>
labels = ['3.8', 'type-bug', 'tests']
title = 'test_json.test_recursion.TestPyRecursion.test_endless_recursion stack overflow in AMD64 Windows8.1 Non-Debug 3.x'
updated_at = <Date 2019-05-21.12:21:28.182>
user = 'https://github.com/tirkarthi'

bugs.python.org fields:

activity = <Date 2019-05-21.12:21:28.182>
actor = 'xtreak'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Tests']
creation = <Date 2019-05-20.17:45:36.362>
creator = 'xtreak'
dependencies = []
files = []
hgrepos = []
issue_num = 36973
keywords = []
message_count = 6.0
messages = ['342937', '343022', '343025', '343028', '343029', '343033']
nosy_count = 4.0
nosy_names = ['vstinner', 'serhiy.storchaka', 'steve.dower', 'xtreak']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue36973'
versions = ['Python 3.8']
@tirkarthi
Copy link
Member Author

test_json results in stack overflow after the commit to implement __repr__ for weakset. This is very much similar to the one consistently occurring on my inspect module PR in Windows tests only for the past two weeks. My change was in changing inspect module and more specifically in importing ast at the top of inspect module. The relevant commit for buildbot failure is in implementing __repr__ for weakset.

Buildbot error : https://buildbot.python.org/all/#/builders/12/builds/2497

Error in my PR that is consistent and same : https://ci.appveyor.com/project/python/cpython/builds/24631185#L1547

@tirkarthi tirkarthi added 3.8 only security fixes tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error labels May 20, 2019
@vstinner
Copy link
Member

Buildbot error : https://buildbot.python.org/all/#/builders/12/builds/2497

It's AMD64 Windows8.1 Non-Debug 3.x:

test_endless_recursion (test.test_json.test_recursion.TestPyRecursion) ... ok
Fatal Python error: Cannot recover from stack overflow.

Current thread 0x00000820 (most recent call first):
File "D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\json\decoder.py", line 83 in py_scanstring
File "D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\json\decoder.py", line 167 in JSONObject
File "D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\json\scanner.py", line 37 in _scan_once
(...)
File "D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\json\scanner.py", line 37 in _scan_once
File "D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\json\decoder.py", line 186 in JSONObject
File "D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\json\scanner.py", line 37 in _scan_once
File "D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\json\decoder.py", line 186 in JSONObject
File "D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\json\scanner.py", line 37 in _scan_once
File "D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\json\decoder.py", line 186 in JSONObject
...
test_highly_nested_objects_decoding (test.test_json.test_recursion.TestPyRecursion) ...
D:\buildarea\3.x.ware-win81-release.nondebug\build>exit /b 0

command timed out: 1200 seconds without output running [b'Tools\\buildbot\\test.bat', b'-x64', b'-j2', b'+d', b'-j4', b'--timeout', b'900'], attempting to kill
SIGKILL failed to kill process

https://ci.appveyor.com/project/python/cpython/builds/24631185#L1547

test_endless_recursion (test.test_json.test_recursion.TestPyRecursion) ... ok
test_highly_nested_objects_decoding (test.test_json.test_recursion.TestPyRecursion) ... Fatal Python error: Cannot recover from stack overflow.
Current thread 0x00000e2c (most recent call first):
File "C:\projects\cpython\lib\json\decoder.py", line 83 in py_scanstring
File "C:\projects\cpython\lib\json\decoder.py", line 167 in JSONObject
File "C:\projects\cpython\lib\json\scanner.py", line 37 in _scan_once
File "C:\projects\cpython\lib\json\decoder.py", line 186 in JSONObject
File "C:\projects\cpython\lib\json\scanner.py", line 37 in _scan_once
File "C:\projects\cpython\lib\json\decoder.py", line 186 in JSONObject
File "C:\projects\cpython\lib\json\scanner.py", line 37 in _scan_once
File "C:\projects\cpython\lib\json\decoder.py", line 186 in JSONObject
File "C:\projects\cpython\lib\json\scanner.py", line 37 in _scan_once
File "C:\projects\cpython\lib\json\decoder.py", line 186 in JSONObject
File "C:\projects\cpython\lib\json\scanner.py", line 37 in _scan_once
File "C:\projects\cpython\lib\json\decoder.py", line 186 in JSONObject
File "C:\projects\cpython\lib\json\scanner.py", line 37 in _scan_once
File "C:\projects\cpython\lib\json\decoder.py", line 186 in JSONObject
File "C:\projects\cpython\lib\json\scanner.py", line 37 in _scan_once
File "C:\projects\cpython\lib\json\decoder.py", line 186 in JSONObject
(...)
File "C:\projects\cpython\lib\json\scanner.py", line 37 in _scan_once
File "C:\projects\cpython\lib\json\decoder.py", line 186 in JSONObject
File "C:\projects\cpython\lib\json\scanner.py", line 37 in _scan_once
File "C:\projects\cpython\lib\json\decoder.py", line 186 in JSONObject
File "C:\projects\cpython\lib\json\scanner.py", line 37 in _scan_once
File "C:\projects\cpython\lib\json\decoder.py", line 186 in JSONObject
File "C:\projects\cpython\lib\json\scanner.py", line 37 in _scan_once
File "C:\projects\cpython\lib\json\decoder.py", line 186 in JSONObject
...
C:\projects\cpython>set lastexitcode=-1073740791
C:\projects\cpython>set 1>C:\Users\appveyor\AppData\Local\Temp\1\tmp4CD0.tmp
C:\projects\cpython>echo C:\projects\cpython 1>C:\Users\appveyor\AppData\Local\Temp\1\tmp4CD1.tmp
C:\projects\cpython>exit /b -1073740791

@vstinner
Copy link
Member

The mashal also caused similar issues with stack overflow in the past:

commit f6c69e6
Author: Steve Dower <steve.dower@microsoft.com>
Date: Sat Nov 1 15:15:16 2014 -0700

bpo-22734 marshal needs a lower stack depth for debug builds on Windows

commit 24e33ac
Author: Victor Stinner <victor.stinner@gmail.com>
Date: Sun Jul 7 02:49:07 2013 +0200

Issue bpo-17206: On Windows, increase the stack size from 2 MB to 4.2 MB to fix
a stack overflow in the marshal module (fix a crash in test_marshal).
Patch written by Jeremy Kloth.

History of PyOS_CheckStack:

commit 5299935
Author: Kristján Valur Jónsson <kristjan@ccpgames.com>
Date: Mon Feb 18 17:40:47 2008 +0000

Perform correct handling of stack overflow for windows: Catch the correct exception code and reset the overflow condition when handled.

commit dc61901
Author: Amaury Forgeot d'Arc <amauryfa@gmail.com>
Date: Sat Nov 22 20:01:18 2008 +0000

bpo-3996: On Windows, PyOS_CheckStack is supposed to protect the interpreter from
stack overflow. But doing this, it always crashes when the stack is nearly full.

Reviewed by Martin von Loewis. Will backport to 2.6.

commit 92e4dd8
Author: Tim Peters <tim.peters@gmail.com>
Date: Sat Oct 5 01:47:34 2002 +0000

s/_alloca/alloca/g; Windows doesn't need the former, at least not unless
__STDC__ is defined (or something like that ...).

commit 399739f
Author: Fred Drake <fdrake@acm.org>
Date: Thu Aug 31 05:52:44 2000 +0000

PyOS_CheckStack():  Better ANSI'fy this while we're at it.

commit 2f15b25
Author: Fredrik Lundh <fredrik@pythonware.com>
Date: Sun Aug 27 19:15:31 2000 +0000

implements PyOS_CheckStack for Windows and MSVC.  this fixes a
couple of potential stack overflows, including bug bpo-110615.

closes patch bpo-101238

@vstinner
Copy link
Member

bpo-25329 was a similar json crash with stack overflow in 2015. The issue gives pointers to:

  • bpo-25222: 3.5.0 regression - Fatal Python error: Cannot recover from stack overflow
  • bpo-25342: test_json segfault on OpenBSD

@vstinner
Copy link
Member

See also:

  • bpo-25240: Stack overflow in reprlib causes a core dump
  • bpo-28913: "Fatal Python error: Cannot recover from stack overflow." from RecursionError in Python 3.5
  • bpo-22583: C stack overflow in the Python 2.7 compiler

@tirkarthi
Copy link
Member Author

I have seen it occurring consistently in my PR 10307 in Travis and AppVeyor. It seems to be reproducible by importing ast at the top in inspect module which seems to have an effect as libregrtest is used to run the test_json. Using unittest module to run test_json doesn't seem to crash the last time I have checked.

@ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
3.8 only security fixes tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error
Projects
JSON issues
Status: No status
Milestone
No milestone
Development

No branches or pull requests
2 participants
@vstinner @tirkarthi