classification
Title: "Fatal Python error: Cannot recover from stack overflow." with SymPy test suite
Type: crash Stage: resolved
Components: Versions: Python 3.2, Python 3.3
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: vstinner Nosy List: Aaron.Meurer, brett.cannon, r.david.murray, serhiy.storchaka, vstinner
Priority: normal Keywords: patch

Created on 2012-04-09 18:48 by Aaron.Meurer, last changed 2014-10-10 23:59 by r.david.murray. This issue is now closed.

Files
File name Uploaded Description Edit
import_pydict_getitem.patch vstinner, 2012-04-09 23:27 review
Messages (11)
msg157876 - (view) Author: Aaron Meurer (Aaron.Meurer) Date: 2012-04-09 18:48
Recently, after a small seemingly unrelated refactoring, the SymPy test suite in Python 3 started dying with "Fatal Python error: Cannot recover from stack overflow."  

Here's how to reproduce the error

git clone git://github.com/sympy/sympy.git # Clone the development version of SymPy
cd sympy
git checkout 0856119bd7399a416c21e1692855a1077164f21c # This is the first commit to exhibit the problem. Do this in case we make an unrelated change that removes the problem.
./bin/use2to3 # Convert the codebase to Python 3
python3 py3k-sympy/setup.py test # Run the tests

The issue is described in more detail at http://groups.google.com/group/sympy/browse_thread/thread/f664fe88e6b4f29d/3a44691c945695db#3a44691c945695db.  Some key points:

- The test that triggers the error is an XFAIL test (test that is expected to fail) that raises RuntimeError: maximum recursion depth exceeded.

- The change that caused the error, 0856119bd7399a416c21e1692855a1077164f21c (see https://github.com/sympy/sympy/commit/0856119bd7399a416c21e1692855a1077164f21c), is seemingly unrelated.  The only thing that I can think of is that it has added another call to the stack.

- This kills Python with Abort Trap: 6 in Mac OS X, which generates a problem report to be sent to Apple.  I have included a copy of it here: https://gist.github.com/2317869.

- Others have reproduced this error as well, as can be seen by our test reporter tool.  See the mailing list thread for more info.

- I tested this in 3.2.3r2, and the error still occurred.  I tried testing in the 3.3 alpha, but I could not get it to compile.
msg157915 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2012-04-09 22:36
It looks like an issue in SymPy, a stack overflow. Why did you report this issue on CPython bug tracker?

Is there an infinite loop in a recursive function call?

Can you try to get the full Python traceback using faulthandler? Use "-X faulthandler" command line option, or use "import faulthandler; faulthandler.enable()" at the beginning of your program. You have to install it manually for Python < 3.3.
msg157917 - (view) Author: Aaron Meurer (Aaron.Meurer) Date: 2012-04-09 22:43
We do have a stack overflow, but this should be raising a RuntimeError, not killing Python.  The way it is now, Python dies completely with abort trap 6 (hence the Mac OS X problem report).  Sorry if I didn't make this clear in the OP.
msg157918 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2012-04-09 22:48
SymPy uses a C extension, numpy. The problem may be related to numpy. CPython tries to catch segmentation fault, but when a C extension is used, CPython may fail to protect your program against stack overflow.
msg157919 - (view) Author: Aaron Meurer (Aaron.Meurer) Date: 2012-04-09 22:50
No it does not.  SymPy is a pure Python library.
msg157923 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2012-04-09 23:27
The (first) Python stack overflow occurs at:
-----------
(gdb) py-bt
Traceback (most recent call first):
  File "/home/haypo/issue14537/sympy/py3k-sympy/sympy/core/expr.py", line 2531, in expand
    from sympy.simplify.simplify import fraction
  File "/home/haypo/issue14537/sympy/py3k-sympy/sympy/core/cache.py", line 91, in wrapper
    func_cache_it_cache[k] = r = func(*args, **kw_args)
  File "/home/haypo/issue14537/sympy/py3k-sympy/sympy/functions/elementary/hyperbolic.py", line 514, in as_real_imag
    return (self.expand(deep, **hints), S.Zero)
  File "/home/haypo/issue14537/sympy/py3k-sympy/sympy/functions/elementary/hyperbolic.py", line 525, in _eval_expand_complex
    re_part, im_part = self.as_real_imag(deep=deep, **hints)
  File "/home/haypo/issue14537/sympy/py3k-sympy/sympy/core/expr.py", line 2551, in expand
    expr = func(deep=deep, **hints)
  File "/home/haypo/issue14537/sympy/py3k-sympy/sympy/core/cache.py", line 91, in wrapper
    func_cache_it_cache[k] = r = func(*args, **kw_args)
  File "/home/haypo/issue14537/sympy/py3k-sympy/sympy/functions/elementary/hyperbolic.py", line 514, in as_real_imag
    return (self.expand(deep, **hints), S.Zero)
  File "/home/haypo/issue14537/sympy/py3k-sympy/sympy/functions/elementary/hyperbolic.py", line 525, in _eval_expand_complex

(gdb) where
#0  _Py_CheckRecursiveCall (where=0x624f4b " in comparison") at Python/ceval.c:736
#1  0x0000000000419e6a in PyObject_RichCompare (v='sympy', w='sympy', op=2) at Objects/object.c:604
#2  0x0000000000419f2d in PyObject_RichCompareBool (v='sympy', w='sympy', op=2) at Objects/object.c:628
#3  0x00000000005e854e in lookdict (mp=0x7ffff1b3b1a8, key='sympy', hash=-191038920480525830) at Objects/dictobject.c:341
#4  0x00000000005e99ca in PyDict_GetItem (op=
    {'sympy.logic.inference': <module at remote 0x7fffeef80ba0>, ..., key='sympy') at Objects/dictobject.c:752
#5  0x00000000004f1510 in import_submodule (mod=None, subname='sympy', fullname='sympy') at Python/import.c:3313
#6  0x00000000004f064b in load_next (mod=None, altmod=None, inputname='sympy.simplify.simplify', p_outputname=0x7fffffe67e28, p_prefix=0x7fffffe67e20)
    at Python/import.c:3150
#7  0x00000000004ef610 in import_module_level (name='sympy.simplify.simplify', globals=
    {...}, locals=None, fromlist=('fraction',), level=0)
    at Python/import.c:2843
-----------

Ok, now I see the problem: import_submodule() uses PyDict_GetItem() whereas PyDict_GetItem() *clears* the exception (!). It should use PyDict_GetItemWithError() instead. Try attached patch. The problem is a generic problem: PyDict_GetItem() should be replaced by PyDict_GetItemWithError() everywhere... But it would be nice to start with import.c :-)
msg211163 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-02-13 19:13
I didn't tested this patch with SymPy, but in general it LGTM.
msg229047 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-10-10 23:36
This code fragment doesn't seem to exist any more.  Since there's no test case, I'm going to close this.  I presume it is no longer a problem for SymPy, since there's been no followup.
msg229049 - (view) Author: Aaron Meurer (Aaron.Meurer) Date: 2014-10-10 23:41
The OP describes how to get the original code. Anyway, the issue was definitely fixed.
msg229050 - (view) Author: Aaron Meurer (Aaron.Meurer) Date: 2014-10-10 23:42
Or do you mean the code in CPython doesn't exist any more?
msg229052 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-10-10 23:59
Yes, it appears that the code does not exist in import.c any more.

When I said "test case", I meant "unit test", sorry.
History
Date User Action Args
2014-10-10 23:59:52r.david.murraysetmessages: + msg229052
2014-10-10 23:42:18Aaron.Meurersetmessages: + msg229050
2014-10-10 23:41:45Aaron.Meurersetmessages: + msg229049
2014-10-10 23:36:04r.david.murraysetstatus: open -> closed

nosy: + brett.cannon, r.david.murray
messages: + msg229047

resolution: out of date
stage: commit review -> resolved
2014-02-13 19:13:13serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg211163

assignee: vstinner
stage: commit review
2012-04-09 23:27:47vstinnersetfiles: + import_pydict_getitem.patch
keywords: + patch
messages: + msg157923
2012-04-09 22:50:37Aaron.Meurersetmessages: + msg157919
2012-04-09 22:48:52vstinnersetmessages: + msg157918
2012-04-09 22:43:26Aaron.Meurersetmessages: + msg157917
2012-04-09 22:36:06vstinnersetmessages: + msg157915
2012-04-09 19:02:21pitrousetnosy: + vstinner

versions: + Python 3.3
2012-04-09 18:48:32Aaron.Meurercreate