classification
Title: Stack overflow exception caused by test_marshal on Windows x64
Type: crash Stage:
Components: Tests, Windows Versions: Python 3.4, Python 2.7, Python 2.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: trent Nosy List: amaury.forgeotdarc, brian.curtin, sable, sbt, terry.reedy, trent
Priority: normal Keywords: 64bit

Created on 2008-03-14 12:57 by trent, last changed 2013-07-02 17:45 by sbt. This issue is now closed.

Messages (11)
msg63523 - (view) Author: Trent Nelson (trent) * (Python committer) Date: 2008-03-14 12:57
S:\src\svn\svn.python.org\projects\python\trunk.x64\PCbuild>amd64\python_d
..\lib\test\test_marshal.py
test_bool (__main__.IntTestCase) ... ok
test_int64 (__main__.IntTestCase) ... ok
test_ints (__main__.IntTestCase) ... ok
test_floats (__main__.FloatTestCase) ... ok
test_buffer (__main__.StringTestCase) ... ok
test_string (__main__.StringTestCase) ... ok
test_unicode (__main__.StringTestCase) ... ok
test_code (__main__.CodeTestCase) ... ok
test_dict (__main__.ContainerTestCase) ... ok
test_list (__main__.ContainerTestCase) ... ok
test_sets (__main__.ContainerTestCase) ... ok
test_tuple (__main__.ContainerTestCase) ... ok
test_exceptions (__main__.ExceptionTestCase) ... ok
test_bug_5888452 (__main__.BugsTestCase) ... ok
test_exact_type_match (__main__.BugsTestCase) ... ok
test_fuzz (__main__.BugsTestCase) ... ok
test_loads_recursion (__main__.BugsTestCase) ...
                                                 ^^^

python_d.exe crashes at this point with a stack overflow.  Just
capturing the issue for now.  Will investigate shortly and hopefully
provide a patch.
msg63536 - (view) Author: Trent Nelson (trent) * (Python committer) Date: 2008-03-14 20:36
Traced the problem down to the following minimal code snippet:

import marshal
s = 'c' + ('X' * 4*4) + '{' * 2**20
marshal.loads(s)

When Python/marshal.c:18 MAX_MARSHAL_STACK_DEPTH is 2000 (which is what
it is currently), marshal.loads() eventually overflows the stack in
r_object().  There is a check in r_object() to avoid this though:

  if (p->depth > MAX_MARSHAL_STACK_DEPTH) {
      p->depth--;
      PyErr_SetString(PyExc_ValueError, "recursion limit exceeded");
      return NULL;
  }

On Windows x64, a value of 1964 raises the recursion limit exception
above (which is what test_marshal is expecting).  With a value of 1965,
a C stack overflow exception is raised.

So, MAX_MARSHAL_STACK_DEPTH needs to be <= 1964 in order to prevent this
particular code from overflowing the stack on Win64 before we can raise
a Python recursion limit exception.

Was there any science behind choosing 2000 as the current value?  Should
a new value (e.g. 1500) be provided for only Win64, leaving everyone
else at 2000?  Interesting that Linux/BSD etc on AMD64 don't run into this.
msg63540 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2008-03-14 22:49
The situation was the same on win32 with the VC2005 debug build, some
months ago.
I think this is because of the extra stack checks added with "recent"
versions of Visual Studio, together with the fact that local variables
are not shared when optimizations are disabled.

Can you try to raise the stack size on x64 builds? If 2Mb is enough for
32bit, 4Mb should be good in your case.
msg63902 - (view) Author: Trent Nelson (trent) * (Python committer) Date: 2008-03-18 07:07
Fixed in r61509; bumped the stack size from 200000 to 210000, which
stops the stack overflow from being triggered.
msg147900 - (view) Author: Sébastien Sablé (sable) Date: 2011-11-18 17:02
I had the exact same problem when compiling the Python 2.7 branch with Visual Studio 2010.

In debug mode and 64 bits, python will crash on test_loads_recursion in test_marshal and on the code provided in msg63536.

Could you please reopen this issue?
msg147907 - (view) Author: Sébastien Sablé (sable) Date: 2011-11-18 17:18
I don't have the issue anymore when bumping the stack size to 2300000.
msg148367 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-11-25 23:18
The 'official' PSF build for 2.7 (and 3.x) is done with VS2008, so this may not be the last problem you have. I am pretty sure this will never change for 2.7, but that is not to say we cannot make a simple change to accommodate VS2010. I am sure we will eventually update VS for 3.x. Can you test 3.2 or even better 3.3?
msg148470 - (view) Author: Sébastien Sablé (sable) Date: 2011-11-28 08:42
Thanks Terry, I am aware of that.
We are working on making Python work with VS2010 in issue 13210.

I will check with the py3k branch soon and report here if the same problem applies.
msg170651 - (view) Author: Trent Nelson (trent) * (Python committer) Date: 2012-09-18 15:37
Closing issue; this has been fixed.
msg192211 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013-07-02 17:13
Reopening because I think this is again a problem for Win64 and 3.x.  The Win64 buildbots always seem to crash on test_marshal (and I do too).

It appears to be BugsTestCase.test_loads_2x_code() which crashes, which is virtually the same as test_loads_recursion().
msg192213 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013-07-02 17:45
Closing because this is caused by #17206 and is already discussed there.
History
Date User Action Args
2013-07-02 17:45:05sbtsetstatus: open -> closed

messages: + msg192213
2013-07-02 17:13:19sbtsetstatus: closed -> open
versions: + Python 3.4
nosy: + sbt

messages: + msg192211
2012-09-18 15:37:49trentsetstatus: open -> closed

messages: + msg170651
2011-11-28 08:42:53sablesetmessages: + msg148470
2011-11-25 23:18:46terry.reedysetnosy: + terry.reedy
messages: + msg148367
2011-11-18 17:20:26brian.curtinsetstatus: closed -> open
nosy: + brian.curtin
components: + Windows
2011-11-18 17:18:37sablesetmessages: + msg147907
2011-11-18 17:02:23sablesetnosy: + sable

messages: + msg147900
versions: + Python 2.7
2008-03-18 07:07:46trentsetstatus: open -> closed
assignee: trent
resolution: fixed
messages: + msg63902
keywords: + 64bit
2008-03-14 22:49:08amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg63540
2008-03-14 20:36:09trentsetmessages: + msg63536
2008-03-14 12:57:55trentcreate