classification
Title: Exceed Recursion Limit in Thread
Type: crash Stage: resolved
Components: Macintosh Versions: Python 3.1, Python 3.2, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: ronaldoussoren Nosy List: jaredlang, lukasz.langa, ned.deily, pitrou, python-dev, r.david.murray, ronaldoussoren, santagada
Priority: normal Keywords: needs review, patch

Created on 2010-08-24 13:23 by jaredlang, last changed 2012-06-30 01:28 by ned.deily. This issue is now closed.

Files
File name Uploaded Description Edit
issue9670.patch ronaldoussoren, 2010-10-20 14:45 review
issue9670-v2.txt ronaldoussoren, 2011-03-14 18:45 review
test_issue9670.py ronaldoussoren, 2011-03-14 18:46
issue9670-v3.patch ned.deily, 2011-04-04 05:52 review
issue9670-v4.patch ronaldoussoren, 2011-05-06 13:26 review
Messages (26)
msg114787 - (view) Author: Jared Lang (jaredlang) Date: 2010-08-24 13:23
Recursion within a thread on OSX can result in a crash by exceeding the systems recursion limit.  Recursion behaves as expected if not in thread, meaning it throws a RunTimeError with the message "maximum recursion depth exceeded."

The crash is able to be avoided if the recursion limit is set to a lower number, ie. 800, via sys.setrecursionlimit.

Example program which crashes on OSX:


>>> def f():
...     return f()
... 
>>> import threading
>>> thread = threading.Thread(target=f)
>>> thread.start()
Bus error


Alternatively, the following works as expected:


>>> import sys
>>> def f():
...     return f()
... 
>>> import threading
>>> thread = threading.Thread(target=f)
>>> sys.setrecursionlimit(800)
>>> thread.start()

>>> Exception in thread Thread-1:
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "<stdin>", line 2, in f
  File "<stdin>", line 2, in f
  File "<stdin>", line 2, in f
  File "<stdin>", line 2, in f
  File "<stdin>", line 2, in f
  File "<stdin>", line 2, in f
...

RuntimeError: maximum recursion depth exceeded
msg114788 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-08-24 13:57
This is not the first recursion limit related problem we've seen with FreeBSD derived systems.  See for example Issue1201456. It would be nice to have an actual fix for this class of problem, but I have no clue what that would look like.  There may also be a specific fix for this issue.  At least, we can hope :)

Python 2.6 is in security-fix-only mode, so I've updated the versions.  I can confirm that this reproduces on 2.7 and py3k trunk on OS/X 10.4 using a non-framework build.  I haven't tested 3.1 but I am sure it must behave the same.
msg114789 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2010-08-24 14:28
This is the same issue as #6763.

The root cause of this issue is that the default stack size for threads other than the main thread is small. 

One way to fix this issue is to increase the default stacksize for new threads.
msg114793 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-08-24 14:43
As I mentioned on the other ticket, I think bumping the default stack size is probably the most useful solution.  This is is a clearer description of the stack problem, since it doesn't get involved in MIMETypes internals, so I'm not closing it as a duplicate.
msg119210 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2010-10-20 14:45
The attached patch explicitly sets the minimal stack size to about 704K, which is the minimal size where 'def f(): f()' doesn't cause a buserror when run in a thread.
msg119214 - (view) Author: Leonardo Santagada (santagada) Date: 2010-10-20 15:41
Why not make it bigger so it doesn't crash for much bigger functions?
msg119217 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2010-10-20 15:48
On 20 Oct, 2010, at 17:41, Leonardo Santagada wrote:

> 
> Leonardo Santagada <santagada@gmail.com> added the comment:
> 
> Why not make it bigger so it doesn't crash for much bigger functions?

I don't mind making it bigger, but a larger size does come with a cost: the larger the initial stack size the less threads you can have without trashing the system (or running out of address space on a 32-bit build). 

I'm fine with increasing it to 1M, which is a nice round number build not larger. 

BTW. I haven't looked up the default stack size for the main thread on OSX.

Ronald
> 
> ----------
> nosy: +santagada
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue9670>
> _______________________________________
msg119219 - (view) Author: Leonardo Santagada (santagada) Date: 2010-10-20 15:57
The default stack size in osx is 8MB. I think it is a safer bet, and no one is supposed to be using more than like 40-50 threads anyway (specially in 32 bit only systems limited to 8 cores).

Is there a user case for hundreds/thousands of threads in python?
msg119222 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-10-20 16:30
It would be nice if we could expand this fix to include FreeBSD, which as I understand it has the same problem.
msg130873 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2011-03-14 18:45
I've slightly updated the patch:

1) Increase default stack size to 1MByte

2) Change the preprocessor guard to also trigger on FreeBSD

Without this patch I get crashes for 32-bit and 64-bit Intel architectures on OSX 10.6, with this patch those crashes go away.

I don't have access to a freebsd box so cannot test there.
msg130874 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2011-03-14 18:46
The attached unittest file triggers the issue until the patch is applied.

I'm not sure if I'd be a good idea to add this test as is to the testsuite, given the hard crash when it fails.
msg132916 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2011-04-04 05:52
Here's an updated combined patch.  I've revised Ronald's test to run via subprocess so it should be safe to add to the standard tests.  It works as expected on OS X.  If there are no objections, I will commit it and monitor the buildbots.
msg133217 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2011-04-07 13:12
Running the test in a separate process is a good idea.

The patch should be fine and fixes a problem on OSX, if the buildbot for FreeBSD starts to complain we can always removing the #if that tests for that platform.
msg133415 - (view) Author: Roundup Robot (python-dev) Date: 2011-04-09 19:48
New changeset b0d2b696da19 by Ned Deily in branch '2.7':
Issue #9670: Increase the default stack size for secondary threads on
http://hg.python.org/cpython/rev/b0d2b696da19

New changeset 378b40d71175 by Ned Deily in branch '3.1':
Issue #9670: Increase the default stack size for secondary threads on
http://hg.python.org/cpython/rev/378b40d71175

New changeset 3fe8fd2fd1d0 by Ned Deily in branch '3.2':
Issue #9670: merge with 3.2
http://hg.python.org/cpython/rev/3fe8fd2fd1d0

New changeset 4c750091d8c5 by Ned Deily in branch 'default':
Issue #9670: merge with current
http://hg.python.org/cpython/rev/4c750091d8c5
msg133417 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2011-04-09 19:53
Applied in 2.7 (for release in 2.7.2), 3.1 (for 3.1.4). 3.2 (for 3.2.1), and default (for 3.3).
msg133418 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2011-04-09 20:14
Looks like the patch breaks the OpenIndiana buildbots:

http://www.python.org/dev/buildbot/all/builders/x86%20OpenIndiana%203.2/builds/168
msg133430 - (view) Author: Roundup Robot (python-dev) Date: 2011-04-09 22:01
New changeset 42d5001e5845 by Ned Deily in branch '3.1':
Issue9670: Back out changeset 378b40d71175; test fails on other platforms
http://hg.python.org/cpython/rev/42d5001e5845

New changeset 54edabf2846d by Ned Deily in branch '3.2':
Issue9670: Merge backout to 3.2.
http://hg.python.org/cpython/rev/54edabf2846d

New changeset b7456c1b4aa4 by Ned Deily in branch 'default':
Issue9670: Merge backout from 3.2.
http://hg.python.org/cpython/rev/b7456c1b4aa4

New changeset 3630bc3d5a88 by Ned Deily in branch '2.7':
Issue9670: Back out changeset b0d2b696da19; test fails on other platforms
http://hg.python.org/cpython/rev/3630bc3d5a88
msg133431 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2011-04-09 22:09
Reverting the patch since it caused failures on failure on some other platform buildbots (for instance, Gentoo and OpenIndiana).  It also fails on OS X buildbots with pydebug enabled, something I hadn't tested:
http://www.python.org/dev/buildbot/all/builders/AMD64%20Leopard%203.2/builds/161

So the patch needs to address the stack size for OS X pydebug builds and what to do about other failing platforms - increase the stack size for each or disable the test there.
msg135271 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2011-05-06 09:41
For pydebug builds the default stacksize should be increased, and prefer to do this by unconditionally changing the stacksize.

W.r.t. the non-osx buildbot failures: it would be better if we could come up with a solution that doesn't disable the test for those platform but I won't do that work. I therefore propose to only enable the test on OSX.
msg135292 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2011-05-06 13:26
Version 4 of the patch has two changes w.r.t version 3:

* Increase the stack size on OSX to ensure that tests pass with
  --with-pydebug

* Only test the recursion behavior on OSX (where we know the bug is
  fixed)
msg137108 - (view) Author: Roundup Robot (python-dev) Date: 2011-05-28 07:50
New changeset ecf0ef85c72a by Ned Deily in branch '2.7':
Issue #9670: Increase the default stack size for secondary threads on
http://hg.python.org/cpython/rev/ecf0ef85c72a

New changeset 0cded2f2cea3 by Ned Deily in branch '3.1':
Issue #9670: Increase the default stack size for secondary threads on
http://hg.python.org/cpython/rev/0cded2f2cea3

New changeset 8a484090da7e by Ned Deily in branch '3.2':
Issue #9670: Increase the default stack size for secondary threads on
http://hg.python.org/cpython/rev/8a484090da7e

New changeset 8220f68f1229 by Ned Deily in branch 'default':
Issue #9670: Increase the default stack size for secondary threads on
http://hg.python.org/cpython/rev/8220f68f1229
msg137109 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2011-05-28 07:53
Version 4 looks good and the tests pass on OS X with pydebug enabled.  Applied in 2.7 (for release in 2.7.2), 3.1 (for 3.1.4). 3.2 (for 3.2.1), and default (for 3.3).
msg164338 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-06-29 17:45
Re-opening. The test fails on 3.2 on the new Lion buildbot (but neither on 3.3 nor 2.7, it seems).
See http://buildbot.python.org/all/builders/AMD64%20Lion%203.2
msg164342 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2012-06-29 18:13
3x is now using clang to compile on 10.7 Lion; 32 is failing back to the Xcode 4 default of llvm-gcc which has proved problematic fox 3x.  Rather than tweak the test again, try appending a CC=clang to the ./configure for the 3.2 buildbot until the clang changes are backported.  The builds should run faster, too, with clang.
msg164344 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-06-29 18:26
> try appending a CC=clang to the ./configure for the 3.2 buildbot until
> the clang changes are backported.

Ok, done that. Can someone watch the next builds?
msg164363 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2012-06-30 01:28
Looks like using clang brought the size back down again.  Closing.
History
Date User Action Args
2012-08-21 03:05:29ned.deilylinkissue15739 superseder
2012-06-30 01:28:56ned.deilysetstatus: open -> closed

messages: + msg164363
2012-06-29 18:26:02pitrousetmessages: + msg164344
2012-06-29 18:13:44ned.deilysetmessages: + msg164342
2012-06-29 17:46:20pitrousetstatus: closed -> open
2012-06-29 17:45:57pitrousetnosy: + pitrou, lukasz.langa
messages: + msg164338
2011-05-28 07:53:22ned.deilysetstatus: open -> closed
resolution: fixed
messages: + msg137109

stage: patch review -> resolved
2011-05-28 07:50:09python-devsetmessages: + msg137108
2011-05-06 13:26:23ronaldoussorensetfiles: + issue9670-v4.patch

messages: + msg135292
2011-05-06 09:41:20ronaldoussorensetmessages: + msg135271
2011-04-09 22:09:59ned.deilysetmessages: + msg133431
stage: resolved -> patch review
2011-04-09 22:01:21python-devsetmessages: + msg133430
2011-04-09 20:14:28ned.deilysetstatus: closed -> open
resolution: fixed -> (no value)
messages: + msg133418
2011-04-09 19:53:02ned.deilysetstatus: open -> closed
resolution: fixed
messages: + msg133417

stage: commit review -> resolved
2011-04-09 19:48:30python-devsetnosy: + python-dev
messages: + msg133415
2011-04-07 13:12:22ronaldoussorensetmessages: + msg133217
2011-04-04 05:52:51ned.deilysetfiles: + issue9670-v3.patch

nosy: + ned.deily
messages: + msg132916

stage: patch review -> commit review
2011-03-14 18:46:36ronaldoussorensetfiles: + test_issue9670.py
nosy: ronaldoussoren, r.david.murray, santagada, jaredlang
messages: + msg130874
2011-03-14 18:45:03ronaldoussorensetfiles: + issue9670-v2.txt
nosy: ronaldoussoren, r.david.murray, santagada, jaredlang
messages: + msg130873
2011-03-14 17:33:14ronaldoussorensetfiles: - smime.p7s
nosy: ronaldoussoren, r.david.murray, santagada, jaredlang
2010-10-20 16:30:31r.david.murraysetmessages: + msg119222
2010-10-20 15:57:58santagadasetmessages: + msg119219
2010-10-20 15:48:41ronaldoussorensetfiles: + smime.p7s

messages: + msg119217
2010-10-20 15:41:38santagadasetnosy: + santagada
messages: + msg119214
2010-10-20 14:45:43ronaldoussorensetkeywords: + patch, needs review
files: + issue9670.patch
messages: + msg119210

stage: needs patch -> patch review
2010-08-24 14:43:59r.david.murraysetmessages: + msg114793
2010-08-24 14:28:11ronaldoussorensetmessages: + msg114789
2010-08-24 13:57:38r.david.murraysetversions: + Python 3.1, Python 2.7, Python 3.2, - Python 2.6
nosy: + r.david.murray

messages: + msg114788

stage: needs patch
2010-08-24 13:23:18jaredlangcreate