msg114787 - (view) |
Author: Jared Lang (jaredlang) |
Date: 2010-08-24 13:23 |
Recursion within a thread on OSX can result in a crash by exceeding the systems recursion limit. Recursion behaves as expected if not in thread, meaning it throws a RunTimeError with the message "maximum recursion depth exceeded."
The crash is able to be avoided if the recursion limit is set to a lower number, ie. 800, via sys.setrecursionlimit.
Example program which crashes on OSX:
>>> def f():
... return f()
...
>>> import threading
>>> thread = threading.Thread(target=f)
>>> thread.start()
Bus error
Alternatively, the following works as expected:
>>> import sys
>>> def f():
... return f()
...
>>> import threading
>>> thread = threading.Thread(target=f)
>>> sys.setrecursionlimit(800)
>>> thread.start()
>>> Exception in thread Thread-1:
Traceback (most recent call last):
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py", line 532, in __bootstrap_inner
self.run()
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py", line 484, in run
self.__target(*self.__args, **self.__kwargs)
File "<stdin>", line 2, in f
File "<stdin>", line 2, in f
File "<stdin>", line 2, in f
File "<stdin>", line 2, in f
File "<stdin>", line 2, in f
File "<stdin>", line 2, in f
...
RuntimeError: maximum recursion depth exceeded
|
msg114788 - (view) |
Author: R. David Murray (r.david.murray) * |
Date: 2010-08-24 13:57 |
This is not the first recursion limit related problem we've seen with FreeBSD derived systems. See for example Issue1201456. It would be nice to have an actual fix for this class of problem, but I have no clue what that would look like. There may also be a specific fix for this issue. At least, we can hope :)
Python 2.6 is in security-fix-only mode, so I've updated the versions. I can confirm that this reproduces on 2.7 and py3k trunk on OS/X 10.4 using a non-framework build. I haven't tested 3.1 but I am sure it must behave the same.
|
msg114789 - (view) |
Author: Ronald Oussoren (ronaldoussoren) * |
Date: 2010-08-24 14:28 |
This is the same issue as #6763.
The root cause of this issue is that the default stack size for threads other than the main thread is small.
One way to fix this issue is to increase the default stacksize for new threads.
|
msg114793 - (view) |
Author: R. David Murray (r.david.murray) * |
Date: 2010-08-24 14:43 |
As I mentioned on the other ticket, I think bumping the default stack size is probably the most useful solution. This is is a clearer description of the stack problem, since it doesn't get involved in MIMETypes internals, so I'm not closing it as a duplicate.
|
msg119210 - (view) |
Author: Ronald Oussoren (ronaldoussoren) * |
Date: 2010-10-20 14:45 |
The attached patch explicitly sets the minimal stack size to about 704K, which is the minimal size where 'def f(): f()' doesn't cause a buserror when run in a thread.
|
msg119214 - (view) |
Author: Leonardo Santagada (santagada) |
Date: 2010-10-20 15:41 |
Why not make it bigger so it doesn't crash for much bigger functions?
|
msg119217 - (view) |
Author: Ronald Oussoren (ronaldoussoren) * |
Date: 2010-10-20 15:48 |
On 20 Oct, 2010, at 17:41, Leonardo Santagada wrote:
>
> Leonardo Santagada <santagada@gmail.com> added the comment:
>
> Why not make it bigger so it doesn't crash for much bigger functions?
I don't mind making it bigger, but a larger size does come with a cost: the larger the initial stack size the less threads you can have without trashing the system (or running out of address space on a 32-bit build).
I'm fine with increasing it to 1M, which is a nice round number build not larger.
BTW. I haven't looked up the default stack size for the main thread on OSX.
Ronald
>
> ----------
> nosy: +santagada
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue9670>
> _______________________________________
|
msg119219 - (view) |
Author: Leonardo Santagada (santagada) |
Date: 2010-10-20 15:57 |
The default stack size in osx is 8MB. I think it is a safer bet, and no one is supposed to be using more than like 40-50 threads anyway (specially in 32 bit only systems limited to 8 cores).
Is there a user case for hundreds/thousands of threads in python?
|
msg119222 - (view) |
Author: R. David Murray (r.david.murray) * |
Date: 2010-10-20 16:30 |
It would be nice if we could expand this fix to include FreeBSD, which as I understand it has the same problem.
|
msg130873 - (view) |
Author: Ronald Oussoren (ronaldoussoren) * |
Date: 2011-03-14 18:45 |
I've slightly updated the patch:
1) Increase default stack size to 1MByte
2) Change the preprocessor guard to also trigger on FreeBSD
Without this patch I get crashes for 32-bit and 64-bit Intel architectures on OSX 10.6, with this patch those crashes go away.
I don't have access to a freebsd box so cannot test there.
|
msg130874 - (view) |
Author: Ronald Oussoren (ronaldoussoren) * |
Date: 2011-03-14 18:46 |
The attached unittest file triggers the issue until the patch is applied.
I'm not sure if I'd be a good idea to add this test as is to the testsuite, given the hard crash when it fails.
|
msg132916 - (view) |
Author: Ned Deily (ned.deily) * |
Date: 2011-04-04 05:52 |
Here's an updated combined patch. I've revised Ronald's test to run via subprocess so it should be safe to add to the standard tests. It works as expected on OS X. If there are no objections, I will commit it and monitor the buildbots.
|
msg133217 - (view) |
Author: Ronald Oussoren (ronaldoussoren) * |
Date: 2011-04-07 13:12 |
Running the test in a separate process is a good idea.
The patch should be fine and fixes a problem on OSX, if the buildbot for FreeBSD starts to complain we can always removing the #if that tests for that platform.
|
msg133415 - (view) |
Author: Roundup Robot (python-dev) |
Date: 2011-04-09 19:48 |
New changeset b0d2b696da19 by Ned Deily in branch '2.7':
Issue #9670: Increase the default stack size for secondary threads on
http://hg.python.org/cpython/rev/b0d2b696da19
New changeset 378b40d71175 by Ned Deily in branch '3.1':
Issue #9670: Increase the default stack size for secondary threads on
http://hg.python.org/cpython/rev/378b40d71175
New changeset 3fe8fd2fd1d0 by Ned Deily in branch '3.2':
Issue #9670: merge with 3.2
http://hg.python.org/cpython/rev/3fe8fd2fd1d0
New changeset 4c750091d8c5 by Ned Deily in branch 'default':
Issue #9670: merge with current
http://hg.python.org/cpython/rev/4c750091d8c5
|
msg133417 - (view) |
Author: Ned Deily (ned.deily) * |
Date: 2011-04-09 19:53 |
Applied in 2.7 (for release in 2.7.2), 3.1 (for 3.1.4). 3.2 (for 3.2.1), and default (for 3.3).
|
msg133418 - (view) |
Author: Ned Deily (ned.deily) * |
Date: 2011-04-09 20:14 |
Looks like the patch breaks the OpenIndiana buildbots:
http://www.python.org/dev/buildbot/all/builders/x86%20OpenIndiana%203.2/builds/168
|
msg133430 - (view) |
Author: Roundup Robot (python-dev) |
Date: 2011-04-09 22:01 |
New changeset 42d5001e5845 by Ned Deily in branch '3.1':
Issue9670: Back out changeset 378b40d71175; test fails on other platforms
http://hg.python.org/cpython/rev/42d5001e5845
New changeset 54edabf2846d by Ned Deily in branch '3.2':
Issue9670: Merge backout to 3.2.
http://hg.python.org/cpython/rev/54edabf2846d
New changeset b7456c1b4aa4 by Ned Deily in branch 'default':
Issue9670: Merge backout from 3.2.
http://hg.python.org/cpython/rev/b7456c1b4aa4
New changeset 3630bc3d5a88 by Ned Deily in branch '2.7':
Issue9670: Back out changeset b0d2b696da19; test fails on other platforms
http://hg.python.org/cpython/rev/3630bc3d5a88
|
msg133431 - (view) |
Author: Ned Deily (ned.deily) * |
Date: 2011-04-09 22:09 |
Reverting the patch since it caused failures on failure on some other platform buildbots (for instance, Gentoo and OpenIndiana). It also fails on OS X buildbots with pydebug enabled, something I hadn't tested:
http://www.python.org/dev/buildbot/all/builders/AMD64%20Leopard%203.2/builds/161
So the patch needs to address the stack size for OS X pydebug builds and what to do about other failing platforms - increase the stack size for each or disable the test there.
|
msg135271 - (view) |
Author: Ronald Oussoren (ronaldoussoren) * |
Date: 2011-05-06 09:41 |
For pydebug builds the default stacksize should be increased, and prefer to do this by unconditionally changing the stacksize.
W.r.t. the non-osx buildbot failures: it would be better if we could come up with a solution that doesn't disable the test for those platform but I won't do that work. I therefore propose to only enable the test on OSX.
|
msg135292 - (view) |
Author: Ronald Oussoren (ronaldoussoren) * |
Date: 2011-05-06 13:26 |
Version 4 of the patch has two changes w.r.t version 3:
* Increase the stack size on OSX to ensure that tests pass with
--with-pydebug
* Only test the recursion behavior on OSX (where we know the bug is
fixed)
|
msg137108 - (view) |
Author: Roundup Robot (python-dev) |
Date: 2011-05-28 07:50 |
New changeset ecf0ef85c72a by Ned Deily in branch '2.7':
Issue #9670: Increase the default stack size for secondary threads on
http://hg.python.org/cpython/rev/ecf0ef85c72a
New changeset 0cded2f2cea3 by Ned Deily in branch '3.1':
Issue #9670: Increase the default stack size for secondary threads on
http://hg.python.org/cpython/rev/0cded2f2cea3
New changeset 8a484090da7e by Ned Deily in branch '3.2':
Issue #9670: Increase the default stack size for secondary threads on
http://hg.python.org/cpython/rev/8a484090da7e
New changeset 8220f68f1229 by Ned Deily in branch 'default':
Issue #9670: Increase the default stack size for secondary threads on
http://hg.python.org/cpython/rev/8220f68f1229
|
msg137109 - (view) |
Author: Ned Deily (ned.deily) * |
Date: 2011-05-28 07:53 |
Version 4 looks good and the tests pass on OS X with pydebug enabled. Applied in 2.7 (for release in 2.7.2), 3.1 (for 3.1.4). 3.2 (for 3.2.1), and default (for 3.3).
|
msg164338 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2012-06-29 17:45 |
Re-opening. The test fails on 3.2 on the new Lion buildbot (but neither on 3.3 nor 2.7, it seems).
See http://buildbot.python.org/all/builders/AMD64%20Lion%203.2
|
msg164342 - (view) |
Author: Ned Deily (ned.deily) * |
Date: 2012-06-29 18:13 |
3x is now using clang to compile on 10.7 Lion; 32 is failing back to the Xcode 4 default of llvm-gcc which has proved problematic fox 3x. Rather than tweak the test again, try appending a CC=clang to the ./configure for the 3.2 buildbot until the clang changes are backported. The builds should run faster, too, with clang.
|
msg164344 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2012-06-29 18:26 |
> try appending a CC=clang to the ./configure for the 3.2 buildbot until
> the clang changes are backported.
Ok, done that. Can someone watch the next builds?
|
msg164363 - (view) |
Author: Ned Deily (ned.deily) * |
Date: 2012-06-30 01:28 |
Looks like using clang brought the size back down again. Closing.
|
|
Date |
User |
Action |
Args |
2022-04-11 14:57:05 | admin | set | github: 53879 |
2012-08-21 03:05:29 | ned.deily | link | issue15739 superseder |
2012-06-30 01:28:56 | ned.deily | set | status: open -> closed
messages:
+ msg164363 |
2012-06-29 18:26:02 | pitrou | set | messages:
+ msg164344 |
2012-06-29 18:13:44 | ned.deily | set | messages:
+ msg164342 |
2012-06-29 17:46:20 | pitrou | set | status: closed -> open |
2012-06-29 17:45:57 | pitrou | set | nosy:
+ pitrou, lukasz.langa messages:
+ msg164338
|
2011-05-28 07:53:22 | ned.deily | set | status: open -> closed resolution: fixed messages:
+ msg137109
stage: patch review -> resolved |
2011-05-28 07:50:09 | python-dev | set | messages:
+ msg137108 |
2011-05-06 13:26:23 | ronaldoussoren | set | files:
+ issue9670-v4.patch
messages:
+ msg135292 |
2011-05-06 09:41:20 | ronaldoussoren | set | messages:
+ msg135271 |
2011-04-09 22:09:59 | ned.deily | set | messages:
+ msg133431 stage: resolved -> patch review |
2011-04-09 22:01:21 | python-dev | set | messages:
+ msg133430 |
2011-04-09 20:14:28 | ned.deily | set | status: closed -> open resolution: fixed -> (no value) messages:
+ msg133418
|
2011-04-09 19:53:02 | ned.deily | set | status: open -> closed resolution: fixed messages:
+ msg133417
stage: commit review -> resolved |
2011-04-09 19:48:30 | python-dev | set | nosy:
+ python-dev messages:
+ msg133415
|
2011-04-07 13:12:22 | ronaldoussoren | set | messages:
+ msg133217 |
2011-04-04 05:52:51 | ned.deily | set | files:
+ issue9670-v3.patch
nosy:
+ ned.deily messages:
+ msg132916
stage: patch review -> commit review |
2011-03-14 18:46:36 | ronaldoussoren | set | files:
+ test_issue9670.py nosy:
ronaldoussoren, r.david.murray, santagada, jaredlang messages:
+ msg130874
|
2011-03-14 18:45:03 | ronaldoussoren | set | files:
+ issue9670-v2.txt nosy:
ronaldoussoren, r.david.murray, santagada, jaredlang messages:
+ msg130873
|
2011-03-14 17:33:14 | ronaldoussoren | set | files:
- smime.p7s nosy:
ronaldoussoren, r.david.murray, santagada, jaredlang |
2010-10-20 16:30:31 | r.david.murray | set | messages:
+ msg119222 |
2010-10-20 15:57:58 | santagada | set | messages:
+ msg119219 |
2010-10-20 15:48:41 | ronaldoussoren | set | files:
+ smime.p7s
messages:
+ msg119217 |
2010-10-20 15:41:38 | santagada | set | nosy:
+ santagada messages:
+ msg119214
|
2010-10-20 14:45:43 | ronaldoussoren | set | keywords:
+ patch, needs review files:
+ issue9670.patch messages:
+ msg119210
stage: needs patch -> patch review |
2010-08-24 14:43:59 | r.david.murray | set | messages:
+ msg114793 |
2010-08-24 14:28:11 | ronaldoussoren | set | messages:
+ msg114789 |
2010-08-24 13:57:38 | r.david.murray | set | versions:
+ Python 3.1, Python 2.7, Python 3.2, - Python 2.6 nosy:
+ r.david.murray
messages:
+ msg114788
stage: needs patch |
2010-08-24 13:23:18 | jaredlang | create | |