classification
Title: sys recursion limit a lot shorter on trunk?
Type: behavior Stage:
Components: Interpreter Core Versions: Python 3.0, Python 2.6
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: gvanrossum Nosy List: amaury.forgeotdarc, benjamin.peterson, brett.cannon, esrever_otua, gvanrossum, pitrou, trent
Priority: release blocker Keywords:

Created on 2008-07-16 04:01 by esrever_otua, last changed 2008-08-21 16:10 by gvanrossum. This issue is now closed.

Files
File name Uploaded Description Edit
rec.py pitrou, 2008-07-17 20:06
Messages (14)
msg69758 - (view) Author: Darryl Dixon (esrever_otua) Date: 2008-07-16 04:01
The system recursion limit seems to be wildly different in its behaviour
on 2.6/trunk versus, for example, 2.5 or 2.4, EG:

On Python 2.4:
Python 2.4.3 (#1, Dec 11 2006, 11:38:52) 
[GCC 4.1.1 20061130 (Red Hat 4.1.1-43)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> class rec(object):
...   child = None
...   def __init__(self, counter):
...     if counter > 0:
...       self.child = rec(counter-1)
... 
>>> mychain = rec(998)
>>>

On Python 2.6/trunk:
Python 2.6b1+ (trunk:64998, Jul 16 2008, 15:50:22) 
[GCC 4.1.1 20070105 (Red Hat 4.1.1-52)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> class rec(object):
...   child = None
...   def __init__(self, counter):
...     if counter > 0:
...       self.child = rec(counter-1)
... 
>>> mychain = rec(249)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 5, in __init__
[...snip...]
  File "<stdin>", line 5, in __init__
RuntimeError: maximum recursion depth exceeded
>>>

In both cases sys.getrecursionlimit() shows 1000.

Is this behaviour intentional? It looks a lot like a regression of some
sort. It appears to be effectively 4x shorter when creating the nested
object graph.
msg69760 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2008-07-16 04:12
A lot of crashers were fixed for 2.6 where the recursion limit was not
being used at all. That is probably what you are seeing.
msg69762 - (view) Author: Darryl Dixon (esrever_otua) Date: 2008-07-16 04:18
Hmmm, I'm not certain I agree; on 2.4/2.5 doing rec(999) hits the
recursion limit, as expected (makes sense that there would be an item or
two on the stack prior to the immediate call to rec() ). This looks more
like the interpreter is adding 4x the number of items to the stack
during the construction of the nested object, which seems pretty
surprising/broken...
msg69766 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2008-07-16 05:28
Well, probably the best way to find out would be to run under gdb and
see who is incrementing the recursion count.
msg69772 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2008-07-16 11:20
Why was 1000 chosen in the first place? If it's just an arbitrary value
then we can bump it to 4000 so that people don't get bad surprises when
upgrading their Python.

> This looks more
> like the interpreter is adding 4x the number of items to the stack
> during the construction of the nested object, which seems pretty
> surprising/broken...

Well PyObject_Call increases the recursion count, and entering __init__
will increase it once more. That explains the 2x, not the 4x though.
msg69810 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2008-07-16 18:25
On Wed, Jul 16, 2008 at 4:20 AM, Antoine Pitrou <report@bugs.python.org> wrote:
>
> Antoine Pitrou <pitrou@free.fr> added the comment:
>
> Why was 1000 chosen in the first place? If it's just an arbitrary value
> then we can bump it to 4000 so that people don't get bad surprises when
> upgrading their Python.
>

It was originally 10,000, but people wanted thread switches to occur more often.

>> This looks more
>> like the interpreter is adding 4x the number of items to the stack
>> during the construction of the nested object, which seems pretty
>> surprising/broken...
>
> Well PyObject_Call increases the recursion count, and entering __init__
> will increase it once more. That explains the 2x, not the 4x though.

As I said, without a comparison of traces this is continue to just be
speculation (and I don't have the time to do that).
msg69812 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2008-07-16 18:28
Brett:
>It was originally 10,000, but people wanted thread switches to occur
>more often.

I thought that was managed by sys.setcheckinterval.
msg69813 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2008-07-16 18:30
On Wed, Jul 16, 2008 at 11:28 AM, Benjamin Peterson
<report@bugs.python.org> wrote:
>
> Benjamin Peterson <musiccomposition@gmail.com> added the comment:
>
> Brett:
>>It was originally 10,000, but people wanted thread switches to occur
>>more often.
>
> I thought that was managed by sys.setcheckinterval.
>

Yes it is; sorry, brain is slow today.

I know the current value usually does not lead to a segfault on any of
the common platforms that Python runs on.
msg69859 - (view) Author: Trent Nelson (trent) * (Python committer) Date: 2008-07-16 23:58
Darryl, was your trunk version built as debug?  If so, can you try 
without?
msg69863 - (view) Author: Darryl Dixon (esrever_otua) Date: 2008-07-17 01:18
Hi Trent,

No, my build did not invoke --with-pydebug. In other words, the process
I used was simply:

svn co http://svn.python.org/projects/python/trunk python-trunk
cd python-trunk
./configure --prefix=/home/dixond/throwaway
make
make install
msg69903 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2008-07-17 20:06
Here is a small script that shows various possibilities depending on how
object creation is done, and here is the output with the trunk:

rec1 stopped at 1000
rec2 stopped at 1000
rec3 stopped at 500
rec4 stopped at 334
rec5 stopped at 334
rec6 stopped at 250

With 2.5, the output is:

rec1 stopped at 1000
rec2 stopped at 1000
rec3 stopped at 500
rec4 stopped at 1000
rec5 stopped at 1000
rec6 stopped at 1000

I think we should just acknowledge that recursion count has gotten
stricter (PyObject_Call() increases it, and then PyEval_EvalFrameEx()
will increase it a second time if Python code is entered), and bump the
default recursion limit.

(the reason calling Python functions directly doesn't increase the
recursion count twice is that there are optimization shortcuts in
ceval.c to avoid calling PyObject_Call() - not the case though when
calling a type object)
msg71601 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2008-08-21 02:49
Guido, can you comment?
msg71636 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2008-08-21 13:27
(FWIW, I just ran Misc/find_recursion_limit.py on a 32-bit Windows box.

With 2.6 I get 5900.
With 3.0 I get 9000.

So the good news is that 3.0 seems to be less stack-hungry :-))
msg71653 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2008-08-21 16:10
I think it's fine as it is. Incrementing the stack level more frequently
is a good thing since there used to be paths that didn't increment it at
all and hence could cause segfaults.  The default is conservative since
increasing it could cause segfaults, and on some systems threads don't
get a very large stack.
History
Date User Action Args
2008-08-21 16:10:21gvanrossumsetstatus: open -> closed
resolution: wont fix
messages: + msg71653
2008-08-21 13:27:22pitrousetmessages: + msg71636
2008-08-21 02:49:23benjamin.petersonsetpriority: high -> release blocker
assignee: gvanrossum
messages: + msg71601
nosy: + gvanrossum
2008-07-21 14:02:40amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
2008-07-17 20:09:16pitrousetpriority: high
versions: + Python 3.0
2008-07-17 20:07:02pitrousetfiles: + rec.py
messages: + msg69903
2008-07-17 01:18:43esrever_otuasetmessages: + msg69863
2008-07-16 23:58:06trentsetnosy: + trent
messages: + msg69859
2008-07-16 18:30:42brett.cannonsetmessages: + msg69813
2008-07-16 18:28:19benjamin.petersonsetnosy: + benjamin.peterson
messages: + msg69812
2008-07-16 18:25:47brett.cannonsetmessages: + msg69810
2008-07-16 11:20:30pitrousetnosy: + pitrou
messages: + msg69772
2008-07-16 05:28:49brett.cannonsetmessages: + msg69766
2008-07-16 04:18:31esrever_otuasetmessages: + msg69762
2008-07-16 04:12:20brett.cannonsetnosy: + brett.cannon
messages: + msg69760
2008-07-16 04:01:52esrever_otuacreate