classification
Title: cPickle segfault with deep recursion
Type: crash Stage:
Components: Library (Lib) Versions: Python 2.6, Python 2.5
process
Status: closed Resolution: fixed
Dependencies: 3640 Superseder:
Assigned To: Nosy List: esrever_otua, facundobatista, jcea, loewis, pitrou
Priority: normal Keywords:

Created on 2008-07-11 04:54 by esrever_otua, last changed 2010-08-03 20:46 by terry.reedy. This issue is now closed.

Messages (11)
msg69532 - (view) Author: Darryl Dixon (esrever_otua) Date: 2008-07-11 04:54
In at least Python 2.4, using cPickle.Pickler to try and pickle a nested
chain of objects more than about 2590 objects deep causes the Python
interpreter to segfault. This doesn't seem to happen when using the pure
Python pickle module.

It is not memory related (witness that the pure Python module can
achieve depths much greater than this just fine), and does not seem to
be directly related to system architecture (happens on both i386 and on
x86_64 (32bit and 64bit)).

Sample interpreter session to replicate:
>>> # Let's cause cPickle to segfault:
>>> from cPickle import Pickler as cPickler
>>> class rec:
...   child = None
...   def __init__(self, counter):
...     if counter > 0:
...       self.child = rec(counter-1)
...
>>> import sys
>>> sys.setrecursionlimit(10000)
>>> mychain = rec(2600)
>>> from cStringIO import StringIO
>>> stream = StringIO()
>>> p = cPickler(stream, 1)
>>> res = p.dump(mychain)
Segmentation fault

And now the same steps again using the pure Python Pickler:
>>> import sys
>>> from pickle import Pickler as pPickler
>>> from cStringIO import StringIO
>>> class rec:
...   child = None
...   def __init__(self, counter):
...     if counter > 0:
...       self.child = rec(counter-1)
... 
>>> sys.setrecursionlimit(20000)
>>> mychain = rec(2600)
>>> stream = StringIO()
>>> p = pPickler(stream, 1)
>>> p.dump(mychain)
>>> len(stream.getvalue())
48676
>>>
msg69534 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2008-07-11 06:49
Can you try this for a newer version? For 2.4, such problems will not be
fixed anymore.
msg69585 - (view) Author: Darryl Dixon (esrever_otua) Date: 2008-07-12 12:37
Happens with Python 2.5.2 on 64bit also:

Python 2.5.2 (r252:60911, Apr 21 2008, 11:17:30) 
[GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import platform
>>> platform.architecture()
('64bit', '')
>>> from cPickle import Pickler
>>> class rec:
...   child = None
...   def __init__(self, counter):
...     if counter > 0:
...       self.child = rec(counter-1)
... 
>>> import sys
>>> sys.setrecursionlimit(10000)
>>> mychain = rec(2600)
>>> from cStringIO import StringIO
>>> stream = StringIO()
>>> p = Pickler(stream, 1)
>>> res = p.dump(mychain)
Segmentation fault
msg69756 - (view) Author: Darryl Dixon (esrever_otua) Date: 2008-07-16 03:27
Hmm, looks like this dup's 2702... Funny how two people find the same
thing within a short window of each other *sighs* so looks like it's
probably fixed. I'll test /trunk against the failing testcase below and
make sure all is OK.

D
msg69759 - (view) Author: Darryl Dixon (esrever_otua) Date: 2008-07-16 04:07
No, I've just tested /trunk, including r64595, and the Segmentation
fault is still present, eg:

Python 2.6b1+ (trunk:64998, Jul 16 2008, 15:50:22) 
[GCC 4.1.1 20070105 (Red Hat 4.1.1-52)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.setrecursionlimit(40000)
>>> class rec(object):
...   child = None
...   def __init__(self, counter):
...     if counter > 0:
...       self.child = rec(counter-1)
... 
>>> mychain = rec(2600)
>>> from cPickle import Pickler
>>> from cStringIO import StringIO
>>> stream = StringIO()
>>> p = Pickler(stream, 1)
>>> p.dump(mychain)
Segmentation fault
msg69803 - (view) Author: Facundo Batista (facundobatista) * (Python committer) Date: 2008-07-16 17:01
Confirmed in... 

Python 2.6b1+ (trunk:65017M, Jul 16 2008, 13:37:00) 
[GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)] on linux2

...with a more simple case:

"""
import sys, cPickle
sys.setrecursionlimit(10405) 

class rec(object):
    child = None
    def __init__(self, counter):
      if counter > 0:
        self.child = rec(counter-1)

mychain = rec(2600)
cPickle.dumps(mychain)
"""

Note that if we put the recursion limit in 10405 we get a segfault, but
if we put it 10404, we get a "RuntimeError: maximum recursion depth
exceeded". 

Considering that 10400 is exactly 2600 * 4, maybe this is a useful hint.

Another behaviour I got: With a recursion limit big big enough, doing
rec(1670) works ok, and rec(1671) segfaults.

And a very nasty one:

I put rec(1671) to see in which recursion limit we have the conversion
of RecursionLimit to SegFault. Testing, I tried with recursion limit in
6700, and sometimes it worked ok, and sometimes it segfaulted. Yes,
*sometimes*, :(
msg69842 - (view) Author: Darryl Dixon (esrever_otua) Date: 2008-07-16 21:39
That is a very interesting observation (x4), especially in light of #3373

Unfortunately I don't really have the (p|g)db -foo to debug either of
these properly :(
msg71656 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2008-08-21 16:24
Well, the standard recursion limit is precisely there to guard against
segfaults when blowing up the C stack, so if you make the recursion
limit much larger, it's quite normal to get segfaults.

Therefore, I don't think this is a real bug.
msg71722 - (view) Author: Darryl Dixon (esrever_otua) Date: 2008-08-22 04:18
Well, it's definitely a bug, or inconsistency, if you like, between
cPickle and pickle.

My gut says that probably there is some fault in cPickle that is causing
this. When pickle.py can recurse to 10,000+ and cPickle segfaults at
2600 on a 64bit machine, something smells wrong.

Especially as there have been other, similar, *fixable* bugs already in
cPickle (cf #2702). Unfortunately I can only report the problem, I do
not have the expertise to debug.

regards,
D
msg71733 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2008-08-22 09:23
> Well, it's definitely a bug, or inconsistency, if you like, between
> cPickle and pickle.

There is clearly a problem with cPickle stack consumption and a new bug has been
opened for this in #3640.

What I don't agree with is your argument that pickle and cPickle should have the
same recursion limits - it's just "foolish consistency" to ask for identical
behaviour on such a low-level and implementation-dependent issue.
msg87936 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-05-16 21:23
Now that #3640 has been fixed, this bug is probably fixed too (at least
in trunk and py3k).
History
Date User Action Args
2010-08-03 20:46:31terry.reedysetstatus: open -> closed
resolution: fixed
2009-05-16 21:23:59pitrousetmessages: + msg87936
2009-05-16 20:35:06ajaksu2setpriority: normal
dependencies: + test_cpickle crash on AMD64 Windows build
versions: + Python 2.6, - Python 2.4
2008-08-22 09:23:45pitrousetmessages: + msg71733
2008-08-22 04:18:08esrever_otuasetmessages: + msg71722
2008-08-21 16:24:20pitrousetnosy: + pitrou
messages: + msg71656
2008-07-16 21:39:59esrever_otuasetmessages: + msg69842
2008-07-16 18:45:41jceasetnosy: + jcea
2008-07-16 17:01:08facundobatistasetnosy: + facundobatista
messages: + msg69803
2008-07-16 04:07:55esrever_otuasetmessages: + msg69759
2008-07-16 03:27:05esrever_otuasetmessages: + msg69756
2008-07-12 12:37:09esrever_otuasetmessages: + msg69585
versions: + Python 2.5
2008-07-11 06:49:03loewissetnosy: + loewis
messages: + msg69534
2008-07-11 04:54:21esrever_otuacreate