classification
Title: segfault on printing deeply nested structures (2.7 only)
Type: crash Stage: needs patch
Components: Interpreter Core Versions: Python 2.7, Python 2.6
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: BreamoreBoy, benjamin.peterson, georg.brandl, haypo, jonaskoelker, r.david.murray, terry.reedy
Priority: normal Keywords:

Created on 2004-11-18 23:31 by jonaskoelker, last changed 2013-11-05 06:10 by benjamin.peterson. This issue is now closed.

Messages (11)
msg60597 - (view) Author: Jonas Kölker (jonaskoelker) Date: 2004-11-18 23:31
## bug1.py
loop = None,
while True: loop = loop, None

bash$ python -i bug1.py
(I press C-c)
>>> print loop
((((((((((((((((( <skipping a large amount of parens
for brevity> (((((((((segfault
bash$ 

bash$ python -i bug1.py
(I press C-c)
>>> str(loop)
segfault
bash$ 

bash$ python -i bug1.py
(I press C-c)
>>> repr(loop)
segfault
bash$

--- ---
the results are the same for bug2.py, bug3.py and
bug.py4 which have the same contents, except that loop
is a list, None is replaced by Ellipsis or both. The
behavior (i.e. segfaulting) is the same.
also, when pressing C-c, python's memory usage remains
constant. gc.garbage is empty; gc.collect() returns 0
and doesn't free memory.

my python and uname:

bash$ python
Python 2.3.4 (#2, Sep 24 2004, 08:39:09)
[GCC 3.3.4 (Debian 1:3.3.4-12)] on linux2

bash$ uname --all
Linux koelker 2.4.18-bf2.4 #1 Son Apr 14 09:53:28 CEST
2002 i686 GNU/Linux

I have also reproduced it on
Python 2.3.3 (#1, May 7 2004, 10:31:40)
[GCC 3.3.3 20040412 (Red Hat Linux 3.3.3-7)] on linux2

bash$ uname --all
Linux horse09 2.6.8-1.521smp #1 SMP Man Aug 16 09:25:06
EDT 2004 i686 i686 i386 GNU/Linux

and on
Python 2.4a3 (#1, Oct  3 2004, 12:05:12)
[GCC 3.3.2] on linux2
(on the same machine)

it didn't reproduce on
Python 1.5.2 (#1, Mar  9 2000, 17:40:34)  [GCC 2.95.2
19991024 (release)] on hp-uxB

bash$ uname -a
HP-UX aragorn B.11.11 U 9000/782 2006786478
unlimited-user license

but it did on
Python 1.5.2 (#1, Jul  5 2001, 03:02:19)  [GCC 2.96
20000731] (Red Hat Linux 7.1 2 on linux-i386

bash$ uname
Linux legolas 2.4.20-20.7smp #1 SMP Mon Aug 18 14:46:14
EDT 2003 i686 unknown
msg60598 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2004-11-24 07:43
Logged In: YES 
user_id=593130

On <<<Python 2.2.1 (#34, Apr  9 2002, 19:34:33) [MSC 32 
bit (Intel)] on win32>>> I got (interactive mode, about 5 
sec before ^C)  appropriate output:

((((((((((((((((((((((((((((((((((((((((((((((((((((((
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
MemoryError: stack overflow

The nested depth was
>>> i=0
>>> while loop: loop,i = loop[0], i+1
...
>>> i
50695

Note: a test case would need a finite loop such as:
for i in range(100000) 
instead of while True (until ^C pressed).
msg60599 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2005-09-15 10:56
Logged In: YES 
user_id=1188172

I can reproduce this here on Linux with 100000x None. Will
try to investigate.
msg74020 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2008-09-29 10:40
This issue is a stack overflow: your code do recursive calls to 
internal_print().

Backtrace from gdb:
--------------------------------------------
#0  0xb7e92064 in _IO_new_file_overflow () 
from /lib/tls/i686/cmov/libc.so.6
#1  0xb7e94bc3 in __overflow () from /lib/tls/i686/cmov/libc.so.6
#2  0xb7e8e03b in fputc () from /lib/tls/i686/cmov/libc.so.6
#3  0x080a2f2d in tupleprint (op=0xb7b5cc2c, fp=0xb7f744e0, flags=0) 
at Objects/tupleobject.c:193
#4  0x0808b9f7 in internal_print (op=0xb7b5cc2c, fp=0xb7f744e0, 
flags=0, nesting=0) at Objects/object.c:309
#5  0x0808ba60 in PyObject_Print (op=0xb7b5cc2c, fp=0xb7f744e0, 
flags=0) at Objects/object.c:324
(...)
#4628 0x0808ba60 in PyObject_Print (op=0xb7b68e4c, fp=0xb7f744e0, 
flags=0) at Objects/object.c:324
#4629 0x080a2f9e in tupleprint (op=0xb7b68e6c, fp=0xb7f744e0, flags=0) 
at Objects/tupleobject.c:201
#4630 0x0808b9f7 in internal_print (op=0xb7b68e6c, fp=0xb7f744e0, 
flags=0, nesting=0) at Objects/object.c:309
#4631 0x0808ba60 in PyObject_Print (op=0xb7b68e6c, fp=0xb7f744e0, 
flags=0) at Objects/object.c:324
(...)
#4638 0x080a2f9e in tupleprint (op=0xb7b68ecc, fp=0xb7f744e0, flags=0) 
at Objects/tupleobject.c:201
#4639 0x0808b9f7 in internal_print (op=0xb7b68ecc, fp=0xb7f744e0, 
flags=0, nesting=0) at Objects/object.c:309
#4640 0x0808ba60 in PyObject_Print (op=0xb7b68ecc, fp=0xb7f744e0, 
flags=0) at Objects/object.c:324
#4641 0x080a2f9e in tupleprint (op=0xb7b68eec, fp=0xb7f744e0, flags=0) 
at Objects/tupleobject.c:201
#4642 0x0808b9f7 in internal_print (op=0xb7b68eec, fp=0xb7f744e0, 
flags=0, nesting=0) at Objects/object.c:309
(...)
--------------------------------------------

Informations using my debugger (python-ptrace):
------------------------------------------------------------
PID: 1201
Signal: SIGSEGV
STACK OVERFLOW! Stack pointer is in 0xbf10d000-0xbf90d000 => [stack] 
(rw-p)
- instruction: CALL 0x80d3305
- mapping: 0xbf10cff0 is not mapped in memory
- register <stack ptr>=0xbf10cff0
------------------------------------------------------------

Some solutions:
 - avoid creation of nested sequence
 - create generic protection against stack overflow (use 
setjmp/longjmp)
 - avoid recursive call to internal_print()
msg75364 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2008-10-30 09:35
I propose a generic solution in issue #3999 to catch stack overflow 
(and be able to continue after such "exception").
msg101503 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2010-03-22 13:02
This issue is 6 years old. Python has some protections against stack overflow, but there is no perfect solution. Since there is no patch, I consider that I can close this issue (as wont fix).
msg101520 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2010-03-22 16:57
General policy is that ordinary code (not using, for instance, ctypes) should not crash or segfault the interpreter.  I believe there is a 'crashers' subdirectory somewhere in the tree for examples that do so that people so inclined can work on them.

The OP reported a crash on 2.3/2.4a on Linux, but not 1.5. I could not reproduce it on 2.2 on Windows. Instead, I (properly) got an exception. Trying again with 3.1, I get a similar exception: RuntimeError: maximum recursion depth exceeded while getting the repr of a tuple. List instead of tuple does similar.

If 
>>> t=None,
>>> for i in range(50000): t = t,None

>>> print(t)

still crashes on 2.6/2.7, at least with Linux, then there is still a bug to be fixed and the issue should be left open. If it now raises an exception as above, then this should be closed as fixed.

I am pretty sure this issue has nothing to do with None and Ellipsis but only with the structure (not necessarily a sequence) being deeply, deeply nested. So I think the title should be: "Segfault on printing deeply nested structures."

I think the deeper issue is the use of recursion on the C stack to print. If the print routine instead usee iteration with an auxiliary Python stack (list), then there should be no stack overflow to worry about. [When in increase the recursion limit to 100000 and try to print the 50000 nested tuples, I get "MemoryError: stack overflow" instead of the RuntimeError above. So there might be more than one fix needed.]
msg101525 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-03-22 18:45
On linux: on Py3 (trunk and 3.1) I get the recursion depth exceeded message, but on py2 trunk I get the segfault if I use 100000 for the range.  So somebody fixed this crasher in py3, somehow.
msg114391 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2010-08-19 18:36
FYI the generic solution mentioned in msg75364 can now be found at #8863
msg202192 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-11-05 03:32
Verified with fresh 2.7.6 build
msg202196 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2013-11-05 06:10
This is wont fix in Python 2.
History
Date User Action Args
2013-11-05 06:10:45benjamin.petersonsetstatus: open -> closed

nosy: + benjamin.peterson
messages: + msg202196

resolution: wont fix
2013-11-05 03:32:50terry.reedysetmessages: + msg202192
title: segfault on printing nested sequences of None/Ellipsis -> segfault on printing deeply nested structures (2.7 only)
2010-08-19 18:36:15BreamoreBoysetnosy: + BreamoreBoy
messages: + msg114391
2010-03-22 18:45:02r.david.murraysetstatus: closed -> open

versions: + Python 2.7
nosy: + r.david.murray

messages: + msg101525
resolution: wont fix -> (no value)
stage: needs patch
2010-03-22 16:57:23terry.reedysetmessages: + msg101520
2010-03-22 13:02:56hayposetstatus: open -> closed
resolution: wont fix
messages: + msg101503
2008-10-30 09:35:01hayposetmessages: + msg75364
2008-09-29 10:40:02hayposetnosy: + haypo
messages: + msg74020
2008-01-20 19:27:12christian.heimessettype: crash
versions: + Python 2.6, - Python 2.3
2004-11-18 23:31:34jonaskoelkercreate