Garbage Collector can cause Segfault whilst iterating dictionary items #74669

jimwright · 2017-05-26T12:48:10Z

BPO	30484
Nosy	@larryhastings, @berkerpeksag, @serhiy-storchaka, @JelleZijlstra
PRs	[3.4] bpo-27945: Fixed various segfaults with dict. (GH-1657) (GH-1678) #2248
Files	segfault.py: segfault.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/larryhastings'
closed_at = <Date 2017-10-26.07:31:16.383>
created_at = <Date 2017-05-26.12:48:10.292>
labels = ['interpreter-core', 'type-crash']
title = 'Garbage Collector can cause Segfault whilst iterating dictionary items'
updated_at = <Date 2017-10-26.07:31:16.380>
user = 'https://bugs.python.org/jimwright'

bugs.python.org fields:

activity = <Date 2017-10-26.07:31:16.380>
actor = 'berker.peksag'
assignee = 'larry'
closed = True
closed_date = <Date 2017-10-26.07:31:16.383>
closer = 'berker.peksag'
components = ['Interpreter Core']
creation = <Date 2017-05-26.12:48:10.292>
creator = 'jimwright'
dependencies = []
files = ['46905']
hgrepos = []
issue_num = 30484
keywords = []
message_count = 8.0
messages = ['294548', '294723', '294741', '294744', '296197', '296198', '298158', '305030']
nosy_count = 5.0
nosy_names = ['larry', 'berker.peksag', 'serhiy.storchaka', 'JelleZijlstra', 'jimwright']
pr_nums = ['2248']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue30484'
versions = ['Python 3.4']

jimwright · 2017-05-26T12:48:08Z

We discovered this issue whilst using h5py (HDF5 python library) under python 3.5.2 on Ubuntu 16.04.2 x86_64. The construct used is very dubious, and I will separately be raising an issue with the h5py team. However I thought you might like to know there is a possible way to cause a segmentation violation using pure python.

There appears to be a new implementation of dictionaries in 3.6 vs 3.5, so I compiled up 3.6.1 and it also segfaults, but not so obviously related to the dictionary iteration (both stack traces are included below).

Regards, Jim.

--------

$ cat segfault.py 
#!/usr/bin/python3

class CyclicObject:
    def __init__(self, key, register):
        self.key = key
        self.self = self
        self.register = register
        self.register[self.key] = None

    def __del__(self):
        del self.register[self.key]

while True:
    register = { }
    objs = set([CyclicObject(i, register) for i in range(10000)])

    while len(objs) > 0:
        objs.remove(next(iter(objs)))

    try:
        list(register.items())
    except RuntimeError as err:
        print(err)

--------

$ python3 --version
Python 3.5.2

$ gdb --args python3 segfault.py
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1

(gdb) r
Starting program: /usr/bin/python3 segfault.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
dictionary changed size during iteration
dictionary changed size during iteration
dictionary changed size during iteration

Program received signal SIGSEGV, Segmentation fault.
dictiter_iternextitem.lto_priv () at ../Objects/dictobject.c:3158
3158 ../Objects/dictobject.c: No such file or directory.

(gdb) bt
#0 dictiter_iternextitem.lto_priv () at ../Objects/dictobject.c:3158
#1 0x000000000059f024 in listextend.lto_priv () at ../Objects/listobject.c:855
#2 0x000000000058fd1c in list_init.lto_priv () at ../Objects/listobject.c:2314
#3 0x000000000055d17c in type_call.lto_priv () at ../Objects/typeobject.c:905
#4 0x00000000005b7167 in PyObject_Call () at ../Objects/abstract.c:2165
#5 0x0000000000528d06 in do_call (nk=<optimised out>, na=<optimised out>, pp_stack=0x7fffffffd9b0, func=<optimised out>) at ../Python/ceval.c:4936
#6 call_function (oparg=<optimised out>, pp_stack=0x7fffffffd9b0) at ../Python/ceval.c:4732
#7 PyEval_EvalFrameEx () at ../Python/ceval.c:3236
#8 0x000000000052d2e3 in _PyEval_EvalCodeWithName () at ../Python/ceval.c:4018
#9 0x000000000052dfdf in PyEval_EvalCodeEx () at ../Python/ceval.c:4039
#10 PyEval_EvalCode (co=<optimised out>, globals=<optimised out>, locals=<optimised out>) at ../Python/ceval.c:777
#11 0x00000000005fd2c2 in run_mod () at ../Python/pythonrun.c:976
#12 0x00000000005ff76a in PyRun_FileExFlags () at ../Python/pythonrun.c:929
#13 0x00000000005ff95c in PyRun_SimpleFileExFlags () at ../Python/pythonrun.c:396
#14 0x000000000063e7d6 in run_file (p_cf=0x7fffffffdc20, filename=0xa73280 L"segfault.py", fp=0xad19a0) at ../Modules/main.c:318
#15 Py_Main () at ../Modules/main.c:768
#16 0x00000000004cfe41 in main () at ../Programs/python.c:65
#17 0x00007ffff7811830 in __libc_start_main (main=0x4cfd60 <main>, argc=2, argv=0x7fffffffde38, init=<optimised out>, fini=<optimised out>, rtld_fini=<optimised out>,
stack_end=0x7fffffffde28) at ../csu/libc-start.c:291
#18 0x00000000005d5f29 in _start ()

--------

$ ./python --version
Python 3.6.1

$ gdb --args ./python ./segfault.py 
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1

(gdb) r
Starting program: ./python ./segfault.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
dictionary changed size during iteration
dictionary changed size during iteration
dictionary changed size during iteration
dictionary changed size during iteration
dictionary changed size during iteration

Program received signal SIGSEGV, Segmentation fault.
_PyObject_Alloc (ctx=0x0, elsize=28, nelem=1, use_calloc=0) at Objects/obmalloc.c:1258
1258 if ((pool->freeblock = *(block **)bp) != NULL) {

(gdb) bt
#0 _PyObject_Alloc (ctx=0x0, elsize=28, nelem=1, use_calloc=0) at Objects/obmalloc.c:1258
#1 _PyObject_Malloc (ctx=0x0, nbytes=28) at Objects/obmalloc.c:1437
#2 0x0000000000490732 in _PyLong_New (size=1) at Objects/longobject.c:196
#3 PyLong_FromLong (ival=<optimised out>) at Objects/longobject.c:254
#4 0x0000000000542fda in _PyEval_EvalFrameDefault (f=<optimised out>, throwflag=<optimised out>) at Python/ceval.c:3060
#5 0x000000000053d671 in PyEval_EvalFrameEx (throwflag=0, f=0x7ffff7fa1648) at Python/ceval.c:718
#6 _PyFunction_FastCall (co=<optimised out>, args=<optimised out>, nargs=1, globals=globals@entry=0x7ffff7f452d0) at Python/ceval.c:4880
#7 0x000000000053e521 in fast_function (kwnames=0x0, nargs=<optimised out>, stack=<optimised out>, func=0x7ffff7eae510) at Python/ceval.c:4915
#8 call_function (pp_stack=pp_stack@entry=0x7fffffffd920, oparg=oparg@entry=1, kwnames=kwnames@entry=0x0) at Python/ceval.c:4819
#9 0x0000000000542c17 in _PyEval_EvalFrameDefault (f=<optimised out>, throwflag=<optimised out>) at Python/ceval.c:3284
#10 0x000000000053e015 in PyEval_EvalFrameEx (throwflag=0, f=0x93a368) at Python/ceval.c:718
#11 _PyEval_EvalCodeWithName (_co=_co@entry=0x7ffff7ec26f0, globals=globals@entry=0x7ffff7f5e078, locals=locals@entry=0x7ffff7ec26f0, args=args@entry=0x0,
argcount=argcount@entry=0, kwnames=kwnames@entry=0x0, kwargs=0x8, kwcount=0, kwstep=2, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0)
at Python/ceval.c:4128
#12 0x000000000053ee43 in PyEval_EvalCodeEx (closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, kwcount=0, kws=0x0, argcount=0, args=0x0, locals=locals@entry=0x7ffff7ec26f0,
globals=globals@entry=0x7ffff7f5e078, _co=_co@entry=0x7ffff7ec26f0) at Python/ceval.c:4149
#13 PyEval_EvalCode (co=co@entry=0x7ffff7ec26f0, globals=globals@entry=0x7ffff7f452d0, locals=locals@entry=0x7ffff7f452d0) at Python/ceval.c:695
#14 0x000000000042740f in run_mod (arena=0x7ffff7f5e078, flags=0x7fffffffdc00, locals=0x7ffff7f452d0, globals=0x7ffff7f452d0, filename=0x7ffff7e844f8, mod=0x933c28)
at Python/pythonrun.c:980
#15 PyRun_FileExFlags (fp=0x93a350, filename_str=<optimised out>, start=<optimised out>, globals=0x7ffff7f452d0, locals=0x7ffff7f452d0, closeit=1, flags=0x7fffffffdc00)
at Python/pythonrun.c:933
#16 0x000000000042763c in PyRun_SimpleFileExFlags (fp=0x93a350, filename=<optimised out>, closeit=1, flags=0x7fffffffdc00) at Python/pythonrun.c:396
#17 0x000000000043b975 in run_file (p_cf=0x7fffffffdc00, filename=0x8f82a0 L"./segfault.py", fp=0x93a350) at Modules/main.c:338
#18 Py_Main (argc=argc@entry=2, argv=argv@entry=0x8f7010) at Modules/main.c:809
#19 0x000000000041dc52 in main (argc=2, argv=<optimised out>) at ./Programs/python.c:69

JelleZijlstra · 2017-05-29T22:50:00Z

I could reproduce this on 3.4, but not on 3.3, 2.7, or master.

jimwright · 2017-05-30T07:02:20Z

Hi Jelle,

I think you're right about master; it looks like this very recent change:

#1657

could have fixed the issue, but I didn't find it whilst doing my original investigation of the issue on our side.

serhiy-storchaka · 2017-05-30T07:47:19Z

3.4 is open only for security fixes. Can this crash be considered a security issue? In that case I can backport the bpo-27945 patch to 3.4.

serhiy-storchaka · 2017-06-16T15:08:20Z

PR 2248 backports the bpo-27945 patch to 3.4. Seems it fixes this issue.

serhiy-storchaka · 2017-06-16T15:17:18Z

Larry, does it worth to fix this issue in 3.4?

larryhastings · 2017-07-11T13:29:55Z

Yes, and thank you for submitting the PR to backport it to 3.4!

(And thank you for backporting it to 3.3, too!)

berkerpeksag · 2017-10-26T07:31:16Z

PR 2248 (3.4) and PR 2396 (3.3) have been merged. I think this issue can be closed now.

jimwright mannequin added interpreter-core (Objects, Python, Grammar, and Parser dirs) type-crash A hard crash of the interpreter, possibly with a core dump labels May 26, 2017

serhiy-storchaka assigned larryhastings Jun 16, 2017

berkerpeksag closed this as completed Oct 26, 2017

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Garbage Collector can cause Segfault whilst iterating dictionary items #74669

Garbage Collector can cause Segfault whilst iterating dictionary items #74669

jimwright mannequin commented May 26, 2017

jimwright mannequin commented May 26, 2017

JelleZijlstra commented May 29, 2017

jimwright mannequin commented May 30, 2017

serhiy-storchaka commented May 30, 2017

serhiy-storchaka commented Jun 16, 2017

serhiy-storchaka commented Jun 16, 2017

larryhastings commented Jul 11, 2017

berkerpeksag commented Oct 26, 2017

Navigation Menu

Garbage Collector can cause Segfault whilst iterating dictionary items #74669

Garbage Collector can cause Segfault whilst iterating dictionary items #74669

Comments

jimwright mannequin commented May 26, 2017

jimwright mannequin commented May 26, 2017

JelleZijlstra commented May 29, 2017

jimwright mannequin commented May 30, 2017

serhiy-storchaka commented May 30, 2017

serhiy-storchaka commented Jun 16, 2017

serhiy-storchaka commented Jun 16, 2017

larryhastings commented Jul 11, 2017

berkerpeksag commented Oct 26, 2017