Active extension modules are MySQL-python, numpy, and crypto.
Here is the output from the non-optimized debug build. Slightly different trace, but still some sort of deallocator crashing AFAIK:
#0 0x000000000046247c in _Py_ForgetReference (op=
<Channel(origin_addr=None, in_window_size=65536, in_window_threshold=6553, lock=<thread.lock at remote 0x571bf90>, _pipe=None, eof_received=0, in_max_packet_size=34816, out_buffer_cv=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0x571bf90>, acquire=<built-in method acquire of thread.lock object at remote 0x571bf90>, _Condition__waiters=[], release=<built-in method release of thread.lock object at remote 0x571bf90>) at remote 0x593d3e0>, event=<_Event(_Verbose__verbose=False, _Event__flag=True, _Event__cond=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0x59138b0>, acquire=<built-in method acquire of thread.lock object at remote 0x59138b0>, _Condition__waiters=[], release=<built-in method release of thread.lock object at remote 0x59138b0>) at remote 0x5a333e0>) at remote 0x593ded0>, transport=<Transport(_Thread__ident=140009885591296, host_key_type='ssh-rsa', _channels=<ChannelMap(_lock=<thread.lock at remote 0x5928f90>, _map=<WeakValueDictionary(_re...(truncated)) at Objects/object.c:2220
#1 0x00000000004624ed in _Py_Dealloc (op=
<Channel(origin_addr=None, in_window_size=65536, in_window_threshold=6553, lock=<thread.lock at remote 0x571bf90>, _pipe=None, eof_received=0, in_max_packet_size=34816, out_buffer_cv=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0x571bf90>, acquire=<built-in method acquire of thread.lock object at remote 0x571bf90>, _Condition__waiters=[], release=<built-in method release of thread.lock object at remote 0x571bf90>) at remote 0x593d3e0>, event=<_Event(_Verbose__verbose=False, _Event__flag=True, _Event__cond=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0x59138b0>, acquire=<built-in method acquire of thread.lock object at remote 0x59138b0>, _Condition__waiters=[], release=<built-in method release of thread.lock object at remote 0x59138b0>) at remote 0x5a333e0>) at remote 0x593ded0>, transport=<Transport(_Thread__ident=140009885591296, host_key_type='ssh-rsa', _channels=<ChannelMap(_lock=<thread.lock at remote 0x5928f90>, _map=<WeakValueDictionary(_re...(truncated)) at Objects/object.c:2240
#2 0x0000000000442244 in list_dealloc (op=0x66d7ab0) at Objects/listobject.c:309
#3 0x00000000004624fa in _Py_Dealloc (op=
[<Channel(origin_addr=None, in_window_size=65536, in_window_threshold=6553, lock=<thread.lock at remote 0x571bf90>, _pipe=None, eof_received=0, in_max_packet_size=34816, out_buffer_cv=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0x571bf90>, acquire=<built-in method acquire of thread.lock object at remote 0x571bf90>, _Condition__waiters=[], release=<built-in method release of thread.lock object at remote 0x571bf90>) at remote 0x593d3e0>, event=<_Event(_Verbose__verbose=False, _Event__flag=True, _Event__cond=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0x59138b0>, acquire=<built-in method acquire of thread.lock object at remote 0x59138b0>, _Condition__waiters=[], release=<built-in method release of thread.lock object at remote 0x59138b0>) at remote 0x5a333e0>) at remote 0x593ded0>, transport=<Transport(_Thread__ident=140009885591296, host_key_type='ssh-rsa', _channels=<ChannelMap(_lock=<thread.lock at remote 0x5928f90>, _map=<WeakValueDictionary(_r...(truncated)) at Objects/object.c:2241
#4 0x0000000000448bc4 in listiter_next (it=0x5d1c530) at Objects/listobject.c:2917
#5 0x00000000004ce425 in PyEval_EvalFrameEx (f=
Frame 0x7f56a050aea0, for file /usr/lib/python2.7/dist-packages/paramiko/transport.py, line 1586, in run (self=<Transport(_Thread__ident=140009885591296, host_key_type='ssh-rsa', _channels=<ChannelMap(_lock=<thread.lock at remote 0x5928f90>, _map=<WeakValueDictionary(_remove=<function at remote 0x56355a0>, data={}) at remote 0x5939588>) at remote 0x5912bc0>, lock=<thread.lock at remote 0x5928d60>, _Thread__started=<_Event(_Verbose__verbose=False, _Event__flag=True, _Event__cond=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0x521ff40>, acquire=<built-in method acquire of thread.lock object at remote 0x521ff40>, _Condition__waiters=[], release=<built-in method release of thread.lock object at remote 0x521ff40>) at remote 0x5223300>) at remote 0x47f56f0>, _channel_counter=22, active=False, _preferred_compression=('none',), server_object=None, kex_engine=None, log_name='paramiko.transport', _x11_handler=None, remote_compression='none', _Thread__initialized=True, server_accepts=[], s...(truncated), throwflag=0) at Python/ceval.c:2497
#6 0x00000000004d41c3 in fast_function (func=<function at remote 0x4716300>, pp_stack=0x7f56977ed400, n=1, na=1, nk=0) at Python/ceval.c:4099
#7 0x00000000004d3ed3 in call_function (pp_stack=0x7f56977ed400, oparg=0) at Python/ceval.c:4034
#8 0x00000000004cec2b in PyEval_EvalFrameEx (f=
Frame 0x5756c40, for file /root/python2.7-2.7.2/Lib/threading.py, line 552, in __bootstrap_inner (self=<Transport(_Thread__ident=140009885591296, host_key_type='ssh-rsa', _channels=<ChannelMap(_lock=<thread.lock at remote 0x5928f90>, _map=<WeakValueDictionary(_remove=<function at remote 0x56355a0>, data={}) at remote 0x5939588>) at remote 0x5912bc0>, lock=<thread.lock at remote 0x5928d60>, _Thread__started=<_Event(_Verbose__verbose=False, _Event__flag=True, _Event__cond=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0x521ff40>, acquire=<built-in method acquire of thread.lock object at remote 0x521ff40>, _Condition__waiters=[], release=<built-in method release of thread.lock object at remote 0x521ff40>) at remote 0x5223300>) at remote 0x47f56f0>, _channel_counter=22, active=False, _preferred_compression=('none',), server_object=None, kex_engine=None, log_name='paramiko.transport', _x11_handler=None, remote_compression='none', _Thread__initialized=True, server_accepts=[], saved_exc...(truncated), throwflag=0) at Python/ceval.c:2666
#9 0x00000000004d41c3 in fast_function (func=<function at remote 0x16ff060>, pp_stack=0x7f56977ed760, n=1, na=1, nk=0) at Python/ceval.c:4099
#10 0x00000000004d3ed3 in call_function (pp_stack=0x7f56977ed760, oparg=0) at Python/ceval.c:4034
#11 0x00000000004cec2b in PyEval_EvalFrameEx (f=
Frame 0x53b9380, for file /root/python2.7-2.7.2/Lib/threading.py, line 525, in __bootstrap (self=<Transport(_Thread__ident=140009885591296, host_key_type='ssh-rsa', _channels=<ChannelMap(_lock=<thread.lock at remote 0x5928f90>, _map=<WeakValueDictionary(_remove=<function at remote 0x56355a0>, data={}) at remote 0x5939588>) at remote 0x5912bc0>, lock=<thread.lock at remote 0x5928d60>, _Thread__started=<_Event(_Verbose__verbose=False, _Event__flag=True, _Event__cond=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0x521ff40>, acquire=<built-in method acquire of thread.lock object at remote 0x521ff40>, _Condition__waiters=[], release=<built-in method release of thread.lock object at remote 0x521ff40>) at remote 0x5223300>) at remote 0x47f56f0>, _channel_counter=22, active=False, _preferred_compression=('none',), server_object=None, kex_engine=None, log_name='paramiko.transport', _x11_handler=None, remote_compression='none', _Thread__initialized=True, server_accepts=[], saved_exception...(truncated), throwflag=0) at Python/ceval.c:2666
#12 0x00000000004d1533 in PyEval_EvalCodeEx (co=0x16cb510, globals=
{'current_thread': <function at remote 0x16ffe28>, '_BoundedSemaphore': <type at remote 0x1680cd0>, 'currentThread': <function at remote 0x16ffe28>, '_Timer': <type at remote 0x1682640>, '_format_exc': <function at remote 0x1538e28>, 'Semaphore': <function at remote 0x16fc840>, 'activeCount': <function at remote 0x1700108>, '_profile_hook': None, '_sleep': <built-in function sleep>, '_trace_hook': None, 'ThreadError': <type at remote 0x165f7f0>, '_enumerate': <function at remote 0x17001b0>, '_start_new_thread': <built-in function start_new_thread>, 'BoundedSemaphore': <function at remote 0x16fcf78>, '_shutdown': <instancemethod at remote 0x14a2960>, '__all__': ['activeCount', 'active_count', 'Condition', 'currentThread', 'current_thread', 'enumerate', 'Event', 'Lock', 'RLock', 'Semaphore', 'BoundedSemaphore', 'Thread', 'Timer', 'setprofile', 'settrace', 'local', 'stack_size'], '_Event': <type at remote 0x1681730>, 'active_count': <function at remote 0x1700108>, '__package__': None, '_Condition': <type at remo...(truncated), locals=0x0, args=0x475f088, argcount=1, kws=0x0, kwcount=0, defs=0x0,
defcount=0, closure=0x0) at Python/ceval.c:3253
#13 0x000000000055c7e9 in function_call (func=<function at remote 0x16feed0>, arg=
(<Transport(_Thread__ident=140009885591296, host_key_type='ssh-rsa', _channels=<ChannelMap(_lock=<thread.lock at remote 0x5928f90>, _map=<WeakValueDictionary(_remove=<function at remote 0x56355a0>, data={}) at remote 0x5939588>) at remote 0x5912bc0>, lock=<thread.lock at remote 0x5928d60>, _Thread__started=<_Event(_Verbose__verbose=False, _Event__flag=True, _Event__cond=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0x521ff40>, acquire=<built-in method acquire of thread.lock object at remote 0x521ff40>, _Condition__waiters=[], release=<built-in method release of thread.lock object at remote 0x521ff40>) at remote 0x5223300>) at remote 0x47f56f0>, _channel_counter=22, active=False, _preferred_compression=('none',), server_object=None, kex_engine=None, log_name='paramiko.transport', _x11_handler=None, remote_compression='none', _Thread__initialized=True, server_accepts=[], saved_exception=exceptions.EOFError(), packetizer=<Packetizer(_Packetizer__remainder='', _Packetizer__block_siz...(truncated), kw=0x0) at Objects/funcobject.c:526
#14 0x0000000000420ae3 in PyObject_Call (func=<function at remote 0x16feed0>, arg=
(<Transport(_Thread__ident=140009885591296, host_key_type='ssh-rsa', _channels=<ChannelMap(_lock=<thread.lock at remote 0x5928f90>, _map=<WeakValueDictionary(_remove=<function at remote 0x56355a0>, data={}) at remote 0x5939588>) at remote 0x5912bc0>, lock=<thread.lock at remote 0x5928d60>, _Thread__started=<_Event(_Verbose__verbose=False, _Event__flag=True, _Event__cond=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0x521ff40>, acquire=<built-in method acquire of thread.lock object at remote 0x521ff40>, _Condition__waiters=[], release=<built-in method release of thread.lock object at remote 0x521ff40>) at remote 0x5223300>) at remote 0x47f56f0>, _channel_counter=22, active=False, _preferred_compression=('none',), server_object=None, kex_engine=None, log_name='paramiko.transport', _x11_handler=None, remote_compression='none', _Thread__initialized=True, server_accepts=[], saved_exception=exceptions.EOFError(), packetizer=<Packetizer(_Packetizer__remainder='', _Packetizer__block_siz...(truncated), kw=0x0) at Objects/abstract.c:2529
#15 0x000000000042b35f in instancemethod_call (func=<function at remote 0x16feed0>, arg=
(<Transport(_Thread__ident=140009885591296, host_key_type='ssh-rsa', _channels=<ChannelMap(_lock=<thread.lock at remote 0x5928f90>, _map=<WeakValueDictionary(_remove=<function at remote 0x56355a0>, data={}) at remote 0x5939588>) at remote 0x5912bc0>, lock=<thread.lock at remote 0x5928d60>, _Thread__started=<_Event(_Verbose__verbose=False, _Event__flag=True, _Event__cond=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0x521ff40>, acquire=<built-in method acquire of thread.lock object at remote 0x521ff40>, _Condition__waiters=[], release=<built-in method release of thread.lock object at remote 0x521ff40>) at remote 0x5223300>) at remote 0x47f56f0>, _channel_counter=22, active=False, _preferred_compression=('none',), server_object=None, kex_engine=None, log_name='paramiko.transport', _x11_handler=None, remote_compression='none', _Thread__initialized=True, server_accepts=[], saved_exception=exceptions.EOFError(), packetizer=<Packetizer(_Packetizer__remainder='', _Packetizer__block_siz...(truncated), kw=0x0) at Objects/classobject.c:2578
#16 0x0000000000420ae3 in PyObject_Call (func=<instancemethod at remote 0x59377e0>, arg=(), kw=0x0) at Objects/abstract.c:2529
#17 0x00000000004d3496 in PyEval_CallObjectWithKeywords (func=<instancemethod at remote 0x59377e0>, arg=(), kw=0x0) at Python/ceval.c:3882
#18 0x000000000051e193 in t_bootstrap (boot_raw=0x3828988) at ./Modules/threadmodule.c:614
#19 0x00007f56b1f17efc in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#20 0x00007f56b15c789d in clone () from /lib/x86_64-linux-gnu/libc.so.6
#21 0x0000000000000000 in ?? ()
The op object is empty, which caused the segfault:
(gdb) f 0
2220 op->_ob_prev->_ob_next != op || op->_ob_next->_ob_prev != op)
(gdb) p *op
$5 = {_ob_next = 0x0, _ob_prev = 0x0, ob_refcnt = 0, ob_type = 0x3c6d9f0}
I believe this is the python decoded version?:
(gdb) p op
$18 =
<Channel(origin_addr=None, in_window_size=65536, in_window_threshold=6553, lock=<thread.lock at remote 0x571bf90>, _pipe=None, eof_received=0, in_max_packet_size=34816, out_buffer_cv=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0x571bf90>, acquire=<built-in method acquire of thread.lock object at remote 0x571bf90>, _Condition__waiters=[], release=<built-in method release of thread.lock object at remote 0x571bf90>) at remote 0x593d3e0>, event=<_Event(_Verbose__verbose=False, _Event__flag=True, _Event__cond=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0x59138b0>, acquire=<built-in method acquire of thread.lock object at remote 0x59138b0>, _Condition__waiters=[], release=<built-in method release of thread.lock object at remote 0x59138b0>) at remote 0x5a333e0>) at remote 0x593ded0>, transport=<Transport(_Thread__ident=140009885591296, host_key_type='ssh-rsa', _channels=<ChannelMap(_lock=<thread.lock at remote 0x5928f90>, _map=<WeakValueDictionary(_re...(truncated
Possibly useful?
(gdb) p (void*)op
$31 = (void *) 0x593bd80
(gdb) p &refchain
$32 = <unknown at remote 0x7e8ca0>
Possibly a double free? |
➜
