My code has recently started triggering a core dump in the Python executable when the VSCode debugger is attached. This doesn't happen right away; it seems to happen more or less _after_ the program is done executing (I just placed a breakpoint and stepped it through).
The program in question is this: https://github.com/hiboxsystems/trac-to-gitlab/blob/master/migrate.py
To help in the debugging of this, I installed python2.7-dbg and gdb-python2 on my Debian machine, and re-ran the script using this version. Here is the GDB output when analyzing the backtrace:
$ gdb /usr/bin/python2.7-dbg core
GNU gdb (Debian 8.1-4+b1) 8.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/python2.7-dbg...done.
[New LWP 19749]
[New LWP 19744]
[New LWP 19747]
[New LWP 19754]
[New LWP 19751]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/python2.7-dbg -m ptvsd --host localhost --port 43959 migrate.py --only'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 PyEval_EvalFrameEx (f=0x7f815c002310, throwflag=0) at ../Python/ceval.c:3347
3347 if (tstate->frame->f_exc_type != NULL)
[Current thread is 1 (Thread 0x7f815bfff700 (LWP 19749))]
The python backtrace looks like this:
(gdb) py-bt
Traceback (most recent call first):
File "/usr/lib/python2.7/threading.py", line 371, in wait
self._acquire_restore(saved_state)
File "/usr/lib/python2.7/Queue.py", line 177, in get
self.not_empty.wait(remaining)
File "/home/per/.vscode/extensions/ms-python.python-2018.8.0/pythonFiles/experimental/ptvsd/ptvsd/_vendored/pydevd/_pydevd_bundle/pydevd_comm.py", line 458, in _on_run
cmd = self.cmdQueue.get(1, 0.1)
File "/home/per/.vscode/extensions/ms-python.python-2018.8.0/pythonFiles/experimental/ptvsd/ptvsd/_vendored/pydevd/_pydevd_bundle/pydevd_comm.py", line 319, in run
self._on_run()
File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 774, in __bootstrap
self.__bootstrap_inner()
And the C-level backtrace:
(gdb) bt
#0 PyEval_EvalFrameEx (f=Frame 0x7f815c002310, for file /usr/lib/python2.7/threading.py, line 371, in wait (), throwflag=0)
at ../Python/ceval.c:3347
#1 0x00005624534af42c in PyEval_EvalCodeEx (co=0x7f816216e7d0,
globals={'current_thread': None, '_BoundedSemaphore': None, 'currentThread': None, '_Timer': None, '_format_exc': None, 'Semaphore': None, '_deque': None, 'activeCount': None, '_profile_hook': None, '_sleep': None, '_trace_hook': None, 'ThreadError': None, '_enumerate': None, '_start_new_thread': None, 'BoundedSemaphore': None, '_shutdown': None, '__all__': None, '_original_start_new_thread': None, '_Event': None, 'active_count': None, '__package__': None, '_Condition': None, '_RLock': None, '_test': None, 'local': None, '__doc__': None, 'Condition': None, '_Verbose': None, '_DummyThread': None, 'Thread': None, 'warnings': None, '__builtins__': {'bytearray': None, 'IndexError': None, 'all': None, 'help': None, 'vars': None, 'SyntaxError': None, 'unicode': None, 'UnicodeDecodeError': None, 'memoryview': None, 'isinstance': None, 'copyright': None, 'NameError': None, 'BytesWarning': None, 'dict': None, 'input': None, 'oct': None, 'bin': None, 'SystemExit': None, 'StandardError': None, 'format': None, 'repr': None, 'sor...(truncated), locals=0x0, args=0x562454463068, argcount=2,
kws=0x562454463078, kwcount=0, defs=0x7f8162116408, defcount=1, closure=0x0) at ../Python/ceval.c:3604
#2 0x00005624534b23a7 in fast_function (func=<function at remote 0x7f81620b76f0>, pp_stack=0x7f815bffd3e8, n=2, na=2, nk=0)
at ../Python/ceval.c:4467
#3 0x00005624534b1f8a in call_function (pp_stack=0x7f815bffd3e8, oparg=1) at ../Python/ceval.c:4392
#4 0x00005624534ac45d in PyEval_EvalFrameEx (
f=Frame 0x562454462eb0, for file /usr/lib/python2.7/Queue.py, line 177, in get (self=<Queue(unfinished_tasks=5, queue=<collections.deque at remote 0x7f8161078f60>, maxsize=0, all_tasks_done=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0x7f816124fea0>, acquire=<built-in method acquire of thread.lock object at remote 0x7f816124fea0>, _Condition__waiters=[], release=<built-in method release of thread.lock object at remote 0x7f816124fea0>) at remote 0x7f81610bc0d0>, mutex=<thread.lock at remote 0x7f816124fea0>, not_full=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0x7f816124fea0>, acquire=<built-in method acquire of thread.lock object at remote 0x7f816124fea0>, _Condition__waiters=[], release=<built-in method release of thread.lock object at remote 0x7f816124fea0>) at remote 0x7f81610bc060>, not_empty=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0x7f816124fea0>, acquire=<built-in method acquire of thread.lock object at r...(truncated), throwflag=0) at ../Python/ceval.c:3009
#5 0x00005624534af42c in PyEval_EvalCodeEx (co=0x7f81620ea5c0,
globals={'LifoQueue': None, 'deque': None, 'heapq': None, 'Full': None, '__all__': None, '_threading': None, '__builtins__': {'bytearray': None, 'IndexError': None, 'all': None, 'help': None, 'vars': None, 'SyntaxError': None, 'unicode': None, 'UnicodeDecodeError': None, 'memoryview': None, 'isinstance': None, 'copyright': None, 'NameError': None, 'BytesWarning': None, 'dict': None, 'input': None, 'oct': None, 'bin': None, 'SystemExit': None, 'StandardError': None, 'format': None, 'repr': None, 'sorted': None, 'False': None, 'RuntimeWarning': None, 'list': None, 'iter': None, 'reload': None, 'Warning': None, '__package__': None, 'round': None, 'dir': None, 'cmp': None, 'set': None, 'bytes': None, 'reduce': None, 'intern': None, 'issubclass': None, 'Ellipsis': None, 'EOFError': None, 'locals': None, 'BufferError': None, 'slice': None, 'FloatingPointError': None, 'sum': None, 'getattr': None, 'abs': None, 'exit': None, 'print': None, 'True': None, 'FutureWarning': None, 'ImportWarning': None, 'None': None, 'hash': None...(truncated), locals=0x0, args=0x7f81500011a0, argcount=3,
kws=0x7f81500011b8, kwcount=0, defs=0x7f8161ce82e0, defcount=2, closure=0x0) at ../Python/ceval.c:3604
#6 0x00005624534b23a7 in fast_function (func=<function at remote 0x7f8161ceb300>, pp_stack=0x7f815bffd9f8, n=3, na=3, nk=0)
at ../Python/ceval.c:4467
#7 0x00005624534b1f8a in call_function (pp_stack=0x7f815bffd9f8, oparg=2) at ../Python/ceval.c:4392
#8 0x00005624534ac45d in PyEval_EvalFrameEx (
f=Frame 0x7f8150000ff0, for file /home/per/.vscode/extensions/ms-python.python-2018.8.0/pythonFiles/experimental/ptvsd/ptvsd/_vendored/pydevd/_pydevd_bundle/pydevd_comm.py, line 458, in _on_run (self=<WriterThread(_Thread__ident=140193571010304, cmdQueue=<Queue(unfinished_tasks=5, queue=<collections.deque at remote 0x7f8161078f60>, maxsize=0, all_tasks_done=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0x7f816124fea0>, acquire=<built-in method acquire of thread.lock object at remote 0x7f816124fea0>, _Condition__waiters=[], release=<built-in method release of thread.lock object at remote 0x7f816124fea0>) at remote 0x7f81610bc0d0>, mutex=<thread.lock at remote 0x7f816124fea0>, not_full=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0x7f816124fea0>, acquire=<built-in method acquire of thread.lock object at remote 0x7f816124fea0>, _Condition__waiters=[], release=<built-in method release of thread.lock object at remote 0x7f816124fea0>) at remote 0x7f81610b...(truncated), throwflag=0) at ../Python/ceval.c:3009
#9 0x00005624534b2291 in fast_function (func=<function at remote 0x7f816122e3a8>, pp_stack=0x7f815bffde78, n=1, na=1, nk=0)
at ../Python/ceval.c:4457
#10 0x00005624534b1f8a in call_function (pp_stack=0x7f815bffde78, oparg=0) at ../Python/ceval.c:4392
#11 0x00005624534ac45d in PyEval_EvalFrameEx (
f=Frame 0x7f8150000dd0, for file /home/per/.vscode/extensions/ms-python.python-2018.8.0/pythonFiles/experimental/ptvsd/ptvsd/_vendored/pydevd/_pydevd_bundle/pydevd_comm.py, line 319, in run (self=<WriterThread(_Thread__ident=140193571010304, cmdQueue=<Queue(unfinished_tasks=5, queue=<collections.deque at remote 0x7f8161078f60>, maxsize=0, all_tasks_done=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0x7f816124fea0>, acquire=<built-in method acquire of thread.lock object at remote 0x7f816124fea0>, _Condition__waiters=[], release=<built-in method release of thread.lock object at remote 0x7f816124fea0>) at remote 0x7f81610bc0d0>, mutex=<thread.lock at remote 0x7f816124fea0>, not_full=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0x7f816124fea0>, acquire=<built-in method acquire of thread.lock object at remote 0x7f816124fea0>, _Condition__waiters=[], release=<built-in method release of thread.lock object at remote 0x7f816124fea0>) at remote 0x7f81610bc060...(truncated), throwflag=0) at ../Python/ceval.c:3009
#12 0x00005624534b2291 in fast_function (func=<function at remote 0x7f816122ac30>, pp_stack=0x7f815bffe2f8, n=1, na=1, nk=0)
at ../Python/ceval.c:4457
#13 0x00005624534b1f8a in call_function (pp_stack=0x7f815bffe2f8, oparg=0) at ../Python/ceval.c:4392
#14 0x00005624534ac45d in PyEval_EvalFrameEx (
f=Frame 0x7f8150000b80, for file /usr/lib/python2.7/threading.py, line 801, in __bootstrap_inner (self=<WriterThread(_Thread__ident=140193571010304, cmdQueue=<Queue(unfinished_tasks=5, queue=<collections.deque at remote 0x7f8161078f60>, maxsize=0, all_tasks_done=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0x7f816124fea0>, acquire=<built-in method acquire of thread.lock object at remote 0x7f816124fea0>, _Condition__waiters=[], release=<built-in method release of thread.lock object at remote 0x7f816124fea0>) at remote 0x7f81610bc0d0>, mutex=<thread.lock at remote 0x7f816124fea0>, not_full=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0x7f816124fea0>, acquire=<built-in method acquire of thread.lock object at remote 0x7f816124fea0>, _Condition__waiters=[], release=<built-in method release of thread.lock object at remote 0x7f816124fea0>) at remote 0x7f81610bc060>, not_empty=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0x7...(truncated), throwflag=0) at ../Python/ceval.c:3009
#15 0x00005624534b2291 in fast_function (func=<function at remote 0x7f81620ba990>, pp_stack=0x7f815bffe778, n=1, na=1, nk=0)
at ../Python/ceval.c:4457
#16 0x00005624534b1f8a in call_function (pp_stack=0x7f815bffe778, oparg=0) at ../Python/ceval.c:4392
#17 0x00005624534ac45d in PyEval_EvalFrameEx (
f=Frame 0x7f81610b9df0, for file /usr/lib/python2.7/threading.py, line 774, in __bootstrap (self=<WriterThread(_Thread__ident=140193571010304, cmdQueue=<Queue(unfinished_tasks=5, queue=<collections.deque at remote 0x7f8161078f60>, maxsize=0, all_tasks_done=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0x7f816124fea0>, acquire=<built-in method acquire of thread.lock object at remote 0x7f816124fea0>, _Condition__waiters=[], release=<built-in method release of thread.lock object at remote 0x7f816124fea0>) at remote 0x7f81610bc0d0>, mutex=<thread.lock at remote 0x7f816124fea0>, not_full=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0x7f816124fea0>, acquire=<built-in method acquire of thread.lock object at remote 0x7f816124fea0>, _Condition__waiters=[], release=<built-in method release of thread.lock object at remote 0x7f816124fea0>) at remote 0x7f81610bc060>, not_empty=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0x7f81612...(truncated), throwflag=0) at ../Python/ceval.c:3009
#18 0x00005624534af42c in PyEval_EvalCodeEx (co=0x7f8162174ca0,
globals={'current_thread': None, '_BoundedSemaphore': None, 'currentThread': None, '_Timer': None, '_format_exc': None, 'Semaphore': None, '_deque': None, 'activeCount': None, '_profile_hook': None, '_sleep': None, '_trace_hook': None, 'ThreadError': None, '_enumerate': None, '_start_new_thread': None, 'BoundedSemaphore': None, '_shutdown': None, '__all__': None, '_original_start_new_thread': None, '_Event': None, 'active_count': None, '__package__': None, '_Condition': None, '_RLock': None, '_test': None, 'local': None, '__doc__': None, 'Condition': None, '_Verbose': None, '_DummyThread': None, 'Thread': None, 'warnings': None, '__builtins__': {'bytearray': None, 'IndexError': None, 'all': None, 'help': None, 'vars': None, 'SyntaxError': None, 'unicode': None, 'UnicodeDecodeError': None, 'memoryview': None, 'isinstance': None, 'copyright': None, 'NameError': None, 'BytesWarning': None, 'dict': None, 'input': None, 'oct': None, 'bin': None, 'SystemExit': None, 'StandardError': None, 'format': None, 'repr': None, 'sor...(truncated), locals=0x0, args=0x7f81610b4788, argcount=1, kws=0x0,
kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../Python/ceval.c:3604
#19 0x00005624533e31c3 in function_call (func=<function at remote 0x7f81620ba840>,
arg=(<WriterThread(_Thread__ident=140193571010304, cmdQueue=<Queue(unfinished_tasks=5, queue=<collections.deque at remote 0x7f8161078f60>, maxsize=0, all_tasks_done=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0x7f816124fea0>, acquire=<built-in method acquire of thread.lock object at remote 0x7f816124fea0>, _Condition__waiters=[], release=<built-in method release of thread.lock object at remote 0x7f816124fea0>) at remote 0x7f81610bc0d0>, mutex=<thread.lock at remote 0x7f816124fea0>, not_full=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0x7f816124fea0>, acquire=<built-in method acquire of thread.lock object at remote 0x7f816124fea0>, _Condition__waiters=[], release=<built-in method release of thread.lock object at remote 0x7f816124fea0>) at remote 0x7f81610bc060>, not_empty=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0x7f816124fea0>, acquire=<built-in method acquire of thread.lock object at remote 0x7f816124fea0>, _Con...(truncated), kw=0x0) at ../Objects/funcobject.c:523
#20 0x00005624533a3730 in PyObject_Call (func=<function at remote 0x7f81620ba840>,
arg=(<WriterThread(_Thread__ident=140193571010304, cmdQueue=<Queue(unfinished_tasks=5, queue=<collections.deque at remote 0x7f8161078f60>, maxsize=0, all_tasks_done=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0x7f816124fea0>, acquire=<built-in method acquire of thread.lock object at remote 0x7f816124fea0>, _Condition__waiters=[], release=<built-in method release of thread.lock object at remote 0x7f816124fea0>) at remote 0x7f81610bc0d0>, mutex=<thread.lock at remote 0x7f816124fea0>, not_full=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0x7f816124fea0>, acquire=<built-in method acquire of thread.lock object at remote 0x7f816124fea0>, _Condition__waiters=[], release=<built-in method release of thread.lock object at remote 0x7f816124fea0>) at remote 0x7f81610bc060>, not_empty=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0x7f816124fea0>, acquire=<built-in method acquire of thread.lock object at remote 0x7f816124fea0>, _Con...(truncated), kw=0x0) at ../Objects/abstract.c:2544
#21 0x00005624533bf50c in instancemethod_call (func=<function at remote 0x7f81620ba840>,
arg=(<WriterThread(_Thread__ident=140193571010304, cmdQueue=<Queue(unfinished_tasks=5, queue=<collections.deque at remote 0x7f8161078f60>, maxsize=0, all_tasks_done=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0x7f816124fea0>, acquire=<built-in method acquire of thread.lock object at remote 0x7f816124fea0>, _Condition__waiters=[], release=<built-in method release of thread.lock object at remote 0x7f816124fea0>) at remote 0x7f81610bc0d0>, mutex=<thread.lock at remote 0x7f816124fea0>, not_full=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0x7f816124fea0>, acquire=<built-in method acquire of thread.lock object at remote 0x7f816124fea0>, _Condition__waiters=[], release=<built-in method release of thread.lock object at remote 0x7f816124fea0>) at remote 0x7f81610bc060>, not_empty=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0x7f816124fea0>, acquire=<built-in method acquire of thread.lock object at remote 0x7f816124fea0>, _Con...(truncated), kw=0x0) at ../Objects/classobject.c:2600
#22 0x00005624533a3730 in PyObject_Call (func=<instancemethod at remote 0x7f81610b7260>, arg=(), kw=0x0) at ../Objects/abstract.c:2544
#23 0x00005624534b14d3 in PyEval_CallObjectWithKeywords (func=<instancemethod at remote 0x7f81610b7260>, arg=(), kw=0x0)
at ../Python/ceval.c:4241
#24 0x00005624535ad097 in t_bootstrap (boot_raw=0x7f81615efaa8) at ../Modules/threadmodule.c:620
#25 0x00007f8162ac6f2a in start_thread (arg=0x7f815bfff700) at pthread_create.c:463
#26 0x00007f816263dedf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
The failing tstate object being mentioned initially looks like this:
(gdb) print *tstate
$1 = {next = 0x562453b13590, interp = 0x56245391a280, frame = 0x0, recursion_depth = 8, tracing = 0, use_tracing = 0,
c_profilefunc = 0x0, c_tracefunc = 0x0, c_profileobj = 0x0, c_traceobj = 0x0, curexc_type = <type at remote 0x562453633660>,
curexc_value = "local variable 'self' referenced before assignment", curexc_traceback = <traceback at remote 0x7f8162130768>,
exc_type = 0x0, exc_value = 0x0, exc_traceback = 0x0, dict = 0x0, tick_counter = 153, gilstate_counter = 1, async_exc = 0x0,
thread_id = 140193571010304, trash_delete_nesting = 0, trash_delete_later = 0x0}
So the NULL pointer being dereferenced is tstate->frame, as hinted to by GDB.
I do not know the Python internals well enough to dig much deeper than this for now, but maybe this helps? If not, let me know - I have the core dump readily available to debug this further. (Can unfortunately provide a copy of the core dump per se since it likely contains credentials/other sensitive info.)
More details:
VSCode extension version: 2018.8.0
python2.7-dbg Debian package version: 2.7.15-4 |
➜
