classification
Title: SIGSEGV on interpreter shutdown, with daemon threads running wild
Type: crash Stage: resolved
Components: Interpreter Core Versions: Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder: PyEval_AcquireLock() and PyEval_AcquireThread() do not handle runtime finalization properly.
View: 36475
Assigned To: Nosy List: A. Skrobov, pitrou, tim.peters, vstinner
Priority: normal Keywords:

Created on 2015-03-05 19:56 by A. Skrobov, last changed 2020-03-09 10:53 by vstinner. This issue is now closed.

Messages (5)
msg237283 - (view) Author: A. Skrobov (A. Skrobov) * Date: 2015-03-05 19:56
I'm observing that this line of code:

https://hg.python.org/cpython/file/ec9bffc35cad/Python/ceval.c#l3010

-- causes a SIGSEGV on interpreter shutdown, after running some really convoluted Python code with daemon threads running wild.

At the time of the crash, tstate->frame==NULL, and tstate->curexc_type points to an exceptions.UnboundLocalError

I can reliably reproduce the crash, but unfortunately I cannot provide a standalone reproducer.

I'm speculating that this case is due to some daemon threads carrying on running while their variables are being destroyed from underneath them.


The traceback is probably of little use, but adding it nevertheless:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x4dc15940 (LWP 10317)]
0x00000000004d5c00 in PyEval_EvalFrameEx (f=0xc0da70, throwflag=0) at Python/ceval.c:3010
3010	    if (tstate->frame->f_exc_type != NULL)
(gdb) bt                                       
#0  0x00000000004d5c00 in PyEval_EvalFrameEx (f=0xc0da70, throwflag=0) at Python/ceval.c:3010
#1  0x00000000004d6db2 in PyEval_EvalCodeEx (co=0x2aaaabe1d510, globals=0x2aaaab9c62f0, locals=0x0, args=0x2aaaaf6e9bf8, argcount=2, kws=0x2aaaaf6e9c08, kwcount=0, defs=0x2aaaabe2fb78, defcount=1, closure=0x0)
    at Python/ceval.c:3267
#2  0x00000000004d9ac9 in fast_function (func=0x2aaaabe37648, pp_stack=0x4dc13c90, n=2, na=2, nk=0) at Python/ceval.c:4131
#3  0x00000000004d96d8 in call_function (pp_stack=0x4dc13c90, oparg=1) at Python/ceval.c:4056
#4  0x00000000004d4258 in PyEval_EvalFrameEx (f=0x2aaaaf6e9a60, throwflag=0) at Python/ceval.c:2681
#5  0x00000000004d6db2 in PyEval_EvalCodeEx (co=0x2aaaabe26250, globals=0x2aaaab9c62f0, locals=0x0, args=0x2aaaaac81f48, argcount=2, kws=0x2aaaaac81f58, kwcount=0, defs=0x2aaaabe2fe88, defcount=1, closure=0x0)
    at Python/ceval.c:3267
#6  0x00000000004d9ac9 in fast_function (func=0x2aaaabe39108, pp_stack=0x4dc14230, n=2, na=2, nk=0) at Python/ceval.c:4131
#7  0x00000000004d96d8 in call_function (pp_stack=0x4dc14230, oparg=1) at Python/ceval.c:4056
#8  0x00000000004d4258 in PyEval_EvalFrameEx (f=0x2aaaaac81db8, throwflag=0) at Python/ceval.c:2681
#9  0x00000000004d99af in fast_function (func=0x2aaaaee97840, pp_stack=0x4dc14620, n=1, na=1, nk=0) at Python/ceval.c:4121
#10 0x00000000004d96d8 in call_function (pp_stack=0x4dc14620, oparg=0) at Python/ceval.c:4056
#11 0x00000000004d4258 in PyEval_EvalFrameEx (f=0xc47fe0, throwflag=0) at Python/ceval.c:2681
#12 0x00000000004d99af in fast_function (func=0x2aaaabe396f0, pp_stack=0x4dc14a10, n=1, na=1, nk=0) at Python/ceval.c:4121
#13 0x00000000004d96d8 in call_function (pp_stack=0x4dc14a10, oparg=0) at Python/ceval.c:4056
#14 0x00000000004d4258 in PyEval_EvalFrameEx (f=0x2aaaaf67bdf0, throwflag=0) at Python/ceval.c:2681
#15 0x00000000004d6db2 in PyEval_EvalCodeEx (co=0x2aaaabe26880, globals=0x2aaaab9c62f0, locals=0x0, args=0x2aaaaef75fd8, argcount=1, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3267
#16 0x000000000056346e in function_call (func=0x2aaaabe395a0, arg=0x2aaaaef75fb0, kw=0x0) at Objects/funcobject.c:526
#17 0x000000000042182c in PyObject_Call (func=0x2aaaabe395a0, arg=0x2aaaaef75fb0, kw=0x0) at Objects/abstract.c:2529
#18 0x000000000042c0d0 in instancemethod_call (func=0x2aaaabe395a0, arg=0x2aaaaef75fb0, kw=0x0) at Objects/classobject.c:2602
#19 0x000000000042182c in PyObject_Call (func=0x2aaaaeec5060, arg=0x2aaaaaacf060, kw=0x0) at Objects/abstract.c:2529
#20 0x00000000004d8ceb in PyEval_CallObjectWithKeywords (func=0x2aaaaeec5060, arg=0x2aaaaaacf060, kw=0x0) at Python/ceval.c:3904
#21 0x000000000052358b in t_bootstrap (boot_raw=0x2aaaafab9d78) at ./Modules/threadmodule.c:614
#22 0x000000342f00677d in start_thread () from /lib64/libpthread.so.0
#23 0x000000342e4d49ad in clone () from /lib64/libc.so.6
msg237311 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2015-03-06 00:27
Yes, daemon threads are really dangerous because they may keep running while the interpreter has started releasing critical resources. Things have improved in 3.x compared to 2.x, though.
msg237354 - (view) Author: A. Skrobov (A. Skrobov) * Date: 2015-03-06 13:53
That's right; and working around this issue, by taming the daemon threads a bit, wasn't too difficult.

Still, if the daemon threads are part of the language, they shouldn't crash the interpreter process, I suppose?
msg237362 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2015-03-06 15:52
Nothing should ever crash the interpreter :-)  So this is a thoroughly legitimate bug report.

However, there's no way to guess whether _this_ crasher is easy to fix, or next to impossible.  Without a test program to provoke the error, there's little to go on.  If it were a common problem with daemon threads, I'd expect at least several reports of similar behavior over the decades.  So chances seem to favor that there's something unique about the specifics of what your daemon threads were doing to provoke it - and/or timing quirks specific to your platform.
msg363713 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-03-09 10:53
I believe that this issue cannot occur in Python 3.8 thanks for bpo-36475.

According to the backtrace, a thread does crash in PyEval_EvalFrameEx(). I understand that it's a daemon thread. To execute PyEval_EvalFrameEx(), a daemon thread must hold the GIL. With bpo-36475, a daemon thread now exits immediately instead of taking the GIL.
History
Date User Action Args
2020-03-09 10:53:14vstinnersetstatus: open -> closed

superseder: PyEval_AcquireLock() and PyEval_AcquireThread() do not handle runtime finalization properly.

nosy: + vstinner
messages: + msg363713
resolution: fixed
stage: resolved
2015-03-06 15:52:20tim.peterssetnosy: + tim.peters
messages: + msg237362
2015-03-06 13:53:59A. Skrobovsetmessages: + msg237354
2015-03-06 00:27:16pitrousetnosy: + pitrou
messages: + msg237311
2015-03-05 19:56:55A. Skrobovcreate