classification
Title: Crash in gen_send_ex(): _PyErr_GetTopmostException() returns freed memory
Type: Stage: resolved
Components: Interpreter Core Versions: Python 3.8, Python 3.7
process
Status: closed Resolution: third party
Dependencies: Superseder:
Assigned To: Nosy List: Mark.Shannon, hroncok, ned.deily, serhiy.storchaka, vstinner, yselivanov
Priority: Keywords: 3.7regression

Created on 2018-06-29 12:35 by vstinner, last changed 2018-07-03 15:36 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
dagpool_test.py vstinner, 2018-06-29 12:35
Messages (6)
msg320712 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-06-29 12:35
Attached dagpool_test.py script triggers a crash on Python 3.7, but works well on Python 3.6. The script is a highly simplified code of tests/dagpool_tests.py of eventlet.

To reproduce the crash, you only need a single dependency: greenlet. Example using a venv:

    python3.7 -m venv ENV
    ENV/bin/python -m pip install greenlet

Then run the script:

    $ test_venv/bin/python dagpool_test.py 
    Segmentation fault (core dumped)

eventlet bug report:
https://github.com/eventlet/eventlet/issues/475


I suspected that the bug is caused by the new exc_info attribute of PyThreadState: commit ae3087c6382011c47db82fea4d05f8bbf514265d.

$ gdb -args test_venv/bin/python -X faulthandler dagpool_test.py
(gdb) run

Program received signal SIGSEGV, Segmentation fault.
0x000000000056c9d2 in PyErr_SetObject (exception=<type at remote 0x9709a0>, value=0x0) at Python/errors.c:101
101	        Py_INCREF(exc_value);
(gdb) where
#0  0x000000000056c9d2 in PyErr_SetObject (exception=<type at remote 0x9709a0>, value=0x0) at Python/errors.c:101
#1  0x000000000056cd4e in PyErr_SetNone (exception=<type at remote 0x9709a0>) at Python/errors.c:162
#2  0x000000000067cb0c in gen_send_ex (gen=0x7fffea651d78, arg=0x0, exc=0, closing=0) at Objects/genobject.c:241
#3  0x000000000067dd86 in gen_iternext (gen=0x7fffea651d78) at Objects/genobject.c:542
#4  0x00000000005461b1 in _PyEval_EvalFrameDefault (...)
...

(gdb) p tstate
$1 = (PyThreadState *) 0xa132a0

(gdb) p tstate->exc_info
$2 = (_PyErr_StackItem *) 0x7fffea651930

(gdb) p *tstate->exc_info
$3 = {
  exc_type = 0x0, 
  exc_value = 0x0, 
  exc_traceback = 0x0, 
  previous_item = 0x7fffea651d20
}

(gdb) p *tstate->exc_info->previous_item
$4 = {
  exc_type = <unknown at remote 0xdbdbdbdbdbdbdbdb>, 
  exc_value = <unknown at remote 0xdbdbdbdbdbdbdbdb>, 
  exc_traceback = <unknown at remote 0xdbdbdbdbdbdbdbdb>, 
  previous_item = 0xdbdbdbdbdbdbdbdb
}
msg320714 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-06-29 12:40
> I suspected that the bug is caused by the new exc_info attribute of PyThreadState: commit ae3087c6382011c47db82fea4d05f8bbf514265d.

The change comes from bpo-25612.
msg320832 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2018-07-01 08:44
Strange. 
I could only reproduce this intermittently with 3.7 build.
But with a debug build, I cannot reproduce this at all.
Rebuilding Python (optimised, but without PGO) I cannot reproduce at all now.
msg320833 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2018-07-01 09:04
This looks like a Greenlet bug to me.
Possibly https://github.com/python-greenlet/greenlet/commit/780a12b51fcef9adcc4f2c9a4cc5b05c2d652ba4 is incomplete.

It is not clear to me why greenlets hold their own exception state, but I suspect that if they didn't then this issue might go away.
msg320854 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-07-02 08:21
> I could only reproduce this intermittently with 3.7 build.

Oh, I forgot to mention that I have to run the test many times to reproduce the crash. Between 1 and 10 times.

I used the master branch compiled in debug mode to report this issue.
msg320987 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-07-03 15:36
It's not a bug in Python but in greenlet:

* greenlet bug: https://github.com/python-greenlet/greenlet/issues/131
* my greenlet fix: https://github.com/python-greenlet/greenlet/pull/132
* Red Hat bug: https://bugzilla.redhat.com/show_bug.cgi?id=1594248
History
Date User Action Args
2018-07-03 15:36:31vstinnersetstatus: open -> closed
priority: release blocker ->
messages: + msg320987

resolution: third party
stage: resolved
2018-07-02 08:21:50vstinnersetmessages: + msg320854
2018-07-01 09:04:37Mark.Shannonsetmessages: + msg320833
2018-07-01 08:44:06Mark.Shannonsetmessages: + msg320832
2018-07-01 08:32:12hroncoksetnosy: + hroncok
2018-06-29 12:52:25serhiy.storchakasetnosy: + serhiy.storchaka
2018-06-29 12:40:51vstinnersetmessages: + msg320714
2018-06-29 12:35:51vstinnercreate