classification
Title: Ansible: _PyImport_LoadDynamicModuleWithSpec() crash on an invalid object (UNREF invalid object)
Type: crash Stage: resolved
Components: Interpreter Core Versions: Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: mdk, vstinner
Priority: normal Keywords:

Created on 2018-07-24 09:58 by mdk, last changed 2018-11-06 12:56 by mdk. This issue is now closed.

Messages (4)
msg322285 - (view) Author: Julien Palard (mdk) * (Python committer) Date: 2018-07-24 09:58
On a freshly installed python3.6.6 using "pyenv install --debug 3.6.6", on a freshly created venv, after running a `pip install ansible`, I'm getting exactly those dependencies:

ansible==2.6.1
asn1crypto==0.24.0
bcrypt==3.1.4
cffi==1.11.5
cryptography==2.3
idna==2.7
Jinja2==2.10
MarkupSafe==1.0
paramiko==2.4.1
pyasn1==0.4.3
pycparser==2.18
PyNaCl==1.2.1
PyYAML==3.13
six==1.11.0

If I run:

    $ python $(which ansible-playbook)

I'm getting:

    * ob
    object  : <refcnt 0 at 0x7f2600ecc820>
    type    : bytes
    refcount: 0
    address : 0x7f2600ecc820
    * op->_ob_prev->_ob_next
    object  : <refcnt 0 at 0x7f2600ecc81f>Segmentation fault

An excerpt of the stack trace:

#0  _PyObject_Dump (op=0x7ffff0e667bf) at Objects/object.c:450
#1  0x000055555566a4cf in _Py_ForgetReference (op=0x7ffff0e667c0) at Objects/object.c:1767
#2  0x000055555566a579 in _Py_Dealloc (op=0x7ffff0e667c0) at Objects/object.c:1790
#3  0x00005555557982c2 in _PyImport_LoadDynamicModuleWithSpec (spec=0x7ffff130ec20, fp=0x0) at ./Python/importdl.c:227
#4  0x0000555555797439 in _imp_create_dynamic_impl (module=0x7ffff7f3e558, spec=0x7ffff130ec20, file=0x0) at Python/import.c:1994
#5  0x0000555555791760 in _imp_create_dynamic (module=0x7ffff7f3e558, args=0x7ffff1307e90) at Python/clinic/import.c.h:289
#6  0x00005555556618fc in PyCFunction_Call (func=0x7ffff7f43aa8, args=0x7ffff1307e90, kwds=0x7ffff126bb20) at Objects/methodobject.c:114
#7  0x0000555555768e90 in do_call_core (func=0x7ffff7f43aa8, callargs=0x7ffff1307e90, kwdict=0x7ffff126bb20) at Python/ceval.c:5095
#8  0x0000555555760fab in _PyEval_EvalFrameDefault (f=0x7ffff0ecb058, throwflag=0) at Python/ceval.c:3397
#9  0x000055555574e22a in PyEval_EvalFrameEx (f=0x7ffff0ecb058, throwflag=0) at Python/ceval.c:754


And what I see in gdb:

(gdb) p (PyObject*)(op)
$1 = (PyObject *) 0x7ffff0e667bf
(gdb) p *(PyObject*)(op)
$2 = {_ob_next = 0x7ffff1307e90fb, _ob_prev = 0x7ffff12df95000, ob_refcnt = 0, ob_type = 0x555555ad9f8000}
(gdb) p (((PyObject*)(op))->ob_type)
$3 = (struct _typeobject *) 0x555555ad9f8000
(gdb) p *(((PyObject*)(op))->ob_type)
Cannot access memory at address 0x555555ad9f8000

I now have to go eat, but don't hesitate to ask me further investigations.
msg322286 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-07-24 10:11
Recently, there was a discussion to find to detect earlier inconsistent objects:
https://mail.python.org/pipermail/python-dev/2018-June/153857.html

In the meanwhile, if you have time, you can try to run Ansible with a Python compiled in debug mode (./configure --with-pydebug).
msg322295 - (view) Author: Julien Palard (mdk) * (Python committer) Date: 2018-07-24 12:18
I think it's already compiled in debug mode (https://github.com/pyenv/pyenv/blob/c057a80c8296a7c694e4ef80ecbac0d0c169df7a/plugins/python-build/bin/python-build#L2050).

`-X dev` won't help (it's a 3.6.6 not a 3.7).

The debug come from _Py_ForgetReference which prints op, op->_ob_prev->_ob_next and op->_ob_next->_ob_prev, crash occurs on the 2nd one, which make sense according to:

    (gdb) p *(((PyObject*)(op)))
    $12 = {_ob_next = 0x7ffff1307e90, _ob_prev = 0x7ffff12df950, ob_refcnt = 0, ob_type = 0x555555ad9f80 <PyBytes_Type>}
    (gdb) p *(((PyObject*)(op->_ob_prev)))
    $13 = {_ob_next = 0x7ffff0e667bf, _ob_prev = 0x7ffff10c58b0, ob_refcnt = 1, ob_type = 0x555555ae89a0 <PyLong_Type>}
    (gdb) p *(((PyObject*)(op->_ob_prev->_ob_next)))
    $14 = {_ob_next = 0x7ffff1307e90fb, _ob_prev = 0x7ffff12df95000, ob_refcnt = 0, ob_type = 0x555555ad9f8000}
msg329353 - (view) Author: Julien Palard (mdk) * (Python committer) Date: 2018-11-06 12:56
Can't reproduce in 3.6.7.
History
Date User Action Args
2018-11-06 12:56:13mdksetstatus: open -> closed
resolution: fixed
messages: + msg329353

stage: resolved
2018-07-24 12:18:41mdksetmessages: + msg322295
2018-07-24 10:11:37vstinnersetmessages: + msg322286
2018-07-24 10:10:12vstinnersettitle: Segmentation fault in _PyObject_Dump -> Ansible: _PyImport_LoadDynamicModuleWithSpec() crash on an invalid object (UNREF invalid object)
2018-07-24 09:58:44mdkcreate