Title: Segfault in gcmodule.c:360 visit_decref
Type: Stage: resolved
Components: Interpreter Core Versions: Python 2.7
Status: closed Resolution: works for me
Dependencies: Superseder:
Assigned To: Nosy List: lgarrett, mdk, ned.deily, skrah, tjh
Priority: normal Keywords:

Created on 2016-07-17 11:31 by mdk, last changed 2019-01-20 14:22 by tjh. This issue is now closed.

Messages (12)
msg270633 - (view) Author: Julien Palard (mdk) * (Python committer) Date: 2016-07-17 11:31
Was running `python -m pip install ansible` when I got a "segmentation fault".

bt is:

(gdb) bt
#0  0x00000000004a7ec4 in visit_decref () at ../Modules/gcmodule.c:360
#1  0x00000000004a7fa9 in dict_traverse () at ../Objects/dictobject.c:2144
#2  0x00000000004a6f88 in subtract_refs () at ../Modules/gcmodule.c:385
#3  collect.lto_priv () at ../Modules/gcmodule.c:925
#4  0x0000000000500d2e in PyGC_Collect () at ../Modules/gcmodule.c:1440
#5  0x0000000000500184 in Py_Finalize () at ../Python/pythonrun.c:448
#6  0x0000000000525148 in Py_Exit (sts=0) at ../Python/pythonrun.c:1783
#7  0x0000000000522553 in handle_system_exit () at ../Python/pythonrun.c:1151
#8  0x0000000000521e46 in PyErr_PrintEx () at ../Python/pythonrun.c:1161
#9  0x0000000000520dc9 in RunModule.lto_priv.1258 () at ../Modules/main.c:194
#10 0x000000000049de26 in Py_Main () at ../Modules/main.c:587
#11 0x00007fddb03e25f0 in __libc_start_main (main=0x49d710 <main>, argc=6, argv=0x7ffdfcb8cfe8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, 
    stack_end=0x7ffdfcb8cfd8) at libc-start.c:291
#12 0x000000000049d639 in _start ()

Problem looks obvious:
(gdb) p (((PyObject*)(op))->ob_type)
$86202 = (struct _typeobject *) 0x0

But `subtract_refs()` from `Modules/gcmodule.c:385` is iterating a very long chained list, and I'm not sure it's of any help iterating over it.

Can provide core dump if of any help.

Python version: Python 2.7.12
PIP version: pip 8.1.2 from /usr/lib/python2.7/dist-packages (python 2.7)

Looks reproductible:

$ python -m pip install --user ansible
Collecting ansible
Collecting jinja2 (from ansible)
  Using cached Jinja2-2.8-py2.py3-none-any.whl
Collecting setuptools (from ansible)
  Using cached setuptools-24.0.3-py2.py3-none-any.whl
Collecting PyYAML (from ansible)
Collecting pycrypto>=2.6 (from ansible)
Collecting paramiko (from ansible)
  Using cached paramiko-2.0.1-py2.py3-none-any.whl
Collecting MarkupSafe (from jinja2->ansible)
Collecting cryptography>=1.1 (from paramiko->ansible)
Collecting pyasn1>=0.1.7 (from paramiko->ansible)
  Using cached pyasn1-0.1.9-py2.py3-none-any.whl
Collecting enum34 (from cryptography>=1.1->paramiko->ansible)
  Using cached enum34-1.1.6-py2-none-any.whl
Collecting idna>=2.0 (from cryptography>=1.1->paramiko->ansible)
  Using cached idna-2.1-py2.py3-none-any.whl
Collecting cffi>=1.4.1 (from cryptography>=1.1->paramiko->ansible)
  Using cached cffi-1.7.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting ipaddress (from cryptography>=1.1->paramiko->ansible)
  Using cached ipaddress-1.0.16-py27-none-any.whl
Collecting six>=1.4.1 (from cryptography>=1.1->paramiko->ansible)
  Using cached six-1.10.0-py2.py3-none-any.whl
Collecting pycparser (from cffi>=1.4.1->cryptography>=1.1->paramiko->ansible)
Installing collected packages: MarkupSafe, jinja2, setuptools, PyYAML, pycrypto, enum34, idna, pycparser, cffi, ipaddress, six, pyasn1, cryptography, paramiko, ansible
Successfully installed MarkupSafe-0.23 PyYAML-3.11 ansible- cffi-1.7.0 cryptography-1.4 enum34-1.1.6 idna-2.1 ipaddress-1.0.16 jinja2-2.8 paramiko-2.0.1 pyasn1-0.1.9 pycparser-2.14 pycrypto-2.6.1 setuptools-24.0.3 six-1.10.0
Segmentation fault (core dumped)

I tried installing only a few packages to see if one in particular causes the bug and yes, cffi may be the one:

$ python -m pip install --user cffi
Collecting cffi
  Using cached cffi-1.7.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting pycparser (from cffi)
Installing collected packages: pycparser, cffi
Successfully installed cffi-1.7.0 pycparser-2.14
Segmentation fault (core dumped)

But not pycparser alone:

$ python -m pip install --user pycparser
Collecting pycparser
Installing collected packages: pycparser
Successfully installed pycparser-2.14
msg270657 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-07-17 18:04
Try installing cffi from source rather than using the pre-compiled wheel:

python -m pip uninstall cffi
python -m pip install -v --no-use-wheel --user cffi

If that works, there could be an incompatibility between the Python 2.7.12 you are using and the C extensions in the wheel.  What platform are you on and what is the source of the Python 2.7?
msg270659 - (view) Author: Julien Palard (mdk) * (Python committer) Date: 2016-07-17 18:14

Despite the segfault, cffi installs well, the segfault occurs during a garbage collect, very late, inside Py_Exit according to the backtrace, so it has no impact.

> What platform are you on and what is the source of the Python 2.7

Debian stretch, python (and sources) 2.7.12-1 from Debian testing packages, pip 8.1.2 from Debian packages too, version 8.1.2-2.
msg270660 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-07-17 18:21
It would still be interesting to know whether you see the same behavior with building from source.  If a cffi extension module is invoked during the install process, it might screw things up.  Otherwise, unless you can reproduce the problem with a vanilla Python 2.7.12 built from scratch, you'll probably have to pursue this with the Debian folks.
msg270661 - (view) Author: Julien Palard (mdk) * (Python committer) Date: 2016-07-17 18:24
@ned Oh ok. I just tried, with `--no-use-wheel`: no segfault.
msg270662 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2016-07-17 18:28
Which makes me think that --no-use-wheel should be the default in pip ...

As a Linux user I'm *very* uneasy about this whole binary wheel thing.
msg270663 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-07-17 18:29
Thanks for trying!  Perhaps the first thing to do is to check with the cffi project; perhaps Armin will recognize something.  I'm going to mark this issue as "closed" but feel free to re-open it if a problem with Python turns up.
msg270664 - (view) Author: Julien Palard (mdk) * (Python committer) Date: 2016-07-17 18:41
CFFI issue, like for the record:
msg270794 - (view) Author: Julien Palard (mdk) * (Python committer) Date: 2016-07-18 22:01
At cffi they thinks the bug may be from cpython or pip, as

$ python -m pip install --user cffi
Collecting cffi
  Using cached cffi-1.7.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting pycparser (from cffi)
Installing collected packages: pycparser, cffi
Successfully installed cffi-1.7.0 pycparser-2.14
Segmentation fault (core dumped)

may not import cffi.
msg270796 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-07-18 22:48
IMO, there are too many possible scenarios here where a segfault could be caused by the Debian patches, by pip, by wheel, or by cffi - too many to assume a CPython problem.

FWIW, I am unable to reproduce the segfault on a pretty current Debian testing system using a vanilla built-from-source Python 2.7.12 and the currently available wheel on PyPI and, for that matter, I'm unable to reproduce using a current Debian python2.7.12-1/pip 8.1.2-2/wheel 0.29.0-1.  Unless someone can provide a reproducible test case that points to a Python problem using a vanilla Python 2.7.12, the issue will have to remain closed as we just don't have the resources to try to isolate every problem like this.  Sorry!
msg280612 - (view) Author: Lee Garrett (lgarrett) Date: 2016-11-11 21:17
In case someone reaches this bug report via search engine:

apt-get remove python-cffi-backend

fixed this problem for me.
msg334084 - (view) Author: tjh (tjh) Date: 2019-01-20 14:22
If you're here looking for a solution, deleting previously installed deps helped me. In my case this was:

$ rm -rf ~/.local/lib/python2.7/site-packages/  # path may differ depending where you install these
$ pip install cffi
Date User Action Args
2019-01-20 14:22:44tjhsetnosy: + tjh
messages: + msg334084
2016-11-11 21:17:40lgarrettsetnosy: + lgarrett
messages: + msg280612
2016-07-18 22:48:49ned.deilysetstatus: open -> closed
resolution: third party -> works for me
messages: + msg270796
2016-07-18 22:01:22mdksetstatus: closed -> open

messages: + msg270794
2016-07-17 18:41:35mdksetmessages: + msg270664
2016-07-17 18:29:29ned.deilysetstatus: open -> closed
resolution: third party
messages: + msg270663

stage: resolved
2016-07-17 18:28:04skrahsetnosy: + skrah
messages: + msg270662
2016-07-17 18:24:14mdksetmessages: + msg270661
2016-07-17 18:21:33ned.deilysetmessages: + msg270660
2016-07-17 18:14:20mdksetmessages: + msg270659
2016-07-17 18:04:43ned.deilysetnosy: + ned.deily
messages: + msg270657
2016-07-17 11:31:26mdkcreate