classification
Title: Segfault in gcmodule.c:360 visit_decref
Type: Stage: resolved
Components: Interpreter Core Versions: Python 2.7
process
Status: closed Resolution: works for me
Dependencies: Superseder:
Assigned To: Nosy List: lgarrett, mdk, ned.deily, skrah, tjh, vstinner
Priority: normal Keywords:

Created on 2016-07-17 11:31 by mdk, last changed 2019-10-08 08:06 by mdk. This issue is now closed.

Messages (15)
msg270633 - (view) Author: Julien Palard (mdk) * (Python committer) Date: 2016-07-17 11:31
Was running `python -m pip install ansible` when I got a "segmentation fault".

bt is:

(gdb) bt
#0  0x00000000004a7ec4 in visit_decref () at ../Modules/gcmodule.c:360
#1  0x00000000004a7fa9 in dict_traverse () at ../Objects/dictobject.c:2144
#2  0x00000000004a6f88 in subtract_refs () at ../Modules/gcmodule.c:385
#3  collect.lto_priv () at ../Modules/gcmodule.c:925
#4  0x0000000000500d2e in PyGC_Collect () at ../Modules/gcmodule.c:1440
#5  0x0000000000500184 in Py_Finalize () at ../Python/pythonrun.c:448
#6  0x0000000000525148 in Py_Exit (sts=0) at ../Python/pythonrun.c:1783
#7  0x0000000000522553 in handle_system_exit () at ../Python/pythonrun.c:1151
#8  0x0000000000521e46 in PyErr_PrintEx () at ../Python/pythonrun.c:1161
#9  0x0000000000520dc9 in RunModule.lto_priv.1258 () at ../Modules/main.c:194
#10 0x000000000049de26 in Py_Main () at ../Modules/main.c:587
#11 0x00007fddb03e25f0 in __libc_start_main (main=0x49d710 <main>, argc=6, argv=0x7ffdfcb8cfe8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, 
    stack_end=0x7ffdfcb8cfd8) at libc-start.c:291
#12 0x000000000049d639 in _start ()

Problem looks obvious:
(gdb) p (((PyObject*)(op))->ob_type)
$86202 = (struct _typeobject *) 0x0

But `subtract_refs()` from `Modules/gcmodule.c:385` is iterating a very long chained list, and I'm not sure it's of any help iterating over it.

Can provide core dump if of any help.

Python version: Python 2.7.12
PIP version: pip 8.1.2 from /usr/lib/python2.7/dist-packages (python 2.7)

Looks reproductible:

```
$ python -m pip install --user ansible
Collecting ansible
Collecting jinja2 (from ansible)
  Using cached Jinja2-2.8-py2.py3-none-any.whl
Collecting setuptools (from ansible)
  Using cached setuptools-24.0.3-py2.py3-none-any.whl
Collecting PyYAML (from ansible)
Collecting pycrypto>=2.6 (from ansible)
Collecting paramiko (from ansible)
  Using cached paramiko-2.0.1-py2.py3-none-any.whl
Collecting MarkupSafe (from jinja2->ansible)
Collecting cryptography>=1.1 (from paramiko->ansible)
Collecting pyasn1>=0.1.7 (from paramiko->ansible)
  Using cached pyasn1-0.1.9-py2.py3-none-any.whl
Collecting enum34 (from cryptography>=1.1->paramiko->ansible)
  Using cached enum34-1.1.6-py2-none-any.whl
Collecting idna>=2.0 (from cryptography>=1.1->paramiko->ansible)
  Using cached idna-2.1-py2.py3-none-any.whl
Collecting cffi>=1.4.1 (from cryptography>=1.1->paramiko->ansible)
  Using cached cffi-1.7.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting ipaddress (from cryptography>=1.1->paramiko->ansible)
  Using cached ipaddress-1.0.16-py27-none-any.whl
Collecting six>=1.4.1 (from cryptography>=1.1->paramiko->ansible)
  Using cached six-1.10.0-py2.py3-none-any.whl
Collecting pycparser (from cffi>=1.4.1->cryptography>=1.1->paramiko->ansible)
Installing collected packages: MarkupSafe, jinja2, setuptools, PyYAML, pycrypto, enum34, idna, pycparser, cffi, ipaddress, six, pyasn1, cryptography, paramiko, ansible
Successfully installed MarkupSafe-0.23 PyYAML-3.11 ansible-2.1.0.0 cffi-1.7.0 cryptography-1.4 enum34-1.1.6 idna-2.1 ipaddress-1.0.16 jinja2-2.8 paramiko-2.0.1 pyasn1-0.1.9 pycparser-2.14 pycrypto-2.6.1 setuptools-24.0.3 six-1.10.0
Segmentation fault (core dumped)
```

I tried installing only a few packages to see if one in particular causes the bug and yes, cffi may be the one:

```
$ python -m pip install --user cffi
Collecting cffi
  Using cached cffi-1.7.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting pycparser (from cffi)
Installing collected packages: pycparser, cffi
Successfully installed cffi-1.7.0 pycparser-2.14
Segmentation fault (core dumped)
```

But not pycparser alone:

```
$ python -m pip install --user pycparser
Collecting pycparser
Installing collected packages: pycparser
Successfully installed pycparser-2.14
```
msg270657 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-07-17 18:04
Try installing cffi from source rather than using the pre-compiled wheel:


python -m pip uninstall cffi
python -m pip install -v --no-use-wheel --user cffi

If that works, there could be an incompatibility between the Python 2.7.12 you are using and the C extensions in the wheel.  What platform are you on and what is the source of the Python 2.7?
msg270659 - (view) Author: Julien Palard (mdk) * (Python committer) Date: 2016-07-17 18:14
@Ned

Despite the segfault, cffi installs well, the segfault occurs during a garbage collect, very late, inside Py_Exit according to the backtrace, so it has no impact.

> What platform are you on and what is the source of the Python 2.7

Debian stretch, python (and sources) 2.7.12-1 from Debian testing packages, pip 8.1.2 from Debian packages too, version 8.1.2-2.
msg270660 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-07-17 18:21
It would still be interesting to know whether you see the same behavior with building from source.  If a cffi extension module is invoked during the install process, it might screw things up.  Otherwise, unless you can reproduce the problem with a vanilla Python 2.7.12 built from scratch, you'll probably have to pursue this with the Debian folks.
msg270661 - (view) Author: Julien Palard (mdk) * (Python committer) Date: 2016-07-17 18:24
@ned Oh ok. I just tried, with `--no-use-wheel`: no segfault.
msg270662 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2016-07-17 18:28
Which makes me think that --no-use-wheel should be the default in pip ...

As a Linux user I'm *very* uneasy about this whole binary wheel thing.
msg270663 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-07-17 18:29
Thanks for trying!  Perhaps the first thing to do is to check with the cffi project; perhaps Armin will recognize something.  I'm going to mark this issue as "closed" but feel free to re-open it if a problem with Python turns up.
msg270664 - (view) Author: Julien Palard (mdk) * (Python committer) Date: 2016-07-17 18:41
CFFI issue, like for the record: https://bitbucket.org/cffi/cffi/issues/272/segfault-while-installing-via-pip
msg270794 - (view) Author: Julien Palard (mdk) * (Python committer) Date: 2016-07-18 22:01
At cffi they thinks the bug may be from cpython or pip, as

```
$ python -m pip install --user cffi
Collecting cffi
  Using cached cffi-1.7.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting pycparser (from cffi)
Installing collected packages: pycparser, cffi
Successfully installed cffi-1.7.0 pycparser-2.14
Segmentation fault (core dumped)
```

may not import cffi.
c.f.: https://bitbucket.org/cffi/cffi/issues/272/segfault-while-installing-via-pip#comment-29256302
msg270796 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-07-18 22:48
IMO, there are too many possible scenarios here where a segfault could be caused by the Debian patches, by pip, by wheel, or by cffi - too many to assume a CPython problem.

FWIW, I am unable to reproduce the segfault on a pretty current Debian testing system using a vanilla built-from-source Python 2.7.12 and the currently available wheel on PyPI and, for that matter, I'm unable to reproduce using a current Debian python2.7.12-1/pip 8.1.2-2/wheel 0.29.0-1.  Unless someone can provide a reproducible test case that points to a Python problem using a vanilla Python 2.7.12, the issue will have to remain closed as we just don't have the resources to try to isolate every problem like this.  Sorry!
msg280612 - (view) Author: Lee Garrett (lgarrett) Date: 2016-11-11 21:17
In case someone reaches this bug report via search engine:

apt-get remove python-cffi-backend

fixed this problem for me.
msg334084 - (view) Author: tjh (tjh) Date: 2019-01-20 14:22
If you're here looking for a solution, deleting previously installed deps helped me. In my case this was:

$ rm -rf ~/.local/lib/python2.7/site-packages/  # path may differ depending where you install these
$ pip install cffi
msg354155 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-10-08 00:33
Julien: I'm working on enhancement of debug traces on visit_decref(). Are you still able to reproduce the crash in 2019?

I failed to reproduce the bug in a virtual environment with:

* Python 2.7.14
* pip 8.1.2
* cffi 1.7.0

$ virtualenv -p python2 venv
Running virtualenv with interpreter /usr/bin/python2
New python executable in /home/haypo/venv/bin/python2
Also creating executable in /home/haypo/venv/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.

$ venv/bin/python -m pip install pip==8.1.2
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting pip==8.1.2
  Using cached https://files.pythonhosted.org/packages/9c/32/004ce0852e0a127f07f358b715015763273799bd798956fa930814b60f39/pip-8.1.2-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 19.2.3
    Uninstalling pip-19.2.3:
      Successfully uninstalled pip-19.2.3
Successfully installed pip-8.1.2

$ venv/bin/python -m pip install cffi==1.7.0
Collecting cffi==1.7.0
  Using cached https://files.pythonhosted.org/packages/46/26/2413bf923b1de47d9e9cf48f248dec1a16027a7773452315f34746253c35/cffi-1.7.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting pycparser (from cffi==1.7.0)
Installing collected packages: pycparser, cffi
Successfully installed cffi-1.7.0 pycparser-2.19
You are using pip version 8.1.2, however version 19.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
msg354174 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-10-08 07:54
> Julien: I'm working on enhancement of debug traces on visit_decref(). Are you still able to reproduce the crash in 2019?

I asked him on IRC. No, he cannot reproduce the bug anymore.
msg354175 - (view) Author: Julien Palard (mdk) * (Python committer) Date: 2019-10-08 08:06
I tried again with the same version of pip and cffi on cpython 2.7.16 (no 13, I have no VM atm to compile a 2.7.13) but no segfault this time.
History
Date User Action Args
2019-10-08 08:06:51mdksetmessages: + msg354175
2019-10-08 07:54:13vstinnersetmessages: + msg354174
2019-10-08 00:33:14vstinnersetnosy: + vstinner
messages: + msg354155
2019-01-20 14:22:44tjhsetnosy: + tjh
messages: + msg334084
2016-11-11 21:17:40lgarrettsetnosy: + lgarrett
messages: + msg280612
2016-07-18 22:48:49ned.deilysetstatus: open -> closed
resolution: third party -> works for me
messages: + msg270796
2016-07-18 22:01:22mdksetstatus: closed -> open

messages: + msg270794
2016-07-17 18:41:35mdksetmessages: + msg270664
2016-07-17 18:29:29ned.deilysetstatus: open -> closed
resolution: third party
messages: + msg270663

stage: resolved
2016-07-17 18:28:04skrahsetnosy: + skrah
messages: + msg270662
2016-07-17 18:24:14mdksetmessages: + msg270661
2016-07-17 18:21:33ned.deilysetmessages: + msg270660
2016-07-17 18:14:20mdksetmessages: + msg270659
2016-07-17 18:04:43ned.deilysetnosy: + ned.deily
messages: + msg270657
2016-07-17 11:31:26mdkcreate