classification
Title: segfault causing regression from PEP 573 implementation (PyQt5)
Type: Stage: resolved
Components: C API Versions: Python 3.9
process
Status: closed Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: corona10, petr.viktorin, shihai1991, tcaswell, vstinner
Priority: normal Keywords:

Created on 2020-05-09 04:08 by tcaswell, last changed 2020-05-12 18:07 by tcaswell. This issue is now closed.

Messages (12)
msg368499 - (view) Author: Thomas Caswell (tcaswell) * Date: 2020-05-09 04:08
https://github.com/python/cpython/commit/e1becf46b4e3ba6d7d32ebf4bbd3e0804766a423 causes pyqt5 to segfault an accessing an attribute

To reproduce this:


pip install pyqt5
pip install sip
python -c "import PyQt5.QtCore; PyQt5.QtCore.Qt.Key_Control"  # this segfaults

Setting up faulthandler gives:

jupiter@23:54  ➤  python -X faulthandler -c "import PyQt5.QtCore; PyQt5.QtCore.Qt.Key_Control"
Fatal Python error: Segmentation fault

Current thread 0x00007f52111cd740 (most recent call first):
<no Python frame>
Segmentation fault (core dumped)

It is likely that this should also be reported to riverbank as a pyqt/sip bug, but I'm starting here.  I apologize for not having a simpler reproducing case.




jupiter@00:01  ➤ git bisect log
git bisect start
# good: [2d8757758d0d75882fef0fe0e3c74c4756b3e81e] bpo-40286: Remove C implementation of Random.randbytes() (GH-19797)
git bisect good 2d8757758d0d75882fef0fe0e3c74c4756b3e81e
# bad: [d10091aa171250c67a5079abfe26b8b3964ea39a] bpo-40502: Initialize n->n_col_offset (GH-19988)
git bisect bad d10091aa171250c67a5079abfe26b8b3964ea39a
# good: [c3f001461d5794c81cf5f70e08ae5435fe935ceb] bpo-40491: Fix typo in syntax error for numeric literals (GH-19893)
git bisect good c3f001461d5794c81cf5f70e08ae5435fe935ceb
# good: [c21c51235aa8061da6b0593d6f857f42fd92fd8b] bpo-40355: Improve error messages in ast.literal_eval with malformed Dict nodes (GH-19868)
git bisect good c21c51235aa8061da6b0593d6f857f42fd92fd8b
# good: [c1c7d8ead9eb214a6149a43e31a3213c52448877] bpo-40397: Refactor typing._GenericAlias (GH-19719)
git bisect good c1c7d8ead9eb214a6149a43e31a3213c52448877
# bad: [c068b53a0ca6ebf740d98e422569d2f705e54f93] bpo-38787: Update structures.rst docs (PEP 573) (GH-19980)
git bisect bad c068b53a0ca6ebf740d98e422569d2f705e54f93
# good: [4638c6429575bd6de26b12b2af5df74d6568b553] bpo-40334: Error message for invalid default args in function call (GH-19973)
git bisect good 4638c6429575bd6de26b12b2af5df74d6568b553
# bad: [8963a7f1f84a05412178b56629508b660d38861b] bpo-40545: Export _PyErr_GetTopmostException() function (GH-19978)
git bisect bad 8963a7f1f84a05412178b56629508b660d38861b
# bad: [e1becf46b4e3ba6d7d32ebf4bbd3e0804766a423] bpo-38787: C API for module state access from extension methods (PEP 573) (GH-19936)
git bisect bad e1becf46b4e3ba6d7d32ebf4bbd3e0804766a423
# first bad commit: [e1becf46b4e3ba6d7d32ebf4bbd3e0804766a423] bpo-38787: C API for module state access from extension methods (PEP 573) (GH-19936)
msg368500 - (view) Author: Thomas Caswell (tcaswell) * Date: 2020-05-09 04:14
Sorry, forgot to add this is on Linux.
msg368575 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-05-10 09:16
I tried to install pyqt5 on Python compiled from source:

make
./python -m venv env
env/bin/python -m pip install sip
env/bin/python -m pip install pyqt5


But I failed to install pyqt5:

vstinner@apu$ env/bin/python -m pip install pyqt5
Collecting pyqt5
/home/vstinner/python/master/env/lib/python3.9/site-packages/pip/_vendor/msgpack/fallback.py:133: DeprecationWarning: encoding is deprecated, Use raw=False instead.
  unpacker = Unpacker(None, max_buffer_size=len(packed), **kwargs)
  Using cached https://files.pythonhosted.org/packages/4d/81/b9a66a28fb9a7bbeb60e266f06ebc4703e7e42b99e3609bf1b58ddd232b9/PyQt5-5.14.2.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... error
    ERROR: Command errored out with exit status 1:
     command: /home/vstinner/python/master/env/bin/python /home/vstinner/python/master/env/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpzl766kxn
         cwd: /tmp/pip-install-a2jym86c/pyqt5
    Complete output (33 lines):
    Traceback (most recent call last):
      File "/home/vstinner/python/master/env/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py", line 64, in prepare_metadata_for_build_wheel
        hook = backend.prepare_metadata_for_build_wheel
    AttributeError: module 'sipbuild.api' has no attribute 'prepare_metadata_for_build_wheel'
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/home/vstinner/python/master/env/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py", line 207, in <module>
        main()
      File "/home/vstinner/python/master/env/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py", line 197, in main
        json_out['return_val'] = hook(**hook_input['kwargs'])
      File "/home/vstinner/python/master/env/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py", line 66, in prepare_metadata_for_build_wheel
        return _get_wheel_metadata_from_wheel(backend, metadata_directory,
      File "/home/vstinner/python/master/env/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py", line 95, in _get_wheel_metadata_from_wheel
        whl_basename = backend.build_wheel(metadata_directory, config_settings)
      File "/tmp/pip-build-env-w2h9381n/overlay/lib/python3.9/site-packages/sipbuild/api.py", line 51, in build_wheel
        project = AbstractProject.bootstrap('pep517')
      File "/tmp/pip-build-env-w2h9381n/overlay/lib/python3.9/site-packages/sipbuild/abstract_project.py", line 82, in bootstrap
        project.setup(pyproject, tool, tool_description)
      File "/tmp/pip-build-env-w2h9381n/overlay/lib/python3.9/site-packages/sipbuild/project.py", line 410, in setup
        self.apply_user_defaults(tool)
      File "project.py", line 62, in apply_user_defaults
        super().apply_user_defaults(tool)
      File "/tmp/pip-build-env-w2h9381n/overlay/lib/python3.9/site-packages/pyqtbuild/project.py", line 86, in apply_user_defaults
        super().apply_user_defaults(tool)
      File "/tmp/pip-build-env-w2h9381n/overlay/lib/python3.9/site-packages/sipbuild/project.py", line 215, in apply_user_defaults
        self.builder.apply_user_defaults(tool)
      File "/tmp/pip-build-env-w2h9381n/overlay/lib/python3.9/site-packages/pyqtbuild/builder.py", line 67, in apply_user_defaults
        raise PyProjectOptionException('qmake',
    sipbuild.pyproject.PyProjectOptionException
    /home/vstinner/python/master/Lib/tempfile.py:818: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpqj0vy32i'>
      _warnings.warn(warn_message, ResourceWarning)
    ----------------------------------------
ERROR: Command errored out with exit status 1: /home/vstinner/python/master/env/bin/python /home/vstinner/python/master/env/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpzl766kxn Check the logs for full command output.
msg368576 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-05-10 09:35
Ah, if I install Python, I can install PyQt5. I installed PyQt5:
   python -X faulthandler -c "import PyQt5.QtCore; PyQt5.QtCore.Qt.Key_Control""
doesn't crash.

I also tested the following hello world, it doesn't crash neither.
---
# https://pythonprogramminglanguage.com/pyqt5-hello-world/
import sys
from PyQt5 import QtCore, QtWidgets
from PyQt5.QtWidgets import QMainWindow, QLabel, QGridLayout, QWidget
from PyQt5.QtCore import QSize    

class HelloWindow(QMainWindow):
    def __init__(self):
        QMainWindow.__init__(self)

        self.setMinimumSize(QSize(640, 480))    
        self.setWindowTitle("Hello world - pythonprogramminglanguage.com") 

        centralWidget = QWidget(self)          
        self.setCentralWidget(centralWidget)   

        gridLayout = QGridLayout(self)     
        centralWidget.setLayout(gridLayout)  

        title = QLabel("Hello World from PyQt", self) 
        title.setAlignment(QtCore.Qt.AlignCenter) 
        gridLayout.addWidget(title, 0, 0)

if __name__ == "__main__":
    app = QtWidgets.QApplication(sys.argv)
    mainWin = HelloWindow()
    mainWin.show()
    sys.exit( app.exec_() )
---

I built Python with:

./configure --cache-file=../python-config.cache --with-pydebug CFLAGS=-O0 --prefix=/opt/py39 --with-system-expat --with-system-ffi
make
make install

What is your OS? How did you build Python? Which SIP and PyQt5 versions did you try?
msg368579 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-05-10 09:46
I also with:

   ./configure --prefix=/opt/py39 --with-system-expat --with-system-ffi

I still cannot reproduce the crash. I tested Python at commit 1c2fa781560608aa4be50c748d4b3f403cfa5035. I tested on Fedora 32 (GCC 10.0.1).

$ /opt/py39/bin/python3.9 -m pip freeze
pyflakes==2.2.0
pyperf==2.0.0
PyQt5==5.14.2
PyQt5-sip==12.7.2

$ /opt/py39/bin/python3.9 -VV
Python 3.9.0a6+ (heads/master:1c2fa78156, May 10 2020, 11:37:38) 
[GCC 10.0.1 20200430 (Red Hat 10.0.1-0.14)]
msg368588 - (view) Author: Thomas Caswell (tcaswell) * Date: 2020-05-10 15:59
The script I used for the bisect was:

---

TARGET_ENV=bisect_env


rm -r ~/.pybuild/$TARGET_ENV || true
git clean -xfd
./configure --prefix=/home/tcaswell/.pybuild/$TARGET_ENV
make -j 9
make install
~/.pybuild/$TARGET_ENV/bin/python3 -m venv --copies --clear ~/.virtualenvs/$TARGET_ENV

popd

source ~/.virtualenvs/$TARGET_ENV/bin/activate
pip install --upgrade pip
echo $PATH

pushd ../sip    # this was at "tip" in the hg repo
pip install .
pip install pyqt5
popd

python -c "import PyQt5.QtCore; PyQt5.QtCore.Qt.Key_Control"
retVal=$?
if [ $retVal -ne 0 ]; then
    exit 1
fi

---


I'm on a relatively up-to-date Arch 

➤  gcc --version
gcc (Arch Linux 9.3.0-1) 9.3.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


---

updating pip is a key step (that I thoughtlessly carried over from my normal build process) because that lets it see manylinux2014 wheels and does not try to build pyqt5 from source.

python3.9 -m pip freeze
packaging==20.3
pyparsing==2.4.7
PyQt5==5.14.2
PyQt5-sip==12.7.2
sip==5.2.0
six==1.14.0
toml==0.10.0

➤  python3.9 -VV
Python 3.9.0a6+ (heads/master-7-g1c2fa78156:1c2fa78156, May 10 2020, 11:11:03) 
[GCC 9.3.0]


----

I'm also having issues getting pyqt5 to compile from the tarball, what do you mean by "Ah, if I install Python, I can install PyQt5. I installed PyQt5:"?
msg368592 - (view) Author: Thomas Caswell (tcaswell) * Date: 2020-05-10 18:46
I think I have figured out the problem.  I had a locally built and cached wheel of PyQt5-sip from before PEP573 went in.  If that wheel is used for later commits I get the segfault, if I rebuilt the wheel from source it works.

I am not sure if this is an expected sharp edge (wheels are be expected to be good between commits prior to a release) or not.


-----


Below is a script that will build everything (sip, pyqt5, pyqt5-sip) from source and install them into a virtual env.

----
#! /usr/bin/bash
set -e
set -o xtrace



rm -r ~/.pybuild/py39 || true
pushd cpython
git clean -xfd
./configure --prefix=/home/tcaswell/.pybuild/py39
make -j 9
make install
popd

~/.pybuild/py39/bin/python3 -m venv --copies --clear ~/.virtualenvs/py39



source ~/.virtualenvs/py39/bin/activate
pip install --upgrade pip
echo $PATH

pushd PyQt5
pip install sip --no-binary sip
rm -rf PyQt5-5.14.2
tar -xzf PyQt5-5.14.2.tar.gz

pushd PyQt5-5.14.2
python configure.py --confirm-license
# patch the one palce that does not respect the venv settings
sed -i 's|$(INSTALL_ROOT)/usr|$(INSTALL_ROOT)/$(HOME)/.virtualenvs/py39/usr|g' Makefile
make -j 9
make install
popd
popd
pip install PyQt5-sip --no-binary PyQt5-sip

python -c "import PyQt5.QtCore; PyQt5.QtCore.Qt.Key_Control"


----
msg368657 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-05-11 21:26
> I think I have figured out the problem.  I had a locally built and cached wheel of PyQt5-sip from before PEP573 went in.  If that wheel is used for later commits I get the segfault, if I rebuilt the wheel from source it works.

Hum, I'm not sure that I understand well. Is there a bug in Python or not?

It seems when in my tests, pip of Python 3.9 installed wheel package marked as "python38". I understand that the ABI didn't change and we are all good.

I suggest to close the issue.
msg368675 - (view) Author: Thomas Caswell (tcaswell) * Date: 2020-05-11 22:45
The path is

 - on a commit prior to e1becf46b4e3ba6d7d32ebf4bbd3e0804766a423 install pyqt-sip.  pip will build a wheel for you called PyQt5_sip-12.7.2-cp39-cp39-linux_x86_64.whl
 - on a commit after e1becf46b4e3ba6d7d32ebf4bbd3e0804766a423 if you do `pip install pyqt-sip` pip will discover the wheel from the previous install in it's cache and use it, but the ABI has changed (?) which leads to a segfault.  If you do `pip install pyqt5-sip --no-binary pyqt5-sip` or clear the cache then things work correctly

I suspect that this also means that wheels made with the early alphas will not work future 3.9 (pre-)releases, however I am not sure if that is a problem or not (given that it is all pre-releases).

I am going to close this as I think if there is a bug, it would be better addressed in the packaging space.
msg368687 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-05-12 00:26
The stable ABI should not change between Python 3.8 and 3.9.

In practice, it seems like something changed. But without any gdb traceback, I cannot tell what.

I suggest to try again when beta1 will be released. The ABI should be way more stable after beta1.
msg368730 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2020-05-12 13:43
So, the failure is expected. Python's ABI can change until the 3.9.0 final release, so wheels built for different commits can be incompatible.
This applies to alphas/betas as well, as you say.

There is the PEP 384 stable ABI, which is much stricter (and more limited), but sip doesn't use it -- its wheels are specific to cp36/cp37/cp38: https://pypi.org/project/sip/#files

Thanks for testing, though! Perhaps we need to communicate better that for the alphas/betas, everything needs to be rebuilt.
msg368741 - (view) Author: Thomas Caswell (tcaswell) * Date: 2020-05-12 18:07
That seems reasonable.

To be pedantic, it is pyqt5-sip (not sip) that was the source of the problem.

I am going to open an issue with pip to disable caching locally built wheels for pre-released versions of Python.
History
Date User Action Args
2020-05-12 18:07:39tcaswellsetmessages: + msg368741
2020-05-12 13:43:02petr.viktorinsetmessages: + msg368730
2020-05-12 00:26:22vstinnersetmessages: + msg368687
2020-05-11 22:45:18tcaswellsetstatus: open -> closed

messages: + msg368675
stage: resolved
2020-05-11 21:26:59vstinnersetmessages: + msg368657
2020-05-10 18:46:41tcaswellsetmessages: + msg368592
2020-05-10 15:59:00tcaswellsetmessages: + msg368588
2020-05-10 13:51:40shihai1991setnosy: + shihai1991
2020-05-10 09:46:48vstinnersetmessages: + msg368579
2020-05-10 09:38:33vstinnersettitle: segfault causing regression from PEP 573 implementation -> segfault causing regression from PEP 573 implementation (PyQt5)
2020-05-10 09:35:24vstinnersetmessages: + msg368576
2020-05-10 09:16:51vstinnersetmessages: + msg368575
2020-05-10 09:06:09vstinnersetnosy: + vstinner, petr.viktorin, corona10
2020-05-09 04:14:47tcaswellsetmessages: + msg368500
2020-05-09 04:08:44tcaswellcreate