This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: pythin is core dumping with SIGILL in pytest of the jupyter-server
Type: crash Stage: resolved
Components: Versions: Python 3.8
process
Status: closed Resolution: third party
Dependencies: Superseder:
Assigned To: Nosy List: christian.heimes, kloczek
Priority: normal Keywords:

Created on 2021-12-23 18:53 by kloczek, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
python_testing_jupyter-server_pytest.log kloczek, 2021-12-23 18:53
Messages (3)
msg409098 - (view) Author: Tomasz Kloczko (kloczek) Date: 2021-12-23 18:53
I;m packaging python modules as rpm packages and just found that package jupyter-server (1.13.1) when pytest was executed python crashed with call trace

+ /usr/bin/pytest -ra jupyter_server
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.12, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/jupyter_server-1.13.1, configfile: pyproject.toml
plugins: cov-3.0.0, console-scripts-1.2.0, mock-3.6.1, tornasync-0.6.0.post2, anyio-3.3.4
collected 581 items

jupyter_server/auth/security.py Fatal Python error: Illegal instruction

Current thread 0x00007f480bd08740 (most recent call first):
  File "/usr/lib64/python3.8/site-packages/argon2/low_level.py", line 112 in hash_secret
  File "/usr/lib64/python3.8/site-packages/argon2/_password_hasher.py", line 133 in hash
  File "/home/tkloczko/rpmbuild/BUILD/jupyter_server-1.13.1/jupyter_server/auth/security.py", line 73 in passwd
  File "<doctest jupyter_server.auth.security.passwd[0]>", line 1 in <module>
  File "/usr/lib64/python3.8/doctest.py", line 1336 in __run
  File "/usr/lib64/python3.8/doctest.py", line 1483 in run
  File "/usr/lib64/python3.8/doctest.py", line 1844 in run
  File "/usr/lib/python3.8/site-packages/_pytest/doctest.py", line 287 in runtest
  File "/usr/lib/python3.8/site-packages/_pytest/runner.py", line 162 in pytest_runtest_call
  File "/usr/lib/python3.8/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/usr/lib/python3.8/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/usr/lib/python3.8/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/usr/lib/python3.8/site-packages/_pytest/runner.py", line 255 in <lambda>
  File "/usr/lib/python3.8/site-packages/_pytest/runner.py", line 311 in from_call
  File "/usr/lib/python3.8/site-packages/_pytest/runner.py", line 254 in call_runtest_hook
  File "/usr/lib/python3.8/site-packages/_pytest/runner.py", line 215 in call_and_report
  File "/usr/lib/python3.8/site-packages/_pytest/runner.py", line 126 in runtestprotocol
  File "/usr/lib/python3.8/site-packages/_pytest/runner.py", line 109 in pytest_runtest_protocol
  File "/usr/lib/python3.8/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/usr/lib/python3.8/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/usr/lib/python3.8/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/usr/lib/python3.8/site-packages/_pytest/main.py", line 348 in pytest_runtestloop
  File "/usr/lib/python3.8/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/usr/lib/python3.8/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/usr/lib/python3.8/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/usr/lib/python3.8/site-packages/_pytest/main.py", line 323 in _main
  File "/usr/lib/python3.8/site-packages/_pytest/main.py", line 269 in wrap_session
  File "/usr/lib/python3.8/site-packages/_pytest/main.py", line 316 in pytest_cmdline_main
  File "/usr/lib/python3.8/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/usr/lib/python3.8/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/usr/lib/python3.8/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/usr/lib/python3.8/site-packages/_pytest/config/__init__.py", line 162 in main
  File "/usr/lib/python3.8/site-packages/_pytest/config/__init__.py", line 185 in console_main
  File "/usr/bin/pytest", line 33 in <module>
/var/tmp/rpm-tmp.WmHwan: line 49: 2150294 Illegal instruction     (core dumped) PATH="/home/tkloczko/rpmbuild/BUILDROOT/python-jupyter-server-1.13.1-2.fc35.x86_64/usr/bin:$PATH" PYTHONDONTWRITEBYTECODE=1 SETUPTOOLS_SCM_PRETEND_VERSION=1.13.1 PYTHONPATH="${PYTHONPATH:-/home/tkloczko/rpmbuild/BUILDROOT/python-jupyter-server-1.13.1-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-jupyter-server-1.13.1-2.fc35.x86_64/usr/lib/python3.8/site-packages}" /usr/bin/pytest -ra jupyter_server

I'm using my own Linux distribution (which I';m supporting for may clients) and I'm still using python 3.8.12.

Here is list of modules installed in build env
[tkloczko@ss-desktop SPECS]$ pip list
Package                       Version
----------------------------- -------------------
alabaster                     0.7.12
anyio                         3.3.4
appdirs                       1.4.4
argon2-cffi                   21.1.0
attrs                         21.2.0
Babel                         2.9.1
backcall                      0.2.0
bleach                        4.0.0
build                         0.7.0
cffi                          1.15.0
chardet                       4.0.0
charset-normalizer            2.0.9
coverage                      6.2
cycler                        0.11.0
dbus-python                   1.2.18
decorator                     5.1.0
defusedxml                    0.7.1
distro                        1.6.0
docutils                      0.17.1
entrypoints                   0.3
extras                        1.0.0
fixtures                      3.0.0
fonttools                     4.28.5
gpg                           1.16.0-unknown
html5lib                      1.1
idna                          3.3
imagesize                     1.2.0
importlib-metadata            4.10.0
importlib-resources           5.4.0
iniconfig                     1.1.1
ipykernel                     6.6.0
ipython                       7.30.1
ipython_genutils              0.2.0
ipywidgets                    7.6.4
jedi                          0.18.1
Jinja2                        3.0.3
jsonschema                    4.3.2
jupyter-client                7.1.0
jupyter-core                  4.9.1
jupyterlab-pygments           0.1.2
kiwisolver                    1.3.2
libcomps                      0.1.18
lxml                          4.7.1
m2r                           0.2.1
Mako                          1.1.6.dev0
Markdown                      3.3.6
markdown-it-py                1.1.0
MarkupSafe                    2.0.1
matplotlib                    3.5.1
matplotlib-inline             0.1.2
mdit-py-plugins               0.3.0
meson                         0.60.2
mistune                       0.8.4
mock                          4.0.3
myst-parser                   0.16.1
nbclient                      0.5.9
nbconvert                     6.1.0
nbformat                      5.1.3
nest-asyncio                  1.5.1
notebook                      6.4.2
numpy                         1.21.4
olefile                       0.46
packaging                     21.3
pandocfilters                 1.5.0
parso                         0.8.3
pbr                           5.6.0
pep517                        0.12.0
pexpect                       4.8.0
pickleshare                   0.7.5
Pillow                        8.4.0
pip                           21.3.1
pluggy                        1.0.0
ply                           3.11
prometheus-client             0.12.0
prompt-toolkit                3.0.24
ptyprocess                    0.7.0
py                            1.11.0
pycparser                     2.21
Pygments                      2.10.0
PyGObject                     3.42.0
pyparsing                     3.0.6
pyrsistent                    0.18.0
pytest                        6.2.5
pytest-console-scripts        1.2.0
pytest-cov                    3.0.0
pytest-mock                   3.6.1
pytest-tornasync              0.6.0.post2
python-dateutil               2.8.2
pytz                          2021.1
PyYAML                        6.0
pyzmq                         22.2.1
requests                      2.26.0
requests-unixsocket           0.2.0
rpm                           4.17.0
scour                         0.38.2
Send2Trash                    1.8.0
setuptools                    60.0.0.post20211220
six                           1.16.0
smartypants                   2.0.1
sniffio                       1.2.0
snowballstemmer               2.1.0
Sphinx                        4.3.2.dev20211220
sphinxcontrib-applehelp       1.0.2.dev20210802
sphinxcontrib-devhelp         1.0.2.dev20210801
sphinxcontrib_github_alt      1.2
sphinxcontrib-htmlhelp        2.0.0
sphinxcontrib-httpdomain      1.7.0
sphinxcontrib-jsmath          1.0.1.dev20210801
sphinxcontrib-openapi         0.7.0
sphinxcontrib-qthelp          1.0.3.dev20210824
sphinxcontrib-serializinghtml 1.1.5
sphinxemoji                   0.1.9
terminado                     0.12.1
testpath                      0.5.0
testtools                     2.5.0
toml                          0.10.2
tomli                         2.0.0
tornado                       6.1
traitlets                     5.1.1
typogrify                     2.0.7
urllib3                       1.26.7
wcwidth                       0.2.5
webencodings                  0.5.1
websocket-client              1.2.3
wheel                         0.37.0
widgetsnbextension            3.5.1
youtube-dl                    2021.6.6
zipp                          3.6.0

I made terminal dump of the gdb session using script command so by "cat python_testing_jupyter-server_pytest.log" on current termilnal you may see full gdb back trace with original colouring.

Please let me know what I can do more to diagnose that issue or if you need more details.

I can add that exactly that python binary was up to now rock solid on building, minstalling and testing +800 python modules as rpm packages.
msg409118 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-12-24 01:11
The traceback indicates that it's a problem with a 3rd party extension module and not a bug in CPython core. The C stack suggests that your copy of libargon2 is the root cause.
msg409120 - (view) Author: Tomasz Kloczko (kloczek) Date: 2021-12-24 04:07
OK accorting to your expertise this could be somethning wrong in some of the DSOs ..

May I ask for a little help with pointing on possible module which i doing sometbing wrong?

Just looking on top of the call trace 

                #0  0x00007f480be7a54c __pthread_kill_implementation (libc.so.6 + 0x8f54c)
                #1  0x00007f480be2d696 raise (libc.so.6 + 0x42696)
                #2  0x00007f480be2d740 __restore_rt (libc.so.6 + 0x42740)
                #3  0x00007f47f9fbe8ff blake2b_init (libargon2.so.1 + 0x38ff)
                #4  0x00007f47f9fbfb0e argon2_ctx.part.0 (libargon2.so.1 + 0x4b0e)
                #5  0x00007f47f9fc0a9f argon2_hash (libargon2.so.1 + 0x5a9f)
                #6  0x00007f47f9fc8f16 _cffi_f_argon2_hash (_ffi.abi3.so + 0x2f16)
                #7  0x00007f480c0a0077 cfunction_call_varargs (libpython3.8.so.1.0 + 0xb7077)
                #8  0x00007f480c10febb _PyObject_MakeTpCall (libpython3.8.so.1.0 + 0x126ebb)
                #9  0x00007f480c05eee4 _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x75ee4)
                #10 0x00007f480c1781b7 _PyEval_EvalCodeWithName (libpython3.8.so.1.0 + 0x18f1b7)
                #11 0x00007f480c1787e3 _PyFunction_Vectorcall (libpython3.8.so.1.0 + 0x18f7e3)
                #12 0x00007f480c05d42c _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x7442c)
                #13 0x00007f480c05897b function_code_fastcall (libpython3.8.so.1.0 + 0x6f97b)
                #14 0x00007f480c05d8e3 _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x748e3)
                #15 0x00007f480c1781b7 _PyEval_EvalCodeWithName (libpython3.8.so.1.0 + 0x18f1b7)
                #16 0x00007f480c1787e3 _PyFunction_Vectorcall (libpython3.8.so.1.0 + 0x18f7e3)
                #17 0x00007f480c05d3e0 _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x743e0)
                #18 0x00007f480c1781b7 _PyEval_EvalCodeWithName (libpython3.8.so.1.0 + 0x18f1b7)
                #19 0x00007f480c1784e7 PyEval_EvalCode (libpython3.8.so.1.0 + 0x18f4e7)
                #20 0x00007f480c1d9295 builtin_exec (libpython3.8.so.1.0 + 0x1f0295)
                #21 0x00007f480c176c37 cfunction_vectorcall_FASTCALL (libpython3.8.so.1.0 + 0x18dc37)
                #22 0x00007f480c05d3e0 _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x743e0)
                #23 0x00007f480c05897b function_code_fastcall (libpython3.8.so.1.0 + 0x6f97b)
                #24 0x00007f480c05d8e3 _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x748e3)
                #25 0x00007f480c1781b7 _PyEval_EvalCodeWithName (libpython3.8.so.1.0 + 0x18f1b7)
                #26 0x00007f480c1787e3 _PyFunction_Vectorcall (libpython3.8.so.1.0 + 0x18f7e3)
                #27 0x00007f480c05e93d _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x7593d)
                #28 0x00007f480c1781b7 _PyEval_EvalCodeWithName (libpython3.8.so.1.0 + 0x18f1b7)
                #29 0x00007f480c1787e3 _PyFunction_Vectorcall (libpython3.8.so.1.0 + 0x18f7e3)
                #30 0x00007f480c132c9a method_vectorcall (libpython3.8.so.1.0 + 0x149c9a)
                #31 0x00007f480c05d42c _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x7442c)
                #32 0x00007f480c05897b function_code_fastcall (libpython3.8.so.1.0 + 0x6f97b)
                #33 0x00007f480c05d8e3 _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x748e3)
                #34 0x00007f480c05897b function_code_fastcall (libpython3.8.so.1.0 + 0x6f97b)
                #35 0x00007f480c1100bd PyVectorcall_Call (libpython3.8.so.1.0 + 0x1270bd)
                #36 0x00007f480c05ada7 _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x71da7)
                #37 0x00007f480c1781b7 _PyEval_EvalCodeWithName (libpython3.8.so.1.0 + 0x18f1b7)
                #38 0x00007f480c1787e3 _PyFunction_Vectorcall (libpython3.8.so.1.0 + 0x18f7e3)
                #39 0x00007f480c05e93d _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x7593d)
                #40 0x00007f480c05897b function_code_fastcall (libpython3.8.so.1.0 + 0x6f97b)
                #41 0x00007f480c132c9a method_vectorcall (libpython3.8.so.1.0 + 0x149c9a)
                #42 0x00007f480c05e93d _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x7593d)
                #43 0x00007f480c1781b7 _PyEval_EvalCodeWithName (libpython3.8.so.1.0 + 0x18f1b7)
                #44 0x00007f480c1787e3 _PyFunction_Vectorcall (libpython3.8.so.1.0 + 0x18f7e3)
                #45 0x00007f480c110246 _PyObject_FastCallDict (libpython3.8.so.1.0 + 0x127246)
                #46 0x00007f480c11042d _PyObject_Call_Prepend (libpython3.8.so.1.0 + 0x12742d)
                #47 0x00007f480c110501 slot_tp_call (libpython3.8.so.1.0 + 0x127501)
                #48 0x00007f480c0a01f4 PyObject_Call (libpython3.8.so.1.0 + 0xb71f4)
                #49 0x00007f480c05ada7 _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x71da7)
                #50 0x00007f480c1781b7 _PyEval_EvalCodeWithName (libpython3.8.so.1.0 + 0x18f1b7)
                #51 0x00007f480c1787e3 _PyFunction_Vectorcall (libpython3.8.so.1.0 + 0x18f7e3)
                #52 0x00007f480c05d3e0 _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x743e0)
                #53 0x00007f480c1781b7 _PyEval_EvalCodeWithName (libpython3.8.so.1.0 + 0x18f1b7)
                #54 0x00007f480c1787e3 _PyFunction_Vectorcall (libpython3.8.so.1.0 + 0x18f7e3)
                #55 0x00007f480c132c9a method_vectorcall (libpython3.8.so.1.0 + 0x149c9a)
                #56 0x00007f480c05d42c _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x7442c)
                #57 0x00007f480c1781b7 _PyEval_EvalCodeWithName (libpython3.8.so.1.0 + 0x18f1b7)
                #58 0x00007f480c1787e3 _PyFunction_Vectorcall (libpython3.8.so.1.0 + 0x18f7e3)
                #59 0x00007f480c1100bd PyVectorcall_Call (libpython3.8.so.1.0 + 0x1270bd)
                #60 0x00007f480c05ada7 _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x71da7)
                #61 0x00007f480c1781b7 _PyEval_EvalCodeWithName (libpython3.8.so.1.0 + 0x18f1b7)
                #62 0x00007f480c1787e3 _PyFunction_Vectorcall (libpython3.8.so.1.0 + 0x18f7e3)
                #63 0x00007f480c05d3e0 _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x743e0)


I don't see any DSO module code on that call trace (?!?)
May I ask help with assesment which one module may be messing here?

Thank you for any help
History
Date User Action Args
2022-04-11 14:59:53adminsetgithub: 90323
2021-12-24 04:07:42kloczeksetmessages: + msg409120
2021-12-24 01:11:27christian.heimessetstatus: open -> closed

type: crash

nosy: + christian.heimes
messages: + msg409118
resolution: third party
stage: resolved
2021-12-23 18:53:27kloczekcreate