classification
Title: readline module core dumps Python 3.8.1 when calling exit()
Type: crash Stage: resolved
Components: Extension Modules Versions: Python 3.8
process
Status: closed Resolution: third party
Dependencies: Superseder:
Assigned To: Nosy List: ammar2, fhstoica, vstinner
Priority: normal Keywords:

Created on 2020-01-17 11:17 by fhstoica, last changed 2020-01-17 12:42 by fhstoica. This issue is now closed.

Messages (7)
msg360170 - (view) Author: Horace Stoica (fhstoica) Date: 2020-01-17 11:17
Built Python 3.8.1 from source on Fedora 30:

kernel: 5.1.8-300.fc30.x86_64

Python 3.8.1 (default, Jan 15 2020, 08:49:34)
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux

Installed the readline module as the arrows do not work when running Python interactively.

~/local/bin/pip3.8 install readline
Collecting readline
  Using cached https://files.pythonhosted.org/packages/f4/01/2cf081af8d880b44939a5f1b446551a7f8d59eae414277fd0c303757ff1b/readline-6.2.4.1.tar.gz

However, after installing readline Python core-dumps on Crtl+D or exit():

>>> exit()
munmap_chunk(): invalid pointer
Aborted (core dumped)

I uninstalled the readline module and now it no longer core dumps.
msg360171 - (view) Author: Ammar Askar (ammar2) * (Python committer) Date: 2020-01-17 11:29
> Installed the readline module as the arrows do not work when running Python interactively.

This is not a suggested way to fix the issue, especially given that the module you installed is marked as deprecated: https://pypi.org/project/readline/

> Built Python 3.8.1 from source on Fedora 30:

Instead, when you compile, look for a line like this

    Failed to find the necessary bits to build these modules:
    ...
    readline

or

    Failed to build these modules:
    ...
    readline

and then scan through the build log to either find the compiler error or you're most probably missing a dependency like readline-devel
msg360173 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-01-17 12:01
I'm able to reproduce the issue on Fedora 31, but only if the 3rd party readline module is loaded, instead of the stdlib "readline" module.

The problem is that https://pypi.org/project/readline/ misuses the Python memory allocator API. The GIL must be held to call PyMem_Malloc(). This project should be modified to use PyMem_RawMalloc().

https://docs.python.org/dev/c-api/memory.html#raw-memory-interface
vs
https://docs.python.org/dev/c-api/memory.html#memory-interface

It's a bug in a 3rd party module, I close the issue.


== packaged Python 3.8 ==

vstinner@apu$ python3.8 -m pip install --user readline

Python still uses the stdlib module by default (sys.path order):

vstinner@apu$ python3.8 
Python 3.8.1 (default, Dec 19 2019, 00:00:00) 
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import readline
>>> readline
<module 'readline' from '/usr/lib64/python3.8/lib-dynload/readline.cpython-38-x86_64-linux-gnu.so'>

I'm using uninstall just to know the path, but then cancel:

vstinner@apu$ python3.8 -m pip uninstall readline
Uninstalling readline-6.2.4.1:
  Would remove:
    /home/vstinner/.local/lib/python3.8/site-packages/readline-6.2.4.1-py3.8.egg-info
    /home/vstinner/.local/lib/python3.8/site-packages/readline.cpython-38-x86_64-linux-gnu.so
Proceed (y/n)? n

I reproduce the issue if I force to prefer the 3rd party readline:

vstinner@apu$ PYTHONPATH=/home/vstinner/.local/lib/python3.8/site-packages/ python3.8 
Python 3.8.1 (default, Dec 19 2019, 00:00:00) 
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import readline
munmap_chunk(): invalid pointer
Abandon (core dumped)

== Python 3.9 compiled manually in debug mode ==

(...)
vstinner@apu$ PYTHONPATH=/home/vstinner/python/master/env/lib/python3.9/site-packages/ gdb -args env/bin/python 
GNU gdb (GDB) Fedora 8.3.50.20190824-26.fc31
(gdb) run

>>> import readline
Fatal Python error: Python memory allocator called without holding the GIL
Python runtime state: initialized

Current thread 0x00007ffff7c1f740 (most recent call first):
<no Python frame>

Program received signal SIGABRT, Aborted.
0x00007ffff7c60625 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: dnf debuginfo-install libxcrypt-4.4.10-2.fc31.x86_64 ncurses-libs-6.1-12.20190803.fc31.x86_64
(gdb) where
#0  0x00007ffff7c60625 in raise () from /lib64/libc.so.6
#1  0x00007ffff7c498d9 in abort () from /lib64/libc.so.6
#2  0x0000000000560b43 in fatal_error (prefix=0x0, msg=0x6b4c40 "Python memory allocator called without holding the GIL", status=-1) at Python/pylifecycle.c:2249
#3  0x0000000000560b6f in Py_FatalError (msg=0x6b4c40 "Python memory allocator called without holding the GIL") at Python/pylifecycle.c:2259
#4  0x000000000047d988 in _PyMem_DebugCheckGIL () at Objects/obmalloc.c:2320
#5  0x000000000047d9a0 in _PyMem_DebugMalloc (ctx=0x7cd950 <_PyMem_Debug+48>, nbytes=17) at Objects/obmalloc.c:2328
#6  0x000000000047b958 in PyMem_Malloc (size=17) at Objects/obmalloc.c:605
#7  0x00007fffeaa48dc0 in call_readline (sys_stdin=0x7ffff7de67e0 <_IO_2_1_stdin_>, sys_stdout=0x7ffff7de7500 <_IO_2_1_stdout_>, prompt=0x7fffeaa9c480 ">>> ")
    at Modules/3.x/readline.c:1122
#8  0x000000000069e8a4 in PyOS_Readline (sys_stdin=0x7ffff7de67e0 <_IO_2_1_stdin_>, sys_stdout=0x7ffff7de7500 <_IO_2_1_stdout_>, prompt=0x7fffeaa9c480 ">>> ")
    at Parser/myreadline.c:358
#9  0x0000000000615155 in tok_nextc (tok=0x988f00) at Parser/tokenizer.c:848
#10 0x0000000000615f7d in tok_get (tok=0x988f00, p_start=0x7fffffffd118, p_end=0x7fffffffd110) at Parser/tokenizer.c:1128
#11 0x0000000000617876 in PyTokenizer_Get (tok=0x988f00, p_start=0x7fffffffd118, p_end=0x7fffffffd110) at Parser/tokenizer.c:1774
#12 0x0000000000612862 in parsetok (tok=0x988f00, g=0x7f1ce0 <_PyParser_Grammar>, start=256, err_ret=0x7fffffffd280, flags=0x7fffffffd27c) at Parser/parsetok.c:251
#13 0x0000000000612672 in PyParser_ParseFileObject (fp=0x7ffff7de67e0 <_IO_2_1_stdin_>, filename='<stdin>', enc=0x7fffeab24340 "utf-8", g=0x7f1ce0 <_PyParser_Grammar>, start=256, 
    ps1=0x7fffeaa9c480 ">>> ", ps2=0x7fffeaa9c520 "... ", err_ret=0x7fffffffd280, flags=0x7fffffffd27c) at Parser/parsetok.c:186
#14 0x0000000000567716 in PyParser_ASTFromFileObject (fp=0x7ffff7de67e0 <_IO_2_1_stdin_>, filename='<stdin>', enc=0x7fffeab24340 "utf-8", start=256, ps1=0x7fffeaa9c480 ">>> ", 
    ps2=0x7fffeaa9c520 "... ", flags=0x7fffffffd470, errcode=0x7fffffffd314, arena=0x7fffeab2d7c0) at Python/pythonrun.c:1387
#15 0x0000000000564fc1 in PyRun_InteractiveOneObjectEx (fp=0x7ffff7de67e0 <_IO_2_1_stdin_>, filename='<stdin>', flags=0x7fffffffd470) at Python/pythonrun.c:239
#16 0x0000000000564cbd in PyRun_InteractiveLoopFlags (fp=0x7ffff7de67e0 <_IO_2_1_stdin_>, filename_str=0x69f43a "<stdin>", flags=0x7fffffffd470) at Python/pythonrun.c:121
#17 0x0000000000564b80 in PyRun_AnyFileExFlags (fp=0x7ffff7de67e0 <_IO_2_1_stdin_>, filename=0x69f43a "<stdin>", closeit=0, flags=0x7fffffffd470) at Python/pythonrun.c:80
#18 0x000000000041e910 in pymain_run_stdin (config=0x819240, cf=0x7fffffffd470) at Modules/main.c:467
#19 0x000000000041ec10 in pymain_run_python (exitcode=0x7fffffffd4bc) at Modules/main.c:556
#20 0x000000000041ece6 in Py_RunMain () at Modules/main.c:632
#21 0x000000000041ed60 in pymain_main (args=0x7fffffffd520) at Modules/main.c:662
#22 0x000000000041edda in Py_BytesMain (argc=1, argv=0x7fffffffd648) at Modules/main.c:686
#23 0x000000000041d796 in main (argc=1, argv=0x7fffffffd648) at ./Programs/python.c:16
msg360175 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-01-17 12:12
Oh right, Ammar is right, you miss a dependency. Try maybe:

sudo dnf install -y dnf-plugins-core
sudo dnf builddep -y python3

https://cpython-core-tutorial.readthedocs.io/en/latest/build_cpython_linux.html

... but why are you building Python from source? Fedora provides python3.8 ;-) At least, on Fedora 31, Python 3.8 and 3.9 are packaged ;-)

If you built it to contribute to Python, that's great!
msg360176 - (view) Author: Horace Stoica (fhstoica) Date: 2020-01-17 12:33
A big "thank you" to everyone, I installed readline-develop and rebuilt from source, and now the problem is fixed.

The reason I am building from source is that Fedora 30 provides Python 3.7.3 and I wanted to try out the new features (like the walrus operator) that were introduced in version 3.8.

I hope to get to the point of contributing to Python itself, for now I am looking mainly at extending Python with C++ and embedding Python in C++ applications. 

Thanks again, 
Horace
msg360178 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-01-17 12:36
> The reason I am building from source is that Fedora 30 provides Python 3.7.3 and I wanted to try out the new features (like the walrus operator) that were introduced in version 3.8.

Did you try "dnf install python38"? It's available on Fedora 30. FYI I'm part of the package maintainers of Python packages on Fedora and RHEL ;-)
msg360179 - (view) Author: Horace Stoica (fhstoica) Date: 2020-01-17 12:42
Tried dnf install python38 and it works. Thank you!
History
Date User Action Args
2020-01-17 12:42:52fhstoicasetmessages: + msg360179
2020-01-17 12:36:11vstinnersetmessages: + msg360178
2020-01-17 12:33:41fhstoicasetmessages: + msg360176
2020-01-17 12:12:51vstinnersetmessages: + msg360175
2020-01-17 12:01:14vstinnersetstatus: open -> closed

nosy: + vstinner
messages: + msg360173

resolution: third party
stage: resolved
2020-01-17 11:29:00ammar2setnosy: + ammar2
messages: + msg360171
2020-01-17 11:17:46fhstoicacreate