classification
Title: Py_GetPath, Py_SetPath memory corruption due to mixed PyMem_New micex with PyMem_Raw_Free
Type: crash Stage: resolved
Components: Interpreter Core Versions: Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: benjamin.peterson, htgoebel, nurelin, vstinner
Priority: normal Keywords: patch

Created on 2017-09-20 14:38 by htgoebel, last changed 2017-09-21 09:23 by nurelin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 3679 merged nurelin, 2017-09-20 19:48
PR 3681 merged python-dev, 2017-09-21 06:08
Messages (5)
msg302624 - (view) Author: Hartmut Goebel (htgoebel) Date: 2017-09-20 14:38
When using

Py_GetPath();
Py_SetPath(pypath_w);

near the beginning of a program, Py_SetPath crashes with memory corruption if run on a systems with glibc's MALLOC_CHECK enabled.

The reason is: Py_GetPath and Py_SetPath use different memory interfaces.

* Py_GetPath() calls calculate_path(), which uses PyMem_New() to allocate the memory (see https://github.com/python/cpython/blob/v3.6.0/Modules/getpath.c#L738, assigned to `buf` first)

* But Py_SetPath() uses PyMem_RawFree() to free the memory (see https://github.com/python/cpython/blob/v3.6.0/Modules/getpath.c#L830).


This error DOES NOT occur on Windows, since for win32 there is a separate implementation, which uses PyMem_RawMalloc (see https://github.com/python/cpython/blob/v3.6.0/PC/getpathp.c#L378).

This error also DOES NOT occur in Python <= 3.5, since PYMEM_FUNCS have been the same as PYRAW_FUNCS (see https://github.com/python/cpython/blob/v3.5.0/Objects/obmalloc.c#L148). This was changed in v3.6.0: PYMEM_FUNCS now is PYOBJ_FUNCS, see https://github.com/python/cpython/blob/v3.6.0/Objects/obmalloc.c#L159.

This error only occurs when running on a system with glibc's MALLOC_CHECK enabled, which seems to be the default st least on CentOS, Fedora and ArchLinux.

Example code and relevant source see below.


Example stack trace
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

*** glibc detected *** ./dist/jcollect3: double free or corruption (out): 0x00007fde271ab030 ***
======= Backtrace: =========
/lib64/libc.so.6[0x37f0675e66]
/lib64/libc.so.6[0x37f06789b3]
/tmp/_MEIhKg3kx/libpython3.6m.so.1.0(Py_SetPath+0x15)[0x7fde2d20e825]
./dist/jcollect3[0x4043e2]
./dist/jcollect3[0x402db2]
./dist/jcollect3[0x402fb0]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x37f061ed5d]
./dist/jcollect3[0x401a9e]
======= Memory map: ========


How to reproduce
~~~~~~~~~~~~~~~~~~~~~~~~~

An example for this problem is the bootloader of PyInstaller (as of 2017-08-01).

* Make sure you are running on a Linux distribution having glibc's MALLOC_CHECK enabled.

* Set up a virtual environment (just too keep your system clean)

pip install https://github.com/pyinstaller/pyinstaller/archive/develop@%7B2017-08-01%7D.zip
echo 'print("Hello")' > test.py
pyinstaller test.py
dist/test/test  # run the frozen script

The relevant source of PyInstaller's bootloader is at <https://github.com/pyinstaller/pyinstaller/blob/develop@%7B2017-08-01%7D/bootloader/src/pyi_pythonlib.c#L466>
msg302668 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2017-09-21 06:08
New changeset 3d1e2ab584ed0175592b5be2a0bc98dc1723776a by Benjamin Peterson (nurelin) in branch 'master':
bpo-31532: Fix memory corruption due to allocator mix (#3679)
https://github.com/python/cpython/commit/3d1e2ab584ed0175592b5be2a0bc98dc1723776a
msg302669 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2017-09-21 06:47
New changeset 88d0663005d258526496d1f8ee0acb7103c69e80 by Benjamin Peterson (Miss Islington (bot)) in branch '3.6':
[3.6] closes bpo-31532: Fix memory corruption due to allocator mix (GH-3679) (#3681)
https://github.com/python/cpython/commit/88d0663005d258526496d1f8ee0acb7103c69e80
msg302675 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-09-21 09:09
I'm curious. Would you mind to try to run your frozen application using PYTHONMALLOC=debug environment variable on a Python without the fix, to see if you get an error? I'm not sure that the debug hooks on memory allocators are set early enough to detect your specific bug.

https://docs.python.org/dev/using/cmdline.html#envvar-PYTHONMALLOC
msg302676 - (view) Author: Vincent Gatine (nurelin) * Date: 2017-09-21 09:23
Tried your suggestion. Nothing is printed and program still crash.
History
Date User Action Args
2017-09-21 09:23:48nurelinsetmessages: + msg302676
2017-09-21 09:09:32vstinnersetnosy: + vstinner
messages: + msg302675
2017-09-21 06:47:14benjamin.petersonsetstatus: open -> closed
resolution: fixed
messages: + msg302669

stage: patch review -> resolved
2017-09-21 06:08:39python-devsetpull_requests: + pull_request3670
2017-09-21 06:08:25benjamin.petersonsetnosy: + benjamin.peterson
messages: + msg302668
2017-09-20 19:50:14nurelinsetnosy: + nurelin
2017-09-20 19:48:06nurelinsetkeywords: + patch
stage: patch review
pull_requests: + pull_request3668
2017-09-20 14:38:42htgoebelcreate