classification
Title: Debug hooks on memory allocators are not thread safe (serialno variable)
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.8, Python 3.7, Python 3.6, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: vstinner, xiang.zhang
Priority: normal Keywords:

Created on 2017-09-14 17:36 by xiang.zhang, last changed 2019-05-15 02:45 by vstinner. This issue is now closed.

Messages (7)
msg302187 - (view) Author: Xiang Zhang (xiang.zhang) * (Python committer) Date: 2017-09-14 17:36
PyMem_Raw* APIs should be thread safe. But in debug mode, they are not that simple and increment a global variable *serialno* and the incrementing action is not thread safe.
msg330947 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-12-03 15:19
I looked at _Py_atomic_address to avoid atomic "serialno++", but we don't have atomic_fetch_add(). We could implement it using a loop and atomic_compare_exchange_strong()... but we don't have atomic_compare_exchange_strong() neither.

I tried to add a mutex, but there are some pratical issues:

* bpo-35388: question about calling Py_Initialize() / Py_Finalize() multiple times
* I modified _PyRuntimeState_Init() to initialize the lock.

_PyRuntimeState_Init() calls PyThread_acquire_lock() which calls PyMem_RawMalloc(). Problem: PyMem_RawMalloc() requires the lock. I worked around the isuse using "if (_PyRuntime.mem.mutex != NULL) {".
msg330948 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-12-03 15:21
By the way, the previous attempt to add _PyRuntime.mem had to be reverted. See bpo-32096, bpo-30860 and the commit:

commit 9e87e7776f7ace66baaf7247233afdabd00c2b44
Author: Victor Stinner <victor.stinner@gmail.com>
Date:   Fri Nov 24 12:09:24 2017 +0100

    bpo-32096: Remove obj and mem from _PyRuntime (#4532)
    
    bpo-32096, bpo-30860:  Partially revert the commit
    2ebc5ce42a8a9e047e790aefbf9a94811569b2b6:
    
    * Move structures back from Include/internal/mem.h to
      Objects/obmalloc.c
    * Remove _PyObject_Initialize() and _PyMem_Initialize()
    * Remove Include/internal/pymalloc.h
    * Add test_capi.test_pre_initialization_api():
       Make sure that it's possible to call Py_DecodeLocale(), and then call
       Py_SetProgramName() with the decoded string, before Py_Initialize().
    
    PyMem_RawMalloc() and Py_DecodeLocale() can be called again before
    _PyRuntimeState_Init().
    
    Co-Authored-By: Eric Snow <ericsnowcurrently@gmail.com>
msg330949 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-12-03 15:22
See also bpo-35265: "Internal C API: pass the memory allocator in a context".

The most complex part is the Python initialization which changes the memory allocator *and* allocate memory. We have to remain which allocator has been used to allocate data, to be able to release memory at exit.
msg330957 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-12-03 16:41
Note: PTHREAD_MUTEX_INITIALIZER can be used to statically initialize a mutex.
msg330959 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-12-03 16:45
See also bpo-35368: [2.7] Make PyMem_Malloc() thread-safe in debug mode.
msg342538 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-05-15 02:45
Python 3.8 has been fixed. I disabled serialno field by default: PYMEM_DEBUG_SERIALNO is not defined by default. You have to opt-in to get this bug :-) I don't see any easy fix older Python versions. I close the issue.
History
Date User Action Args
2019-05-15 02:45:20vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg342538

stage: needs patch -> resolved
2018-12-03 16:45:31vstinnersetmessages: + msg330959
2018-12-03 16:43:57vstinnersettitle: PyMem_Raw* API in debug mode are not thread safe -> Debug hooks on memory allocators are not thread safe (serialno variable)
versions: + Python 2.7, Python 3.8
2018-12-03 16:41:18vstinnersetmessages: + msg330957
2018-12-03 15:22:40vstinnersetmessages: + msg330949
2018-12-03 15:21:01vstinnersetmessages: + msg330948
2018-12-03 15:19:07vstinnersetmessages: + msg330947
2017-09-14 17:36:34xiang.zhangcreate