classification
Title: PyMem_Malloc(): check that the GIL is hold in debug hooks
Type: Stage:
Components: Versions: Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: python-dev, vstinner
Priority: normal Keywords: patch

Created on 2016-03-14 22:52 by vstinner, last changed 2016-03-18 10:04 by python-dev. This issue is now closed.

Files
File name Uploaded Description Edit
pymem_gil.patch vstinner, 2016-03-14 22:52 review
Messages (11)
msg261787 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-03-14 22:52
With the issue #26558, debug hooks of PyObject_Malloc() now checks that the GIL is hold.

I left PyMem_Malloc() unchanged, because I am not 100% sure that it's ok yet to implement the same check in PyMem_Malloc(). So I opened this issue.

Python 3 doc explicitly asks that the GIL is hold to call PyMem_Malloc():
https://docs.python.org/dev/c-api/memory.html#memory-interface

Python 2 doc doesn't say anything about the GIL:
https://docs.python.org/2/c-api/memory.html#memory-interface

Usually, functions prefixed by "Py" require the GIL to be hold.

In practice, currently PyMem_Malloc() is implemented with malloc() which is thread-safe.

In the issue #26249, I tested numpy, lxml, Pillow and cryptography with all debug hooks enabled, including GIL checks in PyMem_Malloc() and PyObject_Malloc(). I only found one bug in numpy: https://github.com/numpy/numpy/pull/7404

Attached patch changes debug hooks on PyMem_Malloc() to ensure that the GIL is hold.
msg261850 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-03-16 11:17
New changeset 58644086c195 by Victor Stinner in branch 'default':
Fail if PyMem_Malloc() is called without holding the GIL
https://hg.python.org/cpython/rev/58644086c195
msg261851 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-03-16 11:17
I pushed the patch (I added an unit test).

If too many users complain, we can revert the change since it's now a small change.
msg261854 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-03-16 14:36
New changeset ae76a1046bb9 by Victor Stinner in branch 'default':
Fix usage of PyMem_Malloc() in os.stat()
https://hg.python.org/cpython/rev/ae76a1046bb9
msg261865 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-03-16 22:21
New changeset 5b4f173f0690 by Victor Stinner in branch 'default':
faulthandler now works in non-Python threads
https://hg.python.org/cpython/rev/5b4f173f0690

New changeset d6d64168ee8c by Victor Stinner in branch 'default':
Py_FatalError: disable faulthandler earlier
https://hg.python.org/cpython/rev/d6d64168ee8c
msg261866 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-03-16 22:27
New changeset 959e58cfbde9 by Victor Stinner in branch 'default':
Fix usage of PyMem_Malloc() in overlapped.c
https://hg.python.org/cpython/rev/959e58cfbde9
msg261867 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-03-16 22:45
New changeset 73d8adc0d5ea by Victor Stinner in branch '3.5':
Fix usage of PyMem_Malloc() in overlapped.c
https://hg.python.org/cpython/rev/73d8adc0d5ea
msg261874 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-03-17 00:43
Reopen. I found and fixed two bugs in os and _overlapped modules (_overlapped is used by asyncio on Windows).
msg261875 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-03-17 00:46
New changeset ce62df22f3bf by Victor Stinner in branch '3.5':
Py_FatalError: disable faulthandler earlier
https://hg.python.org/cpython/rev/ce62df22f3bf
msg261902 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-03-17 08:19
Hum ok, all buildbots look to be happy again :-) I close the issue (again).
msg261954 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-03-18 10:04
New changeset 7b079adb0774 by Victor Stinner in branch 'default':
Enhance documentation on malloc debug hooks
https://hg.python.org/cpython/rev/7b079adb0774
History
Date User Action Args
2016-03-18 10:04:44python-devsetmessages: + msg261954
2016-03-17 08:20:02vstinnersetstatus: open -> closed
resolution: fixed
2016-03-17 08:19:56vstinnersetmessages: + msg261902
2016-03-17 00:46:28python-devsetmessages: + msg261875
2016-03-17 00:43:33vstinnersetstatus: closed -> open
resolution: fixed -> (no value)
messages: + msg261874
2016-03-16 22:45:40python-devsetmessages: + msg261867
2016-03-16 22:27:39python-devsetmessages: + msg261866
2016-03-16 22:21:23python-devsetmessages: + msg261865
2016-03-16 14:36:17python-devsetmessages: + msg261854
2016-03-16 11:17:47vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg261851
2016-03-16 11:17:03python-devsetnosy: + python-dev
messages: + msg261850
2016-03-14 22:54:38vstinnersettitle: PyMem_Malloc(): check that the GIL is held in debug hooks -> PyMem_Malloc(): check that the GIL is hold in debug hooks
2016-03-14 22:52:53vstinnercreate