classification
Title: Order of decrementing reference counts in meth_dealloc
Type: crash Stage: resolved
Components: C API, Interpreter Core Versions: Python 3.10, Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Henry Schreiner, YannickJadoul, miss-islington, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2020-10-12 14:23 by YannickJadoul, last changed 2020-10-13 05:55 by serhiy.storchaka. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 22670 merged YannickJadoul, 2020-10-12 16:10
PR 22674 merged miss-islington, 2020-10-12 21:06
Messages (7)
msg378500 - (view) Author: Yannick Jadoul (YannickJadoul) * Date: 2020-10-12 14:23
In Python 3.9, the line `Py_XDECREF(PyCFunction_GET_CLASS(m));` was added to `meth_dealloc` (in `methodobject.c`). Unfortunately for pybind11, it's inserted exactly two lines too low, since it accesses the `PyMethodDef` and we store the `PyMethodDef` instance in the capsule that's used as `self`-argument of the `PyCFunction`.

Result: UB, since `Py_XDECREF(m->m_self);` brings down the refcount of the capsule to 0 and (indirectly) frees the `PyMethodDef`, while its contents are now still accessed.

From the pybind11 perspective, it would be optimal if this could be fixed in CPython itself, by moving up this one `Py_XDECREF` 2 lines. This would a) be more efficient than creating a workaround, and b) allow old, existing versions of pybind11 to work with Python 3.9 (well, 3.9.1, then, hopefully); the user base of pybind11 has grown quite a bit and now includes giants like scipy or some Google libraries.
I will make a PR implementing this, soon.

If there's a different, recommended way of creating these `PyCFunctionObject`s dynamically and cleaning up the `PyMethodDef`, we'd be interested as well, to make sure these kinds of breakages are avoided in the future.

Apologies for only figuring out now how to debug this, using valgrind. Up until yesterday, we only saw some failures in CI on macOS, but it was hard to reproduce and debug locally.


- Related issue: https://bugs.python.org/issue41237
- pybind11 issue: https://github.com/pybind/pybind11/issues/2558
- pybind11 PR: https://github.com/pybind/pybind11/pull/2576
msg378502 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-10-12 15:26
Do you mind to create a PR Yannick?
msg378506 - (view) Author: Yannick Jadoul (YannickJadoul) * Date: 2020-10-12 16:12
Yes, sorry for the delay; I got caught up in something else.

Meanwhile, https://github.com/python/cpython/pull/22670 should solve our issues. I think Henry confirmed this locally?
msg378511 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-10-12 17:51
Thank you Yannick for your report and PR!
msg378525 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-10-12 21:06
New changeset 04b8631d84a870dda456ef86039c1baf34d08500 by Yannick Jadoul in branch 'master':
bpo-42015: Reorder dereferencing calls in meth_dealloc, to make sure m_self is kept alive long enough (GH-22670)
https://github.com/python/cpython/commit/04b8631d84a870dda456ef86039c1baf34d08500
msg378528 - (view) Author: miss-islington (miss-islington) Date: 2020-10-12 21:29
New changeset 8a12503b4532e33d590ecea7eb94cd0e6b0f1488 by Miss Skeleton (bot) in branch '3.9':
bpo-42015: Reorder dereferencing calls in meth_dealloc, to make sure m_self is kept alive long enough (GH-22670)
https://github.com/python/cpython/commit/8a12503b4532e33d590ecea7eb94cd0e6b0f1488
msg378529 - (view) Author: Henry Schreiner (Henry Schreiner) * Date: 2020-10-12 21:51
I tested before the patch, and I got 17 segfaults running a pybind11 module 20 times. After the patch, I ran about 50 times and had no segfaults!
History
Date User Action Args
2020-10-13 05:55:13serhiy.storchakasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2020-10-12 21:51:24Henry Schreinersetmessages: + msg378529
2020-10-12 21:29:10miss-islingtonsetmessages: + msg378528
2020-10-12 21:06:43miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request21648
2020-10-12 21:06:27serhiy.storchakasetmessages: + msg378525
2020-10-12 17:51:38serhiy.storchakasetmessages: + msg378511
2020-10-12 16:12:05YannickJadoulsetmessages: + msg378506
2020-10-12 16:10:49YannickJadoulsetkeywords: + patch
stage: patch review
pull_requests: + pull_request21643
2020-10-12 15:26:28serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg378502
2020-10-12 14:54:47Henry Schreinersetnosy: + Henry Schreiner
2020-10-12 14:23:44YannickJadoulcreate