This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author YannickJadoul
Recipients YannickJadoul
Date 2020-10-12.14:23:43
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1602512624.15.0.894803367223.issue42015@roundup.psfhosted.org>
In-reply-to
Content
In Python 3.9, the line `Py_XDECREF(PyCFunction_GET_CLASS(m));` was added to `meth_dealloc` (in `methodobject.c`). Unfortunately for pybind11, it's inserted exactly two lines too low, since it accesses the `PyMethodDef` and we store the `PyMethodDef` instance in the capsule that's used as `self`-argument of the `PyCFunction`.

Result: UB, since `Py_XDECREF(m->m_self);` brings down the refcount of the capsule to 0 and (indirectly) frees the `PyMethodDef`, while its contents are now still accessed.

From the pybind11 perspective, it would be optimal if this could be fixed in CPython itself, by moving up this one `Py_XDECREF` 2 lines. This would a) be more efficient than creating a workaround, and b) allow old, existing versions of pybind11 to work with Python 3.9 (well, 3.9.1, then, hopefully); the user base of pybind11 has grown quite a bit and now includes giants like scipy or some Google libraries.
I will make a PR implementing this, soon.

If there's a different, recommended way of creating these `PyCFunctionObject`s dynamically and cleaning up the `PyMethodDef`, we'd be interested as well, to make sure these kinds of breakages are avoided in the future.

Apologies for only figuring out now how to debug this, using valgrind. Up until yesterday, we only saw some failures in CI on macOS, but it was hard to reproduce and debug locally.


- Related issue: https://bugs.python.org/issue41237
- pybind11 issue: https://github.com/pybind/pybind11/issues/2558
- pybind11 PR: https://github.com/pybind/pybind11/pull/2576
History
Date User Action Args
2020-10-12 14:23:44YannickJadoulsetrecipients: + YannickJadoul
2020-10-12 14:23:44YannickJadoulsetmessageid: <1602512624.15.0.894803367223.issue42015@roundup.psfhosted.org>
2020-10-12 14:23:44YannickJadoullinkissue42015 messages
2020-10-12 14:23:43YannickJadoulcreate