classification
Title: Convert PyTuple_GET_ITEM() macro to a function call with additional checks in debug mode
Type: Stage: resolved
Components: Interpreter Core Versions: Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: rhettinger, vstinner
Priority: normal Keywords: patch

Created on 2018-11-09 15:32 by vstinner, last changed 2018-11-10 01:43 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 10434 merged vstinner, 2018-11-09 15:38
PR 10435 closed vstinner, 2018-11-09 15:58
Messages (9)
msg329523 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-11-09 15:32
Currently, even when Python is compiled in debug mode, PyTuple_GET_ITEM() doesn't check that the first argument is a tuple objet and that the second argument is valid index. It can lead to a crash and Python doesn't help debugging.

I propose to convert the macro to a function call and use regular assertions to abort Python if the C API is misused.

I propose to use a function call rather than abusing the preprocessor syntax like (assert(...),expr) syntax used in unicodeobject.h, because I the preprocessor causes complex bugs (difficult to understand and to fix/work around) and because later I would like to experiment to be able to compile C extensions to always use function calls, but get a different implementation depending on the "Python runtime".

I elaborated this idea on this website:

* https://pythoncapi.readthedocs.io/runtimes.html#debug-build
* https://pythoncapi.readthedocs.io/

I am working on an implementation.
msg329525 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-11-09 15:40
Currently, PyTuple_GET_ITEM() can be used with &PyTuple_GET_ITEM(). If we convert PyTuple_GET_ITEM() to a function returning PyObject*, it's no longer possible. My PR 10434 prepares the code for that.
msg329529 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-11-09 15:56
New changeset d17a693fa08ce9f2d35acbb1f76e20bdae3e01da by Victor Stinner in branch 'master':
bpo-35199: Add an internal _PyTuple_ITEMS() macro (GH-10434)
https://github.com/python/cpython/commit/d17a693fa08ce9f2d35acbb1f76e20bdae3e01da
msg329530 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-11-09 16:05
Copy of my comment:
https://github.com/python/cpython/pull/10435#issuecomment-437405620

This change breaks the backward compatibility when a C extension is compiled with Py_DEBUG. So I'm not sure that it should be merged into Python 3.8. Maybe we should add a new opt-in experimental option to enable it, instead of Py_DEBUG?
msg329536 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2018-11-09 16:47
I don't think this should be merged.  A lot of code of uses PyTuple_GET_ITEM().  Also, the "problem" your solving doesn't seem to exist in practice.  Further, I worry that a compiler may choose not to inline on occasion, leading to performance regressions in code that has already been finely tuned.
msg329570 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-11-09 23:48
> I don't think this should be merged.  A lot of code of uses PyTuple_GET_ITEM(). Also, the "problem" your solving doesn't seem to exist in practice.

Sorry my comment on the PR, a friend contacted me because (...) he misused PyTuple_GET_ITEM() :-)

> Further, I worry that a compiler may choose not to inline on occasion, leading to performance regressions in code that has already been finely tuned.

Oh, maybe I didn't explain properly my change. I don't want to change the "default" API nor the "default" implementation. My change is enabled enabled if you compile Python in debug mode, if Py_DEBUG is defined.

In fact, it's an early attempt to see how we can introduce a new C API in Python. It seems like abusing Py_DEBUG was a bad idea, so I close my PR and I will close this issue as well. I will start a discussion on python-dev later, when I will have a plan.

See also the discussion on the PR:
https://github.com/python/cpython/pull/10435#issuecomment-437530430
msg329572 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2018-11-10 00:10
Thanks.  I don't really know enough about compilers and required-vs-optional C99 semantics to know whether these changes are safe.  These are very old APIs and are used throughout the entire Python ecosystem.  It would pay to be somewhat cautious.   

The whole C-API project is predicated on giving a 2x speed-up in exchange for destabilizing changes.  These cross-module inline function-for-macro swaps aren't aligned with that goal.  At the very best, the swaps will be performance neutral.  At worst, they will invisibly degrade a huge swath of code, someone of which is finely-tuned (people generally only use PyTuple_GET_ITEM() in places where they want better performance than the existing function based alternative)
msg329577 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-11-10 00:36
Oops, I forgot to close the issue.

This change was an early attempt (tagged as WIP/DO-NOT-MERGE) to expriment changing the implementation without touching the API. But I was wrong, it does change the API (&PyTuple_GET_ITEM(ob, i) causes a compilation error) and so I closed my PR.

Instead, I opened a thread on python-dev to better explain the whole context and my intent:
https://mail.python.org/pipermail/python-dev/2018-November/155702.html

I invite everyone to discuss there instead :-)
msg329585 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-11-10 01:43
I created bpo-35206 "Add a new experimental _Py_CAPI2 API".
History
Date User Action Args
2018-11-10 01:43:18vstinnersetmessages: + msg329585
2018-11-10 00:36:49vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg329577

stage: patch review -> resolved
2018-11-10 00:10:36rhettingersetmessages: + msg329572
2018-11-09 23:48:50vstinnersetmessages: + msg329570
2018-11-09 16:47:38rhettingersetnosy: + rhettinger
messages: + msg329536
2018-11-09 16:05:18vstinnersetmessages: + msg329530
2018-11-09 15:58:12vstinnersetpull_requests: + pull_request9709
2018-11-09 15:56:52vstinnersetmessages: + msg329529
2018-11-09 15:40:52vstinnersetmessages: + msg329525
2018-11-09 15:38:19vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request9708
2018-11-09 15:32:02vstinnercreate