New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Convert PyTuple_GET_ITEM() macro to a function call with additional checks in debug mode #79380
Comments
Currently, even when Python is compiled in debug mode, PyTuple_GET_ITEM() doesn't check that the first argument is a tuple objet and that the second argument is valid index. It can lead to a crash and Python doesn't help debugging. I propose to convert the macro to a function call and use regular assertions to abort Python if the C API is misused. I propose to use a function call rather than abusing the preprocessor syntax like (assert(...),expr) syntax used in unicodeobject.h, because I the preprocessor causes complex bugs (difficult to understand and to fix/work around) and because later I would like to experiment to be able to compile C extensions to always use function calls, but get a different implementation depending on the "Python runtime". I elaborated this idea on this website: I am working on an implementation. |
Currently, PyTuple_GET_ITEM() can be used with &PyTuple_GET_ITEM(). If we convert PyTuple_GET_ITEM() to a function returning PyObject*, it's no longer possible. My PR 10434 prepares the code for that. |
Copy of my comment: This change breaks the backward compatibility when a C extension is compiled with Py_DEBUG. So I'm not sure that it should be merged into Python 3.8. Maybe we should add a new opt-in experimental option to enable it, instead of Py_DEBUG? |
I don't think this should be merged. A lot of code of uses PyTuple_GET_ITEM(). Also, the "problem" your solving doesn't seem to exist in practice. Further, I worry that a compiler may choose not to inline on occasion, leading to performance regressions in code that has already been finely tuned. |
Sorry my comment on the PR, a friend contacted me because (...) he misused PyTuple_GET_ITEM() :-)
Oh, maybe I didn't explain properly my change. I don't want to change the "default" API nor the "default" implementation. My change is enabled enabled if you compile Python in debug mode, if Py_DEBUG is defined. In fact, it's an early attempt to see how we can introduce a new C API in Python. It seems like abusing Py_DEBUG was a bad idea, so I close my PR and I will close this issue as well. I will start a discussion on python-dev later, when I will have a plan. See also the discussion on the PR: |
Thanks. I don't really know enough about compilers and required-vs-optional C99 semantics to know whether these changes are safe. These are very old APIs and are used throughout the entire Python ecosystem. It would pay to be somewhat cautious. The whole C-API project is predicated on giving a 2x speed-up in exchange for destabilizing changes. These cross-module inline function-for-macro swaps aren't aligned with that goal. At the very best, the swaps will be performance neutral. At worst, they will invisibly degrade a huge swath of code, someone of which is finely-tuned (people generally only use PyTuple_GET_ITEM() in places where they want better performance than the existing function based alternative) |
Oops, I forgot to close the issue. This change was an early attempt (tagged as WIP/DO-NOT-MERGE) to expriment changing the implementation without touching the API. But I was wrong, it does change the API (&PyTuple_GET_ITEM(ob, i) causes a compilation error) and so I closed my PR. Instead, I opened a thread on python-dev to better explain the whole context and my intent: I invite everyone to discuss there instead :-) |
I created bpo-35206 "Add a new experimental _Py_CAPI2 API". |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: