classification
Title: PEP 353: Drop support for PyArg_ParseTuple() "#" formats when PY_SSIZE_T_CLEAN is not defined
Type: Stage: resolved
Components: C API Versions: Python 3.10
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: methane, miss-islington, serhiy.storchaka, vstinner
Priority: normal Keywords: patch

Created on 2020-06-10 16:09 by vstinner, last changed 2021-05-10 21:29 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 20781 merged vstinner, 2020-06-10 16:09
PR 20784 merged vstinner, 2020-06-10 16:38
PR 25937 merged methane, 2021-05-06 09:03
PR 25961 merged miss-islington, 2021-05-07 02:56
Messages (11)
msg371216 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-10 16:09
Follow-up of bpo-36381: In Python 3.8, PyArg_ParseTuple() and Py_BuildValue() formats using "int" when PY_SSIZE_T_CLEAN is not defined, but Py_ssize_t when PY_SSIZE_T_CLEAN is defined, were deprecated by:

commit d3c72a223a5f771f964fc34557c55eb5bfa0f5a0
Author: Inada Naoki <songofacandy@gmail.com>
Date:   Sat Mar 23 21:04:40 2019 +0900

    bpo-36381: warn when no PY_SSIZE_T_CLEAN defined (GH-12473)
    
    We will remove int support from 3.10 or 4.0.

I propose to drop support for these formats in Python 3.10. It is a backward incompatible change on purpose, to ensure that all C extensions are compatible with objects larger than 2 GB, and that all C extensions behave the same.

I'm not sure of the effects of this issue on bpo-27499 "PY_SSIZE_T_CLEAN conflicts with Py_LIMITED_API".
msg371218 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-10 16:38
New changeset d36cf5f1d20ce9f111a8fc997104785086e8eee6 by Victor Stinner in branch 'master':
bpo-40943: Replace PY_FORMAT_SIZE_T with "z" (GH-20781)
https://github.com/python/cpython/commit/d36cf5f1d20ce9f111a8fc997104785086e8eee6
msg371219 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-10 16:39
I started to write a long rationale to explain why "#" variants of formats must be deprecated, before I noticed that they are already deprecated! To not lost it, here is my rationale :-)


The C code base of Python switched from int to Py_ssize_t with PEP 353. For not break the backward compatibility, C extension modules have to opt-in for Py_ssize_t by defining PY_SSIZE_T_CLEAN macro in their code base. It affects a few format PyArg_ParseTuple() and Py_BuildValue():
https://docs.python.org/dev/c-api/arg.html

Currently, the documentation says "This behavior will change in a future Python version to only support Py_ssize_t and drop int support. It is best to always define PY_SSIZE_T_CLEAN."

Continue to use int by default prevents to accept content larger than 2 GB and causes issues with limited C API: bpo-27499.

I propose to raise a DeprecationWarning on C extension modules built without the PY_SSIZE_T_CLEAN macro defined.

#warning could be used to emit a warning when building a C extension without PY_SSIZE_T_CLEAN. The problem is that the compiler output is hidden by "pip install", only few developers pay attention to such warnings, and also we should not bother C extensions which don't use PyArg_ParseTuple() and Py_BuildValue().

I prefer to raise a DeprecationWarning exception at runtime since this warning can be made an error when using -Werror: it eases detection of deprecated modules without preventing to build or use a C extension using the deprecated functions.

Two releases after the DeprecationWarning is introduced, we can consider to always raise an exception. Again, I dislike the idea of always require PY_SSIZE_T_CLEAN macro when including "Python.h" header file. I suggest to only raise an exception at runtime when the few affected functions are called.

For example, PyArg_ParseTuple(args, "y*", &buffer) always Py_buffer which uses Py_ssize_t: the behavior does not depend on PY_SSIZE_T_CLEAN.

An alternative is to even deprecate the few "#" formats which are affected by PY_SSIZE_T_CLEAN.
msg371862 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-19 09:45
New changeset 37bb2895561d3e63a631f10875567b4e33b30c07 by Victor Stinner in branch 'master':
bpo-40943: PY_SSIZE_T_CLEAN required for '#' formats (GH-20784)
https://github.com/python/cpython/commit/37bb2895561d3e63a631f10875567b4e33b30c07
msg380656 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-11-10 11:48
libxml2 is hit by this issue:

* https://gitlab.gnome.org/GNOME/libxml2/-/issues/203
* https://gitlab.gnome.org/GNOME/libxml2/-/merge_requests/87 (my proposed fix)
msg393064 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2021-05-06 07:02
There is a still warning, not error.

https://github.com/python/cpython/blob/985ac016373403e8ad41f8d563c4355ffa8d49ff/Python/getargs.c#L2535-L2542
msg393067 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-05-06 07:14
Why PR 20784 has been merged? Was not the plan to emit a deprecation warning first?
msg393068 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2021-05-06 07:19
>Why PR 20784 has been merged? Was not the plan to emit a deprecation warning first?

We had been emitted DeprecationWarning since Python 3.8. See GH-12473.
GH-20784 changed the DeprecationWarning to SystemError.
msg393069 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-05-06 07:25
Sorry, I was confused by Victor's long msg371219.
msg393165 - (view) Author: miss-islington (miss-islington) Date: 2021-05-07 03:18
New changeset 569ca81adf0be92be8752f6cc6492117f9ef3c0b by Miss Islington (bot) in branch '3.10':
bpo-40943: Fix skipitem() didn't raise SystemError (GH-25937)
https://github.com/python/cpython/commit/569ca81adf0be92be8752f6cc6492117f9ef3c0b
msg393434 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-05-10 21:29
commit 4ebf4a6bfad4afcbab3baf9c0159c7767e2a64c0
Author: Inada Naoki <songofacandy@gmail.com>
Date:   Fri May 7 11:56:48 2021 +0900

    bpo-40943: Fix skipitem() didn't raise SystemError (GH-25937)
    
    `convertitem()` raises `SystemError` when '#' is used without `PY_SSIZE_T_CLEAN`.
    This commit makes `skipitem()` raise it too.
History
Date User Action Args
2021-05-10 21:29:50vstinnersetmessages: + msg393434
2021-05-07 03:29:07methanesetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2021-05-07 03:18:45miss-islingtonsetmessages: + msg393165
2021-05-07 02:56:58miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request24620
2021-05-06 09:03:28methanesetstage: resolved -> patch review
pull_requests: + pull_request24605
2021-05-06 07:25:32serhiy.storchakasetmessages: + msg393069
2021-05-06 07:19:57methanesetmessages: + msg393068
2021-05-06 07:14:36serhiy.storchakasetmessages: + msg393067
2021-05-06 07:02:17methanesetstatus: closed -> open
resolution: fixed -> (no value)
messages: + msg393064
2020-11-10 11:48:47vstinnersetmessages: + msg380656
2020-06-19 11:49:25vstinnersetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2020-06-19 09:45:39vstinnersetmessages: + msg371862
2020-06-10 16:48:54vstinnersettitle: PEP 353: Drop support for PyArg_ParseTuple() format when PY_SSIZE_T_CLEAN is not defined -> PEP 353: Drop support for PyArg_ParseTuple() "#" formats when PY_SSIZE_T_CLEAN is not defined
2020-06-10 16:39:46vstinnersetmessages: + msg371219
2020-06-10 16:38:28vstinnersetpull_requests: + pull_request19980
2020-06-10 16:38:13vstinnersetmessages: + msg371218
2020-06-10 16:24:12vstinnersettitle: Drop support for PyArg_ParseTuple() format when PY_SSIZE_T_CLEAN is not defined -> PEP 353: Drop support for PyArg_ParseTuple() format when PY_SSIZE_T_CLEAN is not defined
2020-06-10 16:09:22vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request19978
2020-06-10 16:09:12vstinnercreate