This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: Schedule Py_UNICODE API removal
Type: Stage: resolved
Components: Library (Lib) Versions: Python 3.9
Status: closed Resolution: duplicate
Dependencies: Superseder: Prepare for removing the legacy Unicode C API
View: 36346
Assigned To: Nosy List: methane, serhiy.storchaka, vstinner
Priority: normal Keywords:

Created on 2019-10-27 16:02 by vstinner, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Messages (6)
msg355475 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-10-27 16:02
Python 3.3 deprecated the C API functions using Py_UNICODE type. Examples in the doc:


Currently, functions removal is scheduled for Python 4.0 but I would prefer that Python 4.0 doesn't have a long list of removed features, but no more than usual. So I'm trying to remove a few functions from Python 3.9, and try to prepare removal for others.

Py_UNICODE C API was mostly kept for backward compatibility with Python 2. Since Python 2 support ends at the end of the year, can we start to organize Py_UNICODE C API removal?

There are multiple questions:

* Should we drop the whole API at once? Or can we/should we start by removing a few functions, and then the others?
* Deprecation warnings are emitted at compilation. But I'm not aware of DeprecationWarning emited at runtime. IMHO we should emit DesprecationWarning at runtime during at least one release, so most developers ignore compilation warnings.

I propose to:

* (Right now) write an exhaustive list of all deprecated APIs: functions, constants, types, etc.
* Modify C code to emit DeprecationWarning at runtime in Python 3.9
* Experiment a modified Python without these APIs and test how many projects are broken by this removal: see PEP 608
* Schedule the actual removal of all these APIS from Python 3.10

Honestly, if the removal is causing too much issues, I'm fine to make slowdown the removal. It's just a matter of clearly communicating our intent.

Maybe we should also announce the scheduled removal in What's in Python 3.9 and in the capi-sig mailing list.
msg355478 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-10-27 16:15
> (Right now) write an exhaustive list of all deprecated APIs: functions, constants, types, etc.

I searched "4.0" in the documentation:

* Py_UNICODE type
* array.array: "u" type
* PyArg_ParseTuple, Py_BuildValue: "u", "u#", "Z", "Z#" formats

* PyUnicode_FromUnicode()
* PyUnicode_GetSize(), PyUnicode_GET_SIZE()
* PyUnicode_AsUnicode(), PyUnicode_AS_UNICODE(), PyUnicode_AS_DATA()
* PyUnicode_AsUnicodeAndSize()
* PyUnicode_AsUnicodeCopy()

* PyUnicode_FromObject()
* PyLong_FromUnicode()
* PyUnicode_TransformDecimalToASCII()

* PyUnicode_Encode()
* PyUnicode_EncodeUTF7()
* PyUnicode_EncodeUTF8()
* PyUnicode_EncodeUTF32()
* PyUnicode_EncodeUTF16()
* PyUnicode_EncodeUnicodeEscape()
* PyUnicode_EncodeRawUnicodeEscape()
* PyUnicode_EncodeLatin1()
* PyUnicode_EncodeASCII()
* PyUnicode_EncodeMBCS()
* PyUnicode_EncodeCharmap()
* PyUnicode_TranslateCharmap()
msg355524 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-10-28 11:06
A preleminary step was to modify PyUnicode_AsWideChar() and PyUnicode_AsWideCharString() to remove the internal caching: it has been done in Python 3.8.0 with bpo-30863.
msg355528 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2019-10-28 11:20
I want to remove them in 3.10 too.
If we chose the annual release cycle, I'm OK to postpone the removal to 3.11.

FWIW, ujson is apopular extension that uses these APIs.
But it is not maintained for a long time, and there are many alternative JSON libraries.
msg355532 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-10-28 11:28
See issue36346.

First we need to add compile-time deprecation warnings to all C API. Then add runtime deprecation warnings. And since this is a major compatibility breakage, it can take longer deprecation period.
msg355534 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-10-28 11:34
> See issue36346.

Oh, I failed to find this issue. Ok, I close mine as a duplicate.
Date User Action Args
2022-04-11 14:59:22adminsetgithub: 82785
2019-10-28 11:34:46vstinnersetstatus: open -> closed
superseder: Prepare for removing the legacy Unicode C API
messages: + msg355534

resolution: duplicate
stage: resolved
2019-10-28 11:28:03serhiy.storchakasetmessages: + msg355532
2019-10-28 11:20:00methanesetmessages: + msg355528
2019-10-28 11:06:13vstinnersetnosy: + methane, serhiy.storchaka
messages: + msg355524
2019-10-27 16:15:56vstinnersetmessages: + msg355478
2019-10-27 16:02:05vstinnercreate