This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: libpython.so 3.9.8 drops symbol used by third party extension module(s)
Type: crash Stage: resolved
Components: Interpreter Core Versions: Python 3.10, Python 3.9
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: christian.heimes, doko, hroncok, lukasz.langa, pablogsal, serhiy.storchaka, vstinner
Priority: release blocker Keywords: 3.9regression

Created on 2021-11-10 12:11 by doko, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Messages (8)
msg406085 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2021-11-10 12:11
[reported in Debian as https://bugs.debian.org/998854]

3.9.8, compared to 3.9.7 introduces a regression, making at least one third party extension fail at runtime. The symbol changes in 3.9.8 are:

- _PyUnicode_DecodeUnicodeEscape@Base
+ _PyUnicode_DecodeRawUnicodeEscapeStateful@Base
+ _PyUnicode_DecodeUnicodeEscapeInternal@Base
+ _PyUnicode_DecodeUnicodeEscapeStateful@Base

Affected at least is the typed-ast extension.  Yes, you can rebuild the extension with 3.9.8, but then it doesn't work with earlier 3.9 versions.  Just dropping the symbol on a stable branch is suboptimal.
msg406086 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-11-10 12:25
Aren't these private symbols?
msg406087 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-11-10 12:27
Yes, they are private symbols. 3rd parties should not use them.

The symbol _PyUnicode_DecodeUnicodeEscape was replace in GH-28953: "bpo-45467: Fix IncrementalDecoder and StreamReader in the "raw-unicode-escape" codec".
msg406088 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-11-10 12:29
Yeah, the ABI guarantees are for the public C-API (both the restricted/limited and the general one) but not for private symbols.
msg406089 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-11-10 12:34
Matthias mentions the upstream tickets

  https://github.com/python/typed_ast/issues/169
  https://github.com/python/typed_ast/issues/170

on the BDO. The issue affects typed-ast and indirectly older versions of black. Upstream is considering to EOL typed-ast package.
msg406091 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2021-11-10 12:39
the two users of typed-ast (at least in Debian) are black and mypy.  Apparently black stopped using typed-ast recently.
msg406100 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-11-10 14:02
I suggest to close this issue. We are free to remove private functions in minor releases (3.9.x) without any warning. Private functions are excluded from the backward compatibility warranty.

For example: "Names prefixed by an underscore, such as _Py_InternalState, are private API that can change without notice even in patch releases."
https://docs.python.org/dev/c-api/stable.html#stable

In C, it's easy to call the decode() method of a string object and pass the expected encoding and error handler.
msg406101 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-11-10 14:05
- _PyUnicode_DecodeUnicodeEscape@Base

Removed by:

commit 7c722e32bf582108680f49983cf01eaed710ddb9
Author: Serhiy Storchaka <storchaka@gmail.com>
Date:   Thu Oct 14 20:03:29 2021 +0300

    [3.9] bpo-45461: Fix IncrementalDecoder and StreamReader in the "unicode-escape" codec (GH-28939) (GH-28945)
    
    They support now splitting escape sequences between input chunks.
    
    Add the third parameter "final" in codecs.unicode_escape_decode().
    It is True by default to match the former behavior.
    (cherry picked from commit c96d1546b11b4c282a7e21737cb1f5d16349656d)
    
    Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
History
Date User Action Args
2022-04-11 14:59:52adminsetgithub: 89936
2021-11-10 14:11:03lukasz.langasetstatus: open -> closed
resolution: wont fix
stage: resolved
2021-11-10 14:05:12vstinnersetmessages: + msg406101
2021-11-10 14:02:59vstinnersetnosy: + serhiy.storchaka
2021-11-10 14:02:39vstinnersetnosy: + vstinner
messages: + msg406100
2021-11-10 13:25:19hroncoksetversions: + Python 3.10
2021-11-10 13:25:10hroncoksetnosy: + hroncok
2021-11-10 12:39:12dokosetmessages: + msg406091
2021-11-10 12:34:42christian.heimessetmessages: + msg406089
2021-11-10 12:29:25pablogsalsetmessages: + msg406088
2021-11-10 12:27:48christian.heimessetnosy: + christian.heimes
messages: + msg406087
2021-11-10 12:25:22pablogsalsetnosy: + pablogsal
messages: + msg406086
2021-11-10 12:11:49dokocreate