classification
Title: Removing old buffer support
Type: Stage: resolved
Components: C API Versions: Python 3.11, Python 3.10
process
Status: open Resolution: wont fix
Dependencies: Superseder:
Assigned To: petr.viktorin Nosy List: hroncok, lukasz.langa, methane, miss-islington, petr.viktorin, tarun.johar, vstinner
Priority: normal Keywords: patch

Created on 2020-06-24 13:43 by methane, last changed 2021-08-05 08:33 by petr.viktorin.

Pull Requests
URL Status Linked Edit
PR 21117 merged methane, 2020-06-24 13:58
PR 27437 merged methane, 2021-07-29 07:06
PR 27441 merged miss-islington, 2021-07-29 10:46
Messages (12)
msg372251 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2020-06-24 13:43
https://docs.python.org/3/c-api/objbuffer.html
Old buffer protocol has been deprecated since Python 3.0.
It was useful to make transition from Python 2 easy.
But it's time to remove.
msg372391 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2020-06-25 23:07
New changeset 6f8a6ee59cb7f99f68df8ee9c3e8c8cf19af3eed by Inada Naoki in branch 'master':
bpo-41103: Remove old buffer protocol support (#21117)
https://github.com/python/cpython/commit/6f8a6ee59cb7f99f68df8ee9c3e8c8cf19af3eed
msg381854 - (view) Author: Miro Hrončok (hroncok) * Date: 2020-11-25 17:22
I've just seen a fix of PyQt4 that basically copy pastes (some of) the removed code to PyQt4:

https://src.fedoraproject.org/rpms/PyQt4/pull-request/2#request_diff

What is the benefit of removing this? Is copy pasting the compatibility layer to (possibly many) different projects worth the "cleanup"?

In Fedora, at least 7 packages are broken so far:

PyQt4: https://bugzilla.redhat.com/show_bug.cgi?id=1895298
libsolv: https://bugzilla.redhat.com/show_bug.cgi?id=1896411
m2crypto: https://bugzilla.redhat.com/show_bug.cgi?id=1897148
apsw: https://bugzilla.redhat.com/show_bug.cgi?id=1897500
djvulibre: https://bugzilla.redhat.com/show_bug.cgi?id=1897558
wsaccel: https://bugzilla.redhat.com/show_bug.cgi?id=1899550
webassets: https://bugzilla.redhat.com/show_bug.cgi?id=1899555
msg381865 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-11-25 21:24
PyObject_AsCharBuffer() is dangerous: it returns a dangling pointer by design. PyObject_GetBuffer() design is safer: the API ensures that the buffer remains valid until PyBuffer_Release() is called.

PyQt5 was updated to use the safe PyObject_GetBuffer()/PyBuffer_Release(). PyQt4 is no longer updated, that's why I proposed a downstream fix which copy/paste the old code. Changing PyQt4 to use the safe API was not worth it, since it's complex to redesign the impacted function qpycore_encode().
msg381873 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2020-11-26 02:35
Thank you for reporting. I removed these APIs to get such feedback as early as possible.

We are free to revive these APIs if it breaks too much and some projects can not be fixed before Python 3.10 release.

Some project maintainers ignore deprecations and wait compile errors to fix soemthing. (e.g. https://github.com/rogerbinns/apsw/issues/288#issuecomment-644442322)
That's why we need to break some projects during alpha phase.


> What is the benefit of removing this? Is copy pasting the compatibility layer to (possibly many) different projects worth the "cleanup"?

Oh, no need to copy-paste compatibility layer for all projects. They can migrate to new APIs.
msg398459 - (view) Author: Tarun Johar (tarun.johar) Date: 2021-07-29 06:22
Also filed under https://bugs.python.org/issue44609

PEP 384 and PEP 652 define a stable ABI to be used with Python 3.2 and later.  On Windows, symbols for the stable ABI are exported from the python3.dll shared library.

The following functions are present in Python 3.9 but have been removed from Python 3.10b3:

PyObject_AsCharBuffer()
PyObject_AsReadBuffer()
PyObject_AsWriteBuffer()
PyObject_CheckReadBuffer()

Without these functions, an extension cannot utilize the stable ABI to access the buffer memory of data structures.  The buffer protocol is suggested as an alternative, but the buffer functions PyObject_GetBuffer() and PyBuffer_Release() are not present in the stable ABI.

While these two functions may be added to the stable ABI, removal of the four functions above makes Python 3.10 incompatible with previous versions.  It is requested that the four functions be reinstated and maintained as described in PEP 652.
msg398460 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2021-07-29 06:48
Oh, I didn't notice that APIs deprecated in Python 3.0 are included in stable ABIs defined at Python 3.2!
msg398467 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2021-07-29 07:57
Should I resurrect only function implementation and symbol?
Or may I resurrect definitions in header files too?
msg398476 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-07-29 10:46
New changeset ce5e1a6809b714eb0383219190a076d9f883e008 by Inada Naoki in branch 'main':
bpo-41103: Resurrect the old buffer protocol. (GH-27437)
https://github.com/python/cpython/commit/ce5e1a6809b714eb0383219190a076d9f883e008
msg398477 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-07-29 11:32
New changeset 6b922dae6cae9f212d678a36e8b41f2287cf5441 by Miss Islington (bot) in branch '3.10':
bpo-41103: Resurrect the old buffer protocol. (GH-27437) (GH-27441)
https://github.com/python/cpython/commit/6b922dae6cae9f212d678a36e8b41f2287cf5441
msg398560 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-07-30 15:32
Resolved as "wontfix" due to presence in the stable ABI.
msg398974 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2021-08-05 08:33
These should be removed from the *limited API*, but stay for the stable ABI.
(PEP 652 mentions this can happen but doesn't give details, which will be a bit tricky. These are good candidates for figuring out the process.)

I assigned this to myself; I'll try to get to it for 3.11.


> What is the benefit of removing this? Is copy pasting the compatibility layer to (possibly many) different projects worth the "cleanup"?

Yes. PyObject_AsReadBuffer is dangerous *in general*, but in certain specific cases where you're relying on CPython implementations details it can work. For example, borrowing a buffer from a *string* or *array* will work on CPython, and I don't see that changing any time soon. So, using PyObject_AsReadBuffer is still tech debt, but for some projects that's OK.

All this should be documented in the release notes, though, so the projects know what they're doing. (And the release notes should be written *with the change*, not later, so projects testing early can read them.)
History
Date User Action Args
2021-08-05 08:33:02petr.viktorinsetstatus: closed -> open

nosy: + petr.viktorin
messages: + msg398974

assignee: petr.viktorin
2021-07-30 15:32:35lukasz.langasetstatus: open -> closed
resolution: wont fix
messages: + msg398560

stage: patch review -> resolved
2021-07-29 11:32:13lukasz.langasetmessages: + msg398477
2021-07-29 10:46:56miss-islingtonsetnosy: + miss-islington

pull_requests: + pull_request25970
stage: patch review
2021-07-29 10:46:54lukasz.langasetnosy: + lukasz.langa
messages: + msg398476
2021-07-29 07:57:26methanesetstatus: closed -> open
stage: resolved -> (no value)
resolution: fixed -> (no value)
versions: + Python 3.11
2021-07-29 07:57:09methanesetmessages: + msg398467
2021-07-29 07:06:05methanesetpull_requests: + pull_request25967
2021-07-29 06:48:34methanesetmessages: + msg398460
2021-07-29 06:22:15tarun.joharsetnosy: + tarun.johar
messages: + msg398459
2020-11-26 02:35:36methanesetmessages: + msg381873
2020-11-25 21:24:44vstinnersetmessages: + msg381865
2020-11-25 17:22:55hroncoksetnosy: + vstinner, hroncok
messages: + msg381854
2020-06-25 23:07:42methanesetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2020-06-25 23:07:28methanesetmessages: + msg372391
2020-06-24 13:58:20methanesetkeywords: + patch
stage: patch review
pull_requests: + pull_request20280
2020-06-24 13:43:42methanecreate