classification
Title: Remove cross-version binary compatibility
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: barry, ncoghlan, ned.deily, njs, paul.moore, pitrou, scoder, serhiy.storchaka, steve.dower, tim.golden, vstinner, zach.ware
Priority: normal Keywords: patch

Created on 2017-12-20 17:27 by pitrou, last changed 2019-06-12 13:11 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 4944 merged pitrou, 2017-12-20 17:29
Messages (12)
msg308753 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-12-20 17:27
See original discussion on python-dev:
https://mail.python.org/pipermail/python-dev/2017-December/151328.html

Summary: binary compatibility for C extensions which don't use the stable ABI is currently "best effort" and uses distinguished flags in the tp_flags field of type objects to indicate whether a field is physically present or not.  Unfortunately, tp_flags is 32 bits and therefore a scarce resource.  Also, binary compatibility is 1) not guaranteed anyway 2) of less use nowadays.

So we propose to remove the binary compatibility requirement when creating static type objects (i.e. not using the stable ABI).
msg308881 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-12-21 16:51
Totally okay with this in theory, but wanted to clarify that the linked PR is really just taking advantage of removing the guarantee?

That is, the change isn't necessary to remove the requirement.
msg308882 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-12-21 16:54
That's basically exact.  Actually, there is no ABI guarantee at all (when not using the stable ABI), just a best effort for advanced users who know what they're doing.

Actually, this PR stems from Nathaniel's message here:
https://mail.python.org/pipermail/python-dev/2017-December/151345.html
msg308884 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-12-21 17:01
Good to know. As I said on the other issue, Windows won't load extension modules built for a different version anyway unless they use the stable ABI, so there's no real reason these flags need to be binary compatible between 3.x versions.
msg313182 - (view) Author: Stefan Behnel (scoder) * (Python committer) Date: 2018-03-03 08:01
I'm ok with "removing" this "guarantee" (although it seems too late to apply this specific change to 3.7 now).

While Cython users do ask for a way to build cross-version binaries from time to time, it's neither supported nor planned, and the underlying problem is IMHO solved better via tooling like the manylinux wheel build.

I also generally consider it good to let user code take advantage of new features in new CPython releases, even if it means that they need to retranslate their Cython code from time to time. The 1.5 years release cycle of CPython should exceed that of most "still in maintenance" extension modules out there, and it's long enough to not impose a real burden on their maintainers.
msg313351 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2018-03-06 18:18
Yes, let's retarget to 3.8.
msg319944 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-06-19 09:05
In which version of Python tp_finalize slot has been added? Does it mean that Python 3.8 wil crash on loading C extensions compiled on Python older than this version?
msg319947 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2018-06-19 09:11
> Does it mean that Python 3.8 wil crash on loading C extensions compiled on Python older than this version?

What makes you think that it would be otherwise be able to load and execute such C extensions without any bugs?

There is no ABI except the stable ABI, and tp_flags isn't part of the stable ABI. 

PS : tp_finalize is in Python 3.4, but that doesn't really matter.
msg341560 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2019-05-06 16:34
I updated the PR (sorry to let this sleep).
msg343918 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2019-05-29 20:12
New changeset ada319bb6d0ebcc68d3e0ef2b4279ea061877ac8 by Antoine Pitrou in branch 'master':
bpo-32388: Remove cross-version binary compatibility requirement in tp_flags (GH-4944)
https://github.com/python/cpython/commit/ada319bb6d0ebcc68d3e0ef2b4279ea061877ac8
msg343919 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2019-05-29 20:13
PR is merged now!
msg345339 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-12 13:11
About ABI "foward compatibility", Python 3.8 introduced another backward incompatible change: bpo-37250 "C files generated by Cython set tp_print to NULL: PyTypeObject.tp_print removed, replaced with tp_vectorcall_offset".
History
Date User Action Args
2019-06-12 13:11:44vstinnersetmessages: + msg345339
2019-05-29 20:13:09pitrousetstatus: open -> closed
resolution: fixed
messages: + msg343919

stage: patch review -> resolved
2019-05-29 20:12:41pitrousetmessages: + msg343918
2019-05-06 16:34:45pitrousetmessages: + msg341560
2018-06-19 09:11:17pitrousetmessages: + msg319947
2018-06-19 09:05:50vstinnersetnosy: + vstinner
messages: + msg319944
2018-03-06 18:18:41pitrousetmessages: + msg313351
versions: + Python 3.8, - Python 3.7
2018-03-03 08:01:35scodersetmessages: + msg313182
2018-02-19 11:16:53scodersetnosy: + scoder
2017-12-21 20:14:49ned.deilysetstage: patch review
2017-12-21 20:14:32ned.deilysetnosy: + njs, serhiy.storchaka
2017-12-21 20:14:05ned.deilysetnosy: + ncoghlan, ned.deily, - njs, serhiy.storchaka
stage: patch review -> (no value)
2017-12-21 17:01:13steve.dowersetmessages: + msg308884
2017-12-21 16:54:34pitrousetmessages: + msg308882
2017-12-21 16:51:10steve.dowersetmessages: + msg308881
2017-12-20 17:55:26pitrousetnosy: + serhiy.storchaka
2017-12-20 17:29:43pitrousetkeywords: + patch
stage: patch review
pull_requests: + pull_request4836
2017-12-20 17:29:26pitrousetnosy: + njs
2017-12-20 17:27:58pitroucreate