Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove cross-version binary compatibility #76569

Closed
pitrou opened this issue Dec 20, 2017 · 12 comments
Closed

Remove cross-version binary compatibility #76569

pitrou opened this issue Dec 20, 2017 · 12 comments
Labels
3.8 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) type-bug An unexpected behavior, bug, or error

Comments

@pitrou
Copy link
Member

pitrou commented Dec 20, 2017

BPO 32388
Nosy @warsaw, @pfmoore, @ncoghlan, @pitrou, @scoder, @vstinner, @tjguk, @ned-deily, @njsmith, @zware, @serhiy-storchaka, @zooba
PRs
  • bpo-32388: Remove cross-version binary compatibility requirement in tp_flags #4944
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2019-05-29.20:13:09.708>
created_at = <Date 2017-12-20.17:27:58.959>
labels = ['interpreter-core', 'type-bug', '3.8']
title = 'Remove cross-version binary compatibility'
updated_at = <Date 2019-06-12.13:11:44.251>
user = 'https://github.com/pitrou'

    bugs.python.org fields:

    activity = <Date 2019-06-12.13:11:44.251>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-05-29.20:13:09.708>
closer = 'pitrou'
components = ['Interpreter Core']
creation = <Date 2017-12-20.17:27:58.959>
creator = 'pitrou'
dependencies = []
files = []
hgrepos = []
issue_num = 32388
keywords = ['patch']
message_count = 12.0
messages = ['308753', '308881', '308882', '308884', '313182', '313351', '319944', '319947', '341560', '343918', '343919', '345339']
nosy_count = 12.0
nosy_names = ['barry', 'paul.moore', 'ncoghlan', 'pitrou', 'scoder', 'vstinner', 'tim.golden', 'ned.deily', 'njs', 'zach.ware', 'serhiy.storchaka', 'steve.dower']
pr_nums = ['4944']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue32388'
versions = ['Python 3.8']
    @pitrou
    Copy link
    Member Author

    pitrou commented Dec 20, 2017

    See original discussion on python-dev:
    https://mail.python.org/pipermail/python-dev/2017-December/151328.html

    Summary: binary compatibility for C extensions which don't use the stable ABI is currently "best effort" and uses distinguished flags in the tp_flags field of type objects to indicate whether a field is physically present or not. Unfortunately, tp_flags is 32 bits and therefore a scarce resource. Also, binary compatibility is 1) not guaranteed anyway 2) of less use nowadays.

    So we propose to remove the binary compatibility requirement when creating static type objects (i.e. not using the stable ABI).

    @pitrou pitrou added 3.7 (EOL) end of life interpreter-core (Objects, Python, Grammar, and Parser dirs) type-bug An unexpected behavior, bug, or error labels Dec 20, 2017
    @zooba
    Copy link
    Member

    zooba commented Dec 21, 2017

    Totally okay with this in theory, but wanted to clarify that the linked PR is really just taking advantage of removing the guarantee?

    That is, the change isn't necessary to remove the requirement.

    @pitrou
    Copy link
    Member Author

    pitrou commented Dec 21, 2017

    That's basically exact. Actually, there is no ABI guarantee at all (when not using the stable ABI), just a best effort for advanced users who know what they're doing.

    Actually, this PR stems from Nathaniel's message here:
    https://mail.python.org/pipermail/python-dev/2017-December/151345.html

    @zooba
    Copy link
    Member

    zooba commented Dec 21, 2017

    Good to know. As I said on the other issue, Windows won't load extension modules built for a different version anyway unless they use the stable ABI, so there's no real reason these flags need to be binary compatible between 3.x versions.

    @scoder
    Copy link
    Contributor

    scoder commented Mar 3, 2018

    I'm ok with "removing" this "guarantee" (although it seems too late to apply this specific change to 3.7 now).

    While Cython users do ask for a way to build cross-version binaries from time to time, it's neither supported nor planned, and the underlying problem is IMHO solved better via tooling like the manylinux wheel build.

    I also generally consider it good to let user code take advantage of new features in new CPython releases, even if it means that they need to retranslate their Cython code from time to time. The 1.5 years release cycle of CPython should exceed that of most "still in maintenance" extension modules out there, and it's long enough to not impose a real burden on their maintainers.

    @pitrou
    Copy link
    Member Author

    pitrou commented Mar 6, 2018

    Yes, let's retarget to 3.8.

    @pitrou pitrou added 3.8 only security fixes and removed 3.7 (EOL) end of life labels Mar 6, 2018
    @vstinner
    Copy link
    Member

    In which version of Python tp_finalize slot has been added? Does it mean that Python 3.8 wil crash on loading C extensions compiled on Python older than this version?

    @pitrou
    Copy link
    Member Author

    pitrou commented Jun 19, 2018

    Does it mean that Python 3.8 wil crash on loading C extensions compiled on Python older than this version?

    What makes you think that it would be otherwise be able to load and execute such C extensions without any bugs?

    There is no ABI except the stable ABI, and tp_flags isn't part of the stable ABI.

    PS : tp_finalize is in Python 3.4, but that doesn't really matter.

    @pitrou
    Copy link
    Member Author

    pitrou commented May 6, 2019

    I updated the PR (sorry to let this sleep).

    @pitrou
    Copy link
    Member Author

    pitrou commented May 29, 2019

    New changeset ada319b by Antoine Pitrou in branch 'master':
    bpo-32388: Remove cross-version binary compatibility requirement in tp_flags (GH-4944)
    ada319b

    @pitrou
    Copy link
    Member Author

    pitrou commented May 29, 2019

    PR is merged now!

    @pitrou pitrou closed this as completed May 29, 2019
    @vstinner
    Copy link
    Member

    About ABI "foward compatibility", Python 3.8 introduced another backward incompatible change: bpo-37250 "C files generated by Cython set tp_print to NULL: PyTypeObject.tp_print removed, replaced with tp_vectorcall_offset".

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    @vstinner vstinner mentioned this issue Jun 21, 2023
    Closed
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.8 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    4 participants
    @vstinner @scoder @zooba @pitrou