Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

distutils creates unreproducible .so files #80483

Closed
bmwiedemann mannequin opened this issue Mar 15, 2019 · 8 comments
Closed

distutils creates unreproducible .so files #80483

bmwiedemann mannequin opened this issue Mar 15, 2019 · 8 comments
Labels
3.9 only security fixes stdlib Python modules in the Lib dir

Comments

@bmwiedemann
Copy link
Mannequin

bmwiedemann mannequin commented Mar 15, 2019

BPO 36302
Nosy @ronaldoussoren, @vstinner, @mcepl, @merwok, @dstufft, @bmwiedemann
PRs
  • bpo-36302: Sort list of sources #12341
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2019-08-01.13:19:58.540>
created_at = <Date 2019-03-15.12:30:33.299>
labels = ['library', '3.9']
title = 'distutils creates unreproducible .so files'
updated_at = <Date 2021-02-02.10:48:12.466>
user = 'https://github.com/bmwiedemann'

    bugs.python.org fields:

    activity = <Date 2021-02-02.10:48:12.466>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-08-01.13:19:58.540>
closer = 'vstinner'
components = ['Distutils']
creation = <Date 2019-03-15.12:30:33.299>
creator = 'bmwiedemann'
dependencies = []
files = []
hgrepos = []
issue_num = 36302
keywords = ['patch']
message_count = 8.0
messages = ['337983', '348852', '348853', '373024', '373106', '373118', '386133', '386134']
nosy_count = 6.0
nosy_names = ['ronaldoussoren', 'vstinner', 'mcepl', 'eric.araujo', 'dstufft', 'bmwiedemann']
pr_nums = ['12341']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue36302'
versions = ['Python 3.9']
    @bmwiedemann
    Copy link
    Mannequin Author

    bmwiedemann mannequin commented Mar 15, 2019

    While working on reproducible builds for openSUSE, I found countless python modules that come with binary .so files that did not build reproducibly from non-deterministic filesystem readdir order.

    One contributing factor is bpo-30461 that will not be fixed.
    I found that a simple patch can be done in distutils
    instead of fixing an infinite number of current and future python modules.

    @bmwiedemann bmwiedemann mannequin added 3.7 (EOL) end of life 3.8 only security fixes stdlib Python modules in the Lib dir labels Mar 15, 2019
    @vstinner
    Copy link
    Member

    vstinner commented Aug 1, 2019

    New changeset 0d30ae1 by Victor Stinner (Bernhard M. Wiedemann) in branch 'master':
    bpo-36302: Sort list of sources (GH-12341)
    0d30ae1

    @vstinner
    Copy link
    Member

    vstinner commented Aug 1, 2019

    I'm not excited about backporting this change to Python 3.8 and older. Python 3.8 also switch to its feature freeze for Python 3.8.0. This change is borderline between bugfix and feature. In case of doubt, I prefer to do nothing.

    See concerns in the PR discussion:
    #12341 (comment)

    @vstinner vstinner added 3.9 only security fixes and removed 3.7 (EOL) end of life 3.8 only security fixes labels Aug 1, 2019
    @vstinner vstinner closed this as completed Aug 1, 2019
    @ronaldoussoren
    Copy link
    Contributor

    An unfortunate side effect of this change is that changes the build order even if the source list order is relevant.

    In particular, I sort the source list by age when working on larger extensions, this makes sure that the source files I edited last get compiled first, which speeds up edit/build cycles.

    @vstinner
    Copy link
    Member

    vstinner commented Jul 6, 2020

    An unfortunate side effect of this change is that changes the build order even if the source list order is relevant.

    That sounds like a new feature. I don't think that it was supported previously. Or maybe it was more an implementation detail.

    Please open a new issue if you would like to add the ability to specify in which order source files must be compiled.

    @ronaldoussoren
    Copy link
    Contributor

    That's the problem with any change to distutils, the API is unclear and that causes any change to be potentially breaking for existing users. That's why distutils has been mostly stagnant for years.

    I have no particular wish w.r.t. changing behaviour, I've a local workaround for now and am hoping someone will write a tool similar to flit for packages that include extensions.

    @mcepl
    Copy link
    Mannequin

    mcepl mannequin commented Feb 2, 2021

    In particular, I sort the source list by age when working on larger extensions, this makes sure that the source files I edited last get compiled first, which speeds up edit/build cycles.

    On the one hand, this sounds creepily like https://xkcd.com/1172, but on the other: why do we need to sort those extensions by name? Wouldn’t sorting by some attribute of those extensions (including their size) be better?

    @vstinner
    Copy link
    Member

    vstinner commented Feb 2, 2021

    Why do we need to sort those extensions by name?

    Because calling sorted() is easy. Any other arbitrary choice would make sense. But giving the ability to build sources in a specific order, add dependencies, or something else, would a new features.

    Also, distutils is now deprecated in the stdlib: the steering council approved https://www.python.org/dev/peps/pep-0632/

    You should now report distutils issues to setuptools.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.9 only security fixes stdlib Python modules in the Lib dir
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    2 participants
    @vstinner @ronaldoussoren