Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Race condition in "make regen-all" when running jobs in parallel #86564

Closed
vstinner opened this issue Nov 18, 2020 · 4 comments
Closed

Race condition in "make regen-all" when running jobs in parallel #86564

vstinner opened this issue Nov 18, 2020 · 4 comments
Labels
3.8 only security fixes 3.9 only security fixes 3.10 only security fixes build The build process and cross-build

Comments

@vstinner
Copy link
Member

BPO 42398
Nosy @vstinner, @miss-islington
PRs
  • bpo-42398: Fix "make regen-all" race condition #23362
  • [3.9] bpo-42398: Fix "make regen-all" race condition (GH-23362) #23367
  • [3.8] bpo-42398: Fix "make regen-all" race condition (GH-23362) (GH-23367) #23371
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2020-11-18.17:23:44.296>
created_at = <Date 2020-11-18.10:47:58.667>
labels = ['build', '3.8', '3.9', '3.10']
title = 'Race condition in "make regen-all" when running jobs in parallel'
updated_at = <Date 2020-11-18.17:23:44.296>
user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2020-11-18.17:23:44.296>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2020-11-18.17:23:44.296>
closer = 'vstinner'
components = ['Build']
creation = <Date 2020-11-18.10:47:58.667>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 42398
keywords = ['patch']
message_count = 4.0
messages = ['381325', '381348', '381355', '381356']
nosy_count = 2.0
nosy_names = ['vstinner', 'miss-islington']
pr_nums = ['23362', '23367', '23371']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue42398'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']
    @vstinner
    Copy link
    Member Author

    When building the Fedora package of Python 3.10, *sometimes* the "make regen-all" fail to build with errors like:
    ------------
    /builddir/build/BUILD/Python-3.9.0/Modules/_weakref.c:131:5: error: '_WEAKREF_GETWEAKREFCOUNT_METHODDEF' undeclared here (not in a function)
    131 | _WEAKREF_GETWEAKREFCOUNT_METHODDEF
    | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    /builddir/build/BUILD/Python-3.9.0/Modules/_weakref.c:132:5: error: expected '}' before '_WEAKREF__REMOVE_DEAD_WEAKREF_METHODDEF'
    132 | _WEAKREF__REMOVE_DEAD_WEAKREF_METHODDEF
    | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    ------------

    Example: https://src.fedoraproject.org/rpms/python3.9/pull-request/40

    The _WEAKREF_GETWEAKREFCOUNT_METHODDEF macro is defined by Modules/clinic/_weakref.c.h which is included by Modules/_weakref.c (at line 13).

    It seems like while Modules/_weakref.o was being built, Modules/clinic/_weakref.c.h was being generated "at the same time".

    The "make regen-all" command runs "make clinic" and "make regen-importlib" targets:

    On Fedora, packages are built using MAKEFLAGS="-j<N>" where <N> is a number, like "-j4". Since there is no dependency between "clinic" and "regen-importlib" targets, these two targets *can* be run in parallel.

    It seems like "make clinic" always rewrites *all* generated files and the half of writes are *not* atomic.

    Attached PR makes all writes of clinic.py atomic *and* also avoids modifying the file modification file if the content does not change, as done in other "make regen-all" targets (using Tools/scripts/update_file.py).

    @vstinner vstinner added 3.8 only security fixes 3.9 only security fixes 3.10 only security fixes build The build process and cross-build labels Nov 18, 2020
    @vstinner
    Copy link
    Member Author

    New changeset 8fba952 by Victor Stinner in branch 'master':
    bpo-42398: Fix "make regen-all" race condition (GH-23362)
    8fba952

    @vstinner
    Copy link
    Member Author

    New changeset c53c3f4 by Victor Stinner in branch '3.9':
    bpo-42398: Fix "make regen-all" race condition (GH-23362) (GH-23367)
    c53c3f4

    @miss-islington
    Copy link
    Contributor

    New changeset 66dd533 by Miss Islington (bot) in branch '3.8':
    bpo-42398: Fix "make regen-all" race condition (GH-23362) (GH-23367)
    66dd533

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.8 only security fixes 3.9 only security fixes 3.10 only security fixes build The build process and cross-build
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    2 participants
    @vstinner @miss-islington