Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

no PGO for built-in modules with make profile-opt #70495

Closed
tzot mannequin opened this issue Feb 8, 2016 · 8 comments
Closed

no PGO for built-in modules with make profile-opt #70495

tzot mannequin opened this issue Feb 8, 2016 · 8 comments
Assignees
@gpshead
Labels
build The build process and cross-build performance Performance or resource usage

Comments

@tzot
Copy link
Mannequin

tzot mannequin commented Feb 8, 2016

BPO 26307
Nosy @gpshead, @stratakis
Files
  • pgofix-cpython2.patch
  • pgofix-cpython3.patch
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/gpshead'
closed_at = <Date 2016-09-08.05:33:30.788>
created_at = <Date 2016-02-08.13:28:26.701>
labels = ['build', 'performance']
title = 'no PGO for built-in modules with `make profile-opt`'
updated_at = <Date 2016-09-08.05:33:30.786>
user = 'https://bugs.python.org/tzot'

    bugs.python.org fields:

    activity = <Date 2016-09-08.05:33:30.786>
actor = 'gregory.p.smith'
assignee = 'gregory.p.smith'
closed = True
closed_date = <Date 2016-09-08.05:33:30.788>
closer = 'gregory.p.smith'
components = ['Build']
creation = <Date 2016-02-08.13:28:26.701>
creator = 'tzot'
dependencies = []
files = ['41999', '42000']
hgrepos = []
issue_num = 26307
keywords = ['patch']
message_count = 8.0
messages = ['259842', '259884', '260270', '260659', '274039', '274676', '274694', '274966']
nosy_count = 5.0
nosy_names = ['gregory.p.smith', 'tzot', 'python-dev', 'alecsandru.patrascu', 'cstratak']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'commit review'
status = 'closed'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue26307'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6']
    @tzot
    Copy link
    Mannequin Author

    tzot mannequin commented Feb 8, 2016

    (related to issue bpo-24915)
    I discovered that make profile-opt does not use the profile information for the builtin-modules (e.g. arraymodule or _pickle) because in the profile-opt target there is the following sequence of actions:

    …
$(MAKE) build_all_merge_profile
@echo "Rebuilding with profile guided optimizations:"
$(MAKE) clean
$(MAKE) build_all_use_profile
…

    The action $(MAKE) clean performs an rm -rf build, destroying among other things all the *.gcda files generated for the built-in modules.
    On my Linux system with gcc, a kludge to Makefile.pre.in that works is:

    …
@echo "Rebuilding with profile guided optimizations:"
find build -name \*.gcda -print | cpio -o >_modules.gcda.cpio # XXX
$(MAKE) clean
cpio -id <_modules.gcda.cpio # XXX
$(MAKE) build_all_use_profile
…

    but, like I said, it's a kludge and it's POSIX-only (-print0 can be avoided since I believe it's guaranteed that there will be no whitespace-containing-filenames).

    Now, if this road is to be taken, I believe the most cross-platform method available to save the profile-generated files is to use a custom python script (at this point, the local profile-generating python executable is functional) and make a tar file, which will be untared back. However, I don't like it much.

    I'm willing to provide any necessary patches if someone gives me a "proper" roadmap as to how to handle this issue.

    @tzot tzot mannequin added build The build process and cross-build performance Performance or resource usage labels Feb 8, 2016
    @alecsandrupatrascu
    Copy link
    Mannequin

    alecsandrupatrascu mannequin commented Feb 8, 2016

    Thank you for noticing and signaling this issue. Since I proposed the PGO patches, I will fix it in another patch, as I don't want it to break any builds.

    @alecsandrupatrascu
    Copy link
    Mannequin

    alecsandrupatrascu mannequin commented Feb 14, 2016

    I've added a fix for the PGO builds after the issue pointed out by you. Can you please test it also?

    @alecsandrupatrascu
    Copy link
    Mannequin

    alecsandrupatrascu mannequin commented Feb 22, 2016

    I've added the patch here also.

    @stratakis
    Copy link
    Mannequin

    stratakis mannequin commented Aug 31, 2016

    Pinging here. Christos could you test the patch?

    @gpshead gpshead self-assigned this Sep 6, 2016
    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Sep 6, 2016

    New changeset 7d9cd4a0d488 by Gregory P. Smith in branch '3.5':
    Fixes bpo-26307: The profile-opt build now applys PGO to the built-in
    https://hg.python.org/cpython/rev/7d9cd4a0d488

    New changeset bdc7292cf87e by Gregory P. Smith in branch 'default':
    Fixes bpo-26307: The profile-opt build now applys PGO to the built-in modules.
    https://hg.python.org/cpython/rev/bdc7292cf87e

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Sep 7, 2016

    New changeset 75dae0b2ccb3 by Gregory P. Smith in branch '2.7':
    Fixes bpo-26307: The profile-opt build now applys PGO to the built-in modules.
    https://hg.python.org/cpython/rev/75dae0b2ccb3

    @gpshead
    Copy link
    Member

    gpshead commented Sep 8, 2016

    My change means that the build/ directory tree exists but won't have any files other than *.gc?? files in it after a make clean. I doubt this will bother anyone. That felt better for clean than just avoiding the removal of build/ entirely.

    @gpshead gpshead closed this as completed Sep 8, 2016
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees

    @gpshead gpshead

    Labels
    build The build process and cross-build performance Performance or resource usage
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    1 participant
    @gpshead