classification
Title: no PGO for built-in modules with `make profile-opt`
Type: performance Stage: commit review
Components: Build Versions: Python 3.6, Python 3.5, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: gregory.p.smith Nosy List: alecsandru.patrascu, cstratak, gregory.p.smith, python-dev, tzot
Priority: normal Keywords: patch

Created on 2016-02-08 13:28 by tzot, last changed 2016-09-08 05:33 by gregory.p.smith. This issue is now closed.

Files
File name Uploaded Description Edit
pgofix-cpython2.patch alecsandru.patrascu, 2016-02-22 06:32 review
pgofix-cpython3.patch alecsandru.patrascu, 2016-02-22 06:32 review
Messages (8)
msg259842 - (view) Author: Χρήστος Γεωργίου (Christos Georgiou) (tzot) * Date: 2016-02-08 13:28
(related to issue #24915)
I discovered that `make profile-opt` does not use the profile information for the builtin-modules (e.g. arraymodule or _pickle) because in the `profile-opt` target there is the following sequence of actions:

    …
    $(MAKE) build_all_merge_profile
    @echo "Rebuilding with profile guided optimizations:"
    $(MAKE) clean
    $(MAKE) build_all_use_profile
    …

The action `$(MAKE) clean` performs an `rm -rf build`, destroying among other things all the *.gcda files generated for the built-in modules.
On my Linux system with gcc, a kludge to `Makefile.pre.in` that works is:

    …
    @echo "Rebuilding with profile guided optimizations:"
    find build -name \*.gcda -print | cpio -o >_modules.gcda.cpio # XXX
    $(MAKE) clean
    cpio -id <_modules.gcda.cpio # XXX
    $(MAKE) build_all_use_profile
    …

but, like I said, it's a kludge and it's POSIX-only (-print0 can be avoided since I believe it's guaranteed that there will be no whitespace-containing-filenames).

Now, if this road is to be taken, I believe the most cross-platform method available to save the profile-generated files is to use a custom python script (at this point, the local profile-generating python executable is functional) and make a tar file, which will be untared back. However, I don't like it much.

I'm willing to provide any necessary patches if someone gives me a "proper" roadmap as to how to handle this issue.
msg259884 - (view) Author: Alecsandru Patrascu (alecsandru.patrascu) * Date: 2016-02-08 20:35
Thank you for noticing and signaling this issue. Since I proposed the PGO patches, I will fix it in another patch, as I don't want it to break any builds.
msg260270 - (view) Author: Alecsandru Patrascu (alecsandru.patrascu) * Date: 2016-02-14 10:21
I've added a fix for the PGO builds after the issue pointed out by you. Can you please test it also?
msg260659 - (view) Author: Alecsandru Patrascu (alecsandru.patrascu) * Date: 2016-02-22 06:32
I've added the patch here also.
msg274039 - (view) Author: Charalampos Stratakis (cstratak) * Date: 2016-08-31 16:42
Pinging here. Christos could you test the patch?
msg274676 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-09-06 23:58
New changeset 7d9cd4a0d488 by Gregory P. Smith in branch '3.5':
Fixes issue26307: The profile-opt build now applys PGO to the built-in
https://hg.python.org/cpython/rev/7d9cd4a0d488

New changeset bdc7292cf87e by Gregory P. Smith in branch 'default':
Fixes issue26307: The profile-opt build now applys PGO to the built-in modules.
https://hg.python.org/cpython/rev/bdc7292cf87e
msg274694 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-09-07 01:06
New changeset 75dae0b2ccb3 by Gregory P. Smith in branch '2.7':
Fixes issue26307: The profile-opt build now applys PGO to the built-in modules.
https://hg.python.org/cpython/rev/75dae0b2ccb3
msg274966 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2016-09-08 05:33
My change means that the build/ directory tree exists but won't have any files other than *.gc?? files in it after a make clean.  I doubt this will bother anyone.  That felt better for clean than just avoiding the removal of build/ entirely.
History
Date User Action Args
2016-09-08 05:33:30gregory.p.smithsetstatus: open -> closed
resolution: fixed
messages: + msg274966

stage: needs patch -> commit review
2016-09-07 01:06:21python-devsetmessages: + msg274694
2016-09-06 23:58:45python-devsetnosy: + python-dev
messages: + msg274676
2016-09-06 22:41:59gregory.p.smithsetassignee: gregory.p.smith

nosy: + gregory.p.smith
2016-08-31 16:42:06cstrataksetnosy: + cstratak
messages: + msg274039
2016-02-22 06:32:54alecsandru.patrascusetfiles: + pgofix-cpython3.patch
2016-02-22 06:32:47alecsandru.patrascusetfiles: + pgofix-cpython2.patch
keywords: + patch
messages: + msg260659
2016-02-20 20:27:59brett.cannonsetstage: needs patch
2016-02-14 10:21:13alecsandru.patrascusetmessages: + msg260270
2016-02-08 20:35:47alecsandru.patrascusetmessages: + msg259884
2016-02-08 20:30:24alecsandru.patrascusetnosy: + alecsandru.patrascu
2016-02-08 13:28:26tzotcreate