Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

optimize compilation options #61981

Closed
pitrou opened this issue Apr 17, 2013 · 17 comments
Closed

optimize compilation options #61981

pitrou opened this issue Apr 17, 2013 · 17 comments
Labels
build The build process and cross-build performance Performance or resource usage

Comments

@pitrou
Copy link
Member

pitrou commented Apr 17, 2013

BPO 17781
Nosy @warsaw, @rhettinger, @terryjreedy, @doko42, @pitrou, @scoder, @vstinner, @tiran, @bitdancer, @zware
Files
  • perflog.txt
  • lto_flag.patch
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2018-09-09.19:05:34.368>
created_at = <Date 2013-04-17.18:40:08.367>
labels = ['build', 'performance']
title = 'optimize compilation options'
updated_at = <Date 2018-09-09.19:05:34.366>
user = 'https://github.com/pitrou'

    bugs.python.org fields:

    activity = <Date 2018-09-09.19:05:34.366>
actor = 'zach.ware'
assignee = 'none'
closed = True
closed_date = <Date 2018-09-09.19:05:34.368>
closer = 'zach.ware'
components = ['Build']
creation = <Date 2013-04-17.18:40:08.367>
creator = 'pitrou'
dependencies = []
files = ['29907', '29920']
hgrepos = []
issue_num = 17781
keywords = ['patch']
message_count = 17.0
messages = ['187192', '187193', '187245', '187246', '248993', '248994', '249103', '249253', '249260', '249261', '251304', '251305', '251335', '251608', '256281', '258629', '324898']
nosy_count = 11.0
nosy_names = ['barry', 'rhettinger', 'terry.reedy', 'doko', 'pitrou', 'scoder', 'vstinner', 'christian.heimes', 'r.david.murray', 'zach.ware', 'alecsandru.patrascu']
pr_nums = []
priority = 'low'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue17781'
versions = ['Python 3.4']
    @pitrou
    Copy link
    Member Author

    pitrou commented Apr 17, 2013

    Ubuntu's system Python 3.3 shows consistently better performance than a vanilla Python 3.3: around 10-15% faster in general (see attached benchmark numbers).

    If this can be attributed to different compilation options, it would be nice to backport those options to our standard build config.

    @pitrou pitrou added build The build process and cross-build performance Performance or resource usage labels Apr 17, 2013
    @doko42
    Copy link
    Member

    doko42 commented Apr 17, 2013

    most of that can be attributed to the pgo build, which is upstream for a long time. the second thing to do is to build with lto, and see what speedups you get in addition. and it certainly helps to build the interpreter statically (without --enable-shared).

    but thanks to confirming my own experience ;)

    @tiran
    Copy link
    Member

    tiran commented Apr 18, 2013

    Here is a patch for -flto. You need to run autoconf to re-generate configure, too.

    @doko42
    Copy link
    Member

    doko42 commented Apr 18, 2013

    the proposed patch is wrong. when linking with -flto, you should pass all the relevant CFLAGS to the linker as well. Also pass -fuse-linker-plugin.

    and this should be an opt-in, not the default. Depending on the architecture and the compiler version, -flto is not as stable as you want it to be.

    and last, this ends up as the default for building third party extensions too, which again, I think should be an opt-in.

    @scoder
    Copy link
    Contributor

    scoder commented Aug 22, 2015

    bpo-24915 suggests PGO and comes with an actual patch. I suggest rejecting this ticket as too broad.

    @pitrou
    Copy link
    Member Author

    pitrou commented Aug 22, 2015

    LTO (Link-Time Optimization) is not the same as PGO, though I guess it can take advantage of PGO for its heuristics.

    @rhettinger
    Copy link
    Contributor

    I would like the see LTO enabled. The intermodule calls to code in abstract.c would become less expensive.

    @pitrou
    Copy link
    Member Author

    pitrou commented Aug 27, 2015

    Note this patch is likely wrong, as it doesn't add the optimization options to the linker invocation. According to the gcc does, """To use the link-time optimizer, -flto and optimization options should be specified at compile time and during the final link""".

    So probably $OPT should be added to $PY_LDFLAGS.

    @pitrou
    Copy link
    Member Author

    pitrou commented Aug 27, 2015

    For the record, the gain for LTO+PGO (with "-flto -O3") over PGO alone seems to be between 0% and 10% for most benchmarks.

    @pitrou
    Copy link
    Member Author

    pitrou commented Aug 27, 2015

    You can test for yourself by passing CFLAGS="-flto -O3" LDFLAGS="-flto -O3" to ./configure (using gcc).

    @vstinner
    Copy link
    Member

    For the record, the gain for LTO+PGO (with "-flto -O3") over PGO alone seems to be between 0% and 10% for most benchmarks.

    Hum, does it make sense to enable LTO without PGO?

    @pitrou
    Copy link
    Member Author

    pitrou commented Sep 22, 2015

    Hum, does it make sense to enable LTO without PGO?

    Probably not.

    By the way, I now have a small ARM system to play with, and there the gain of LTO+PGO over PGO alone is around 10%.

    Also note LTO can make compilation times much longer (it's the linking step actually, which can take minutes).

    @doko42
    Copy link
    Member

    doko42 commented Sep 22, 2015

    On 22.09.2015 12:31, Antoine Pitrou wrote:

    Also note LTO can make compilation times much longer (it's the linking step actually, which can take minutes).

    use -flto=jobserver

    @terryjreedy
    Copy link
    Member

    bpo-24915 is about adding pgo and has a slew of patches.

    @alecsandrupatrascu
    Copy link
    Mannequin

    alecsandrupatrascu mannequin commented Dec 12, 2015

    Hi, I added a dedicated issue just for LTO only when using GCC and CLANG (http://bugs.python.org/issue25702), that works well with PGO also.

    @zware
    Copy link
    Member

    zware commented Jan 19, 2016

    PGO is available as make profile-opt, LTO has a patch in bpo-25702. Are there any other interesting options worth investigating?

    @zware
    Copy link
    Member

    zware commented Sep 9, 2018

    After 2.5 years without response, I think the answer is probably "no" :)

    @zware zware closed this as completed Sep 9, 2018
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    build The build process and cross-build performance Performance or resource usage
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    9 participants
    @vstinner @tiran @bitdancer @scoder @rhettinger @pitrou @zware @doko42 @terryjreedy