classification
Title: optimize compilation options
Type: performance Stage: resolved
Components: Build Versions: Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: alecsandru.patrascu, barry, christian.heimes, doko, pitrou, r.david.murray, rhettinger, scoder, terry.reedy, vstinner, zach.ware
Priority: low Keywords: patch

Created on 2013-04-17 18:40 by pitrou, last changed 2018-09-09 19:05 by zach.ware. This issue is now closed.

Files
File name Uploaded Description Edit
perflog.txt pitrou, 2013-04-17 18:40
lto_flag.patch christian.heimes, 2013-04-18 13:05 review
Messages (17)
msg187192 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-04-17 18:40
Ubuntu's system Python 3.3 shows consistently better performance than a vanilla Python 3.3: around 10-15% faster in general (see attached benchmark numbers).

If this can be attributed to different compilation options, it would be nice to backport those options to our standard build config.
msg187193 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2013-04-17 18:45
most of that can be attributed to the pgo build, which is upstream for a long time. the second thing to do is to build with lto, and see what speedups you get in addition. and it certainly helps to build the interpreter statically (without --enable-shared).

but thanks to confirming my own experience ;)
msg187245 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-04-18 13:05
Here is a patch for -flto. You need to run autoconf to re-generate configure, too.
msg187246 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2013-04-18 13:24
the proposed patch is wrong. when linking with -flto, you should pass all the relevant CFLAGS to the linker as well. Also pass -fuse-linker-plugin.

and this should be an opt-in, not the default. Depending on the architecture and the compiler version, -flto is not as stable as you want it to be.

and last, this ends up as the default for building third party extensions too, which again, I think should be an opt-in.
msg248993 - (view) Author: Stefan Behnel (scoder) * (Python committer) Date: 2015-08-22 19:27
Issue 24915 suggests PGO and comes with an actual patch. I suggest rejecting this ticket as too broad.
msg248994 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2015-08-22 19:29
LTO (Link-Time Optimization) is not the same as PGO, though I guess it can take advantage of PGO for its heuristics.
msg249103 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2015-08-25 03:00
I would like the see LTO enabled.  The intermodule calls to code in abstract.c would become less expensive.
msg249253 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2015-08-27 19:08
Note this patch is likely wrong, as it doesn't add the optimization options to the linker invocation. According to the gcc does, """To use the link-time optimizer, -flto and optimization options should be specified at compile time and during the final link""".

So probably $OPT should be added to $PY_LDFLAGS.
msg249260 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2015-08-27 21:09
For the record, the gain for LTO+PGO (with "-flto -O3") over PGO alone seems to be between 0% and 10% for most benchmarks.
msg249261 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2015-08-27 21:10
You can test for yourself by passing `CFLAGS="-flto -O3" LDFLAGS="-flto -O3"` to ./configure (using gcc).
msg251304 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-09-22 10:28
> For the record, the gain for LTO+PGO (with "-flto -O3") over PGO alone seems to be between 0% and 10% for most benchmarks.

Hum, does it make sense to enable LTO without PGO?
msg251305 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2015-09-22 10:31
> Hum, does it make sense to enable LTO without PGO?

Probably not.

By the way, I now have a small ARM system to play with, and there the gain of LTO+PGO over PGO alone is around 10%.

Also note LTO can make compilation times much longer (it's the linking step actually, which can take minutes).
msg251335 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2015-09-22 17:44
On 22.09.2015 12:31, Antoine Pitrou wrote:
> Also note LTO can make compilation times much longer (it's the linking step actually, which can take minutes).

use -flto=jobserver
msg251608 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2015-09-25 20:26
#24915 is about adding pgo and has a slew of patches.
msg256281 - (view) Author: Alecsandru Patrascu (alecsandru.patrascu) * Date: 2015-12-12 12:05
Hi, I added a dedicated issue just for LTO only when using GCC and CLANG (http://bugs.python.org/issue25702), that works well with PGO also.
msg258629 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2016-01-19 22:37
PGO is available as `make profile-opt`, LTO has a patch in issue25702.  Are there any other interesting options worth investigating?
msg324898 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2018-09-09 19:05
After 2.5 years without response, I think the answer is probably "no" :)
History
Date User Action Args
2018-09-09 19:05:34zach.waresetstatus: pending -> closed
resolution: fixed
messages: + msg324898

stage: resolved
2016-01-19 23:00:27r.david.murraysetstatus: open -> pending
2016-01-19 22:52:56r.david.murraysetstatus: pending -> open
nosy: + r.david.murray
2016-01-19 22:37:50zach.waresetstatus: open -> pending
nosy: + zach.ware
messages: + msg258629

2015-12-12 12:05:05alecsandru.patrascusetnosy: + alecsandru.patrascu
messages: + msg256281
2015-09-25 20:26:52terry.reedysetnosy: + terry.reedy
messages: + msg251608
2015-09-22 17:44:41dokosetmessages: + msg251335
2015-09-22 10:31:19pitrousetmessages: + msg251305
2015-09-22 10:28:13vstinnersetnosy: + vstinner
messages: + msg251304
2015-08-27 21:10:10pitrousetmessages: + msg249261
2015-08-27 21:09:16pitrousetmessages: + msg249260
2015-08-27 19:08:50pitrousetmessages: + msg249253
2015-08-25 03:00:19rhettingersetnosy: + rhettinger
messages: + msg249103
2015-08-22 19:29:52pitrousetmessages: + msg248994
2015-08-22 19:27:53scodersetnosy: + scoder
messages: + msg248993
2014-05-13 22:19:17skrahsetnosy: - skrah
2013-12-17 18:24:34skrahsetnosy: + skrah
2013-04-18 13:24:13dokosetmessages: + msg187246
2013-04-18 13:05:43christian.heimessetfiles: + lto_flag.patch

nosy: + christian.heimes
messages: + msg187245

keywords: + patch
2013-04-17 18:45:41dokosetmessages: + msg187193
2013-04-17 18:40:08pitroucreate