In a discussion on python-ideas about some computed goto stuff, Greg Smith brought up compilation with "profile-opt", which I'd never heard of. I asked Greg about it and he enlightened me. Looking around, I saw no mention of this make target except in the Makefile itself, and a small what's new note for Python 2.6. I was only looking on the 2.7 branch, but aside from a bug fix mention in Misc/NEWS for Python 3.4, it doesn't seem to feature more prominently there.

It would seem to be stable. Should it perhaps be the default when available (presuming the proper configure tests can be written)? Alternatively, it could be featured more prominently in the build instructions. I've attached a simple patch to README for that.

The Makefile's default "make all" target is meant for developers and iterative development.

make profile-opt is not.

If anything, this issue should document it somewhere if we even have docs on how to make "release" builds of something.  Realistically Debian already uses profile-opt for its binary builds.  I _believe_ the rpm based distros do as well (but I haven't checked, someone else can confirm).  Anyone else not doing so, please look at what debian does:

make profile-opt PROFILE_TASK="..."

where ... is regrtest.py given a test exclusion list of some of the longer running more annoying ones.

Not using a profile-opt builds in production is costing you ~10-40% in CPU performance.  This advice applies to all programs, not only python. :)

Also FWIW, I setup a buildbot earlier this year building in profile-opt mode.

Agreed with Greg, this should not be the default.  Changing title and component to reflect that better documentation is the goal here.

Thanks. The doc change occurred to me while I was composing my ticket. It wasn't obvious at that time that profile-opt shouldn't be the default (though I was carrying on an email conversation with Greg at about the same time).

The readme in 3.6.5 has a section on PGO but the sentence "If ran,
``make profile-opt`` will do several steps." can be misunderstood that one has to run this command instead of "make" after "configure --enable-optimizations". Furthermore, a 2015 post suggests that most Linux distributions use a different workload than the one provided for much better results. The readme should talk about this, either clarifying that this issue has been resolved or making recommendations for the workload. (I understand that licence incompatibilities or the size of the workload may be reasons for not including the recommended workload in the source distribution.)

FYI - The test suite minus some of the crazier tests (multiprocessing) that the makefile uses today is the recommended workload and is effectively what linux distros I've looked at use.

It is a myth that the specific application workload matters a great deal for building a Python PGO interpreter.  The layer of indirection provided by an interpreter makes it easier.  All you really need is code that exercises the major important C speed paths.  The eval loop, unicode internals, long bignum internals, marshal, re, pickle, json, xml, and cdecimal are all well exercised by running the test suite.

A specific Python application's actual logic is not going to exercise the internals of those in a meaningfully different manner.

Performance experiments bear this out.

New changeset 93f9a8a5afb58b1d2e4f27bb46d3d4e0fa8c08f0 by Gregory P. Smith in branch 'master':
bpo-24318: Rewrite the README PGO section. (#6863)
https://github.com/python/cpython/commit/93f9a8a5afb58b1d2e4f27bb46d3d4e0fa8c08f0

New changeset 447fdd178f1007294db38bf0392a0cf74a49d460 by Miss Islington (bot) in branch '3.7':
bpo-24318: Rewrite the README PGO section. (GH-6863)
https://github.com/python/cpython/commit/447fdd178f1007294db38bf0392a0cf74a49d460