classification
Title: --with-lto builds segfault in many situations
Type: crash Stage: resolved
Components: Versions: Python 3.6, Python 3.5, Python 2.7
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: gregory.p.smith Nosy List: alecsandru.patrascu, benjamin.peterson, gregory.p.smith, koobs, larry, ned.deily, pitrou, python-dev, sjt
Priority: normal Keywords:

Created on 2016-09-08 21:32 by gregory.p.smith, last changed 2016-11-21 08:19 by gregory.p.smith. This issue is now closed.

Messages (12)
msg275171 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2016-09-08 21:32
configure --with-optimizations is failing on the Debian --with-optimizations buildbot:

http://buildbot.python.org/all/builders/AMD64%20Debian%20PGO%203.x/builds/2448/steps/compile/logs/stdio

The compiled python is segfaulting when it goes to run the interpreter to build modules via setup.py.

this passes manually on ubuntu 16.04.  debian jessie 8.5 is failing here.  older gcc 4.9.2 toolchain.

i suspect --with-lto is not a good thing to enable by default everywhere.

configure.ac should be updated to exclude --with-lto in more situations than just Darwin.  we also know it doesn't work with clang as our configure and makefile are today.  so start with a conditional on gcc version being at least 5.4 (which is what ubuntu 16.04 ships with)?  or is it related to the binutils (ld) version when gcc is present?  or both.  LTO is a maze of twisty passages.

To lower this from "release blocker" we need to make --with-optimizations not enable --with-lto anywhere it is likely to fail.
msg275177 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2016-09-08 21:45
Can we just split --with-optimizations into --with-pgo and --with-lto, and let users choose whichever combination they want?
msg275182 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2016-09-08 22:06
That is effectively what this devolves to.  i'm going to try "gcc and gcc > 5.4" implies --with-lto as part of --with-optimizations for now.  but if that causes problems, i'll just remove the --with-lto implication from --with-optimizations entirely.
msg275194 - (view) Author: Roundup Robot (python-dev) Date: 2016-09-08 22:43
New changeset b77465544f28 by Gregory P. Smith in branch '3.5':
issue28032: never imply --with-lto as part of --with-optimizations.
https://hg.python.org/cpython/rev/b77465544f28

New changeset 4a50d87808a6 by Gregory P. Smith in branch 'default':
issue28032: never imply --with-lto as part of --with-optimizations.
https://hg.python.org/cpython/rev/4a50d87808a6
msg275198 - (view) Author: Roundup Robot (python-dev) Date: 2016-09-08 22:45
New changeset 7e89469e4342 by Gregory P. Smith in branch '2.7':
issue28032: never imply --with-lto as part of --with-optimizations.
https://hg.python.org/cpython/rev/7e89469e4342
msg280630 - (view) Author: Kubilay Kocak (koobs) Date: 2016-11-12 05:15
Could we please:

a) Rename 'optimizations' to 'pgo' or 'profiled'
b) Use enable/disable instead of with/without. The former is used for toggling program features, the latter is used for external dependencies [1][2][3]

Yes Python's configure / autofoo isn't the paragon of conventions, consistency or elegance, but it would be fantastic to make the job of bringing it to (or arguing for) a more standards-oriented state moving forward easier rather than harder.

[1] https://www.gnu.org/software/autoconf/manual/autoconf-2.66/html_node/Package-Options.html
[2] https://www.gnu.org/software/autoconf/manual/autoconf-2.66/html_node/External-Software.html
[3] https://autotools.io/autoconf/arguments.html
msg280631 - (view) Author: Kubilay Kocak (koobs) Date: 2016-11-12 05:16
I also invoke PEP20 (explicit > implicit) in favour of renaming the option :]
msg281300 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2016-11-20 21:23
The configure flag has been renamed to --enable-optimizations in the following commits for 3,5, 3.6, default, & 2.7 branches (everywhere it exists):

remote: notified python-checkins@python.org of incoming changeset c0ea81315fb6
remote: notified python-checkins@python.org of incoming changeset 58c1a49a10b4
remote: notified python-checkins@python.org of incoming changeset 11cacf9f9a33
remote: notified python-checkins@python.org of incoming changeset 0d2b42344ae5
remote: notified python-checkins@python.org of incoming changeset 2d1d70b53376
remote: notified python-checkins@python.org of incoming changeset d5ff5a2f33fd
msg281313 - (view) Author: Stephen J. Turnbull (sjt) * Date: 2016-11-21 01:16
FWIW, XEmacs has used a bit of m4 magic to make --with-* and --enable-* equivalent for 15 years, and nobody has ever complained.  The autotools convention is a distinction without a difference, and confuses users when a program feature depends on an external library (especially where there are alternative implementations).
msg281327 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-11-21 07:09
For 3.6 at least, there are still mentions of --with-optimizations in Doc/whatsnew/3.6.rst and README.
msg281330 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2016-11-21 08:15
doc references fixed.  thanks!

remote: notified python-checkins@python.org of incoming changeset a5e2add2c37b
remote: notified python-checkins@python.org of incoming changeset 6ae0e6d435de
remote: notified python-checkins@python.org of incoming changeset ea9cc29a274b
remote: notified python-checkins@python.org of incoming changeset 38e9761ccc13
msg281332 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2016-11-21 08:19
This is no longer a release blocker as --with-lto is not enabled by default by --enable-optimizations since my commits in September.

Regarding --with-lto itself segfaulting... Fixing compiler+linker toolchains is beyond what CPython itself should do.  But the reason the flag remains named "optimizations" is that it is fair for us to include --with-lto or other future build time optimization in known good configurations detected by the autoconf configure.ac in the future if desired.

I do not think there is anything more for us to do here.  We should explicitly _not_ try to add smarts to --with-lto itself as to whether we believe it will build a broken binary or not.
History
Date User Action Args
2016-11-21 08:19:52gregory.p.smithsetpriority: release blocker -> normal
status: open -> closed
resolution: not a bug
messages: + msg281332
2016-11-21 08:15:08gregory.p.smithsetmessages: + msg281330
2016-11-21 07:09:21ned.deilysetmessages: + msg281327
2016-11-21 01:16:47sjtsetnosy: + sjt
messages: + msg281313
2016-11-20 21:23:51gregory.p.smithsetmessages: + msg281300
2016-11-12 05:16:32koobssetmessages: + msg280631
2016-11-12 05:15:28koobssetstatus: closed -> open

nosy: + koobs
messages: + msg280630

resolution: fixed -> (no value)
2016-09-08 22:45:50gregory.p.smithsetstatus: open -> closed
resolution: fixed
stage: needs patch -> resolved
2016-09-08 22:45:28python-devsetmessages: + msg275198
2016-09-08 22:43:03python-devsetnosy: + python-dev
messages: + msg275194
2016-09-08 22:06:59gregory.p.smithsetmessages: + msg275182
2016-09-08 21:45:24pitrousetnosy: + pitrou
messages: + msg275177
2016-09-08 21:33:11gregory.p.smithsetnosy: + alecsandru.patrascu

type: crash
stage: needs patch
2016-09-08 21:32:32gregory.p.smithcreate