classification
Title: configure --with-lto builds fail when CC=clang on Linux, requires gold linker
Type: compile error Stage: resolved
Components: Build Versions: Python 3.8, Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: benjamin.peterson, cstratak, gregory.p.smith, miss-islington, ned.deily, serge-sans-paille, vstinner
Priority: normal Keywords: patch

Created on 2016-09-08 07:45 by gregory.p.smith, last changed 2018-12-09 08:14 by ned.deily. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 9908 merged python-dev, 2018-10-16 09:15
PR 10089 merged miss-islington, 2018-10-25 00:00
PR 10089 merged miss-islington, 2018-10-25 00:00
PR 10922 merged cstratak, 2018-12-05 15:22
PR 10922 merged cstratak, 2018-12-05 15:22
PR 10922 merged cstratak, 2018-12-05 15:22
Messages (11)
msg274984 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2016-09-08 07:45
The error message you will see when building may look something like:

clang -pthread -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes  -fprofile-instr-generate -flto Parser/acceler.o Parser/grammar1.o Parser/listnode.o Parser/node.o Parser/parser.o Parser/bitset.o Parser/metagrammar.o Parser/firstsets.o Parser/grammar.o Parser/pgen.o Objects/obmalloc.o Python/dynamic_annotations.o Python/mysnprintf.o Python/pyctype.o Parser/tokenizer_pgen.o Parser/printgrammar.o Parser/parsetok_pgen.o Parser/pgenmain.o -lpthread -ldl  -lutil -o Parser/pgen
/usr/bin/ld: /usr/lib/llvm-3.8/bin/../lib/LLVMgold.so: error loading plugin: /usr/lib/llvm-3.8/bin/../lib/LLVMgold.so: cannot open shared object file: No such file or directory
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Makefile:793: recipe for target 'Parser/pgen' failed
make[2]: *** [Parser/pgen] Error 1

as seen on ubuntu 16.04.

Related to https://bugs.launchpad.net/ubuntu/+source/llvm-toolchain-snapshot/+bug/1254970 which suggests we also need -B/usr/lib/gold-ld in our LDFLAGS and/or CFLAGS when using clang on Linux with LTO.

testing with it set in both still runs into errors; someone with clang LTO experience will need to untangle this.

(is this related to why --with-lto doesn't work on MacOS as well?)
msg327816 - (view) Author: (serge-sans-paille) * Date: 2018-10-16 09:36
Looks like a package dependency issue: installing ``llvm-dev`` package should fix the problem. Or in that particular case ``llvm-3.8-dev``.
msg327817 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-10-16 09:41
I confirm that it's currently broken. Test on Fedora 28, clang version 6.0.1 (tags/RELEASE_601/final):

$ ./configure --with-pydebug CC=clang --with-lto  && make
(...)
checking for x64 gcc inline assembler... yes
checking whether float word ordering is bigendian... unknown
configure: error: 

Unknown float word ordering. You need to manually preset
ax_cv_c_float_words_bigendian=no (or yes) according to your system.
msg328403 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-10-24 23:54
New changeset 5ad36f9b21a3aa3b2265b1b43d73522cc3322df2 by Victor Stinner (serge-sans-paille) in branch 'master':
bpo-28015: Support LTO build with clang (GH-9908)
https://github.com/python/cpython/commit/5ad36f9b21a3aa3b2265b1b43d73522cc3322df2
msg328405 - (view) Author: miss-islington (miss-islington) Date: 2018-10-25 00:32
New changeset 69a3f153a92fd8c86080e8da477ee50df18fc0d1 by Miss Islington (bot) in branch '3.7':
bpo-28015: Support LTO build with clang (GH-9908)
https://github.com/python/cpython/commit/69a3f153a92fd8c86080e8da477ee50df18fc0d1
msg328826 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-10-29 13:39
The issue has been fixed in Python 3.7 and master.

If someone is volunteer to backport it to older version, please go ahead :-) But according to serge-sans-paille, it's non trivial and I'm not sure that it's worth it. Sorry, usually we focus on the master branch for "new features".
https://github.com/python/cpython/pull/9908#issuecomment-433896161

So I close the issue.

Thanks serge-sans-paille for this nice enhancement!
msg329002 - (view) Author: Charalampos Stratakis (cstratak) * Date: 2018-10-31 17:45
The issue with the 3.6 branch is https://bugs.python.org/issue31625 which hasn't been backported to 3.6. Would it make sense to backport it?

I have the backports ready locally but not sure if pushing https://bugs.python.org/issue31625 to 3.6 is desirable.
msg329042 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2018-11-01 04:39
Probably that's fine, but can #31625 be easily worked around by, e.g., using make AR=llvm-ar RANLIB=llvm-ranlib?
msg329421 - (view) Author: Charalampos Stratakis (cstratak) * Date: 2018-11-07 15:55
The workaround indeed works.
msg331414 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-12-09 08:08
New changeset f83ee476d48dbeb90ddf3526b04936a49a87973a by Ned Deily (stratakis) in branch '3.6':
bpo-28015: Support LTO build with clang (GH-9908) (GH-10922)
https://github.com/python/cpython/commit/f83ee476d48dbeb90ddf3526b04936a49a87973a
msg331415 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-12-09 08:14
See also Issue31354, Issue35351, and Issue35257.
History
Date User Action Args
2018-12-09 08:14:19ned.deilysetmessages: + msg331415
versions: + Python 3.6
2018-12-09 08:08:46ned.deilysetnosy: + ned.deily
messages: + msg331414
2018-12-05 15:22:16cstrataksetpull_requests: + pull_request10163
2018-12-05 15:22:12cstrataksetpull_requests: + pull_request10162
2018-12-05 15:22:05cstrataksetpull_requests: + pull_request10161
2018-11-07 15:55:51cstrataksetmessages: + msg329421
2018-11-01 04:39:10benjamin.petersonsetnosy: + benjamin.peterson
messages: + msg329042
2018-10-31 17:45:57cstrataksetnosy: + cstratak
messages: + msg329002
2018-10-29 13:39:40vstinnersetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2018-10-29 13:39:33vstinnersetmessages: + msg328826
versions: + Python 3.7, Python 3.8, - Python 2.7, Python 3.5, Python 3.6
2018-10-25 00:32:09miss-islingtonsetnosy: + miss-islington
messages: + msg328405
2018-10-25 00:00:33miss-islingtonsetpull_requests: + pull_request9418
2018-10-25 00:00:29miss-islingtonsetpull_requests: + pull_request9417
2018-10-24 23:54:25vstinnersetmessages: + msg328403
2018-10-16 09:41:56vstinnersetnosy: + vstinner
messages: + msg327817
2018-10-16 09:36:10serge-sans-paillesetnosy: + serge-sans-paille
messages: + msg327816
2018-10-16 09:15:53python-devsetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request9267
2016-09-08 07:45:09gregory.p.smithcreate