classification
Title: Some python extensions can't be compiled with clang 3.4
Type: compile error Stage: resolved
Components: Distutils Versions: Python 3.6, Python 3.5, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Antoine.Brodin.FreeBSD, Arfrever, Evelyn Mitchell, djc, doko, dstufft, eric.araujo, ezio.melotti, haypo, koobs, loewis, matrixise, paul.moore, python-dev, skrah, steve.dower, thomas-petazzoni, tim.golden, zach.ware
Priority: high Keywords: easy, needs review, patch

Created on 2014-02-25 13:33 by Antoine.Brodin.FreeBSD, last changed 2016-11-28 08:07 by skrah. This issue is now closed.

Files
File name Uploaded Description Edit
python-issue20767.diff koobs, 2014-02-28 09:04 review
Messages (27)
msg212176 - (view) Author: Antoine Brodin.FreeBSD (Antoine.Brodin.FreeBSD) Date: 2014-02-25 13:33
Hi,

On FreeBSD -current,  clang 3.4 is now the default compiler.
Clang 3.4 rejects "-R/path/to/lib" flag (previously in version 3.3 it just ignored it).

This leads to some errors with some python extensions:

cc -shared -O2 -pipe -fno-strict-aliasing build/temp.freebsd-11.0-CURRENT-amd64-2.7/_sqlite/cache.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/_sqlite/connection.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/_sqlite/cursor.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/_sqlite/microprotocols.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/_sqlite/module.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/_sqlite/prepare_protocol.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/_sqlite/row.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/_sqlite/statement.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/_sqlite/util.o -L/usr/local/lib -R/usr/local/lib -lsqlite3 -o build/lib.freebsd-11.0-CURRENT-amd64-2.7/_sqlite3.so
cc: error: unknown argument: '-R/usr/local/lib'
error: command 'cc' failed with exit status 1


cc -shared -O2 -pipe -DLDAP_DEPRECATED -fno-strict-aliasing build/temp.freebsd-11.0-CURRENT-amd64-2.7/Modules/LDAPObject.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/Modules/ldapcontrol.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/Modules/common.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/Modules/constants.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/Modules/errors.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/Modules/functions.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/Modules/schema.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/Modules/ldapmodule.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/Modules/message.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/Modules/version.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/Modules/options.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/Modules/berval.o -L/usr/local/lib -L/usr/lib -R/usr/local/lib -R/usr/lib -lldap_r -o build/lib.freebsd-11.0-CURRENT-amd64-2.7/_ldap.so
cc: error: unknown argument: '-R/usr/local/lib'
cc: error: unknown argument: '-R/usr/lib'
error: command 'cc' failed with exit status 1


cc -shared -O2 -pipe -fno-strict-aliasing build/temp.freebsd-11.0-CURRENT-amd64-2.7/Modules/_bsddb.o -L/usr/local/lib -R/usr/local/lib -o build/lib.freebsd-11.0-CURRENT-amd64-2.7/bsddb3/_pybsddb.so -ldb-4.3
cc: error: unknown argument: '-R/usr/local/lib'
error: command 'cc' failed with exit status 1


I found the patch below to help, it is for function runtime_library_dir_option in Lib/distutils/unixccompiler.py
-Wl,-rpath works for both gcc and clang on freebsd

--- ./Lib/distutils/unixccompiler.py.orig
+++ ./Lib/distutils/unixccompiler.py
@@ -228,6 +228,8 @@
         if sys.platform[:6] == "darwin":
             # MacOSX's linker doesn't understand the -R flag at all
             return "-L" + dir
+        elif sys.platform[:7] == "freebsd":
+            return "-Wl,-rpath=" + dir
         elif sys.platform[:5] == "hp-ux":
             if self._is_gcc(compiler):
                 return ["-Wl,+s", "-L" + dir]
msg212177 - (view) Author: Kubilay Kocak (koobs) Date: 2014-02-25 13:40
Details on how clang 3.4 changes behaviour for compiler flags: 

http://llvm.org/releases/3.4/tools/clang/docs/ReleaseNotes.html#new-compiler-flags
msg212185 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2014-02-25 15:48
this looks safe from my point of view.

However the real problem is that you unconditionally add a runtime path for a standard system path.  I think the better way to fix this is not to pass the -L and -R arguments at all if the library is found in a system path.
msg212188 - (view) Author: Antoine Brodin.FreeBSD (Antoine.Brodin.FreeBSD) Date: 2014-02-25 16:45
For the python-ldap extension, this seems to be a buglet in its setup.cfg, it lists /usr/lib in library_dirs and /usr/include in library_dirs

For the others, /usr/local/lib is not in the default library search path (only /lib and /usr/lib) so at least -L has to be specified.
msg212193 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2014-02-25 18:26
doko: how do you know the addition of the -R option is unconditional? and whom do you refer to by "you" who is adding the option?

In any case, the patch is independent of whether the option is added unconditionally, and I agree that the patch looks safe. The question is whether it should be extended to the other *BSDs as well.
msg212415 - (view) Author: Kubilay Kocak (koobs) Date: 2014-02-28 09:04
Attaching patch against default
msg212419 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-02-28 10:32
>  The question is whether it should be extended to the other *BSDs as well.

The change is not needed on Linux (if Clang 3.4 is used to compile extensions too)?
msg212718 - (view) Author: Kubilay Kocak (koobs) Date: 2014-03-04 13:23
It would be great if this could make it into 3.4, extended to include other OS's if necessary at a later date.
msg212719 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * Date: 2014-03-04 13:38
https://bugs.gentoo.org/show_bug.cgi?id=503352 suggests that this problem occurs also on Linux when using Clang.
msg215310 - (view) Author: Dirkjan Ochtman (djc) * (Python committer) Date: 2014-04-01 13:12
Can we have some more feedback on this?
msg215312 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-04-01 13:16
See also issue #21122.
msg215315 - (view) Author: Dirkjan Ochtman (djc) * (Python committer) Date: 2014-04-01 13:29
That does look to be a different issue, though.
msg257021 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2015-12-26 10:27
Over at the llvm bug tracker this is marked as a release blocker:

    https://llvm.org/bugs/show_bug.cgi?id=18164
msg270077 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2016-07-10 08:50
Strange, why can anyone edit the classification? It happens a lot lately.
msg270767 - (view) Author: Stéphane Wirtel (matrixise) * Date: 2016-07-18 15:58
I have checked the status of the FreeBSD issue, and it's "New". What do you think for the fix ?
msg271845 - (view) Author: Evelyn Mitchell (Evelyn Mitchell) * Date: 2016-08-02 19:39
The patch looks reasonable, but needs to be tested on FreeBSD.
msg271850 - (view) Author: Kubilay Kocak (koobs) Date: 2016-08-02 20:56
Nothing is required (as far as I'm aware) other than commit/merge on a two line change scoped only to FreeBSD.

The issue was reported for and on FreeBSD and the patch has been carried locally in all FreeBSD Python ports/packages (2.7, 3.3, 3.4, 3.5) for over two years:

https://svnweb.freebsd.org/ports?view=revision&revision=346428
https://svnweb.freebsd.org/ports?view=revision&revision=351610
msg271851 - (view) Author: Evelyn Mitchell (Evelyn Mitchell) * Date: 2016-08-02 21:00
I think this patch should pass to the next stage.
Thank you, koobs.
msg271852 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2016-08-02 21:08
We can of course commit this, but could you also lobby here?

   https://llvm.org/bugs/show_bug.cgi?id=18164


I'm seeing this quite often that we fix something here and no one insists on the upstream bug trackers.
msg271856 - (view) Author: Evelyn Mitchell (Evelyn Mitchell) * Date: 2016-08-02 21:35
Stefan, I've commented on the llvm bugzilla.

Thanks for suggesting that.
msg271857 - (view) Author: Kubilay Kocak (koobs) Date: 2016-08-02 21:35
@Stefan I've notified our FreeBSD Clang/LLVM people of the upstream bug status, though there are indications that the "-Wl,-rpath" method is considered the recommended/canonical/future way to do things properly.
msg271883 - (view) Author: Roundup Robot (python-dev) Date: 2016-08-03 09:19
New changeset 441bbf4cc914 by Stefan Krah in branch '3.5':
Issue #20767: Fix -R option for FreeBSD/clang.
https://hg.python.org/cpython/rev/441bbf4cc914
msg271884 - (view) Author: Roundup Robot (python-dev) Date: 2016-08-03 09:23
New changeset 65eb8d0ede75 by Stefan Krah in branch '2.7':
Issue #20767: Fix -R option for FreeBSD/clang.
https://hg.python.org/cpython/rev/65eb8d0ede75
msg271885 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2016-08-03 09:26
@Evelyn @koobs Thanks for communicating with the other projects that are involved in this issue!


Could you double-check that especially the 2.7 patch does exactly what you want? For 2.7 it's practically a one time chance.
msg271889 - (view) Author: Kubilay Kocak (koobs) Date: 2016-08-03 10:19
@Stefan I'll touch base with Antoine (OP) and confirm that this is a root-cause, permanent solution
msg271890 - (view) Author: Kubilay Kocak (koobs) Date: 2016-08-03 10:26
Not sure why the stage field changed on last submission. Restore accordingly to previous state
msg281845 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2016-11-28 08:07
I guess the FreeBSD people are happy with the solution.
History
Date User Action Args
2016-11-28 08:07:33skrahsetstatus: open -> closed
resolution: fixed
messages: + msg281845

stage: commit review -> resolved
2016-08-03 10:26:08koobssetmessages: + msg271890
stage: patch review -> commit review
2016-08-03 10:19:02koobssetmessages: + msg271889
stage: commit review -> patch review
2016-08-03 09:28:02skrahsetstage: patch review -> commit review
2016-08-03 09:26:58skrahsetmessages: + msg271885
2016-08-03 09:23:50python-devsetmessages: + msg271884
2016-08-03 09:19:23python-devsetnosy: + python-dev
messages: + msg271883
2016-08-02 21:35:30koobssetmessages: + msg271857
2016-08-02 21:35:04Evelyn Mitchellsetmessages: + msg271856
2016-08-02 21:08:06skrahsetmessages: + msg271852
2016-08-02 21:00:57Evelyn Mitchellsetmessages: + msg271851
2016-08-02 20:56:05koobssetmessages: + msg271850
2016-08-02 19:39:23Evelyn Mitchellsetnosy: + Evelyn Mitchell
messages: + msg271845
2016-07-18 15:58:48matrixisesetnosy: + matrixise
messages: + msg270767
2016-07-10 09:25:45berker.peksagsettype: performance -> compile error
components: - Installation, Interpreter Core, Library (Lib), Tests
versions: - Python 3.3, Python 3.4
2016-07-10 08:50:52skrahsetmessages: + msg270077
2016-07-10 05:26:56mostafa shahdadisettype: compile error -> performance
components: + Installation, Interpreter Core, Library (Lib), Tests
versions: + Python 3.3, Python 3.4
2016-04-12 18:52:57zach.waresethgrepos: - hgrepo339
2016-04-12 16:41:12berker.peksagsetstage: patch review
components: - Build, Extension Modules, Interpreter Core, Tkinter, Unicode, Windows, XML
versions: - Python 3.2, Python 3.3, Python 3.4
2016-04-12 15:46:51supriyantomaftuhsethgrepos: + hgrepo339

nosy: + ezio.melotti, paul.moore, tim.golden, eric.araujo, zach.ware, dstufft, steve.dower
components: + Build, Extension Modules, Interpreter Core, Tkinter, Unicode, Windows, XML
versions: + Python 3.2
2015-12-26 10:27:56skrahsetnosy: + skrah
messages: + msg257021
2015-12-26 02:43:19koobssetkeywords: + easy, needs review
versions: + Python 3.6
2014-04-01 13:29:58djcsetmessages: + msg215315
2014-04-01 13:16:22hayposetmessages: + msg215312
2014-04-01 13:12:52djcsetmessages: + msg215310
2014-04-01 13:12:27djcsetnosy: + djc
2014-03-04 13:38:57Arfreversetmessages: + msg212719
2014-03-04 13:23:03koobssetmessages: + msg212718
2014-02-28 10:32:18hayposetnosy: + haypo
messages: + msg212419
2014-02-28 09:04:49koobssetfiles: + python-issue20767.diff
type: compile error
messages: + msg212415

keywords: + patch
2014-02-26 04:50:02Arfreversetnosy: + Arfrever
2014-02-25 18:26:18loewissetmessages: + msg212193
2014-02-25 16:45:44Antoine.Brodin.FreeBSDsetmessages: + msg212188
2014-02-25 15:48:31dokosetmessages: + msg212185
2014-02-25 15:44:47rhettingersetpriority: normal -> high
2014-02-25 13:40:55koobssetmessages: + msg212177
2014-02-25 13:38:36koobssetversions: + Python 3.3, Python 3.4, Python 3.5
2014-02-25 13:34:27hayposetnosy: + loewis, doko, thomas-petazzoni
2014-02-25 13:33:18Antoine.Brodin.FreeBSDcreate