Hi,

On FreeBSD -current,  clang 3.4 is now the default compiler.
Clang 3.4 rejects "-R/path/to/lib" flag (previously in version 3.3 it just ignored it).

This leads to some errors with some python extensions:

cc -shared -O2 -pipe -fno-strict-aliasing build/temp.freebsd-11.0-CURRENT-amd64-2.7/_sqlite/cache.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/_sqlite/connection.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/_sqlite/cursor.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/_sqlite/microprotocols.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/_sqlite/module.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/_sqlite/prepare_protocol.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/_sqlite/row.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/_sqlite/statement.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/_sqlite/util.o -L/usr/local/lib -R/usr/local/lib -lsqlite3 -o build/lib.freebsd-11.0-CURRENT-amd64-2.7/_sqlite3.so
cc: error: unknown argument: '-R/usr/local/lib'
error: command 'cc' failed with exit status 1


cc -shared -O2 -pipe -DLDAP_DEPRECATED -fno-strict-aliasing build/temp.freebsd-11.0-CURRENT-amd64-2.7/Modules/LDAPObject.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/Modules/ldapcontrol.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/Modules/common.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/Modules/constants.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/Modules/errors.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/Modules/functions.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/Modules/schema.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/Modules/ldapmodule.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/Modules/message.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/Modules/version.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/Modules/options.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/Modules/berval.o -L/usr/local/lib -L/usr/lib -R/usr/local/lib -R/usr/lib -lldap_r -o build/lib.freebsd-11.0-CURRENT-amd64-2.7/_ldap.so
cc: error: unknown argument: '-R/usr/local/lib'
cc: error: unknown argument: '-R/usr/lib'
error: command 'cc' failed with exit status 1


cc -shared -O2 -pipe -fno-strict-aliasing build/temp.freebsd-11.0-CURRENT-amd64-2.7/Modules/_bsddb.o -L/usr/local/lib -R/usr/local/lib -o build/lib.freebsd-11.0-CURRENT-amd64-2.7/bsddb3/_pybsddb.so -ldb-4.3
cc: error: unknown argument: '-R/usr/local/lib'
error: command 'cc' failed with exit status 1


I found the patch below to help, it is for function runtime_library_dir_option in Lib/distutils/unixccompiler.py
-Wl,-rpath works for both gcc and clang on freebsd

--- ./Lib/distutils/unixccompiler.py.orig
+++ ./Lib/distutils/unixccompiler.py
@@ -228,6 +228,8 @@
         if sys.platform[:6] == "darwin":
             # MacOSX's linker doesn't understand the -R flag at all
             return "-L" + dir
+        elif sys.platform[:7] == "freebsd":
+            return "-Wl,-rpath=" + dir
         elif sys.platform[:5] == "hp-ux":
             if self._is_gcc(compiler):
                 return ["-Wl,+s", "-L" + dir]

Details on how clang 3.4 changes behaviour for compiler flags: 

http://llvm.org/releases/3.4/tools/clang/docs/ReleaseNotes.html#new-compiler-flags

this looks safe from my point of view.

However the real problem is that you unconditionally add a runtime path for a standard system path.  I think the better way to fix this is not to pass the -L and -R arguments at all if the library is found in a system path.

For the python-ldap extension, this seems to be a buglet in its setup.cfg, it lists /usr/lib in library_dirs and /usr/include in library_dirs

For the others, /usr/local/lib is not in the default library search path (only /lib and /usr/lib) so at least -L has to be specified.

doko: how do you know the addition of the -R option is unconditional? and whom do you refer to by "you" who is adding the option?

In any case, the patch is independent of whether the option is added unconditionally, and I agree that the patch looks safe. The question is whether it should be extended to the other *BSDs as well.

Attaching patch against default

>  The question is whether it should be extended to the other *BSDs as well.

The change is not needed on Linux (if Clang 3.4 is used to compile extensions too)?

It would be great if this could make it into 3.4, extended to include other OS's if necessary at a later date.

https://bugs.gentoo.org/show_bug.cgi?id=503352 suggests that this problem occurs also on Linux when using Clang.

Can we have some more feedback on this?

See also issue #21122.

That does look to be a different issue, though.

Over at the llvm bug tracker this is marked as a release blocker:

    https://llvm.org/bugs/show_bug.cgi?id=18164

Strange, why can anyone edit the classification? It happens a lot lately.

I have checked the status of the FreeBSD issue, and it's "New". What do you think for the fix ?

The patch looks reasonable, but needs to be tested on FreeBSD.

Nothing is required (as far as I'm aware) other than commit/merge on a two line change scoped only to FreeBSD.

The issue was reported for and on FreeBSD and the patch has been carried locally in all FreeBSD Python ports/packages (2.7, 3.3, 3.4, 3.5) for over two years:

https://svnweb.freebsd.org/ports?view=revision&revision=346428
https://svnweb.freebsd.org/ports?view=revision&revision=351610

I think this patch should pass to the next stage.
Thank you, koobs.

We can of course commit this, but could you also lobby here?

   https://llvm.org/bugs/show_bug.cgi?id=18164


I'm seeing this quite often that we fix something here and no one insists on the upstream bug trackers.

Stefan, I've commented on the llvm bugzilla.

Thanks for suggesting that.

@Stefan I've notified our FreeBSD Clang/LLVM people of the upstream bug status, though there are indications that the "-Wl,-rpath" method is considered the recommended/canonical/future way to do things properly.

New changeset 441bbf4cc914 by Stefan Krah in branch '3.5':
Issue #20767: Fix -R option for FreeBSD/clang.
https://hg.python.org/cpython/rev/441bbf4cc914

New changeset 65eb8d0ede75 by Stefan Krah in branch '2.7':
Issue #20767: Fix -R option for FreeBSD/clang.
https://hg.python.org/cpython/rev/65eb8d0ede75

@Evelyn @koobs Thanks for communicating with the other projects that are involved in this issue!


Could you double-check that especially the 2.7 patch does exactly what you want? For 2.7 it's practically a one time chance.

@Stefan I'll touch base with Antoine (OP) and confirm that this is a root-cause, permanent solution

Not sure why the stage field changed on last submission. Restore accordingly to previous state

I guess the FreeBSD people are happy with the solution.

> I guess the FreeBSD people are happy with the solution.

According to the discussion on https://github.com/python/cpython/pull/5233 : some FreeBSD people are unhappy :-)

Well, they had ample time to articulate themselves. :)

The changeset(s) applied in this issue were/are fine (thus no response on the issue)

It is the *removal* of the changesets included in this issue (as proposed in https://github.com/python/cpython/pull/5233) that would re-break compilation of extensions with clang (requiring re-opening this issue)

The root cause of the current confusion is the special case added here was conditional on the OS ('freebsd'), whereas it should (arguably) have been for a specific compiler (clang). 

In order to do this, this leads to the compiler detection code requiring improvement, see: https://github.com/python/cpython/pull/5233#issuecomment-358906977 as it currently does not detect clang as the default compiler (cc), see: https://github.com/python/cpython/pull/5233#issuecomment-358948425

If the intent is to use this issue to track the resolution of the original issue (clang compilation broken) and cover the new PR (which just moves the code from os detection to compiler detection), then it can remain open. 

Otherwise it can be closed, as the original issue 'has' been fixed.

It's too late to fix this for Python 3.4 and 3.5, as those are now in security-fixes-only mode.  Also, please don't select every possible component that could be remotely connected.

> If the intent is to use this issue to track the resolution of the original issue (clang compilation broken) and cover the new PR (which just moves the code from os detection to compiler detection), then it can remain open. Otherwise it can be closed, as the original issue 'has' been fixed.

Ok, I close the issue.

On https://github.com/python/cpython/pull/5233 bapt asked "code changes", but I don't understand what exactly and I'm not interested to continue to work on these changes. If someone is interested, please open a new issue (pointing to this one).