This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Some python extensions can't be compiled with clang 3.4
Type: compile error Stage: resolved
Components: Extension Modules Versions: Python 3.6, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: docs@python Nosy List: Alex.Willmer, Antoine.Brodin.FreeBSD, Arfrever, Evelyn Mitchell, asvetlov, barry, djc, docs@python, doko, dstufft, eric.araujo, ezio.melotti, koobs, larry, loewis, matrixise, mrabarnett, paul.moore, python-dev, skrah, steve.dower, thomas-petazzoni, tim.golden, vstinner, yselivanov, zach.ware
Priority: normal Keywords: easy, needs review, patch

Created on 2014-02-25 13:33 by Antoine.Brodin.FreeBSD, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
python-issue20767.diff koobs, 2014-02-28 09:04 review
Messages (33)
msg212176 - (view) Author: Antoine Brodin.FreeBSD (Antoine.Brodin.FreeBSD) Date: 2014-02-25 13:33
Hi,

On FreeBSD -current,  clang 3.4 is now the default compiler.
Clang 3.4 rejects "-R/path/to/lib" flag (previously in version 3.3 it just ignored it).

This leads to some errors with some python extensions:

cc -shared -O2 -pipe -fno-strict-aliasing build/temp.freebsd-11.0-CURRENT-amd64-2.7/_sqlite/cache.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/_sqlite/connection.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/_sqlite/cursor.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/_sqlite/microprotocols.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/_sqlite/module.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/_sqlite/prepare_protocol.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/_sqlite/row.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/_sqlite/statement.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/_sqlite/util.o -L/usr/local/lib -R/usr/local/lib -lsqlite3 -o build/lib.freebsd-11.0-CURRENT-amd64-2.7/_sqlite3.so
cc: error: unknown argument: '-R/usr/local/lib'
error: command 'cc' failed with exit status 1


cc -shared -O2 -pipe -DLDAP_DEPRECATED -fno-strict-aliasing build/temp.freebsd-11.0-CURRENT-amd64-2.7/Modules/LDAPObject.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/Modules/ldapcontrol.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/Modules/common.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/Modules/constants.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/Modules/errors.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/Modules/functions.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/Modules/schema.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/Modules/ldapmodule.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/Modules/message.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/Modules/version.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/Modules/options.o build/temp.freebsd-11.0-CURRENT-amd64-2.7/Modules/berval.o -L/usr/local/lib -L/usr/lib -R/usr/local/lib -R/usr/lib -lldap_r -o build/lib.freebsd-11.0-CURRENT-amd64-2.7/_ldap.so
cc: error: unknown argument: '-R/usr/local/lib'
cc: error: unknown argument: '-R/usr/lib'
error: command 'cc' failed with exit status 1


cc -shared -O2 -pipe -fno-strict-aliasing build/temp.freebsd-11.0-CURRENT-amd64-2.7/Modules/_bsddb.o -L/usr/local/lib -R/usr/local/lib -o build/lib.freebsd-11.0-CURRENT-amd64-2.7/bsddb3/_pybsddb.so -ldb-4.3
cc: error: unknown argument: '-R/usr/local/lib'
error: command 'cc' failed with exit status 1


I found the patch below to help, it is for function runtime_library_dir_option in Lib/distutils/unixccompiler.py
-Wl,-rpath works for both gcc and clang on freebsd

--- ./Lib/distutils/unixccompiler.py.orig
+++ ./Lib/distutils/unixccompiler.py
@@ -228,6 +228,8 @@
         if sys.platform[:6] == "darwin":
             # MacOSX's linker doesn't understand the -R flag at all
             return "-L" + dir
+        elif sys.platform[:7] == "freebsd":
+            return "-Wl,-rpath=" + dir
         elif sys.platform[:5] == "hp-ux":
             if self._is_gcc(compiler):
                 return ["-Wl,+s", "-L" + dir]
msg212177 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2014-02-25 13:40
Details on how clang 3.4 changes behaviour for compiler flags: 

http://llvm.org/releases/3.4/tools/clang/docs/ReleaseNotes.html#new-compiler-flags
msg212185 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2014-02-25 15:48
this looks safe from my point of view.

However the real problem is that you unconditionally add a runtime path for a standard system path.  I think the better way to fix this is not to pass the -L and -R arguments at all if the library is found in a system path.
msg212188 - (view) Author: Antoine Brodin.FreeBSD (Antoine.Brodin.FreeBSD) Date: 2014-02-25 16:45
For the python-ldap extension, this seems to be a buglet in its setup.cfg, it lists /usr/lib in library_dirs and /usr/include in library_dirs

For the others, /usr/local/lib is not in the default library search path (only /lib and /usr/lib) so at least -L has to be specified.
msg212193 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2014-02-25 18:26
doko: how do you know the addition of the -R option is unconditional? and whom do you refer to by "you" who is adding the option?

In any case, the patch is independent of whether the option is added unconditionally, and I agree that the patch looks safe. The question is whether it should be extended to the other *BSDs as well.
msg212415 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2014-02-28 09:04
Attaching patch against default
msg212419 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-02-28 10:32
>  The question is whether it should be extended to the other *BSDs as well.

The change is not needed on Linux (if Clang 3.4 is used to compile extensions too)?
msg212718 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2014-03-04 13:23
It would be great if this could make it into 3.4, extended to include other OS's if necessary at a later date.
msg212719 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * (Python triager) Date: 2014-03-04 13:38
https://bugs.gentoo.org/show_bug.cgi?id=503352 suggests that this problem occurs also on Linux when using Clang.
msg215310 - (view) Author: Dirkjan Ochtman (djc) * (Python committer) Date: 2014-04-01 13:12
Can we have some more feedback on this?
msg215312 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-04-01 13:16
See also issue #21122.
msg215315 - (view) Author: Dirkjan Ochtman (djc) * (Python committer) Date: 2014-04-01 13:29
That does look to be a different issue, though.
msg257021 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2015-12-26 10:27
Over at the llvm bug tracker this is marked as a release blocker:

    https://llvm.org/bugs/show_bug.cgi?id=18164
msg270077 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2016-07-10 08:50
Strange, why can anyone edit the classification? It happens a lot lately.
msg270767 - (view) Author: Stéphane Wirtel (matrixise) * (Python committer) Date: 2016-07-18 15:58
I have checked the status of the FreeBSD issue, and it's "New". What do you think for the fix ?
msg271845 - (view) Author: Evelyn Mitchell (Evelyn Mitchell) * (Python triager) Date: 2016-08-02 19:39
The patch looks reasonable, but needs to be tested on FreeBSD.
msg271850 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2016-08-02 20:56
Nothing is required (as far as I'm aware) other than commit/merge on a two line change scoped only to FreeBSD.

The issue was reported for and on FreeBSD and the patch has been carried locally in all FreeBSD Python ports/packages (2.7, 3.3, 3.4, 3.5) for over two years:

https://svnweb.freebsd.org/ports?view=revision&revision=346428
https://svnweb.freebsd.org/ports?view=revision&revision=351610
msg271851 - (view) Author: Evelyn Mitchell (Evelyn Mitchell) * (Python triager) Date: 2016-08-02 21:00
I think this patch should pass to the next stage.
Thank you, koobs.
msg271852 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2016-08-02 21:08
We can of course commit this, but could you also lobby here?

   https://llvm.org/bugs/show_bug.cgi?id=18164


I'm seeing this quite often that we fix something here and no one insists on the upstream bug trackers.
msg271856 - (view) Author: Evelyn Mitchell (Evelyn Mitchell) * (Python triager) Date: 2016-08-02 21:35
Stefan, I've commented on the llvm bugzilla.

Thanks for suggesting that.
msg271857 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2016-08-02 21:35
@Stefan I've notified our FreeBSD Clang/LLVM people of the upstream bug status, though there are indications that the "-Wl,-rpath" method is considered the recommended/canonical/future way to do things properly.
msg271883 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-08-03 09:19
New changeset 441bbf4cc914 by Stefan Krah in branch '3.5':
Issue #20767: Fix -R option for FreeBSD/clang.
https://hg.python.org/cpython/rev/441bbf4cc914
msg271884 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-08-03 09:23
New changeset 65eb8d0ede75 by Stefan Krah in branch '2.7':
Issue #20767: Fix -R option for FreeBSD/clang.
https://hg.python.org/cpython/rev/65eb8d0ede75
msg271885 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2016-08-03 09:26
@Evelyn @koobs Thanks for communicating with the other projects that are involved in this issue!


Could you double-check that especially the 2.7 patch does exactly what you want? For 2.7 it's practically a one time chance.
msg271889 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2016-08-03 10:19
@Stefan I'll touch base with Antoine (OP) and confirm that this is a root-cause, permanent solution
msg271890 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2016-08-03 10:26
Not sure why the stage field changed on last submission. Restore accordingly to previous state
msg281845 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2016-11-28 08:07
I guess the FreeBSD people are happy with the solution.
msg310437 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-01-22 17:35
> I guess the FreeBSD people are happy with the solution.

According to the discussion on https://github.com/python/cpython/pull/5233 : some FreeBSD people are unhappy :-)
msg310446 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2018-01-22 18:27
Well, they had ample time to articulate themselves. :)
msg311315 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2018-01-31 07:48
The changeset(s) applied in this issue were/are fine (thus no response on the issue)

It is the *removal* of the changesets included in this issue (as proposed in https://github.com/python/cpython/pull/5233) that would re-break compilation of extensions with clang (requiring re-opening this issue)

The root cause of the current confusion is the special case added here was conditional on the OS ('freebsd'), whereas it should (arguably) have been for a specific compiler (clang). 

In order to do this, this leads to the compiler detection code requiring improvement, see: https://github.com/python/cpython/pull/5233#issuecomment-358906977 as it currently does not detect clang as the default compiler (cc), see: https://github.com/python/cpython/pull/5233#issuecomment-358948425
msg311316 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2018-01-31 07:51
If the intent is to use this issue to track the resolution of the original issue (clang compilation broken) and cover the new PR (which just moves the code from os detection to compiler detection), then it can remain open. 

Otherwise it can be closed, as the original issue 'has' been fixed.
msg334606 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2019-01-31 06:00
It's too late to fix this for Python 3.4 and 3.5, as those are now in security-fixes-only mode.  Also, please don't select every possible component that could be remotely connected.
msg334620 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-01-31 10:26
> If the intent is to use this issue to track the resolution of the original issue (clang compilation broken) and cover the new PR (which just moves the code from os detection to compiler detection), then it can remain open. Otherwise it can be closed, as the original issue 'has' been fixed.

Ok, I close the issue.

On https://github.com/python/cpython/pull/5233 bapt asked "code changes", but I don't understand what exactly and I'm not interested to continue to work on these changes. If someone is interested, please open a new issue (pointing to this one).
History
Date User Action Args
2022-04-11 14:57:59adminsetgithub: 64966
2019-01-31 22:45:10r.david.murraysetnosy: - r.david.murray
2019-01-31 10:26:46vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg334620
2019-01-31 10:02:57terry.reedysetnosy: - terry.reedy
2019-01-31 06:01:45larrysetpriority: high -> normal
type: resource usage -> compile error
2019-01-31 06:00:11larrysetmessages: + msg334606
components: - Build, Demos and Tools, Distutils, Documentation, IDLE, Installation, Interpreter Core, Library (Lib), Regular Expressions, Tests, Tkinter, Unicode, XML, 2to3 (2.x to 3.x conversion tool), ctypes, IO, Cross-Build, email, asyncio, Argument Clinic, FreeBSD, SSL
versions: - Python 3.4, Python 3.5
2019-01-31 05:57:26Pascal van der Doncksetversions: + Python 3.4
nosy: + terry.reedy, larry, asvetlov, mrabarnett, yselivanov, barry, Alex.Willmer, r.david.murray, docs@python

assignee: docs@python
components: + Build, Demos and Tools, Documentation, Extension Modules, IDLE, Installation, Interpreter Core, Library (Lib), Regular Expressions, Tests, Tkinter, Unicode, XML, 2to3 (2.x to 3.x conversion tool), ctypes, IO, Cross-Build, email, asyncio, Argument Clinic, FreeBSD, SSL
type: compile error -> resource usage
2018-01-31 07:51:48koobssetmessages: + msg311316
2018-01-31 07:48:49koobssetmessages: + msg311315
2018-01-22 18:27:28skrahsetmessages: + msg310446
2018-01-22 17:35:02vstinnersetstatus: closed -> open
resolution: fixed -> (no value)
messages: + msg310437
2016-11-28 08:07:33skrahsetstatus: open -> closed
resolution: fixed
messages: + msg281845

stage: commit review -> resolved
2016-08-03 10:26:08koobssetmessages: + msg271890
stage: patch review -> commit review
2016-08-03 10:19:02koobssetmessages: + msg271889
stage: commit review -> patch review
2016-08-03 09:28:02skrahsetstage: patch review -> commit review
2016-08-03 09:26:58skrahsetmessages: + msg271885
2016-08-03 09:23:50python-devsetmessages: + msg271884
2016-08-03 09:19:23python-devsetnosy: + python-dev
messages: + msg271883
2016-08-02 21:35:30koobssetmessages: + msg271857
2016-08-02 21:35:04Evelyn Mitchellsetmessages: + msg271856
2016-08-02 21:08:06skrahsetmessages: + msg271852
2016-08-02 21:00:57Evelyn Mitchellsetmessages: + msg271851
2016-08-02 20:56:05koobssetmessages: + msg271850
2016-08-02 19:39:23Evelyn Mitchellsetnosy: + Evelyn Mitchell
messages: + msg271845
2016-07-18 15:58:48matrixisesetnosy: + matrixise
messages: + msg270767
2016-07-10 09:25:45berker.peksagsettype: performance -> compile error
components: - Installation, Interpreter Core, Library (Lib), Tests
versions: - Python 3.3, Python 3.4
2016-07-10 08:50:52skrahsetmessages: + msg270077
2016-07-10 05:26:56Pes2009ksettype: compile error -> performance
components: + Installation, Interpreter Core, Library (Lib), Tests
versions: + Python 3.3, Python 3.4
2016-04-12 18:52:57zach.waresethgrepos: - hgrepo339
2016-04-12 16:41:12berker.peksagsetstage: patch review
components: - Build, Extension Modules, Interpreter Core, Tkinter, Unicode, Windows, XML
versions: - Python 3.2, Python 3.3, Python 3.4
2016-04-12 15:46:51supriyantomaftuhsethgrepos: + hgrepo339

nosy: + ezio.melotti, paul.moore, tim.golden, eric.araujo, zach.ware, dstufft, steve.dower
components: + Build, Extension Modules, Interpreter Core, Tkinter, Unicode, Windows, XML
versions: + Python 3.2
2015-12-26 10:27:56skrahsetnosy: + skrah
messages: + msg257021
2015-12-26 02:43:19koobssetkeywords: + easy, needs review
versions: + Python 3.6
2014-04-01 13:29:58djcsetmessages: + msg215315
2014-04-01 13:16:22vstinnersetmessages: + msg215312
2014-04-01 13:12:52djcsetmessages: + msg215310
2014-04-01 13:12:27djcsetnosy: + djc
2014-03-04 13:38:57Arfreversetmessages: + msg212719
2014-03-04 13:23:03koobssetmessages: + msg212718
2014-02-28 10:32:18vstinnersetnosy: + vstinner
messages: + msg212419
2014-02-28 09:04:49koobssetfiles: + python-issue20767.diff
type: compile error
messages: + msg212415

keywords: + patch
2014-02-26 04:50:02Arfreversetnosy: + Arfrever
2014-02-25 18:26:18loewissetmessages: + msg212193
2014-02-25 16:45:44Antoine.Brodin.FreeBSDsetmessages: + msg212188
2014-02-25 15:48:31dokosetmessages: + msg212185
2014-02-25 15:44:47rhettingersetpriority: normal -> high
2014-02-25 13:40:55koobssetmessages: + msg212177
2014-02-25 13:38:36koobssetversions: + Python 3.3, Python 3.4, Python 3.5
2014-02-25 13:34:27vstinnersetnosy: + loewis, doko, thomas-petazzoni
2014-02-25 13:33:18Antoine.Brodin.FreeBSDcreate