This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Investigate using Apple clang for building OS X installers
Type: Stage: resolved
Components: Build, macOS Versions: Python 3.7, Python 3.6, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: ned.deily Nosy List: abarnert, ned.deily, python-dev, r.david.murray, rhettinger, ronaldoussoren, skip.montanaro
Priority: normal Keywords: patch

Created on 2013-09-14 20:31 by ned.deily, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 6469 merged ned.deily, 2018-04-14 14:25
PR 6470 merged ned.deily, 2018-04-14 14:48
Messages (14)
msg197729 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-09-14 20:31
On the python-dev mailing list, Raymond Hettinger asks:
> I was exercising the alpha two release of 3.4 and noticed that
> it is still being built under GCC 4.2.1.  
> 
> Is there any reason we have to use an old compiler?

Yes, kinda.  It's because the 64-bit/32-bit installer we supply is designed to run on multiple versions of OS X: 10.6 (Snow Leopard), 10.7, 10.8, and the upcoming 10.9.  The safest way to ensure that the bits produced (executables, shared libs, and .so bundles) are linked with compatible versions of system shared libraries and frameworks is to build on the lowest supported system, 10.6, or - carefully - with a 10.6 SDK on a newer system.  The standard compiler in the Apple Developer Tools for 10.6 is gcc 4.2.  The most recent Developer Tools for 10.8 and 10.7 systems, Xcode 4.6.x, have a mature clang but do not provide a 10.6 SDK.  Even with using an SDK, it's still possible to end up inadvertently linking with the wrong versions of system libraries.  We have been burned by that in the past.

> I would like to see it built under the latest version of Clang
> (like the other tools on the Mac) or under GCC 4.8.1.
> I've better using the latter for many months.  It works great
> and produces *much* better code (in terms of readability
> for those of us who look at assembly output and in terms
> of speed (approx 20-25% better for cpu bound code)).
> 
> I'm wondering whether is some issue that forces us to use
> GCC 4.2.1 or whether that is just an artifact of continuing
> to do what we've always been doing.

I'd like to move to Apple's latest clang, as well.  (Apple is dropping gcc totally from the next release of Xcode.)  There are a couple of approaches that may work and still reliably support multiple versions of OS X.  Investigating this is on my list of things to do prior to the next alpha.
msg198028 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2013-09-18 17:26
If you find it difficult to find people with 10.6 systems to build a Python dmg for that platform, give me a shout. I'm not planning on replacing my system anytime soon. I have 10.6.8 installed, and what I believe to be the last Xcode version to support it, 3.2.something.
msg198029 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-09-18 17:41
Thanks, Skip. At the moment, finding platforms for building and testing is not a problem: I have at hand running versions of the latest releases for 10.3 onwards. :)  (Buildbots for automated testing are another issue, though. We have a critical need for buildbots running more recent versions of OS X, though.  At the moment, the only OS X buildbot is one running 10.4 (Tiger).  If anyone is able to provide newer ones, please see http://python.org/dev/buildbot/.  Or, if someone would like to donate hardware, I may be able to arrange hosting for it.)
msg198030 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2013-09-18 18:50
I guess snakebite (http://www.snakebite.net/) never materialized?
msg198031 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-09-18 19:05
Snakebite did provide some additional OS X buildbots for a while but, AFAIK, they have been offline for some months now.
msg198032 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-09-18 19:41
Snakebite is live, and python committers can get shell access on the bots, as originally planned.  Some of the bots are offline, but most are running.  I don't see any OS X servers in the list, though, unless Antoine deleted them (but there are several other currently-offline ones he hasn't deleted, so I doubt that).
msg198176 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-09-20 22:29
I've spun up a 10.6.8 OS X buildbot.
msg200367 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013-10-19 04:36
New changeset 5e385dcfbd32 by Ned Deily in branch 'default':
Issue #19019: Change the OS X installer build script to use CFLAGS instead
http://hg.python.org/cpython/rev/5e385dcfbd32
msg201198 - (view) Author: Andrew Barnert (abarnert) * Date: 2013-10-25 00:05
Now that Xcode 5 is out, and does not have anything named "gcc-4.2" (it _does_ still have something called "gcc", which is now a wrapper around clang instead of a wrapper around llvm-gcc-4.2), this seems a lot more critical, because (as far as I can tell) nobody with Xcode 5 can build C extensions for python.org Python installations.

And that also implies that any changes need to be backported to 2.7 and 3.3, unless the plan is that only Python 3.4+ should work with Xcode 5.
msg201199 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-10-25 01:37
"nobody with Xcode 5 can build C extensions for python.org Python installations"

Andrew, can you elaborate on what problems you or others are seeing?  Xcode 5 should not be an issue for users of the current python.org Pythons. When building extension modules, if the compiler Python was built with is not available Distutils in the current 2.7.x, 3.3.x, and 3.4.x release will substitute clang.  This assumes they have the Command Line Tools installed, either through Xcode or the standalone CLT package and assumes that the user is not overriding defaults by setting environment variables like CC and LDSHARED which Distutils will honor without attempting to modify them.  This should all be independent of what compiler Python was built with.
msg201220 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013-10-25 07:54
New changeset 323f1dcceb73 by Ned Deily in branch '2.7':
Issue #19019: Change the OS X installer build script to use CFLAGS instead
http://hg.python.org/cpython/rev/323f1dcceb73

New changeset 689b5c2d6959 by Ned Deily in branch '3.3':
Issue #19019: Change the OS X installer build script to use CFLAGS instead
http://hg.python.org/cpython/rev/689b5c2d6959
msg315293 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-04-14 14:37
New changeset ee8e4b61d6181f5bfadf6ee3d229f0ebda8a6ae9 by Ned Deily in branch '2.7':
[2.7] Backport macOS universal build and installer fixes from 3.6. (GH-6469)
https://github.com/python/cpython/commit/ee8e4b61d6181f5bfadf6ee3d229f0ebda8a6ae9
msg315295 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-04-14 14:56
New changeset 93c91ac8f67ef0819ddacdcca12ef0ae88e5802e by Ned Deily in branch '2.7':
[2.7] Fix errant NEWS item: bpo-19019 -> bpo-17128 (GH-6470)
https://github.com/python/cpython/commit/93c91ac8f67ef0819ddacdcca12ef0ae88e5802e
msg315340 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-04-15 20:41
As of 3.6.5, 2.7.15, and the upcoming 3.7.0. the macOS installers now build with a generic 'gcc' specified, rather than the somewhat confusing 'gcc-4.2' and we also provide a new 10.9+ 64-bit-only installer variant that is built with clang.  So I think we can finally close this issue.  Thanks all!
History
Date User Action Args
2022-04-11 14:57:50adminsetgithub: 63219
2018-04-15 20:41:24ned.deilysetstatus: open -> closed
versions: + Python 2.7, Python 3.6, Python 3.7, - Python 3.4
messages: + msg315340

resolution: fixed
stage: patch review -> resolved
2018-04-14 14:56:20ned.deilysetmessages: + msg315295
2018-04-14 14:48:24ned.deilysetpull_requests: + pull_request6168
2018-04-14 14:37:34ned.deilysetmessages: + msg315293
2018-04-14 14:25:13ned.deilysetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request6166
2013-10-25 07:54:05python-devsetmessages: + msg201220
2013-10-25 01:37:40ned.deilysetmessages: + msg201199
2013-10-25 00:05:40abarnertsetnosy: + abarnert
messages: + msg201198
2013-10-19 04:36:46python-devsetnosy: + python-dev
messages: + msg200367
2013-09-20 23:04:33rhettingersetnosy: + rhettinger
2013-09-20 22:29:58r.david.murraysetmessages: + msg198176
2013-09-20 14:36:55ronaldoussorensetnosy: + ronaldoussoren
2013-09-18 19:41:09r.david.murraysetnosy: + r.david.murray
messages: + msg198032
2013-09-18 19:05:45ned.deilysetmessages: + msg198031
2013-09-18 18:50:33skip.montanarosetmessages: + msg198030
2013-09-18 17:41:51ned.deilysetmessages: + msg198029
2013-09-18 17:26:59skip.montanarosetnosy: + skip.montanaro
messages: + msg198028
2013-09-14 20:31:13ned.deilycreate