This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Extension module builds fail with Xcode 4.3 on OS X 10.7 due to SDK move
Type: Stage: resolved
Components: Distutils, macOS Versions: Python 3.2, Python 3.3, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: ned.deily Nosy List: Laurent.Duchesne, Matthew.Scott, eric.araujo, lukasz.langa, ned.deily, python-dev, ronaldoussoren, s7v7nislands, samueljohn, tarek
Priority: Keywords:

Created on 2012-04-04 22:01 by ned.deily, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Messages (10)
msg157511 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2012-04-04 22:01
With the release of Xcode 4.3 for OS X 10.7, Apple has moved the location of the OS X SDKs from their long-time path of /Developer to within the new Xcode.app bundle itself.  This breaks the building of extension modules with any of Distutils or packaging/Distutils2 when Python was built as as universal build with an SDK in the former location, as is the case with the python.org installers.

A workaround is to either leave the old /Developer directory in place when upgrading to Xcode 4.3 or to install a symlink to the new location:

sudo ln -s /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer /Developer
msg158211 - (view) Author: s7v7nislands (s7v7nislands) Date: 2012-04-13 17:02
maybe you can use xcode-select to set the correct path

xcode-select -print-path
/Applications/Xcode.app/Contents/Developer

Usage: xcode-select -print-path
   or: xcode-select -switch <xcode_folder_path>

   -switch <xcode_folder_path>     Sets the path for the current Xcode 


for detail, you can man xcode-select
msg161941 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2012-05-30 09:24
s7v7nislands: you cannot use Xcode-select to fix this, xcode-select is used to switch between 2 or more installed versions of Xcode.
msg161942 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2012-05-30 09:52
The issue is more annoying than the change of location of the SDK, the path to the compiler has also changed unless users manually install the Unix command-line tools, either using a button in the GUI or by installing a separate DMG.

There seem to be two options to thoroughly fixing this:

1) Tell users they must install the unix tools (cltools*.dmg)

   I'm not 100% sure at this time that this would work, from what I've
   learned from inspecting the installer packages this seems to imply
   "-isysroot /" (that is, no /Developer/SDKS/....)

   A small change to distutils and packaging would switch distutils from
   using "-isysroot /Developer/SDKs/..." to "-isysroot /" when it 
   notices that the system root doesn't exist. This is similar to 
   existing hacks for changing a number of compiler flags on OSX 10.3
   when using the 32-bit universal build.

2) Tweak the configure script, distutils and packaging to look for the
   SDK and compiler in the new location if they cannot be found in the
   default location.

   This would be the most user-friendly as the build would "just work",
   but is more work on our end. I have a 10.8 VM with and Xcode install
   where I haven't installed the unix tools to experiment with this. 

   This is definitely more work than running configure with a
   different SDK and custom values for CC and CXX, we also have to 
   convince configure that it isn't performing a cross-build (it seems
   that some compile checks are performed before we add -isysroot to
   CFLAGS and because I don't have the unix tools installed this results
   in a compile failure early in configure)

   This option might be of limited use though: when you want to build
   an extension that uses a C library that isn't shipped with the OS
   (for example PIL with libjpeg) the build of the python extension
   would "just work", but you'd still have to manually specify the
   compiler and system root for the build of libjpeg.
msg161956 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2012-05-30 14:12
I've been working on this and it does need to be thoroughly fixed.  There are two different aspects to it: (1) being able to build Python using any of the supported development environment options; and (2) support in Distutils and packaging to build extension modules for any of the supported development options.  With the release of the Command Line Tools for 10.7 and beyond (and with the dangling mess of 10.6 with both Xcode 3.2.6 and Xcode 4.2 installations out there), we need to be able to support multiple options for users and builders of universal builds.  Apple does supply tools to deal with development environments installed in arbitrary locations (primarily, xcode-select and xcrun) but there are some gaps in them for supporting command-line-based builds, like a convenient way to find the absolute path to the SDK.  I'm working on patches for all of this right now.  I'm hopeful we will end up with a major increase in usability to users of universal builds.
msg161962 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2012-05-30 15:07
What I'd prefer to look for the compiler:

* in distutils: if $CC is an absolute path and exists, use that

* look for clang on $PATH,  use it if found
   (default configure looks for GCC in preference of other compilers,
    but with Apple's toolchain it is better to use clang)

* look for gcc on $PATH, use it if found
   (needed to use the Fink compiler when Fink is on $PATH, alternative
    for 3.3 is to document that configure will try to pick and Apple
    compiler)

* use $(xcodebuild -find clang)

To look for the SDK:

* in distutils: if -isysroot is set, is not '/' and exists, use that

* in distutils: extract SDK version from -isysroot string (if set),
  then use $(xcode-select) to find the root of the installation and
  construct a path relative to that. If the requested SDK version exists,
  use that

* if osx release < 10.5 and building universal: 

  - use /Developer/SDKs/MacOSX10.4u.sdk
    (needed for universal builds on early releases of OSX 10.4)

* if /usr/include/stdint.h exists: use -isysroot /
  (older Xcode, or Xcode >=4.3 with Command Line Tools installed)

* use $(xcodebuild -showsdks) to find list of SDKs and use most recent
  one.


I haven't tried to capture this in code yet, and haven't tested the procedure on earlier releases (or even a 10.7 system with Xcode 4.3 without unix tools), but this should work.
msg163961 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2012-06-25 12:29
An initial set of changes to support Xcode 4.3+ have been applied for  Python 3.3.0b1 have been applied via Issue13590.  As noted there, a back port of some subset is needed for 3.2 and 2.7.  I'll update this after 3.3.0b1 has been released.
msg200366 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013-10-19 04:36
New changeset 73532f33fbb3 by Ned Deily in branch 'default':
Issue #14499: Fix several problems with OS X universal build support:
http://hg.python.org/cpython/rev/73532f33fbb3
msg302984 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2017-09-25 22:56
Not critical anymore. Probably safe to close?
msg315347 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-04-16 01:03
Update: as of 3.6.5 and 2.7.15 and in the upcoming 3.7.0 release, macSO framework builds with --enable-universalsdk=/ no longer add -isysroot=/ to compiler and linker calls.  They are not necessary and made it more difficult to attempt to override.  Although we still do not officially support building on newer macOS systems for older systems (due to lack of "weak link" testing support and issues with setup.py and configure), it should now be possible to more easily use the Apple build tool chain features documented for xcrun, like the DEVELOPER_RUN and SDKROOT environment variables (see 'man xcrun').  This includes current python.org macOS installer builds.  So I think we can finally close this.
History
Date User Action Args
2022-04-11 14:57:28adminsetgithub: 58704
2018-04-16 01:03:09ned.deilysetstatus: open -> closed
resolution: fixed
messages: + msg315347

stage: needs patch -> resolved
2017-09-25 22:56:57lukasz.langasetpriority: critical ->
nosy: + lukasz.langa
messages: + msg302984

2013-10-19 04:36:46python-devsetnosy: + python-dev
messages: + msg200366
2012-06-25 12:29:22ned.deilysetmessages: + msg163961
2012-06-11 15:51:04samueljohnsetnosy: + samueljohn
2012-06-08 12:06:33Laurent.Duchesnesetnosy: + Laurent.Duchesne
2012-06-01 02:41:19eric.araujosetnosy: + eric.araujo
2012-05-30 15:07:14ronaldoussorensetmessages: + msg161962
2012-05-30 14:12:47ned.deilysetmessages: + msg161956
2012-05-30 09:52:13ronaldoussorensetmessages: + msg161942
2012-05-30 09:24:45ronaldoussorensetnosy: + ronaldoussoren
messages: + msg161941
2012-04-13 17:02:31s7v7nislandssetnosy: + s7v7nislands
messages: + msg158211
2012-04-04 22:06:15Matthew.Scottsetnosy: + Matthew.Scott
2012-04-04 22:01:40ned.deilycreate