classification
Title: Python doesn't find Xcode 7 SDK stub libraries
Type: compile error Stage: resolved
Components: Build, Distutils, macOS Versions: Python 3.6, Python 3.5, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: ned.deily Nosy List: afoglia, brett.cannon, dstufft, eric.araujo, ned.deily, python-dev, ronaldoussoren, tdsmith
Priority: normal Keywords: patch

Created on 2015-09-16 01:03 by tdsmith, last changed 2016-02-24 14:44 by ned.deily. This issue is now closed.

Files
File name Uploaded Description Edit
xcode-stubs.diff tdsmith, 2015-09-16 01:03 Diff against 3.5.0 release review
xcode-stubs-2.7.patch tdsmith, 2015-09-16 04:06 Patch against 2.7 release branch
Messages (6)
msg250813 - (view) Author: Tim Smith (tdsmith) * Date: 2015-09-16 01:02
In Xcode 7, Apple is replacing many of the .dylibs in SDKROOT with textual stubs. [1] These files exist on disk with filenames like:

/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/lib/libz.tbd

They are short YAML documents that look like this: [2]

The same linker invocation that has always worked will continue to work with Xcode 7 (i.e. you still pass `-lz` to the linker), but this disrupts the checks that cpython's setup.py uses to determine if it can build extension modules. The dylibs physically exist on disk in /usr/lib, but since we've set -isysroot to the appropriate SDKROOT in CPPFLAGS, distutils searches for the dylibs in the sysroot path, and does not find them (since they have been replaced with .tbd stubs). Since distutils cannot find the libraries, setup.py declines to attempt to build any of the extension modules that depend on libraries in the OS X SDK, even though it would have succeeded if it had tried. Several Homebrew users have reported this while trialling Xcode 7 [3].

distutils should treat the .tbd files as a "real" library so that compiler.find_library_file succeeds and setup.py will proceed to attempt to build the extension modules.

The attached diff applies against the 3.5.0 release and allows extension modules to be built against Xcode 7 without installing the Command-Line Tools package.

If anyone is experiencing this issue, a workaround is to install the Xcode Command Line Tools package with `xcode-select --install` (which, among other things, installs headers to /usr/include), ensure the CLT is active with `xcode-select -s /Library/Developer/CommandLineTools`, and do not include `-isysroot` in CPPFLAGS when building python.

[1]: https://forums.developer.apple.com/thread/4572
[2]: https://gist.github.com/474233e561e28e1a8866
[3]: https://github.com/Homebrew/homebrew/issues/41085
msg250894 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2015-09-17 15:57
Thanks for the analysis and the patches, Tim. Now that Xcode 7 is released, we'll need to ensure we fully support the new stubs.  But, as you note, the workaround as always is to make sure that the current Command Line Tools are installed, the configuration that we test with.
msg260694 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2016-02-22 17:57
We should update the devguide to specify that the command-line tools need to be installed and either explain or point to documentation on how to install the tools.
msg260803 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-02-24 14:00
New changeset 59d797915dca by Ned Deily in branch '2.7':
Issue #25136: Support Apple Xcode 7's new textual SDK stub libraries.
https://hg.python.org/cpython/rev/59d797915dca

New changeset b0490b8af7aa by Ned Deily in branch '3.5':
Issue #25136: Support Apple Xcode 7's new textual SDK stub libraries.
https://hg.python.org/cpython/rev/b0490b8af7aa

New changeset db6c1308eb31 by Ned Deily in branch 'default':
Issue #25136: merge from 3.5
https://hg.python.org/cpython/rev/db6c1308eb31
msg260804 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-02-24 14:36
New changeset 157eb9d40bf9 by Ned Deily in branch '2.7':
Issue #25136: Add reference to 'xcode-select --install' to Mac README.
https://hg.python.org/cpython/rev/157eb9d40bf9

New changeset 61752c7ea9c7 by Ned Deily in branch '3.5':
Issue #25136: Add reference to 'xcode-select --install' to Mac README.
https://hg.python.org/cpython/rev/61752c7ea9c7

New changeset 2c4448bbed1f by Ned Deily in branch 'default':
Issue #25136: merge from 3.5
https://hg.python.org/cpython/rev/2c4448bbed1f
msg260805 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-02-24 14:44
I've committed Tim's patches with some additional comments for release in 2.7.12, 3.5,2, and 3.6.0 and have added a note about using 'xcode-select --install' to the Mac/README files.  (The Developer's Guide already documents this.)  Thanks again, Tim.
History
Date User Action Args
2016-02-24 14:44:52ned.deilysetstatus: open -> closed
versions: - Python 3.4
title: Python doesn't find Xcode 7 stub libraries -> Python doesn't find Xcode 7 SDK stub libraries
messages: + msg260805

resolution: fixed
stage: patch review -> resolved
2016-02-24 14:36:03python-devsetmessages: + msg260804
2016-02-24 14:00:18python-devsetnosy: + python-dev
messages: + msg260803
2016-02-23 19:22:19afogliasetnosy: + afoglia
2016-02-22 17:57:24brett.cannonsetnosy: + brett.cannon
messages: + msg260694
2016-02-15 08:37:47ned.deilylinkissue26265 superseder
2015-09-17 15:57:57ned.deilysetassignee: ned.deily
stage: patch review
messages: + msg250894
versions: + Python 2.7, Python 3.4, Python 3.5, Python 3.6
2015-09-16 04:06:21tdsmithsetfiles: + xcode-stubs-2.7.patch
2015-09-16 01:03:03tdsmithcreate