classification
Title: setup.py ignores SDK root on OSX
Type: compile error Stage: resolved
Components: Build, macOS Versions: Python 3.2, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: ronaldoussoren Nosy List: benjamin.peterson, ned.deily, pitrou, ronaldoussoren, skrah
Priority: normal Keywords: needs review, patch

Created on 2010-01-17 15:06 by ronaldoussoren, last changed 2010-06-27 12:56 by ronaldoussoren. This issue is now closed.

Files
File name Uploaded Description Edit
issue7724.txt ronaldoussoren, 2010-04-20 20:51
issue7724-v2.patch ronaldoussoren, 2010-05-01 08:32
issue7724-v3.patch ronaldoussoren, 2010-05-13 07:45
Messages (19)
msg97949 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2010-01-17 15:06
On OSX it is possible to compile using an SDK, which is basicly a directory tree containing include files and shared library stubs. When building using an SDK (such as the 10.4u SDK) the compiler looks in the SDK subtree instead of / (that is, look for include files in $SDKROOT/usr/include instead of /usr/include, and simularly for other locations).

Python's setup.py should do the same thing when performing build-time tests, such as looking for header files or libraries.

BTW. I'm planning to work on a patch for this, the issue is mostly here to remind me that something needs to be done.
msg97959 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2010-01-17 18:12
See also newly opened Issue7713 which expands the issue to other platforms .
msg97961 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2010-01-17 18:33
Issue7713 is not the same: it asks for a way to affect the hardcoded paths in setup.py, this issue asks to honor the SDK-root by setup.py.

I've done some further research and this affects distutils in general: the compiler has methods to look for files and those need to honor the SDK root as well.

The hard part will be to describe correctly how Apple's compilers behave when -isysroot is present. Replicating that in our build environment should then be straightforward.   The reason that the specification part is hard is that Apple's document don't fully describe how -isysroot works. That is, '-isysroot,/Some/SDK -I/Users/ronald/OpenSource' works and will look in /Users/ronald/OpenSource, even though '/Users' is not in the SDK directory. This means it is just prepending $UNIVERSALSDK to every filename is not a correct fix.
msg97962 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2010-01-17 18:59
It's not exactly the same issue but I think it is closely related since effectively both document the need for setup.py to build with a non-default system root (or SDK) and some of the hardcoded paths should be being satisfied from the SDK.  So at least parts of the more general solution should apply to OS X builds and potentially vice versa.

Yes, Apple's documentation of -isysroot, -syslibroot, and --sysroot leave a lot to be desired. My guess is that --sysroot was added independently by GNU and doesn't seem to be useful here.  And, as best I can tell on 10.4 to 10.6, -syslibroot gets added automatically when -isysroot is supplied. Is that your understanding as well?  I also see the man page for gcc discusses -isystem which sounds like it might be helpful if it actually works.
msg97964 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2010-01-17 19:04
I agree, our usage of -isysroot seems to be correct.

What's rather annoying is that the documentation seems to claim that the sysroot value gets prepended to every search location, while that is obviously not try.  My guess is that it is only prepended for built-in locations (/usr/{include,lib}, /System/Library/Frameworks, /usr/local/{include,lib} and /Library/Frameworks), but I haven't found documentation yet that confirms this.
msg97967 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2010-01-17 19:28
I suppose all of the relevant setup.py build-time tests could be restructured as autoconf-style tests using gcc & friends with consistent arguments (with the build and with Distutils) so there wouldn't need to be special knowledge in setup.py or configure about what paths are "system directories".  That might also solve the problems for other platforms as well.  A bit messy but maybe the right solution.
msg103752 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2010-04-20 20:51
The attached patch (for the trunk) fixes this issue on my machine.

The patch also fixes issue 8444.

I wouldn't mind some review of the patch, it does affect core bits of setup.py.
msg103768 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2010-04-20 21:13
As a short explanation of OSX SDKs: those are basicly directories containing header files and stub libraries with the same directory structure as a real system. 

As an example, /Developer/SDKs/MacOSX10.4u.sdk contains the 10.4u SDK, that tree contains /Developer/SDKs/MacOSX10.4u.sdk/usr/include containing the header files that would have been in /usr/include on a canonical 10.4 system.

The SDK root is passed to the compiler using the '-isysroot' argument.
msg103769 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-04-20 21:22
There's a bunch of debug stuff in that patch ("if 1", commented out lines...).
Also, the patch should not change behaviour on non-OS X platforms, which is not obvious right now.
msg104697 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2010-05-01 08:32
I've cleaned up the patch and made it clearer that platforms other than OSX aren't affected by rewriting code like this:

f = os.path.join(d, "db.h")
if sys.platform == "darwin" and is_macosx_sdk_path(d):
   f = os.path.join(sysroot, d[1:], "db.h")

The new version of the patch is also more compreshensive, I've added SDK-awareness code for the sqlite and bdb extensions as well, those looked for files without using find_file (because they do more than just look at files).

I can build unix-style and framework builds with this patch, both with and without specifying SDKs. I haven't tested the results on a linux box yet.

I intent to apply this patch on sunday.

Note: the patch intentionally doesn't include an update to Misc/NEWS, I will write that bit when I actually commit. 

I'll also forward port to 3.2 when committing.
msg105269 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2010-05-08 08:45
Committed to the trunk in r80963.

I'm not closing this yet because the patch needs to be ported to 3.2 at least, and preferably 2.6 and 3.1 as well.
msg105298 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-05-08 15:25
Ronald, I've reverted the patch since it broke compilation on almost every buildbot, and we're close to beta2 release.
msg105303 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2010-05-08 16:02
Just to save you some time investigating, it broke the build because
of missing parentheses in setup.py:

if sys.platform == 'darwin' and dir.startswith('/System') or dir.startswith('/usr')
msg105628 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2010-05-13 07:45
The attached version should fix the issue found by Stefan. I'm going to do builds on OSX as well as Linux before committing though.
msg106935 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2010-06-03 09:48
I've applied a new version of the patch in r81662.

I'll be monitoring the buildbot farm to ensure that any issues that might crop up get fixed ASAP.
msg106955 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2010-06-03 14:42
Ported to 3.2 in r81673.
msg106957 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2010-06-03 15:00
Committed for 2.6 in r81674
msg106963 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2010-06-03 16:21
And for 3.1 in r81677.
msg108787 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2010-06-27 12:56
This was committed a while back, I guess I forgot to close this issue.
History
Date User Action Args
2010-06-27 12:56:22ronaldoussorensetstatus: open -> closed
resolution: fixed
messages: + msg108787

stage: commit review -> resolved
2010-06-03 16:21:42ronaldoussorensetmessages: + msg106963
2010-06-03 15:00:28ronaldoussorensetmessages: + msg106957
2010-06-03 14:42:47ronaldoussorensetmessages: + msg106955
2010-06-03 09:48:44ronaldoussorensetmessages: + msg106935
2010-05-13 07:45:41ronaldoussorensetfiles: + issue7724-v3.patch

messages: + msg105628
2010-05-08 16:02:26skrahsetnosy: + skrah
messages: + msg105303
2010-05-08 15:25:09pitrousetnosy: + benjamin.peterson

messages: + msg105298
stage: resolved -> commit review
2010-05-08 08:45:39ronaldoussorensetstage: patch review -> resolved
2010-05-08 08:45:23ronaldoussorensetmessages: + msg105269
2010-05-01 08:32:48ronaldoussorensetfiles: + issue7724-v2.patch
keywords: + patch
messages: + msg104697
2010-04-20 21:22:17pitrousetnosy: + pitrou
messages: + msg103769
2010-04-20 21:13:55ronaldoussorensetmessages: + msg103768
2010-04-20 20:51:34ronaldoussorensetkeywords: + needs review
files: + issue7724.txt
messages: + msg103752

stage: needs patch -> patch review
2010-01-17 19:28:19ned.deilysetmessages: + msg97967
2010-01-17 19:04:54ronaldoussorensetmessages: + msg97964
2010-01-17 18:59:37ned.deilysetmessages: + msg97962
2010-01-17 18:33:50ronaldoussorensetmessages: + msg97961
2010-01-17 18:12:42ned.deilysetnosy: + ned.deily
messages: + msg97959
2010-01-17 15:06:12ronaldoussorencreate