On OSX it is possible to compile using an SDK, which is basicly a directory tree containing include files and shared library stubs. When building using an SDK (such as the 10.4u SDK) the compiler looks in the SDK subtree instead of / (that is, look for include files in $SDKROOT/usr/include instead of /usr/include, and simularly for other locations).

Python's setup.py should do the same thing when performing build-time tests, such as looking for header files or libraries.

BTW. I'm planning to work on a patch for this, the issue is mostly here to remind me that something needs to be done.

See also newly opened bpo-7713 which expands the issue to other platforms .

bpo-7713 is not the same: it asks for a way to affect the hardcoded paths in setup.py, this issue asks to honor the SDK-root by setup.py.

I've done some further research and this affects distutils in general: the compiler has methods to look for files and those need to honor the SDK root as well.

The hard part will be to describe correctly how Apple's compilers behave when -isysroot is present. Replicating that in our build environment should then be straightforward. The reason that the specification part is hard is that Apple's document don't fully describe how -isysroot works. That is, '-isysroot,/Some/SDK -I/Users/ronald/OpenSource' works and will look in /Users/ronald/OpenSource, even though '/Users' is not in the SDK directory. This means it is just prepending $UNIVERSALSDK to every filename is not a correct fix.

It's not exactly the same issue but I think it is closely related since effectively both document the need for setup.py to build with a non-default system root (or SDK) and some of the hardcoded paths should be being satisfied from the SDK. So at least parts of the more general solution should apply to OS X builds and potentially vice versa.

Yes, Apple's documentation of -isysroot, -syslibroot, and --sysroot leave a lot to be desired. My guess is that --sysroot was added independently by GNU and doesn't seem to be useful here. And, as best I can tell on 10.4 to 10.6, -syslibroot gets added automatically when -isysroot is supplied. Is that your understanding as well? I also see the man page for gcc discusses -isystem which sounds like it might be helpful if it actually works.

I agree, our usage of -isysroot seems to be correct.

What's rather annoying is that the documentation seems to claim that the sysroot value gets prepended to every search location, while that is obviously not try. My guess is that it is only prepended for built-in locations (/usr/{include,lib}, /System/Library/Frameworks, /usr/local/{include,lib} and /Library/Frameworks), but I haven't found documentation yet that confirms this.

I suppose all of the relevant setup.py build-time tests could be restructured as autoconf-style tests using gcc & friends with consistent arguments (with the build and with Distutils) so there wouldn't need to be special knowledge in setup.py or configure about what paths are "system directories". That might also solve the problems for other platforms as well. A bit messy but maybe the right solution.

The attached patch (for the trunk) fixes this issue on my machine.

The patch also fixes bpo-8444.

I wouldn't mind some review of the patch, it does affect core bits of setup.py.

As a short explanation of OSX SDKs: those are basicly directories containing header files and stub libraries with the same directory structure as a real system.

As an example, /Developer/SDKs/MacOSX10.4u.sdk contains the 10.4u SDK, that tree contains /Developer/SDKs/MacOSX10.4u.sdk/usr/include containing the header files that would have been in /usr/include on a canonical 10.4 system.

The SDK root is passed to the compiler using the '-isysroot' argument.

There's a bunch of debug stuff in that patch ("if 1", commented out lines...).
Also, the patch should not change behaviour on non-OS X platforms, which is not obvious right now.

I've cleaned up the patch and made it clearer that platforms other than OSX aren't affected by rewriting code like this:

f = os.path.join(d, "db.h")
if sys.platform == "darwin" and is_macosx_sdk_path(d):
   f = os.path.join(sysroot, d[1:], "db.h")

The new version of the patch is also more compreshensive, I've added SDK-awareness code for the sqlite and bdb extensions as well, those looked for files without using find_file (because they do more than just look at files).

I can build unix-style and framework builds with this patch, both with and without specifying SDKs. I haven't tested the results on a linux box yet.

I intent to apply this patch on sunday.

Note: the patch intentionally doesn't include an update to Misc/NEWS, I will write that bit when I actually commit.

I'll also forward port to 3.2 when committing.

Committed to the trunk in r80963.

I'm not closing this yet because the patch needs to be ported to 3.2 at least, and preferably 2.6 and 3.1 as well.

Ronald, I've reverted the patch since it broke compilation on almost every buildbot, and we're close to beta2 release.

Just to save you some time investigating, it broke the build because
of missing parentheses in setup.py:

if sys.platform == 'darwin' and dir.startswith('/System') or dir.startswith('/usr')

The attached version should fix the issue found by Stefan. I'm going to do builds on OSX as well as Linux before committing though.

I've applied a new version of the patch in r81662.

I'll be monitoring the buildbot farm to ensure that any issues that might crop up get fixed ASAP.

Ported to 3.2 in r81673.

Committed for 2.6 in r81674

And for 3.1 in r81677.

This was committed a while back, I guess I forgot to close this issue.