Install the Python-2.7.2 mac installer for Lion on Lion.

Then attempt "easy_install -U psutil".  I get:

za-dc-dev/bin/easy_install -U psutil
install_dir /Users/rich/projects/za-packages/za-dependency-checker/za-dc-dev/lib/python2.7/site-packages/
Searching for psutil
Reading http://pypi.python.org/simple/psutil/
Reading http://code.google.com/p/psutil/
Best match: psutil 0.4.0
Downloading http://psutil.googlecode.com/files/psutil-0.4.0.tar.gz
Processing psutil-0.4.0.tar.gz
Running psutil-0.4.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-7euim1/psutil-0.4.0/egg-dist-tmp-QRoCe6
unable to execute gcc-4.2: No such file or directory
error: Setup script exited with error: command 'gcc-4.2' failed with exit status 1
make: *** [za-dc-dev/lib/python2.7/site-packages/psutil-1.1.2-py2.7.egg] Error 1


There is no binary named "gcc-4.2" on my system.  I'm running the latest Xcode, (4.2.1).  And gcc in my PATH is a 4.2 binary:

rich@fuji-land.noir.com> type gcc
gcc is hashed (/usr/bin/gcc)
rich@fuji-land.noir.com> gcc --version
i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.1.00)
Copyright (C) 2007 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


I see no reference to "gcc-4.2" in psutils source nor in distutils.  From this I guess that the python configuration is looking for the same compiler that was used to produce the package, (presumably on osx-10.6).

Other developers tell me that they have a "gcc-4.2" on osx-10.6.  And indeed, downloading and building python-2.7.2 from source results in a python that can download and compile psutil.

When building and installing C extension modules, distutils attempts to use the same compiler and some compiler options as Python itself was built with.  For the current (3.2.2 and 2.7.2) 64-bit/32-bit python installers, the standard Apple-supplied gcc-4.2 in Xcode 3.2.x on OS X 10.6 was used.  As of Xcode 4.2, now standard for OS X 10.7 and optional for OS X 10.6, Apple no longer ships gcc-4.2 in Xcode, in favor of clang and the transitional llvm-gcc.  Moving the entire Python build to another compiler is a major undertaking, requiring careful testing, which is underway.  A compiler change may well be deemed too risky for a bug-fix release.

In any case, we can no longer assume that the same build compiler will be available on most user systems.  While it is possible for the user to manually override the distutils defaults by setting the CC and LDSHARED environment variables properly, setting the latter is tedious.  For example, to manually override to clang, the following is currently needed:

  CC=clang
  LDSHARED='clang -bundle -undefined dynamic_lookup \
    -arch i386 -arch x86_64 -isysroot /Developer/SDKs/MacOSX10.6.sdk -g'
  python setup.py ...

Attached is a patch that attempts to "do the right thing" for OS X.  First, if the default build compiler is gcc-4.2 and the compiler has not been explicitly overridden by defining CC, distutils will check for the presence of gcc-4.2 on the default path.  If gcc-4.2 is not found and if clang is found, it will automatically substitute clang as the build compiler.  Second, if CC is used to override the compiler selection but LDSHARED is not defined, distutils will substitute the CC value into the default LDSHARED value.  This allows simple one-line compiler overrides, like:

  CC=llvm-gcc python setup.py ...

To minimize the risk of unintended side effects, these changes would apply to OS X only.

I propose applying this patch to 3.2 (for 3.2.3) and 2.7 (for 2.7.3) as well as provisionally to default for 3.3; a second patch will be needed with similar changes to packaging.  After the evaluation of compiler alternatives is complete and we decide what to do for 3.3, this approach might change.

Would you think it acceptable to judge that the problem is Apple’s and that we could do only a doc change with the not-so-hard envvar override?

I understand that you’d like to fix the problem for all OS X users in one go, but the new behavior seems a bit too magical for me.

I think a better solution that declaring it to be apple's bug would be to release one binary for pre-10.7, (or maybe 10.6 with the current xcode), and a different binary for post-10.7.

This isn't an apple "bug" in the sense that there's anything wrong nor in the sense that they would ever "fix" it.  It's simply a difference between xcode versions.  So the choices would seem to be a) code around it or b) release different binaries.

I'm ok with either solution.  I'm not sure what would be best as I'm not sure I know all of the concerns involved.

Éric, no, the problem is not Apple's in the sense that we enforce the use of the build compiler.  Without a fix along this line would mean that essentially *every* user of python.org Pythons on the latest releases of OS X would have to ensure that the environment variable override is in place for every Distutils install of an extension module.  That strikes me as unacceptable.  Distutils already does equally magical things.  I think this is the best solution at the moment.  Ronald, what's your opinion?

> the problem is not Apple's in the sense that we enforce the use of the build compiler.
Well, yes, and this is a known limitation, so we could argue that Apple broke distutils.  But...

> Without a fix along this line would mean that essentially *every* user of python.org Pythons on
> the latest releases of OS X would have to ensure that the environment variable override is in
> place for every Distutils install of an extension module.  That strikes me as unacceptable.
> Distutils already does equally magical things.
Given that we did similar changes to support Debian multiarch instead of having people use environment variables, I change my position and support your patch.  (Comments on Rietveld.)

distutils would not only have to check for gcc-4.2 vs. clang, but also needs to strip "-arch ppc" from the compiler flags when it is present and it cannot use gcc-4.2.   Otherwise you cannot build extensions for the 32-bit python installer on OSX 10.7 with Xcode 4.2.

It is sadly enough not possible to create a binary installer that creates a working installation including building of extensions on all supported versions of OSX without adding special-case code to distutils/packaging.

Note that there already is some special-case code to support the universal binary builds on OSX 10.3.9 (which does not have a compiler that can build universal binaries).


W.r.t. the patch: is it really necessary to use a subshell to look for the compiler? I'd either walk os.environ['PATH'] myself or use xcodebuild to locate binaries.

New changeset 29507a2acdb5 by Ned Deily in branch '2.7':
Issue #13590: On OS X 10.7 and 10.6 with Xcode 4.2, building
http://hg.python.org/cpython/rev/29507a2acdb5

New changeset 5c784b0f263d by Ned Deily in branch '3.2':
Issue #13590: On OS X 10.7 and 10.6 with Xcode 4.2, building
http://hg.python.org/cpython/rev/5c784b0f263d

New changeset 35bd40b16a91 by Ned Deily in branch 'default':
Issue #13590: merge
http://hg.python.org/cpython/rev/35bd40b16a91

Éric: I've replied to your review comments in Rietveld.  Thanks.

Ronald: "distutils would not only have to check for gcc-4.2 vs. clang, but also needs to strip "-arch ppc" from the compiler flags when it is present and it cannot use gcc-4.2.   Otherwise you cannot build extensions for the 32-bit python installer on OSX 10.7 with Xcode 4.2."

The 32-bit-only installers uses gcc-4.0, not -4.2 and it uses the 10.4u SDK, neither of which is available with Xcode 4.*.  The patch does not currently attempt to support building extensions for the 32-bit-only installers.  Besides stripping arch ppc, it would also have to substitute an available SDK (10.6 is the oldest SDK shipping with Xcode 4.2).  CFLAGS and LDFLAGS would also have to be edited to remove ppc and the use of 10.3 as a deployment target results in warning messages:
  #warning Building for Intel with Mac OS X Deployment Target < 10.4 is invalid
even though for simple cases it seems to work.

But I think trying to automatically support the 32-bit-only configuration is too intrusive and unnecessary for most users.  There are disadvantages to trying to use the old 32-bit-only configuration on newer systems, such as using the older, more broken, or less feature-rich system APIs.  One reasonable use case I can think of is for app developers who want to distribute Python-based apps on OS X that run on a range of systems, say 10.4 to 10.7.  For those presumably more sophisticated users, documenting the unsupported environment variable settings to override when attempting to build with Xcode 4 should be sufficient.  (The safer options of building with Xcode 3 on 10.6 (or earlier) or building their own custom Pythons remain, of course.)

"W.r.t. the patch: is it really necessary to use a subshell to look for the compiler? I'd either walk os.environ['PATH'] myself or use xcodebuild to locate binaries."

I suppose that could be done.  I've tried to minimize the performance impact by only performing the check the first time it is needed and caching the result so it's only done once per Python invocation.  I'm concerned about the added complexity of getting the PATH parsing and semantics right; I don't know of any Python code in the standard library that does this.  My immediate concern is for the imminent code cutoffs for Python 3.2.3 and 2.7.3 so I've committed the patch, updated for other review comments, for them.

I'm deferring looking at doing a PATH walk for the 3.3 version of the code which is needed along with corresponding changes to packaging.  There also should be some OS X installer README and/or other documentation of this and other 10.7 and 10.6 issues.  I'm considering how to handle that.

Ping?

In progress; fixes for this and the other Xcode4-related issues will be in for 3.3.0b1.

The attached patch (issue1390-version1.txt) is not tested very well at the moment because I'm running into another problem when building on OSX 10.5, but should improve the build situation

With this patch:

* Configure will automatically pick clang when gcc is llvm-gcc
  (which miscompiles python3.3)
* Configure will use the compiler embedded in Xcode.app when there is
  no compiler in /usr/bin (e.g. when using a clean install of Xcode 4.3)
* Configure picks the most recent available SDK when using
  --with-universalsdk (cannot use '/' as the -isysroot because 
  Xcode4.3 won't install headers in /usr/include unless you install
  the unix tools)
* Distutils uses the configured compiler when available, and uses the
  same algorithm as configure when the configured compiler cannot be
  found
* Distutils will remove '-arch ppc' from the compiler flags when that
  architecture is not supported (such as with recent Xcode 4 releases)

I have build the tree on 10.7 and 10.6 with working tests. I'm now trying to build on 10.5 to generate the 32-bit installer, but ran into problems on my 10.5 machine: python.exe claims it cannot find the std. library. This is likely the known issue of Apple's GCC miscompiling python, but I haven't excluded other possibilities yet.

_binary_on_path looks unnecessary: distutils.util has a function to do that.

distutils.spawn.find_executable*

Thanks, I'll add it to the patch when I have a working build again.

Looks like I get a failure because "pybuilddir.txt" is not created, and that's because "./python.exe setup.py build" crashes when distutils.sysconfig tries to import subprocess (which tries to import time, which cannot be found because it is not build yet.

I'm switching to os.popen in my patch, that should be good enough for that code.

With version 2 of the patch I can bootstrap on OSX 10.5 as well (removes the dependency on subprocess, falls back on os.system instead)

This also removes _binary_on_path as suggested by Éric.

The patch needs more testing: I haven't tried using the binary installers on systems with a different compiler configuration that (that is, 32-bit installer on a 10.7 system, 64-bit installer on a system without "/Developer").

It's currently about 23:30 local time, and I won't be able to continue working on this patch until tomorrow (Saterday) night (19:00 or later CEST)

Thanks, Ronald.  Version 3 addresses various issues, including adding a search of $PATH for clang since xcrun is not useful in the case where the user has installed a standalone Command Line Tools package or has installed a Command Line Tools component from within Xcode but hasn't run xcode-select.  Another problem: the SDK path is likely going to be incorrect in the common case of an installer build on 10.5 or 10.6 but run on 10.7 or later.  It's tricky to get all the edge cases correct for that.  For now, the solution is to delete -sdkroot parameters from the default CFLAGS and friends if the SDK path is invalid; that assumes the Command Line Tools component/package has been installed.  If necessary, the user can override via env variables.  Also, the compiler validity checks are now bypassed if the user has overridden CC.  I'll plan to commit later today for 3.3.0b1 along with some README updates.

New changeset 688d48e434e4 by Ned Deily in branch 'default':
Issue #13590: Improve support for OS X Xcode 4:
http://hg.python.org/cpython/rev/688d48e434e4

New changeset 796d9fc28e95 by Ned Deily in branch 'default':
Issue #13590: Improve support for OS X Xcode 4:
http://hg.python.org/cpython/rev/796d9fc28e95

[wrong issue number in commit]
New changeset 381c7f897207 by Ned Deily in branch 'default':
Issue #13590: Improve support for OS X Xcode 4:
http://hg.python.org/cpython/rev/381c7f897207

I think this is now in decent shape for 3.3.0b1.
Remaining work includes:
1. A backport of a minimal subset to 2.7 and 3.2 so they are usable with
   Xcode 4 as well
2. Further updates to the README and other documentation
3. Support in distutils2 and post-3.3 packaging
4. More robust support in configure
Some of these items may get pushed to separate issues.

This may have caused failures on non-darwin unix OSes: #15184

New changeset 6a1e983647bd by Ned Deily in branch 'default':
Issue #13590: Improve support for OS X Xcode 4:
http://hg.python.org/cpython/rev/6a1e983647bd

Ned: do you have time to backport this to 3.2 and 2.7?

Also, the code might not be entirely correct for all machines: on my machine 'gcc-4.2' still exists in /usr/bin but doesn't work because the actual compiler (/usr/bin/i686-apple-darwin11-gcc-4.2.1) is no longer present. This is with Xcode 4.5.2.

I haven't checked yet if that actually causes a problem with the binary installer for Python 3.3.

> Ned: do you have time to backport this to 3.2 and 2.7?

Yes, I'll be taking care of it shortly.

> Also, the code might not be entirely correct for all machines: on my
> machine 'gcc-4.2' still exists in /usr/bin but doesn't work because the
> actual compiler (/usr/bin/i686-apple-darwin11-gcc-4.2.1) is no longer
> present. This is with Xcode 4.5.2.

AFAIK, the file /usr/bin/gcc-4.2 is not installed with any version of Xcode 4, only Xcode 3.x.  It should have been removed by the command line tools uninstaller that used to be provided in Xcode, prior to the consolidation under Xcode.app in Xcode 4.2(?):

   sudo /Developer/Library/uninstall-devtools --mode=unixdev

Presumably that functionality should now exist in Xcode.app itself.  FWIW, I've not seen /usr/bin/gcc-4.2 left behind with any Xcode 4 installation I've done.

Attached are the back ports to 2.7.x and 3.2.x of the Xcode 4 support changes as released in 3.3.0.  I've built and tested both with various configurations on a variety of systems, both Intel and PPC, and various OS X versions (10.4, 10.5, 10.6, 10.7, and 10.8), including all of the standard installer configurations.  I also tested with just the standalone Command Line Tools package (for 10.7 and 10.8).  With these back ports, extension module builds should once again work out of the box on all supported systems with their most recent versions of Xcode or Command Line Tools.  Unless there are objections, I'll commit these in the next day or two for 2.7.4 and 3.2.4.

New changeset 1d061f83a6bf by Ned Deily in branch '2.7':
Issue #13590: OS X Xcode 4 - improve support for universal extension modules
http://hg.python.org/cpython/rev/1d061f83a6bf

New changeset 502b139b3b19 by Ned Deily in branch '3.2':
Issue #13590: OS X Xcode 4 - improve support for universal extension modules
http://hg.python.org/cpython/rev/502b139b3b19

Backports committed for 2.7 (2.7.4) and 3.2 (3.2.4).

Note that, although the uploaded review patches also included backported changes to ./configure to improve defaults for building Python itself under Xcode 4, I decided not to commit them here as they might introduce a small incompatibility and are more directly associated with Issue11485.