classification
Title: Distutils is reporting OSX 10.6 w/ XCode 4 as "universal"
Type: behavior Stage: needs patch
Components: Distutils, Macintosh Versions: Python 3.3, Python 3.1, Python 3.2, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: ronaldoussoren Nosy List: James.Tatum, eric.araujo, jlindenbaum, ned.deily, ronaldoussoren, santoso.wijaya, sorin, tarek
Priority: normal Keywords:

Created on 2011-03-21 16:43 by jlindenbaum, last changed 2011-05-30 06:08 by sorin.

Messages (16)
msg131673 - (view) Author: Johannes Lindenbaum (jlindenbaum) Date: 2011-03-21 16:43
This bug is a crosspost from /setuptools.
(http://bugs.python.org/setuptools/issue122)

Summary:
OSX 10.6.x with Xcode 4 installed. Xcode 4 removes the PPC assembler from GCC. I attempted to install Fabric-1.0.0 which depended on pycrypto, during the pycrypto build there was a failure regarding "Broken Pipe".

I downloaded the package separately and attempted to do the install there. Same error.

I was pointed in the right direction that the broken pipe error was coming from the '-arch ppc' flag during build.


Expected Result
10.6.x should not report itself as "universal" when it's no longer possible to compile for PPC.


Actual Result
Attempted PPC compilation fails.


More detail:
http://superuser.com/questions/259278/python-2-6-1-pycrypto-2-3-pypi-package-broken-pipe-during-build/260106#260106

Let me know if you require more information.

Kind Regards,
Johannes
msg131708 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-03-21 23:28
Thanks for the report.  Can you try to reproduce the bug with a distutils setup script to confirm the bug and get an error log?  You may have to apply the patch from #11599 to get a complete error message.  (Also, superuser rights are not required for build, so let’s remove that too.)

I only know the basics about C compilation, and I only know that Mac OS is a strange and peculiar system, but let’s try to work on this bug :)  The distutils.unixccompiler module already has code to work around things in various Mac OS versions, so if there is an easy way to detect “when it’s no longer possible to compile for PPC”, there is hope.
msg131720 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2011-03-22 02:11
The release of Xcode 4 for Mac OS X 10.6 has complicated things for Python builds.  We need to sort out what the problems are and possible solutions for them.  I suggest this issue be reassigned to Ronald and me.

In the meantime, the workaround is to continue to use Xcode 3 (which is still fully supported) on OS X 10.6.  If you feel you absolutely need to have Xcode 4 installed *and* you still need to build C extension modules for the currently available Pythons for OS X (i.e. the python.org installers or even the Apple-supplied Pythons in 10.60), there are unofficial suggestions out on the web on how one can have Xcode 3 and 4 installed side-by-side, for example:
   http://hints.macworld.com/article.php?story=20110318050811544
(Note, I have not tried this and am not recommending it.)
msg131724 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2011-03-22 07:15
Issue11485 is related to this.

As Ned noted Ned and I will look into this.
msg131747 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-03-22 13:47
Thanks for taking care of this.  I’ll be here to review distutils-related changes, if any are needed.
msg131749 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2011-03-22 14:03
I haven't put much thought in a solution yet, but at this point I'd go for three changes:

1) Give a clear warning when python was configured for i386/ppc 
   and Xcode4 is installed (instead of giving a vague compiler crash 
   due to crapping out on ppc code)

2) Add an option that forces distutils to compile for i386 only

   This should not be the default because that would result in broken
   packages for users that want to redistribute binaries to ppc 
   machines.

3) The i386/ppc build should use the system headers instead of the
   10.4u SDK (change in configure.in), with a fallback to the 10.4u
   SDK when building extensions on a 10.4 machine (because the 
   "-isysroot /" cannot build universal binaries on all such machines)

BTW. distutils does not have code to work around OSX issues, it has code
to make use of OSX specific compilation features (such as having universal binaries and SDKs). ;-) ;-)
msg132071 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2011-03-25 06:17
A similar issue on StackOverflow reminded me that Distutils has support for the ARCHFLAGS environment variable on Mac OS X which you can use when building a C extension module to override the ARCH values that Python was built with.  I don't have Xcode 4 installed yet to verify this but you may be able to work around the problem by something like:

ARCHFLAGS='-arch i386 -arch x86_64' /usr/bin/python2.6 setup.py install

when using the Apple-supplied Python 2.6 or:

ARCHFLAGS='-arch i386' /usr/local/bin/pythonx.x setup.py install

when using a 32-bit-only python.org Python.
msg137138 - (view) Author: sorin (sorin) Date: 2011-05-28 15:58
I think that distutils must me modified to auto detect supported architectures and use them, this without requiring you to hack ARCHFLAGS. 

The number of developers that need to build PPC binaries is at least one order of magnitude lower than the number of users in need of building/installing python packages.
msg137159 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2011-05-28 22:38
All that said, the best solution to the problem is to use an appropriately configured Python for the task at hand.  If you want to build Python executables that are compatible with older OS X versions and Mac hardware (in particular, all machines capable of running OS X 10.4, 10.5, and 10.6), then you should be using something like the traditional python.org 32-bit-only installer configuration (i386/ppc, deployment target=10.3). And to build that Python or extension modules for it on OS X 10.6, you need to have Xcode 3 installed.  End of story.

However, if you don't need that kind of backward compatibility - either you're just running on your own 10.6 (or later) system or are only targeting 10.6+ systems, then you should be using something like the python.org 64-/32- installer configuration (x86_64/i386, deployment target=10.6).  For that you should be able to build Python or extension modules with either Xcode 4 or Xcode 3.

If you are targeting for 10.5 Intel-only and above and want to use Xcode 4, you would need a Python with a slightly different configuration (i.e. with a 10.5 deployment target); we don't currently supply an installer that meets that specialized (and presumably rapidly diminishing) need for x86_64 on 10.5 but it isn't that difficult to build yourself if you really need it.

So, if you choose the right binary installer for the task, there *shouldn't* be any issues with using Xcode 4.  If you need to build a Python from source, make sure you do not use the default ./configure values.  In particular, explicitly set --enable-universalsdk, --with-universal-archs, and MACOSX_DEPLOYMENT_TARGET.  For example,

  ./configure \
      --enable-universalsdk=/Developer/SDKs/MacOSX10.6.sdk \
      --with-universal-archs=intel \
      --enable-framework \
      MACOSX_DEPLOYMENT_TARGET=10.6

and any other options you might want like:
     --prefix=/path/to/bin
     --enable-framework=/path/to/framework
     --with-framework-name=whatever
         #which also influences the /Applications directory name

And, yes, longer term, we should do provide more appropriate defaults for these newer environments.  As we don't have a lot of experience yet with Xcode 4 and 10.7 has not been released yet, feedback is appreciated.
msg137168 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2011-05-29 07:06
I don't agree that distutils should autodetect architectures, that can result in silent behavior changes when you built extensions and assume ppc support will get compiled in.

We should probably start to mention the intel only builds more prominently though, that's the one that will be useful to most people at this time.
msg137172 - (view) Author: sorin (sorin) Date: 2011-05-29 08:14
Let's try to find a good compromise regarding this. I hope the we all agree that we should not put our personal needs or preferences on the first place, it's important to improve the overall experience for most users.

First, we do not want to fail to build and install a package if you are on specific case (xcode3, xcode4, ...).

I think that most people do not have `ARCHFLAGS` env var defined and they should not even care about it. Compilers are smart enough to pick the right architecture for them (xcode3 will build ppc/i386/x64 and xcode4 i386/x64, and who known maybe next version will include armX).

As for package maintainers and they build machines, they are more likely to be already aware about what they want to distribute. For this reason I consider that they can hack the build environment to meet their distribution needs (whatever env vars they want, blend xcode4 with xcode3,...). 

Still as stated in a previous comment, we do not want publish Universal binaries that will miss PPC architecture, in order to prevent this distutils should throw an warning if ppc architecture is not supported (xcode4).

Also if the ARCHFLAGS is defined and includes PPC, the build is supposed to fail.

Some other considerations: think that "some" people may want to install packages to their OS provided Python and in this case the python that comes with 10.6 doesn't have a PPC arch.
msg137177 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2011-05-29 10:39
"Compilers are smart enough to pick the right architecture for them (xcode3 will build ppc/i386/x64 and xcode4 i386/x64"

Actually, that's not the case for Python builds.  The architecture selection is based on what the builder of the Python interpreter specified on the ./configure command and/or the defaults programmed into ./configure which look at other settings like deployment target. For extension module builds, Distutils attempts to use compile settings that are compatible with those the interpreter was built with, allowing some values to be influenced by environment variable settings.  In either case, the compiler has no say in the selection.

"Some other considerations: think that "some" people may want to install packages to their OS provided Python and in this case the python that comes with 10.6 doesn't have a PPC arch."

Actually, that's also not the case.  The Apple-supplied Pythons in OS X 10.6 do have a PPC arch; try "file /usr/bin/python2.6" in a shell and you'll see there are three archs.  It will even run on 10.6 in PPC emulation under Rosetta (try "arch -ppc /usr/bin/python2.6"),

But I sense a misconception here.  The python.org project has no say about what Apple does or doesn't do with regard to the system Pythons they supply in OS X. Python is an open-source project and many distributors (like Apple) build and release their own versions of Python. The PSF python.org project also builds and releases our own Windows and OS X binary installers.  But what is shipped with OS X is up to Apple.  For 10.6 the system python2.6 has a number of Apple-developed changes and add-ons.  The fact that Xcode 4 has complicated extension module builds with the system Python 2.6 is an issue to be taken up with Apple; there's nothing we here can do about that other than to suggest some workarounds.  And based on past experience, with the imminent release of 10.7, it's highly unlikely Apple will be making any non-security related changes to their Pythons in 10.6.  We'll have to see what 10.7 brings.  For the system Pythons on 10.6, the choice seems clear: either stick with Xcode 3 or resort to workarounds.

What we can do here is try to make things better for users of the python.org source or binary installers for currently supported versions of Python.  Documenting the tradeoffs of our current installer options is one step towards that.  Hope that makes things clearer!
msg137179 - (view) Author: sorin (sorin) Date: 2011-05-29 10:50
You are right about Python from 10.6, it has PPC, but 10.7 doesn't.

bash-3.2$ file /Volumes/107/usr/bin/python2.6
/Volumes/107/usr/bin/python2.6: Mach-O universal binary with 2 architectures
/Volumes/107/usr/bin/python2.6 (for architecture x86_64):	Mach-O 64-bit executable x86_64
/Volumes/107/usr/bin/python2.6 (for architecture i386):	Mach-O executable i386
msg137180 - (view) Author: sorin (sorin) Date: 2011-05-29 10:52
And in addition to 2.5 and 2.6, 10.7 includes 2.7.
msg137244 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2011-05-30 05:36
What 10.7? ;-) Please remember 10.7 is under NDA, and as such shouldn't be discussed in public.

As I mentioned earlier: the 32-bit build should be deprecated and is only needed for older machines, anyone running modern machines can install OSX 10.6 (or later) and can use the intel-only build.
msg137245 - (view) Author: sorin (sorin) Date: 2011-05-30 06:08
FYI, I got the above output from 10.7 from a friend, I do not have access to the seed, so the information could be wrong.
History
Date User Action Args
2011-05-30 06:08:42sorinsetmessages: + msg137245
2011-05-30 05:36:34ronaldoussorensetmessages: + msg137244
2011-05-29 10:52:01sorinsetmessages: + msg137180
2011-05-29 10:50:23sorinsetmessages: + msg137179
2011-05-29 10:39:32ned.deilysetmessages: + msg137177
2011-05-29 08:14:50sorinsetmessages: + msg137172
2011-05-29 07:06:37ronaldoussorensetmessages: + msg137168
2011-05-28 22:38:59ned.deilysetmessages: + msg137159
2011-05-28 15:58:43sorinsetnosy: + sorin
messages: + msg137138
2011-05-07 01:30:41James.Tatumsetnosy: + James.Tatum
2011-03-25 06:17:07ned.deilysetmessages: + msg132071
2011-03-22 14:03:05ronaldoussorensetnosy: ronaldoussoren, tarek, ned.deily, eric.araujo, santoso.wijaya, jlindenbaum
messages: + msg131749
stage: needs patch
2011-03-22 13:47:20eric.araujosetassignee: tarek -> ronaldoussoren
messages: + msg131747
nosy: ronaldoussoren, tarek, ned.deily, eric.araujo, santoso.wijaya, jlindenbaum
2011-03-22 07:15:24ronaldoussorensetnosy: ronaldoussoren, tarek, ned.deily, eric.araujo, santoso.wijaya, jlindenbaum
messages: + msg131724
2011-03-22 02:11:12ned.deilysetnosy: + ned.deily, ronaldoussoren
messages: + msg131720
components: + Macintosh
2011-03-21 23:28:21eric.araujosetnosy: tarek, eric.araujo, santoso.wijaya, jlindenbaum
messages: + msg131708
versions: + Python 3.1, Python 2.7, Python 3.2, Python 3.3, - Python 2.6
2011-03-21 18:32:08santoso.wijayasetnosy: + santoso.wijaya
2011-03-21 16:43:39jlindenbaumcreate