Created on 2009-09-21 11:05 by ned.deily, last changed 2009-09-29 19:35 by ronaldoussoren. This issue is now closed.
|issue-6957-force-gcc-4.0.txt||ned.deily, 2009-09-26 09:29|
|msg92930 - (view)||Author: Ned Deily (ned.deily) *||Date: 2009-09-21 11:05|
Potential 2.6.3 release blocker On OS X 10.6 (Snow Leopard), if you attempt to install a package with a extension module using a Python from a python.org OS X installer (say, 2.6.x or 3.1.x), the c compilation steps will likely fail in one of two ways: 1. Recent python.org installers are built using the OS X 10.4 SDK so that one installer image will work on 10.4, 10.5 and now 10.6 (in theory, 10.3.x as well). Building of extension modules also require gcc and the SDKs, which are included as part of Apple's Xcode tools. However, with 10.6 Xcode, the 10.4 SDK is not installed by default so the extension module build will fail due to missing include files. Once the problem is known, it can be solved easily by installing the SDK from the release DVD that comes with the system or upgrade. 2. Even with the 10.4 SDK installed on 10.6, the compilation fails with: /Developer/SDKs/MacOSX10.4u.sdk/usr/include/stdarg.h:4:25: error: stdarg.h: No such file or directory This can be especially baffling to those unfamiliar with nested include files because the base file stdarg.h does exist in that directory. The root cause here is that Snow Leopard now uses gcc-4.2 by default but the 10.4 SDK does not include headers for 4.2. The solution is to force use of gcc-4.0, the default on 10.4 and 10.5 and included in 10.6, by defining the CC environment variable: $ export CC=/usr/bin/gcc-4.0 $ python setup.py install or $ easy_install xxx Even for experienced developers, these can be non-trivial problems to resolve although the solutions, once known, are easy to implement. Unfortunately, all types of python users can run into these issues simply by trying to install and use a package that includes an extension module. There are a growing number of hits on the web from people trying to install popular packages such as MySQLdb, PIL, and lxml using a python.org Python on 10.6. Note this isn't a problem for users of the Apple-supplied Pythons (2.6 and 2.5) on 10.6 since they do not use the 10.4 SDK. There are two separate issues here I think: 1. What, if anything, to do for users of installers already out in the field? 2. What to do to mitigate the problems for yet-to-be released installers (e.g. 2.6.3)? Documenting and publicizing the problems somehow may be the only viable option for 1. For 2, there is the possibility of some additional actions, perhaps a warning in the installer readme or some such. It probably would make sense to contact the maintainers of such packages so that they can add something to their web pages and documentation.
|msg92973 - (view)||Author: Ronald Oussoren (ronaldoussoren) *||Date: 2009-09-22 06:04|
It should be possible to tweak distutils to do the right thing for upcoming releases, distutils already contains some special code to allow building extensions for a univeral build on 10.3.9 (where the compiler doesn't support universal builds at all) and we could extend that for the compiler version on SL. Implementation sketch: * Record the compiler version during build (using configure) * Store the compiler version as a variable in Makefile.pre.in * Teach distutils to read that variable and adjust the compiler path ($CC,$LD,...) This bit would only be active on OSX.
|msg93005 - (view)||Author: Ned Deily (ned.deily) *||Date: 2009-09-22 17:50|
Tweaking distutils as you propose sounds like a good idea to help with future releases. Also, the proposed installer variant to support 64-bit would not have this problem as it would only work with 10.5 and above. (Perhaps there's a warning in all this that trying to support too many releases with one installer is not a good thing.)
|msg93009 - (view)||Author: Ronald Oussoren (ronaldoussoren) *||Date: 2009-09-22 19:10|
There is a much easier solution for the 2.6.3 release: ensure that CC=gcc- 4.0 when building the installer. That requires minimal changes to the build machinery and should be safe enough. The only change to distutils I do want to make for 2.6.3 is to detect compilation with an SDK that is not present and remove the -isysroot option in that case.
|msg93011 - (view)||Author: Ronald Oussoren (ronaldoussoren) *||Date: 2009-09-22 19:31|
|msg93155 - (view)||Author: Ned Deily (ned.deily) *||Date: 2009-09-26 09:29|
Explicitly defining CC during the installer build does seem to fix the extension build problem, at least for well-behaved extensions. I successfully tested building simplejson (using both easy_install and direct setup.py installs) and MySQLdb, both on 10.6 using a fat installer built on 10.5 with DEPTARGET=10.3 and 10.4u SDK, test cases that previously failed. I've attached a patch to build-installer.py to support setting CC to appropriate values for distutils based on the value of deployment target: gcc-4.0 for DEPTARGET in [10.3, 10.4, 10.5], gcc-4.2 for 10.6. The patch also includes changes to the build process itself to use the appropriate gcc, including the recipes for the 3rd-party libs. This is in support of building installers on 10.6 with gcc-4.0 and 10.4u SDKs (however, there are still problems with this - see below). It also includes fixes for two unrelated build problems noted during testing: (1) the doc html files are installed in the wrong location in the framework (this was fixed in 3.x but not yet backported); and (2) the recently changed test for 10.4 or later was off by one (causing building to fail on 10.4 systems). Those fixes should go into 2.6.3 regardless. There are some open issues with the patch: 1. Should CXX (the distutils environ value for the c++ compiler) also be set? There are multiple paths through configure involving CC and CXX. 2. I chose not to implement a --cc argument to build-installer.py figuring it would be safer to hard code the appropriate CC values into the script for the time being. 3. While the script as is does not cause regressions for installer builds on 10.5 or 10.4 (that is, assuming the remaining builds and tests in progress complete normally - I'll update the status when they finish), as it stands I am still not able to build "standard" fat installers using the 10.4u SDK on 10.6. There seem to be problems up front in configure: a number of build options are coming out differently when using 10.4u on 10.6 than on 10.5, presumably causing the cc compile errors. It's not a release-blocker problem, anyway, since the plan is to build the installer on 10.5 but we do need to get a better idea what's going wrong there. (Unfortunately, I don't expect to have time to investigate this further prior to the 2.6.3 release.)
|msg93162 - (view)||Author: Ned Deily (ned.deily) *||Date: 2009-09-26 16:23|
> (that is, assuming the remaining builds and tests in progress complete normally - I'll update the status when they finish) No regressions noted: - DEPTARGET=10.3, 2-way i386/ppc, built on 10.5: regrtest on 10.4 ppc, 10.5 ppc, 10.6 i386 - DEPTARGET=10.3, 2-way i386/ppc, built on 10.4: regrtest on 10.4 ppc, 10.5 ppc, 10.6 i386
|msg93321 - (view)||Author: Barry A. Warsaw (barry) *||Date: 2009-09-29 19:31|
Please apply this for 2.6.3rc1
|msg93322 - (view)||Author: Ronald Oussoren (ronaldoussoren) *||Date: 2009-09-29 19:35|
|2009-09-29 19:35:28||ronaldoussoren||set||status: open -> closed|
messages: + msg93322
|2009-09-29 19:31:11||barry||set||priority: release blocker|
messages: + msg93321
|2009-09-26 16:23:08||ned.deily||set||messages: + msg93162|
messages: + msg93155
|2009-09-22 19:31:52||ronaldoussoren||set||messages: + msg93011|
|2009-09-22 19:10:45||ronaldoussoren||set||messages: + msg93009|
|2009-09-22 17:50:11||ned.deily||set||messages: + msg93005|
|2009-09-22 06:04:15||ronaldoussoren||set||messages: + msg92973|