Extension modules fail to build on OS X 10.6 using python.org 2.x/3.x #51206

ned-deily · 2009-09-21T11:05:58Z

BPO	6957
Nosy	@warsaw, @ronaldoussoren, @ned-deily
Files	issue-6957-force-gcc-4.0.txt

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ronaldoussoren'
closed_at = <Date 2009-09-29.19:35:28.140>
created_at = <Date 2009-09-21.11:05:58.321>
labels = ['extension-modules', 'OS-mac', 'release-blocker']
title = 'Extension modules fail to build on OS X 10.6 using python.org 2.x/3.x'
updated_at = <Date 2009-09-29.19:35:28.139>
user = 'https://github.com/ned-deily'

bugs.python.org fields:

activity = <Date 2009-09-29.19:35:28.139>
actor = 'ronaldoussoren'
assignee = 'ronaldoussoren'
closed = True
closed_date = <Date 2009-09-29.19:35:28.140>
closer = 'ronaldoussoren'
components = ['Extension Modules', 'macOS']
creation = <Date 2009-09-21.11:05:58.321>
creator = 'ned.deily'
dependencies = []
files = ['14981']
hgrepos = []
issue_num = 6957
keywords = []
message_count = 9.0
messages = ['92930', '92973', '93005', '93009', '93011', '93155', '93162', '93321', '93322']
nosy_count = 3.0
nosy_names = ['barry', 'ronaldoussoren', 'ned.deily']
pr_nums = []
priority = 'release blocker'
resolution = 'accepted'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue6957'
versions = ['Python 2.6', 'Python 3.1']

ned-deily · 2009-09-21T11:05:56Z

Potential 2.6.3 release blocker

On OS X 10.6 (Snow Leopard), if you attempt to install a package with a
extension module using a Python from a python.org OS X installer (say,
2.6.x or 3.1.x), the c compilation steps will likely fail in one of two
ways:

Recent python.org installers are built using the OS X 10.4 SDK so
that one installer image will work on 10.4, 10.5 and now 10.6 (in
theory, 10.3.x as well). Building of extension modules also require gcc
and the SDKs, which are included as part of Apple's Xcode tools.
However, with 10.6 Xcode, the 10.4 SDK is not installed by default so
the extension module build will fail due to missing include files. Once
the problem is known, it can be solved easily by installing the SDK from
the release DVD that comes with the system or upgrade.
Even with the 10.4 SDK installed on 10.6, the compilation fails with:

/Developer/SDKs/MacOSX10.4u.sdk/usr/include/stdarg.h:4:25: error:
stdarg.h: No such file or directory

This can be especially baffling to those unfamiliar with nested include
files because the base file stdarg.h does exist in that directory.

The root cause here is that Snow Leopard now uses gcc-4.2 by default but
the 10.4 SDK does not include headers for 4.2. The solution is to force
use of gcc-4.0, the default on 10.4 and 10.5 and included in 10.6, by
defining the CC environment variable:

$ export CC=/usr/bin/gcc-4.0
$ python setup.py install
or
$ easy_install xxx

Even for experienced developers, these can be non-trivial problems to
resolve although the solutions, once known, are easy to implement.
Unfortunately, all types of python users can run into these issues
simply by trying to install and use a package that includes an extension
module. There are a growing number of hits on the web from people
trying to install popular packages such as MySQLdb, PIL, and lxml using
a python.org Python on 10.6. Note this isn't a problem for users of the
Apple-supplied Pythons (2.6 and 2.5) on 10.6 since they do not use the
10.4 SDK.

There are two separate issues here I think:

What, if anything, to do for users of installers already out in the
field?
What to do to mitigate the problems for yet-to-be released installers
(e.g. 2.6.3)?

Documenting and publicizing the problems somehow may be the only viable
option for 1. For 2, there is the possibility of some additional
actions, perhaps a warning in the installer readme or some such. It
probably would make sense to contact the maintainers of such packages so
that they can add something to their web pages and documentation.

ronaldoussoren · 2009-09-22T06:04:14Z

It should be possible to tweak distutils to do the right thing for
upcoming releases, distutils already contains some special code to allow
building extensions for a univeral build on 10.3.9 (where the compiler
doesn't support universal builds at all) and we could extend that for
the compiler version on SL.

Implementation sketch:

Record the compiler version during build (using configure)
Store the compiler version as a variable in Makefile.pre.in
Teach distutils to read that variable and adjust the compiler
path ($CC,$LD,...)

This bit would only be active on OSX.

ned-deily · 2009-09-22T17:50:11Z

Tweaking distutils as you propose sounds like a good idea to help with
future releases. Also, the proposed installer variant to support 64-bit
would not have this problem as it would only work with 10.5 and above.
(Perhaps there's a warning in all this that trying to support too many
releases with one installer is not a good thing.)

ronaldoussoren · 2009-09-22T19:10:45Z

There is a much easier solution for the 2.6.3 release: ensure that CC=gcc-
4.0 when building the installer.

That requires minimal changes to the build machinery and should be safe
enough.

The only change to distutils I do want to make for 2.6.3 is to detect
compilation with an SDK that is not present and remove the -isysroot
option in that case.

ronaldoussoren · 2009-09-22T19:31:53Z

r75022 (trunk) and r75023 (2.6) add the other half of my proposal:
distutils will ignore -isysroot SDKPATH when SDKPATH is not present on the
current system.

That combined with explicit compilation using gcc-4.0 should solve these
build issues with the OSX binary installer.

ned-deily · 2009-09-26T09:29:32Z

Explicitly defining CC during the installer build does seem to fix the
extension build problem, at least for well-behaved extensions. I
successfully tested building simplejson (using both easy_install and
direct setup.py installs) and MySQLdb, both on 10.6 using a fat
installer built on 10.5 with DEPTARGET=10.3 and 10.4u SDK, test cases
that previously failed.

I've attached a patch to build-installer.py to support setting CC to
appropriate values for distutils based on the value of deployment
target: gcc-4.0 for DEPTARGET in [10.3, 10.4, 10.5], gcc-4.2 for 10.6.

The patch also includes changes to the build process itself to use the
appropriate gcc, including the recipes for the 3rd-party libs. This is
in support of building installers on 10.6 with gcc-4.0 and 10.4u SDKs
(however, there are still problems with this - see below).

It also includes fixes for two unrelated build problems noted during
testing: (1) the doc html files are installed in the wrong location in
the framework (this was fixed in 3.x but not yet backported); and (2)
the recently changed test for 10.4 or later was off by one (causing
building to fail on 10.4 systems). Those fixes should go into 2.6.3
regardless.

There are some open issues with the patch:

Should CXX (the distutils environ value for the c++ compiler) also be
set? There are multiple paths through configure involving CC and CXX.
I chose not to implement a --cc argument to build-installer.py
figuring it would be safer to hard code the appropriate CC values into
the script for the time being.
While the script as is does not cause regressions for installer
builds on 10.5 or 10.4 (that is, assuming the remaining builds and tests
in progress complete normally - I'll update the status when they
finish), as it stands I am still not able to build "standard" fat
installers using the 10.4u SDK on 10.6. There seem to be problems up
front in configure: a number of build options are coming out differently
when using 10.4u on 10.6 than on 10.5, presumably causing the cc compile
errors. It's not a release-blocker problem, anyway, since the plan is
to build the installer on 10.5 but we do need to get a better idea
what's going wrong there. (Unfortunately, I don't expect to have time
to investigate this further prior to the 2.6.3 release.)

ned-deily · 2009-09-26T16:23:08Z

(that is, assuming the remaining builds and tests
in progress complete normally - I'll update the status when they
finish)

No regressions noted:

DEPTARGET=10.3, 2-way i386/ppc, built on 10.5:
regrtest on 10.4 ppc, 10.5 ppc, 10.6 i386
DEPTARGET=10.3, 2-way i386/ppc, built on 10.4:
regrtest on 10.4 ppc, 10.5 ppc, 10.6 i386

warsaw · 2009-09-29T19:31:12Z

Please apply this for 2.6.3rc1

ronaldoussoren · 2009-09-29T19:35:28Z

Committed in r75147 (trunk) and r75148 (2.6)

ned-deily assigned ronaldoussoren Sep 21, 2009

ned-deily added extension-modules C modules in the Modules dir OS-mac labels Sep 21, 2009

warsaw added the release-blocker label Sep 29, 2009

ronaldoussoren closed this as completed Sep 29, 2009

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Extension modules fail to build on OS X 10.6 using python.org 2.x/3.x #51206

Extension modules fail to build on OS X 10.6 using python.org 2.x/3.x #51206

ned-deily commented Sep 21, 2009

ned-deily commented Sep 21, 2009

ronaldoussoren commented Sep 22, 2009

ned-deily commented Sep 22, 2009

ronaldoussoren commented Sep 22, 2009

ronaldoussoren commented Sep 22, 2009

ned-deily commented Sep 26, 2009

ned-deily commented Sep 26, 2009

warsaw commented Sep 29, 2009

ronaldoussoren commented Sep 29, 2009

Extension modules fail to build on OS X 10.6 using python.org 2.x/3.x #51206

Extension modules fail to build on OS X 10.6 using python.org 2.x/3.x #51206

Comments

ned-deily commented Sep 21, 2009

ned-deily commented Sep 21, 2009

ronaldoussoren commented Sep 22, 2009

ned-deily commented Sep 22, 2009

ronaldoussoren commented Sep 22, 2009

ronaldoussoren commented Sep 22, 2009

ned-deily commented Sep 26, 2009

ned-deily commented Sep 26, 2009

warsaw commented Sep 29, 2009

ronaldoussoren commented Sep 29, 2009