classification
Title: python.exe on OS X shared-llbrary build erroneously linked to MacPorts python library
Type: Stage: patch review
Components: Build, macOS Versions: Python 3.4, Python 3.3, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: ned.deily Nosy List: eduardocereto, eric.araujo, esc24, koobs, ned.deily, ronaldoussoren, samueljohn
Priority: normal Keywords: needs review

Created on 2011-03-08 22:24 by skip.montanaro, last changed 2014-08-04 03:36 by koobs.

Files
File name Uploaded Description Edit
issue-11445-linkflags.txt ronaldoussoren, 2013-07-06 15:25
Messages (11)
msg130379 - (view) Author: Skip Montanaro (skip.montanaro) * Date: 2011-03-08 22:24
I routinely configure Python like so on my Mac (10.5.8):

    ./configure  --prefix=/Users/skip/local --enable-shared LDFLAGS=-L/opt/local/lib CPPFLAGS=-I/opt/local/include

This has always worked for me.  Now, after installing from my Mercurial
sandbox I have to set PYTHONPATH to get my <prefix>/python2.7/site-packages
directory in sys.path.  Here's sys.path in a vanilla python2.7 session when
installed from a svn sandbox:

    ['/Users/skip/misc/python/python2', '/Users/skip/misc/python', '',
    '/Users/skip/local/lib/python2.7/site-packages/ZODB3-3.8.1b8-py2.7-macosx-10.3-i386.egg',
    '/Users/skip/local/lib/python2.7/site-packages/zdaemon-2.0.2-py2.7.egg',
    '/Users/skip/local/lib/python2.7/site-packages/ZConfig-2.6.0-py2.7.egg',
    '/Users/skip/local/lib/python2.7/site-packages/zope.testing-3.7.0-py2.7.egg',
    '/Users/skip/local/lib/python2.7/site-packages/zope.proxy-3.4.2-py2.7-macosx-10.3-i386.egg',
    '/Users/skip/local/lib/python2.7/site-packages/zope.interface-3.4.1-py2.7-macosx-10.3-i386.egg',
    '/Users/skip/local/lib/python2.7/site-packages/yolk-0.4.1-py2.7.egg',
    '/Users/skip/local/lib/python2.7/site-packages/SQLAlchemy-0.5.0rc2-py2.7.egg',
    '/Users/skip/local/lib/python2.7/site-packages/decorator-2.3.1-py2.7.egg',
    '/Users/skip/local/lib/python2.7/site-packages/dnspython-1.6.0-py2.7.egg',
    '/Users/skip/local/lib/python2.7/site-packages/spambayes-1.1b1-py2.7.egg',
    '/Users/skip/local/lib/python2.7/site-packages/py2app-0.3.6-py2.7.egg',
    '/Users/skip/local/lib/python2.7/site-packages/bdist_mpkg-0.4.3-py2.7.egg',
    '/Users/skip/local/lib/python2.7/site-packages/macholib-1.1-py2.7.egg',
    '/Users/skip/local/lib/python2.7/site-packages/modulegraph-0.7-py2.7.egg',
    '/Users/skip/local/lib/python2.7/site-packages/altgraph-0.6.7-py2.7.egg',
    '/Users/skip/local/lib/python2.7/site-packages/python_dateutil-1.4.1-py2.7.egg',
    '/Users/skip/local/lib/python2.7/site-packages/coverage-2.85-py2.7.egg',
    '/Users/skip/local/lib/python2.7/site-packages/pycallgraph-0.5.1-py2.7.egg',
    '/Users/skip/local/lib/python2.7/site-packages/see-0.4.0-py2.7.egg',
    '/Users/skip/local/lib/python2.7/site-packages/mercurial-unknown-py2.7-macosx-10.3-i386.egg',
    '/Users/skip/local/lib/python2.7/site-packages/Pyjamas-0.5-py2.7.egg',
    '/Users/skip/local/lib/python2.7/site-packages/Cheetah-2.0.1-py2.7-macosx-10.3-i386.egg',
    '/Users/skip/local/lib/python2.7/site-packages/mock-0.4.0-py2.7.egg',
    '/Users/skip/local/lib/python2.7/site-packages/pydns-2.3.3-py2.7.egg',
    '/Users/skip/local/lib/python2.7/site-packages/Importing-1.9.2-py2.7.egg',
    '/Users/skip/local/lib/python2.7/site-packages/docutils-0.6-py2.7.egg',
    '/Users/skip/local/lib/python2.7/site-packages/MiniMock-1.2.5-py2.7.egg',
    '/Users/skip/local/lib/python2.7/site-packages/nose-0.11.2.dev-py2.7.egg',
    '/Users/skip/local/lib/python2.7/site-packages/pytz-2010b-py2.7.egg',
    '/Users/skip/local/lib/python2.7/site-packages/pip-0.6.3-py2.7.egg',
    '/Users/skip/local/lib/python2.7/site-packages/xlrd-0.7.1-py2.7.egg',
    '/Users/skip/local/lib/python2.7/site-packages/apipkg-1.0-py2.7.egg',
    '/Users/skip/local/lib/python2.7/site-packages/argparse-1.1-py2.7.egg',
    '/Users/skip/local/lib/python2.7/site-packages/virtualenv-1.5.1-py2.7.egg',
    '/Users/skip/local/lib/python2.7/site-packages/tox-0.9-py2.7.egg',
    '/Users/skip/local/lib/python2.7/site-packages/py-1.4.0-py2.7.egg',
    '/Users/skip/local/lib/python2.7/site-packages/virtualenv5-1.3.4.5-py2.7.egg',
    '/Users/skip/local/lib/python2.7/site-packages/pylint-0.22.0-py2.7.egg',
    '/Users/skip/local/lib/python2.7/site-packages/logilab_astng-0.21.0-py2.7.egg',
    '/Users/skip/local/lib/python2.7/site-packages/logilab_common-0.53.0-py2.7.egg',
    '/Users/skip/local/lib/python2.7/site-packages/unittest2-0.5.1-py2.7.egg',
    '/Users/skip/local/lib/python2.7/site-packages/PIL-1.1.7-py2.7-macosx-10.4-i386.egg',
    '/Users/skip/local/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg',
    '/Users/skip/local/lib/python2.7',
    '/Users/skip/local/lib/python2.7/plat-darwin',
    '/Users/skip/local/lib/python2.7/plat-mac',
    '/Users/skip/local/lib/python2.7/plat-mac/lib-scriptpackages',
    '/Users/skip/local/lib/python2.7/lib-tk',
    '/Users/skip/local/lib/python2.7/lib-old',
    '/Users/skip/local/lib/python2.7/lib-dynload',
    '/Users/skip/.local/lib/python2.7/site-packages',
    '/Users/skip/local/lib/python2.7/site-packages',
    '/Users/skip/local/lib/python2.7/site-packages/PIL']

Here it is when installed from a Mercurial sandbox:

    ['/Users/skip/misc/python/python2', '/Users/skip/misc/python', '',
    '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7',
    '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
    '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
    '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
    '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
    '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload',
    '/Users/skip/.local/lib/python2.7/site-packages',
    '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages']

Note that every directory in sys.path involving <prefix> has been completely
muffed (last element in sys.path).  That /opt/local/Library/... directory
does exist some some sort-of-recent build by MacPorts, not me.  That
shouldn't impact the installation of Python into my own directory space
however.

I've confirmed that identical configure commands were used for both the svn
and hg builds.

Skip
msg130394 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2011-03-09 01:14
Are you sure you're not really using the MacPorts python?  What's the value of sys.executable?
msg130395 - (view) Author: Skip Montanaro (skip.montanaro) * Date: 2011-03-09 01:33
Ned> Are you sure you're not really using the MacPorts python?  What's
    Ned> the value of sys.executable?

Yup, pretty sure. :-)  Here it is run from my hg sandbox:

    % pwd
    /Users/skip/src/hgpython/2.7
    % ./python.exe
    Python 2.7.1 (r271:86832, Feb  2 2011, 06:56:19) 
    [GCC 4.0.1 (Apple Inc. build 5493)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import sys
    >>> sys.executable
    '/Users/skip/src/hgpython/2.7/python.exe'
    >>> sys.path
    ['/Users/skip/misc/python/python2', '/Users/skip/misc/python', '', '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7', '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin', '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac', '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages', '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk', '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload', '/Users/skip/.local/lib/python2.7/site-packages', '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages']

Skip
msg130405 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2011-03-09 02:56
I can reproduce this.  Chances are you'll see that the python.exe has been dynamically linked to the MacPorts Python instead of the just produced libpython2.7.dylib :

$ otool -L ./python.exe
./python.exe:
	/opt/local/Library/Frameworks/Python.framework/Versions/2.7/Python (compatibility version 2.7.0, current version 2.7.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 111.1.5)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 476.19.0)
	/usr/lib/libmx.A.dylib (compatibility version 1.0.0, current version 47.1.0)

No doubt you can work around it by removing the --enable-shared and/or removing the MacPorts Python2.7 from the picture.  It's not yet clear to me why it's getting linked that way (--enable-shared on OS X doesn't get tested all that much as we normally do framework builds) but I'm very doubtful it has anything directly to do with hg vs svn checkout.
msg130418 - (view) Author: Skip Montanaro (skip.montanaro) * Date: 2011-03-09 04:07
Ned> No doubt you can work around it by removing the --enable-shared
    Ned> and/or removing the MacPorts Python2.7 from the picture. 

I don't rightly recall why I use --enable-shared, but hopefully I can get
rid of it.  MacPorts must have installed Python 2.7 for some reason of its
own, so I don't want to mess with that.

Any idea why --enable-shared didn't hose up my svn sandbox build?

Skip
msg130420 - (view) Author: Skip Montanaro (skip.montanaro) * Date: 2011-03-09 04:16
> Any idea why --enable-shared didn't hose up my svn sandbox build?

I take that back.  I looked in config.status.  I didn't use --enable-shared
in my svn sandbox build.  I misread the output of grep.

So, it's clearly the --enable-shared that's the culprit.  The library search
path built when I used LDFLAGS=-L/opt/local/lib must somehow search
/opt/local/lib before searching $PWD.

Would be nice if we could fix that.  I can't see why a third party library
search directory (generally intended to point the linker at stuff like
libreadline or libjpeg) should be searched before the directory in which
Python is being built.

S
msg130455 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2011-03-09 14:16
It is --enable-shared that is the culprit, but in the negated form...

The OSX linker will search the entire link path for a shared library before trying to look for a static library. As a workaround you could use '--enable-shared', and that should ensure that you get linked to the python version you're actually building as it is earlier on the path.

A proper fix is to add "-Wl,-search_paths_first" to the linker flags on OSX, with that flag the linker behaves just like the linker on any other unix platform.
msg130490 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2011-03-10 06:58
I see the problem now. Using a --enable-shared configure similar to Skip's, the gcc step that builds python.exe is:

  gcc -L/opt/local/lib -u _PyMac_Error -o python.exe \
      Modules/python.o \
      -L. -lpython2.7 -ldl  -framework CoreFoundation

What I failed to notice originally is that the MacPorts python27 port, which both Skip and I have installed, adds a link in /opt/local/lib to its framework lib:

/opt/local/lib/libpython2.7.dylib@ -> /opt/local/Library/Frameworks/Python.framework/Versions/2.7/Python

So since /opt/local/lib comes first on the lib list, that libpython2.7 is going to be found before the build directory's dylib.  Moving -L /opt/local/lib to after -L. does seem to fix the problem.

In the non-shared case, the gcc link step is:

  gcc -L/opt/local/lib -u _PyMac_Error -o python.exe \
      Modules/python.o libpython2.7.a \
      -ldl  -framework CoreFoundation

so the local static lib is explicitly loaded and there shouldn't be a problem.

The shared-lib case is nasty in that you can easily link to the wrong lib without realizing it.  The Makefile (for 2.7 - py3k is similar) is:

# Build the interpreter
$(BUILDPYTHON):	Modules/python.o $(LIBRARY) $(LDLIBRARY)
		$(LINKCC) $(LDFLAGS) $(LINKFORSHARED) -o $@ \
			Modules/python.o \
			$(BLDLIBRARY) $(LIBS) $(MODLIBS) $(SYSLIBS) $(LDLAST)

I wonder whether $(LDFLAGS) can be safely moved to after $(BLDLIBRARY) without breaking some platform. And I suspect the problem is not unique to OS X but perhaps more likely there.
msg131134 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2011-03-16 16:16
The attached patch fixes the issue by moveing LDFLAGS after BLDLIBRARY in the linking step.

I'm not committing this yet though as this will affect all platforms that use Makefiles to build, and I'm not sure if this change save for all compilers we effectively support.

Fixing this completely for OSX will require another change as well: we'd have to add "-Wl,-search_paths_first" to ensure that the build will pick up the first libpython on the search path, otherwise we'd still have a problem if you try to build a staticly linked binary (as this would still cause the linker to find macports if the additional flag isn't used).
msg145713 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-10-17 15:27
> I'm not committing this yet though as this will affect all platforms
> that use Makefiles to build, and I'm not sure if this change save for
> all compilers we effectively support.
I can test with GCC on Debian (linux kernel); I don’t know if it would be easy to ask all buildbots to test your branch, if not you could call for volunteers on python-dev.
msg192453 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-07-06 15:25
I've attached an updated patch (against the 2.7 branch, the same idea should work for 3.3 and default). 

This does two things:

1) Explicity add '-Wl,-search_paths_first' to LDFLAGS on Mac OS X.

   This ensures that the linker behaves like other platforms: in every
   directory on the search path look for both shared and static libraries,
   the default (upto xcode 4) was to first walk the entire path looking
   for dylibs, then walk the path again looking for static libraries.

2) Change the order of arguments when linking $(BUILDPYTHON),
   the -L flags in $(LDFLAGS) were first added before the "-L." flag, 
   and are now after that flag. This ensures that the correct python 
   library will be found, even if there is a libpython in one of the
   directories on the search path that's added by $(LDFLAGS).

This should work fine on OSX (obviously) and Linux, but I'm not 100% that
moving $(LDFLAGS) will work with the vendor compilers on commercial unix
systems.

(Adding 3.3 and 3.4 to the versions because those should also be affected by this issue)
History
Date User Action Args
2014-08-04 03:36:12koobssetnosy: + koobs
2013-07-06 18:53:03esc24setnosy: + esc24
2013-07-06 15:25:14ronaldoussorensetkeywords: - patch
files: + issue-11445-linkflags.txt
messages: + msg192453

versions: + Python 3.3, Python 3.4
2013-07-06 15:18:11ronaldoussorensetfiles: - issue-11445.txt
2012-07-13 08:20:06eduardoceretosetnosy: + eduardocereto
2012-06-11 13:45:03samueljohnsetnosy: + samueljohn
2011-10-17 15:27:40eric.araujosetmessages: + msg145713
2011-03-19 19:07:55skip.montanarosetnosy: - skip.montanaro
2011-03-16 16:16:05ronaldoussorensetfiles: + issue-11445.txt
nosy: skip.montanaro, ronaldoussoren, ned.deily, eric.araujo
messages: + msg131134

keywords: + patch, needs review
stage: needs patch -> patch review
2011-03-10 06:58:06ned.deilysetnosy: skip.montanaro, ronaldoussoren, ned.deily, eric.araujo
messages: + msg130490
2011-03-09 14:16:59ronaldoussorensetnosy: skip.montanaro, ronaldoussoren, ned.deily, eric.araujo
messages: + msg130455
2011-03-09 06:22:10ned.deilysetnosy: + ronaldoussoren

stage: needs patch
components: + macOS
title: Something changed w.r.t. <prefix>/pythonN.M/site-packages in the Hg switch -> python.exe on OS X shared-llbrary build erroneously linked to MacPorts python library
2011-03-09 04:16:44skip.montanarosetnosy: skip.montanaro, ned.deily, eric.araujo
messages: + msg130420
2011-03-09 04:07:09skip.montanarosetnosy: skip.montanaro, ned.deily, eric.araujo
messages: + msg130418
2011-03-09 02:56:43ned.deilysetversions: + Python 2.7
nosy: skip.montanaro, ned.deily, eric.araujo
messages: + msg130405

assignee: ned.deily
components: + Build
2011-03-09 01:33:31skip.montanarosetnosy: skip.montanaro, ned.deily, eric.araujo
messages: + msg130395
2011-03-09 01:14:53ned.deilysetnosy: + ned.deily
messages: + msg130394
2011-03-08 22:32:55eric.araujosetnosy: + eric.araujo
2011-03-08 22:24:52skip.montanarocreate