classification
Title: Is shared lib building broken on trunk for Mac OS X?
Type: Stage:
Components: Build, macOS Versions: Python 3.0, Python 3.1, Python 2.7, Python 2.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: ronaldoussoren Nosy List: akitada, ronaldoussoren, rpetrov, skip.montanaro
Priority: normal Keywords: patch

Created on 2008-11-30 16:37 by skip.montanaro, last changed 2009-01-10 16:33 by skip.montanaro. This issue is now closed.

Files
File name Uploaded Description Edit
issue4472.patch ronaldoussoren, 2008-12-30 15:49
py-issue-4472-makefile.patch rpetrov, 2009-01-01 22:36
issue4472-v2.patch ronaldoussoren, 2009-01-02 10:48
Messages (19)
msg76642 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2008-11-30 16:37
I have tried several different combinations of configure args on my Mac in
the past couple days in a so far fruitless attempt to generate a
libpython.2.7.dylib file.  All it will ever generate is a .a file.  I've
come to the conclusion that building a shared Python library is broken, at
least on Macs.  I only even noticed this because I wanted to bundle up a
trivial little Python script as a Mac app build.  py2app requires a dylib
file to generate the app bundle (to include?).

I build in separate build directories so I can have normal, debug and
framework builds.  Here's the configure command from my framework build:

    ../configure --prefix=/Users/skip/local --enable-shared \
        '--enable-framework=/Users/skip/Applications' \
        'CPPFLAGS=-I/Users/skip/local/include -I/opt/local/include' \
        'LDFLAGS=-L/Users/skip/local/lib -L/opt/local/lib'

I've tried taking out the --prefix and/or --enable-shared flags thinking
maybe the --enable-framework flag was sufficient.  I've tried making the
sharedinstall target.  I never see anything like -fPIC or -fpic in the gcc
command line args.  In fact, I don't see "dylib" or "pic" mentioned in the
Makefile at all.

I gave up building in a subdirectory then whittled the configure command
down to this:

    ./configure --prefix=/Users/skip/local --enable-shared \
        '--enable-framework=/Users/skip/Applications'

make clean.  make.  Still no libpython.2.7.dylib file.

Skip
msg76669 - (view) Author: Akira Kitada (akitada) * Date: 2008-11-30 22:07
OS X 10.5.5 seems to have the problem just as described, 
whereas there seems no problem on FreeBSD 6.3.
msg78536 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2008-12-30 14:47
The issue is caused by lack of support for building .dylibs in 
configure.in and Makefile.pre.in. AFAIK this has never worked on OSX.

Fixing that is not entirely trivial (did I mention configure.in is rather 
crufty?).

The attached patch should fix that (the patch touched configure.in, you'll 
have to run autoconf after applying the patch).

Could you please test if the patch fixes the issue for you?
msg78539 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2008-12-30 15:07
Never mind, the patch won't work. Python's build procedure on unix-y 
platforms assumes that the suffix for shared libraries is the same as that 
for python extentions. That's not true on OSX :-(

I'm working on an updated patch.
msg78541 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2008-12-30 15:49
I've uploaded a new version of issue4472.patch that seems to do the trick:  
this supports --enable-shared and also still works without that flag.
msg78543 - (view) Author: Roumen Petrov (rpetrov) * Date: 2008-12-30 16:20
What is reason to change SO to ".dylib" in configure script and later
Lib/distutils/sysconfig.py to restore it to ".so" ?

Also other builds use $(LDSHARED) to create python shared library. Why
proposed patch introduce inconsistency with other builds and ignore
LDSHARED set by configure script ?
msg78544 - (view) Author: Roumen Petrov (rpetrov) * Date: 2008-12-30 16:24
Skip,
Why the issue title is without platform? Shared build is fine on Linux
and Cygwin too.
msg78545 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2008-12-30 16:32
"SO=.dylib" is needed in the makefile to ensure that "make install" 
works and to ensure that the libpython shared library is build with the 
right suffix (OSX uses .dylib for shared libraries). 

The code in distutils.sysconfig is necessary to ensure that extentions 
are build with a ".so" suffix because the import machinery expected 
'.so' as the suffix.

BTW. I intent to commit the attached patch later this week.
msg78551 - (view) Author: Roumen Petrov (rpetrov) * Date: 2008-12-30 18:00
Ronald, 

In proposed patch we see (insert about line 740):
+    	LDLIBRARY='libpython$(VERSION).dylib'
+	BLDLIBRARY='-L. -lpython$(VERSION)'
+	RUNSHARED='DYLD_LIBRARY_PATH=`pwd`:${DYLD_LIBRARY_PATH}'


Let see comment in configure script(about line 585):
# MacOSX framework builds need more magic. LDLIBRARY is the dynamic
# library that we build, but we do not want to link against it (we
# will find it with a -framework option). For this reason there is an
# extra variable BLDLIBRARY against which Python and the extension
# modules are linked, BLDLIBRARY. This is normally the same as
# LDLIBRARY, but empty for MacOSX framework builds.


Next on line about 675 we see:
# MacOSX framework builds need more magic. LDLIBRARY is the dynamic
# library that we build, but we do not want to link against it (we
# will find it with a -framework option). For this reason there is an
# extra variable BLDLIBRARY against which Python and the extension
# modules are linked, BLDLIBRARY. This is normally the same as
# LDLIBRARY, but empty for MacOSX framework builds.
if test "$enable_framework"
then
 LDLIBRARY='$(PYTHONFRAMEWORKDIR)/Versions/$(VERSION)/$(PYTHONFRAMEWORK)'
  RUNSHARED=DYLD_FRAMEWORK_PATH="`pwd`:$DYLD_FRAMEWORK_PATH"
  BLDLIBRARY=''
else
  BLDLIBRARY='$(LDLIBRARY)'
fi  

So proposed patch is not consistent with comments. Is expected shared
build to ignore framework build ?


About "SO" - as I understand python build system it is reserved for
module suffix. So there is no reason to change it.

About .dylib - I'm sure that issue can be resolved with appropriate 
update without SO to be changed by configure.
msg78553 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2008-12-30 18:29
A --enable-framework and --enable-shared are mutable exclusive. You 
either have a regular unix build or a Python.framework.

As to the SO update: the only way to avoid my change is yet another 
variable in the makefile. The "altbininstall" target assumes that it 
will install a shared library named "libpython$(VERSION)$(SO)". I 
haven't looked for other uses of the SO variable in the Makefiles yet.

Is there any documentation on what the role of the "SO" variable means. 
If there isn't I propose to leave my patch as it is. The build 
infrastructure is complicated enough as it is.
msg78561 - (view) Author: Roumen Petrov (rpetrov) * Date: 2008-12-30 22:38
> A --enable-framework and --enable-shared are mutable exclusive. You 
> either have a regular unix build or a Python.framework.
May be configure has to block user if both are enabled.

> Is there any documentation on what the role of the "SO" variable means. 
Reading the code (distutils), comments configure script we may get some
information.

From configure (line about 1600) :
AC_DEFINE_UNQUOTED(SHLIB_EXT, "$SO", [Define this to be extension of
shared libraries (including the dot!).])
Actually this is suffix for python module (as example for windows ".pyd"
or "_d.pyd")

If SO is changed by to .dylib for OS X in configure above line will
define SHLIB_EXT to ".dylib" in "pyconfig.h". 

Ronald, you patch restore indirectly shared_lib_extension (distutils) to
".so", but define SHLIB_EXT from "pyconfig.h" will be ".dylib". This may
impact other projects.
msg78602 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2008-12-31 12:48
The patch seemed to work for me.  Should I worry that I don't see
-fPIC or -fpic in the compile commands?  Also, running "make test"
before at least installing libpython2.7.dylib appears to be impossible:

% otool -L python.exe 
python.exe:
	/Users/skip/local/lib/libpython2.7.dylib (compatibility version 
2.7.0, current version 2.7.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current 
version 111.1.3)
regular% ./python.exe 
dyld: Library not loaded: /Users/skip/local/lib/libpython2.7.dylib
  Referenced from: /Users/skip/src/python/trunk/regular/./python.exe
  Reason: image not found
Trace/BPT trap

Is there a way to work around that?
msg78609 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2008-12-31 14:27
Skip: 
* GCC always generates position-independent code on OSX

* I'll look in the test issue, that's probably caused by a broken 
definition for RUNSHARED.

Roumen:

* The SHLIB_EXT definition in pyconfig.h is an issue. And I have to 
agree that "SO" is meant to be the suffix for python extension, given 
the comment about LDSHARED in configure.in (just below the definition of 
SHLIB_EXT). 

I'll work on a different way to deal with the .dylib vs. .so problem.
msg78733 - (view) Author: Roumen Petrov (rpetrov) * Date: 2009-01-01 22:36
I just finish test for altbininstall make target on linux and cygwin.

Ronald,
About the SO issue - in makefile we may use $(LDLIBRARY) instead of
libpython$(VERSION)$(SO) (see py-issue-4472-makefile.patch).
For cygwin libpython$(VERSION)$(SO) is equal to $(DLLLIBRARY) and
DLLLIBRARY is set only on cygwin.
msg78776 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2009-01-02 10:48
LDLIBRARY is indeed a good variable to use. 

I've applied issue447-v2.patch in r68146. Backported to 2.6.x in r68147.
msg78809 - (view) Author: Roumen Petrov (rpetrov) * Date: 2009-01-02 14:58
Ronald,
cygwin is a special case too.
LDLIBRARY is so called import library !

So the make target can be (as I propose in py-issue-4472-makefile.patch)
+		if test -n "$(DLLLIBRARY)"; then \
+			$(INSTALL_SHARED) $(DLLLIBRARY) $(DESTDIR)$(BINDIR); \

or if new check is not ok then we has to restore old code .
-		if test "$(SO)" = .dll; then \
-			$(INSTALL_SHARED) libpython$(VERSION)$(SO) $(DESTDIR)$(BINDIR); \

More info for cygwin:
- SO = .dll
- DLLLIBRARY is set only for cygwin
- libpython$(VERSION)$(SO) = $(DLLLIBRARY)
- but $(LDLIBRARY) = libpython$(VERSION).dll.a !!!
msg78812 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2009-01-02 15:06
I've applied this patch. If I understand things correctly that should 
fix the cygwin issue.

Index: Makefile.pre.in
===================================================================
--- Makefile.pre.in	(revision 68149)
+++ Makefile.pre.in	(working copy)
@@ -773,8 +773,8 @@
 	done
 	$(INSTALL_PROGRAM) $(BUILDPYTHON) 
$(DESTDIR)$(BINDIR)/python$(VERSION)$(EXE)
 	if test -f $(LDLIBRARY); then \
-		if test "$(SO)" = .dll; then \
-			$(INSTALL_SHARED) $(LDLIBRARY) 
$(DESTDIR)$(BINDIR); \
+		if test -n "$(DLLLIBRARY)" ; then \
+			$(INSTALL_SHARED) $(DLLLIBRARY) 
$(DESTDIR)$(BINDIR); \
 		else \
 			$(INSTALL_SHARED) $(LDLIBRARY) 
$(DESTDIR)$(LIBDIR)/$(INSTSONAME); \
 			if test $(LDLIBRARY) != $(INSTSONAME); then \
msg78856 - (view) Author: Roumen Petrov (rpetrov) * Date: 2009-01-02 18:51
10x cygwin is now not impacted.
msg79553 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2009-01-10 16:33
Works for me.  Thanks Ronald.  Closing...
History
Date User Action Args
2009-01-10 16:33:51skip.montanarosetstatus: pending -> closed
assignee: ronaldoussoren
messages: + msg79553
2009-01-02 18:51:33rpetrovsetmessages: + msg78856
2009-01-02 15:06:49ronaldoussorensetmessages: + msg78812
2009-01-02 14:58:42rpetrovsetmessages: + msg78809
2009-01-02 10:48:46ronaldoussorensetstatus: open -> pending
resolution: accepted -> fixed
2009-01-02 10:48:12ronaldoussorensetfiles: + issue4472-v2.patch
messages: + msg78776
2009-01-01 22:36:17rpetrovsetfiles: + py-issue-4472-makefile.patch
messages: + msg78733
2008-12-31 14:27:52ronaldoussorensetmessages: + msg78609
2008-12-31 12:48:12skip.montanarosetmessages: + msg78602
2008-12-30 22:38:16rpetrovsetmessages: + msg78561
2008-12-30 18:29:38ronaldoussorensetmessages: + msg78553
2008-12-30 18:00:54rpetrovsetmessages: + msg78551
2008-12-30 16:32:40ronaldoussorensetresolution: accepted
messages: + msg78545
versions: + Python 2.6, Python 3.0, Python 3.1, Python 2.7
2008-12-30 16:31:12skip.montanarosettitle: Is shared lib building broken on trunk? -> Is shared lib building broken on trunk for Mac OS X?
2008-12-30 16:24:09rpetrovsetmessages: + msg78544
2008-12-30 16:20:09rpetrovsetnosy: + rpetrov
messages: + msg78543
2008-12-30 15:49:07ronaldoussorensetfiles: + issue4472.patch
messages: + msg78541
2008-12-30 15:15:51ronaldoussorensetfiles: - issue4472.patch
2008-12-30 15:07:17ronaldoussorensetmessages: + msg78539
2008-12-30 14:47:14ronaldoussorensetkeywords: + patch
files: + issue4472.patch
messages: + msg78536
nosy: + ronaldoussoren
2008-11-30 22:07:10akitadasetnosy: + akitada
messages: + msg76669
components: + Build, macOS
2008-11-30 16:37:54skip.montanarocreate