classification
Title: test_get_outputs (test_distutils) failure with --enable-shared on Mac OS X
Type: Stage: resolved
Components: Distutils, Distutils2, macOS Versions: Python 3.2, Python 3.3, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: ned.deily Nosy List: alexis, eric.araujo, ned.deily, python-dev, ronaldoussoren, tarek, toggtc
Priority: normal Keywords:

Created on 2012-01-29 08:46 by toggtc, last changed 2012-02-05 11:23 by python-dev. This issue is now closed.

Messages (8)
msg152219 - (view) Author: (toggtc) Date: 2012-01-29 08:46
Current 2.7.2 + building on OS X 10.7.2 and gcc 4.2.1 (Apple build 5666.3):

i686-apple-darwin11-gcc-4.2.1: /private/var/folders/jy/dhptnvj90b34s0135sb_g6w80000gn/T/tmpAfN6sj/foo.so: No such file or directory

======================================================================
ERROR: test_get_outputs (distutils.tests.test_build_ext.BuildExtTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/toggtc/.pythonbrew/pythons/Python-2.7.2/lib/python2.7/distutils/tests/test_build_ext.py", line 291, in test_get_outputs
    cmd.run()
  File "/Users/toggtc/.pythonbrew/pythons/Python-2.7.2/lib/python2.7/distutils/command/build_ext.py", line 340, in run
    self.build_extensions()
  File "/Users/toggtc/.pythonbrew/pythons/Python-2.7.2/lib/python2.7/distutils/command/build_ext.py", line 449, in build_extensions
    self.build_extension(ext)
  File "/Users/toggtc/.pythonbrew/pythons/Python-2.7.2/lib/python2.7/distutils/command/build_ext.py", line 531, in build_extension
    target_lang=language)
  File "/Users/toggtc/.pythonbrew/pythons/Python-2.7.2/lib/python2.7/distutils/ccompiler.py", line 741, in link_shared_object
    extra_preargs, extra_postargs, build_temp, target_lang)
  File "/Users/toggtc/.pythonbrew/pythons/Python-2.7.2/lib/python2.7/distutils/unixccompiler.py", line 258, in link
    raise LinkError, msg
LinkError: command 'gcc' failed with exit status 1
----

"DYLD_LIBRARY_PATH" is not defined in default environment of OS X.
but, ./configure and ./configure.in are:
    Darwin*)
    	LDLIBRARY='libpython$(VERSION).dylib'
	BLDLIBRARY='-L. -lpython$(VERSION)'
	RUNSHARED='DYLD_LIBRARY_PATH=`pwd`:${DYLD_LIBRARY_PATH}'

=> RUNSHARED=DYLD_LIBRARY_PATH=`pwd`:

and, test_build_ext.py:
 53     def _fixup_command(self, cmd):
 ...
 63         if (sysconfig.get_config_var('Py_ENABLE_SHARED') and
 64             not sys.platform.startswith('win')):
 65             runshared = sysconfig.get_config_var('RUNSHARED')
 66             if runshared is None:
 67                 cmd.library_dirs = ['.']
 68             else:
 69                 name, equals, value = runshared.partition('=')
 70                 cmd.library_dirs = value.split(os.pathsep)

=> library_dirs=['`pwd`', '']

link command via unixccompier:
gcc-4.2 ... /foo.o -L`pwd` -L (empty!) -o /private/var/folder.../foo.so

=> "No such file or directory"
---

(for 2.7.2) https://raw.github.com/toggtc/python-patch/master/2.7.2/distutils_test_fixup_command_2.7.2.patch
(for 3.2.2) https://raw.github.com/toggtc/python-patch/master/3.2.2/distutils_test_fixup_command_3.2.2.patch
msg152301 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2012-01-30 01:58
On OS X, the linker includes in executables the absolute path to referenced shared libraries and normally --enable-shared builds are only usable from their installed location, i.e. after doing 'make install'.  RUNSHARED is defined as it is on OS X so that during the build of the standard library, the newly built interpreter uses its shared library before being installed.  After a 'make install', RUNSHARED should not be used on OS X; the shared library will always be found via the absolute path linked in the executable.  So I think the right solution for the problem here is to bypass the fixup code, so something like this (note the current 2.7.x is now similar to 3.2.x and differs from 2.7.2):


diff --git a/Lib/distutils/tests/support.py b/Lib/distutils/tests/support.py
--- a/Lib/distutils/tests/support.py
+++ b/Lib/distutils/tests/support.py
@@ -211,5 +211,8 @@
         if runshared is None:
             cmd.library_dirs = ['.']
         else:
-            name, equals, value = runshared.partition('=')
-            cmd.library_dirs = value.split(os.pathsep)
+            if sys.platform == 'darwin':
+                cmd.library_dirs = []
+            else:
+                name, equals, value = runshared.partition('=')
+                cmd.library_dirs = value.split(os.pathsep)
msg152326 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-01-30 15:20
Thanks for the analysis Ned.  Can you apply your patch to distutils and packaging?  Also, what do you think about adding a buildbot configured with --enable-shared?
msg152328 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2012-01-30 16:43
I'd prefer a buildbot with --enable-framework and --enable-universalsdk (that is, one closer to the options used to build the installer for OSX).
msg152476 - (view) Author: (toggtc) Date: 2012-02-02 21:06
Thank you for analysis and explanations, Ned. 
In addition, the -L(whitespace) is not allowed in Apple's GCC. GNU's GCC is OK. (I checked it using both GCC 4.2)
So, your solution is right.
msg152492 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2012-02-03 01:52
New changeset 41cabdff2686 by Ned Deily in branch '2.7':
Issue #13901: Prevent test_distutils failures on OS X with --enable-shared.
http://hg.python.org/cpython/rev/41cabdff2686

New changeset 6f6100a752ba by Ned Deily in branch '3.2':
Issue #13901: Prevent test_distutils failures on OS X with --enable-shared.
http://hg.python.org/cpython/rev/6f6100a752ba

New changeset 84be86af9161 by Ned Deily in branch 'default':
Issue #13901: Prevent test_packaging failures on OS X with --enable-shared.
http://hg.python.org/cpython/rev/84be86af9161
msg152493 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2012-02-03 01:58
Modified support.py in test_distutils for 2.7 (for 2.7.3), 3.2 (for 3.2.3), and 3.3, as well as in test_packaging for 3.3, to skip shared library fixing for Mac OS X.
msg152676 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2012-02-05 11:23
New changeset e6a28ae0dfd6 by Éric Araujo in branch 'default':
Port OS X --enable-shared fix from packaging (#13901; untested)
http://hg.python.org/distutils2/rev/e6a28ae0dfd6

New changeset ea717d8e71d0 by Éric Araujo in branch 'python3':
Merge fixes for #13901, #11805, #13712 and other improvements
http://hg.python.org/distutils2/rev/ea717d8e71d0
History
Date User Action Args
2012-02-05 11:23:57python-devsetmessages: + msg152676
2012-02-03 01:58:53ned.deilysetstatus: open -> closed
messages: + msg152493

assignee: tarek -> ned.deily
resolution: fixed
stage: resolved
2012-02-03 01:52:36python-devsetnosy: + python-dev
messages: + msg152492
2012-02-02 21:06:05toggtcsetmessages: + msg152476
2012-01-30 16:43:16ronaldoussorensetmessages: + msg152328
2012-01-30 15:20:07eric.araujosetnosy: + alexis
messages: + msg152326
components: + Distutils2
2012-01-30 01:58:52ned.deilysetnosy: + ronaldoussoren, ned.deily

messages: + msg152301
versions: + Python 3.3
2012-01-29 08:46:44toggtccreate