classification
Title: distutils build_ext fails to set library_dirs in 2.7.2 on Linux
Type: compile error Stage: resolved
Components: Distutils Versions: Python 3.2, Python 3.3, Python 3.4, Python 2.7
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: eric.araujo Nosy List: Andy.Salnikov, eric.araujo, tarek, vinay.sajip, zach.ware
Priority: normal Keywords: patch

Created on 2012-10-25 19:44 by Andy.Salnikov, last changed 2020-04-26 17:12 by zach.ware. This issue is now closed.

Files
File name Uploaded Description Edit
distutils-sysexecutable.patch Andy.Salnikov, 2012-10-30 20:46 review
Messages (11)
msg173796 - (view) Author: Andy Salnikov (Andy.Salnikov) Date: 2012-10-25 19:44
Hi,

when trying to build extension modules with distutils I ran into
a problem that linking fails with an errors like:

gcc -pthread -shared -L build/temp.linux-x86_64-2.7/h5py/defs.o -L/reg/g/psdm/sw/external/hdf5/1.8.4p1/x86_64-rhel6-gcc44-opt/lib -L. -Wl,-R/reg/g/psdm/sw/external/hdf5/1.8.4p1/x86_64-rhel6-gcc44-opt/lib -lhdf5 -lpython2.7 -o build/lib.linux-x86_64-2.7/h5py/defs.so
/usr/bin/ld: cannot find -lpython2.7
collect2: ld returned 1 exit status

For some reason location of the python library is not added to the 
command line with -L option.

I tracked the reason down to a particular environment that we have, 
in out environment python executable found in a $PATH is a symbolic link
to a binary installed in some non-standard location. I believe this 
piece of code in build_ext.py fails to realize this:

            if sys.executable.startswith(os.path.join(sys.exec_prefix, "bin")):
                # building third party extensions
                self.library_dirs.append(sysconfig.get_config_var('LIBDIR'))
            else:
                # building python standard extensions
                self.library_dirs.append('.')

apparently sys.executable in our case refers to a symlink path, while 
sys.exec_prefix refers to actual installation directory.

I think fix for our case should be easy (I can't say about other cases
which may be broken by this logic), one just need to apply os.path.realpath()
to sys.executable before comparing it to sys.exec_prefix.

Andy
msg173953 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-10-27 18:04
Would you like to work on a patch?
msg174050 - (view) Author: Andy Salnikov (Andy.Salnikov) Date: 2012-10-28 14:42
I never submitted any patch to Python, but unless somebody more
experienced wants to contribute I can try.
msg174096 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-10-29 01:59
Great!  http://docs.python.org/devguide contains all the info needed to get the code and make a patch.  If you apply your suggestion to the code and it fixes your build, I will commit it.  A small unit test to check the new behavior of build_ext and avoid regressions will be needed, but it’s not always easy to write these tests, so depending on your time I will be able to provide guidance or write the test myself.
msg174225 - (view) Author: Andy Salnikov (Andy.Salnikov) Date: 2012-10-30 20:46
Hi Éric,

I am attaching a patch that fixes the problem. The patch is tiny, basically 1-line. This replaces the direct use of sys.executable with the symlink-resolved version of the same path. I made the change for linux/unix platforms and also for cygwin/atheos (I'm sure cygwin has symlinks, not sure if atheos does but resolving symlinks can't hurt in general). 

The patch was created from default hg branch (3.4.0a0 I guess), I have built it and tested in my simple setup. The problem that we have (in 2.7) is indeed reproducible without this patch and it is fixed with this patch applied. 

Concerning the unit test - I'm not sure how to write one but if you have suggestions I could try. The complications in this case are that python needs to be installed in its configured location and the symlink needs to be created outside python install directory which points to the installed interpreter. If unit test could handle this then it might be possible.

I did not update any documentation, could not find any place to mention this change. Sure you will know better what else is needed to be updated. I'd be happy to help you with whatever else is necessary to commit this patch.

Cheers,
Andy
msg177187 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-12-09 00:24
Vinay, do you think dereferencing sys.executable could lead to trouble with venvs?
msg177209 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2012-12-09 11:38
> Vinay, do you think dereferencing sys.executable could lead to trouble with venvs?

It could - the venv code looks for a venv configuration file relative to sys.executable, which could be a symlink into a system-wide Python installation. Resolving the symlink would mean that the venv can't be found.
msg177379 - (view) Author: Andy Salnikov (Andy.Salnikov) Date: 2012-12-12 16:15
OK, I see the problem. Do you think it would help if it tested both
sys.executable and its symlynk-resolved path against sys.exec_prefix
like this:

            if sys.executable.startswith(os.path.join(sys.exec_prefix, "bin")) or
                os.path.realpath(sys.executable).startswith(os.path.join(sys.exec_prefix, "bin")):
                # building third party extensions
                self.library_dirs.append(sysconfig.get_config_var('LIBDIR'))
            else:
                # building python standard extensions
                self.library_dirs.append('.')

Alternatively one can reverse the test. I guess that 'else:' is supposed
to apply when one builds new Python installation? Where does the 
sys.executable points to in this case? Is there any other (more reliable)
way to figure out that the standard extensions are being built instead of
third-party modules?

Andy
msg177396 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2012-12-13 00:35
In terms of the venv code, I don't see how doing the test in that way would cause problems - as long as the value of sys.executable doesn't change, then as I see it, the venv code should operate as it's meant to.
msg213226 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2014-03-12 09:13
FTR a patch in #18976 is said to also fix this one.
msg367321 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2020-04-26 17:12
Given that #18976 was said to have fixed this and is now closed as "fixed", and every tagged version is now EOL, I'm closing the issue.
History
Date User Action Args
2020-04-26 17:12:02zach.waresetstatus: open -> closed

nosy: + zach.ware
messages: + msg367321

resolution: out of date
stage: test needed -> resolved
2014-03-12 09:13:16eric.araujosetmessages: + msg213226
2012-12-13 00:35:45vinay.sajipsetmessages: + msg177396
2012-12-12 16:15:53Andy.Salnikovsetmessages: + msg177379
2012-12-09 11:38:22vinay.sajipsetmessages: + msg177209
2012-12-09 00:24:10eric.araujosetnosy: + vinay.sajip
messages: + msg177187
2012-10-30 20:46:31Andy.Salnikovsetfiles: + distutils-sysexecutable.patch
keywords: + patch
messages: + msg174225
2012-10-29 01:59:25eric.araujosetmessages: + msg174096
2012-10-28 14:42:12Andy.Salnikovsetmessages: + msg174050
2012-10-27 18:04:11eric.araujosetstage: test needed
messages: + msg173953
versions: + Python 3.2, Python 3.3, Python 3.4
2012-10-25 19:44:24Andy.Salnikovcreate