classification
Title: ctypes.util.find_library should examine binary's RPATH on Solaris
Type: enhancement Stage:
Components: ctypes Versions: Python 3.3, Python 3.4, Python 2.7, Python 2.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: automatthias, jcea, martin.panter, risto3
Priority: normal Keywords: needs review

Created on 2013-10-20 15:31 by automatthias, last changed 2016-04-28 06:43 by martin.panter.

Files
File name Uploaded Description Edit
find_library_looks_into_rpath.patch automatthias, 2013-10-20 17:19 Example implementation review
Messages (12)
msg200593 - (view) Author: Maciej Bliziński (automatthias) Date: 2013-10-20 15:31
On Solaris, when you want to link shared libraries from custom directories, you most often don't modify the system search path, but instead set RPATH in your binaries. For example, OpenCSW packages Python into /opt/csw, and sets Python executable's RPATH to /opt/csw/lib. Therefore, dynamically opening shared libraries will by default look into /opt/csw/lib first, and find_library should do the same. I wrote a sample implementation.
msg200863 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2013-10-22 02:56
Patch looks good. Any other reviewer?

Maciej, do you know about any generally available library in Solaris with SONAME and RPATH?. A test would be nice.

Would this patch solve the "locate the C library" using ctypes, in Solaris? :-).
msg200864 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2013-10-22 03:01
"/usr/bin/chkpass.so" has both SONAME and RPATH.

Would you mind, anyway, to elaborate a bit your use case, including some example?. Thanks!.
msg200865 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2013-10-22 03:04
I mean "/usr/lib/chkpass.so".
msg200889 - (view) Author: Maciej Bliziński (automatthias) Date: 2013-10-22 07:52
The specific case that breaks for me is this:

OpeCSW Python package includes:

/opt/csw/bin/python2.6 (also 2.7, 3.3, etc)
/opt/csw/lib/libpython2.6.so.1.0
(other files)

On the operating system there is only:
/usr/lib/libpython2.4.so.1.0

Let's say there's libmagic installed in /opt/csw/lib, and there's no libmagic in /usr/lib. Let's suppose you run this:

ctypes.cdll.LoadLibrary('libmagic.so.1')

The OpenCSW Python will successfully load libmagic.so.1. But without the patch, find_library will not find libmagic.so.1, it will fail to translate 'magic' to 'libmagic.so.1', even though the libmagic.so symlink is present in /opt/csw/lib.

For the patch, the RPATH of the library itself doesn't matter, the important one is the RPATH of the Python executable itself, which influences what _ctypes.dlopen() does when looking for a library.

I could write a test by providing providing a sample /usr/ccs/bin/dump output and mocking out some libraries. Would that be good?
msg256730 - (view) Author: Richard PALO (risto3) Date: 2015-12-19 15:00
I tried this patch out on pkgsrc, it does seem reasonable and appropriate. So +1 from me.

It does only look for libraries the actual $PREFIX directory used by packaging systems such as pkgsrc and csw. (typically /usr/local, /opt/local or /opt/csw in the case of csw)

So for completeness, perhaps consideration should be made some time for cases such as when environment variables like LD_LIBRARY_PATH*, LD_CONFIG* et cetera are used... BTW, LD_LIBRARY_PATH is also used on other ELF platforms.
msg264147 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-04-25 02:46
find_library() is documented as emulating the build-time linker, and I suspect the RPATH of the Python executable is not relevant at build time. See also Issue 9998 and Issue 18502.
msg264349 - (view) Author: Richard PALO (risto3) Date: 2016-04-27 05:14
There *is* a feature with linking called $ORIGIN.
msg264355 - (view) Author: Richard PALO (risto3) Date: 2016-04-27 06:15
oups... I meant to add the comment about $ORIGIN (not really useful here) but also the fact that the binary python is built with the dependencies found via the library path (-L, for example) and the eventual run-paths (-R or -runpath) when not in the system paths.

As indicated by automatthias@, package managers such as pkgsrc, CSW, *ports and the like have their $PREFIX off of the system main path..  usually in /usr/pkg, /usr/local, /opt/local or something like that
(for unices and linux sort of systems, anyway).

My pkgsrc python2.7 binary has the following runpath:
```
richard@omnis:/home/richard$ dump -Lpv /opt/local/bin/python2.7 |grep RPATH
[12]	RPATH           /opt/pbulk32/gcc49/i486-sun-solaris2.11/lib/.:/opt/pbulk32/gcc49/lib/.:/opt/local/lib
```

What is being requested greatly simplifies things, and can probably be generalised to other ELF based systems too.

BTW I came across the issue as well with py-magic.
msg264412 - (view) Author: Richard PALO (risto3) Date: 2016-04-28 04:44
An example:

richard@omnis:/home/richard$ python2.7
Python 2.7.11 (default, Apr 27 2016, 04:35:25) 
[GCC 4.9.3] on sunos5
Type "help", "copyright", "credits" or "license" for more information.
>>> from ctypes import *
>>> from ctypes.util import find_library
>>> find_library('magic')
>>> cdll.LoadLibrary('libmagic.so')
<CDLL 'libmagic.so', handle fe240840 at fe7879ac>

Finally, as you can see above, LoadLibrary() works fine to load '/opt/local/libmagic.so' because of the runpath in the python binary, but find_library() does not because the runpath is ignored.

This should probably be considered as 'unexpected' behaviour.
msg264413 - (view) Author: Richard PALO (risto3) Date: 2016-04-28 04:47
[fingers not yet warmed up] that is '/opt/local/lib/libmagic.so'
msg264419 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-04-28 06:43
I am realizing that many people like to use find_library() as a way of converting a portable name like “magic” (from building with -lmagic) into a platform-specific name (libmagic.so.1, libmagic.dylib, magic.dll, etc), and then pass this to LoadLibrary() or equivalent. So searching Python’s runpath would probably be valid for that use case.

IMO the extra searching is inefficient, and not robust if there is more than one version of the library. Personally I would be more interested in adding an alternative function, maybe make_library_name("magic", "1") -> "libmagic.so.1", or cdll.LoadLibraryByPortableName("magic", "1").
History
Date User Action Args
2016-04-28 06:43:36martin.pantersetmessages: + msg264419
2016-04-28 04:47:16risto3setmessages: + msg264413
2016-04-28 04:44:47risto3setmessages: + msg264412
2016-04-27 06:15:40risto3setmessages: + msg264355
2016-04-27 05:14:25risto3setmessages: + msg264349
2016-04-25 02:46:47martin.pantersetnosy: + martin.panter
messages: + msg264147
2015-12-19 15:00:05risto3setnosy: + risto3
messages: + msg256730
2013-10-22 07:52:39automatthiassetmessages: + msg200889
2013-10-22 03:04:18jceasetmessages: + msg200865
2013-10-22 03:01:11jceasetmessages: + msg200864
2013-10-22 02:56:41jceasetkeywords: + needs review, - patch
2013-10-22 02:56:20jceasetmessages: + msg200863
2013-10-21 18:28:37ned.deilysetnosy: + jcea
2013-10-21 10:00:00automatthiassetversions: + Python 2.6, Python 2.7
2013-10-20 17:19:41automatthiassetfiles: - find_library_looks_into_rpath.patch
2013-10-20 17:19:32automatthiassetfiles: + find_library_looks_into_rpath.patch
2013-10-20 15:43:42automatthiassetfiles: + find_library_looks_into_rpath.patch
2013-10-20 15:38:55automatthiassetfiles: - find_library_looks_into_rpath.patch
2013-10-20 15:35:31automatthiassetfiles: + find_library_looks_into_rpath.patch
2013-10-20 15:32:24automatthiassetfiles: - find_library_looks_into_rpath.patch
2013-10-20 15:31:38automatthiascreate