It's come up on occasion (#2936, http://osdir.com/ml/python.ctypes/2008-05/msg00046.html) that ctypes find_library doesn't search LD_LIBRARY_PATH for libraries, which is different behaviour from the runtime linker. The attached patch adds this search.

Unfortunately I can't conceive of a reasonable unit test for this (compiling a shared library in setUp seems a bit overkill.)

Hey I have this problem too.  I would love to see this fixed.

This causes problem for Freetype Python bindings on Linux

Yaroslav: does the patch cause problems, or the original issue? If the former, could you be specific so I can try and fix it?

Sorry for confusion, I meant the original problem causes problems. I haven't tested the patch because my target machines don't have gcc

There is a problem in this area of the code, not especially with the patch but with how gcc works (or doesn't). To illustrate:

-----------------------
Version info
-----------------------
vinay@eta-hardy:/tmp$ uname -a
Linux eta-hardy 2.6.24-29-generic #1 SMP Wed Aug 10 16:34:32 UTC 2011 i686 GNU/Linux
vinay@eta-hardy:/tmp$ gcc --version
gcc (GCC) 4.2.4 (Ubuntu 4.2.4-1ubuntu4)
Copyright (C) 2007 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
-----------------------
Create an empty file:
-----------------------
vinay@eta-hardy:/tmp$ touch dummy.c
-----------------------
Compile a shared library from it:
-----------------------
vinay@eta-hardy:/tmp$ gcc -shared -o libdummy.so dummy.c
-----------------------
Invoke gcc with patched command line:
-----------------------
vinay@eta-hardy:/tmp$ gcc -L /tmp -Wl,t -o /dev/null -ldummy
/usr/bin/ld: t: No such file: No such file or directory
collect2: ld returned 1 exit status
-----------------------
It's not patch related, here's what happens with the unpatched line
-----------------------
vinay@eta-hardy:/tmp$ gcc -Wl,t -o /dev/null -ldummy
/usr/bin/ld: t: No such file: No such file or directory
collect2: ld returned 1 exit status
-----------------------
Calling ld directly works as expected. With no search path:
-----------------------
vinay@eta-hardy:/tmp$ ld -t -o /dev/null -ldummy
ld: mode elf_i386
ld: cannot find -ldummy
-----------------------
With the search path supplied:
-----------------------
vinay@eta-hardy:/tmp$ ld -t -L /tmp -o /dev/null -ldummy
ld: mode elf_i386
-ldummy (/tmp/libdummy.so)
ld: warning: cannot find entry symbol _start; not setting start address
vinay@eta-hardy:/tmp$ 

So, ISTM that the find_library code needs changing to use ld, else it will not work on all platforms.

I added an updated patch which uses ld, and added a test (Linux/Unix only).

What changed since http://bugs.python.org/issue2936?

> What changed since http://bugs.python.org/issue2936?

Well for one thing, #2936 was a bug report ("behavior") but this is an enhancement request. It seems that if you don't know exactly what a library is called, you can't use find_library to locate it even if it happens to be on LD_LIBRARY_PATH - and if you don't have the name, you can't load it using cdll.LoadLibrary.

ISTM LoadLibrary wants the full library name (including extension, on Linux); find_library helps determine that in a cross-platform way (since you can just give it the stem of the library rather than the whole filename).

Fair enough. From what I've gathered though, it looks like the window for inclusion in 3.3 is already closed, right?

Any chance of this making it into 3.4? This is a rather annoying deficiency of find_library().

Since 3.4 entered beta last week, it is now in feature freeze, so only bug fixes are allowed, unfortunately.

Since I did the ld patch I didn't want to commit it before getting another developer to review it, and it looks as if it just dropped under everyone's radar.

Any progress on this?
It also affects installation of rtree (https://github.com/Toblerity/rtree/issues/56)

I tend to agree with the comment at <https://bugs.python.org/issue2936#msg67505>. Find_library() is documented as emulating a build-time linker, not run-time. Build-time linking ignores LD_LIBRARY_PATH.

Maybe it would make more sense to pass in an optional list of -L directories to search, but it really depends on what the use cases are.

Pau: What is wrong with the more direct CDLL("libspatialindex_c.so") proposal that bypasses find_library()?

> find_library() is documented as emulating a build-time linker, not run-time

It may be documented as that, but is emulating a build-time linker the most useful thing? In the context of Python binding to external libraries, why is build-time linking behaviour better than run-time linking behaviour? This is an enhancement request, not a bug request: so if a change was to be applied, the documentation could be updated to indicate any change in behaviour.

The use case is that a shared library needed by a Python extension is available on LD_LIBRARY_PATH such that a non-ctypes linking operation would find it, but ctypes.util.find_library() won't, so a user of the extension (who may not be its developer) can't load the extension easily. See also my comment http://bugs.python.org/issue9998#msg165806

I also found Issue 18502 essentially about the same incompatibility between CDLL() and find_library().

The problem I see with using find_library() to blindly load a library is that you could be loading an incompatible version (wrong soname). That is why I asked Pau why they couldn’t give the proper library name to CDLL().

I noticed that Python’s own “uuid” module calls CDLL(find_library("uuid")); see <https://hg.python.org/cpython/annotate/v3.5.1/Lib/uuid.py#l459>. On my Linux computer, the full library name is libuuid.so.1, and according to online man pages, this is also the case on Free BSD. So I wonder if the “uuid” module should call CDLL("libuuid.so.1") directly. If somebody invented a new incompatible version libuuid.so.2, Python’s uuid module might crash, or become subtly broken, but programs that linked directly to libuuid.so.1 would continue to work.

Does anyone have any valid use cases where they want to use a shared library on LD_LIBRARY_PATH or similar, but cannot use CDLL() or LoadLibrary() directly? If people really want a way to load a library given its compile-time name, maybe changing find_library() is a valid way forward. But to me the use case does not seem robust or worth blessing in Python’s standard library.

> Pau: What is wrong with the more direct CDLL("libspatialindex_c.so") proposal that bypasses find_library()?

I don't know much about library loading, but from what I understand at https://github.com/Toblerity/rtree/issues/56 , if they hardcode "libspatialindex_c.so" in CDLL call, then it breaks in OSX, because its uses .dylib extension. And they don't want to reimplement find_library to choose the right extension for each platform.

> Does anyone have any valid use cases where they want to use a shared library on LD_LIBRARY_PATH or similar

I presume that would include this issue's creator and other people who have commented here about what they see as a drawback in find_library's current behaviour.

Pau Tallada's point about wanting to use a cross-platform invocation also seems reasonable. Remember, if you know the exact library you want to use, you don't *need* find_library: and this issue is about making find_library useful in a wider set of cases than it currently is.

> The problem I see with using find_library() to blindly load a library

Nobody is saying that the result of find_library() has to be used to blindly load a library. The point you make about the code in the uuid module is orthogonal to the proposal in this issue - even the behaviour of find_library never changed, that code could break for the reasons you give. For that, it's not unreasonable to raise a separate issue about possible fragility of the code in uuid.

I asked a question which I think is relevant to this enhancement request - "is emulating a build-time linker the most useful thing? In the context of Python binding to external libraries, why is build-time linking behaviour better than run-time linking behaviour?"

Do you have an answer to that?

Thanks Pau, for some reason I didn’t pick up the dylib OS X problem last time I looked at that link. This is a quick summary of the library names searched on different platforms:

Windows: {name} and {name}.dll, via %PATH%
OS X: lib{name}.dylib, {name}.dylib and {name}.framework/{name}, via dyld_find()
BSD: lib{name}.* via ldconfig (choose highest ABI version) and cc
Solaris: lib{name}.so via crle, and lib{name}.* via cc
Gnu: lib{name}.* via ldconfig and cc

Already, these cases seem to be half emulating the run-time linker and half the build-time linker. I don’t have a good answer about which is “better” (it would depend on the use case). But since we already have a mix, maybe changing towards run-time would not be such a problem as I first thought.

A half-cooked idea of mine is a simpler function that just accounts for common library naming conventions on various platforms, but does not extract sonames or spawn compilers. I think this could work with what seems to be a common use case of passing the result directly to CDLL(), which will do the real search. For the Windows and OS X cases, a loop might be required to attempt CDLL() with the different possibilities.

But I agree this is straying from the topic of this bug report. Perhaps I should accept that people want to find libraries by just the build-time name, but that are accessible at runtime.

More related reports about find_library() vs CDLL():
Issue 19317: Search Python’s RPATH
Issue 21622: Work without DT_SONAME

In https://bugs.python.org/issue26439 I have been working on this for AIX - as the default behavior was to depend on two things:
a) ldconfig -p (which generally does not exist on AIX, and I doubt it will know about the non-gnu libraries
b) that the objects are files rather than members of an archive

For python cdll() to work (as find_library generally did not) - the work-around was to extract archive members into standard directories and/or hard-code unique AIX names. But again, always as files, as CDLL did not modify the dlopen() mode to permit loading a shared library from an archive.

re: this issue and behavioral differences between find_library and cdll() (rather dlopen()) searches - a linker is different from a dlopen. When compiling an argument such as -lc -- on AIX -- tells the linker to look in libc.a for any member (of the right size) that satisfies an unknown symbol. That archive (base) and member (member) name is remembered in the "loader section" during linking. Note that static members have no loader section of their own, nor do they add one to the object (program or shared library) being linked.
As a compatibility feature - when a member is not found in an archive the AIX rtl (run-time-linker) also looks for a file named libFOO.so (e.g., libc.so) - this grace behavior is why CDLL() has worked somewhat using default behavior in Lib/ctypes.

As to LD_LIBRARY_PATH and LIBPATH. In AIX terms, these are environment variables to supplement the library search path - only when no PATH information is provided (i.e., cdll("./libc.so") will only look in the current directory of the process, and only for the file libc.so)
The default search path is a list of the directories used to link the executable (e.g., python). For me that list is:                         ***Import File Strings***
INDEX  PATH                          BASE                MEMBER
0      /usr/vac/lib:/usr/lib:/lib

In other words, python, by default, is only going to look in very standard locations - even though it knows it's prefix is not /usr.

There are two solutions - and I will make changes as appropriate.
a) automate setting an environment variable if it is not already set to something like 
 _os.environ['LIBPATH'] = "%s/lib" % _sys.prefix
and not something "arbitrary" such as /usr/local/lib
b) modify the LDFLAGS when packaging to include
-L${prefix}/lib which will have the effect of changing the default search path via -W,-blibpath=/usr/vac/lib:/usr/lib:/lib:${prefix}/lib


In any case, in my patch for AIX in ctypes the two will work identical (cdll() even calls aix.find_library() to simplify portability between GNU/posix environments that expect or support only filenames to work with AIX archive packaging.

My apologies for the wall of text. This is not a simple subject.

Getting back to this summary:
Windows: {name} and {name}.dll, via %PATH%
OS X: lib{name}.dylib, {name}.dylib and {name}.framework/{name}, via dyld_find()
BSD: lib{name}.* via ldconfig (choose highest ABI version) and cc
Solaris: lib{name}.so via crle, and lib{name}.* via cc
Gnu: lib{name}.* via ldconfig and cc

The "Gnu" selection is what is used for "None of the above", so also for AIX - which generally has neither ldconfig nor cc installed.

Having worked on a patch for AIX so that something that is (aka should) always there (dump) for examining archives for dlopen() openable objects.

In another issue I have had the benefit of lots of feedback from Martin Panter - most of which I agree with.

As this issue is about enhancement my feedback re: AIX behavior of dlopen() is that dlopen() takes LD_LIBRARY_PATH (when LIBPATH is not defined) into account.

In general find_library() is used as: CDLL(find_library("xxx")) - as in I do not care if the result is None or libxxx.so.y.z - just do it. However, if find_library() and CDLL() are written so that they "find" the same library object then it does become possible for a programmer to verify that a supported version is what will be loaded.

The return value from find_library() should be related to it's argument - just as dlopen() would 'react' (better, search *PATH*). If no path element ("./*" | "../*", or "/*") then no path element should be in the return value. The case for using path elements is because find_library() could be used to verity it's existence before calling CDLL() - what I did not know initially, but consider vital for proper behavior is that CDLL(NULL) just returns 'python' itself.

To underline that there are many issues that have been left unaddressed for years (this discussion here is nearly 6 years old) I mention:
"The case of python calling find_library("uuid") - on AIX this is probably NULL, as libuuid.so(.X) is not native to AIX - and even if it is present will not be found via a non-existent ldconfig or cc (only thing the "Gnu" option even considers).

IMHO: find_library should reflect dlopen() - with find_library being, as documented.

From the Python2.6 documentation (when introduced?)
"""The purpose of the find_library() function is to locate a library in a way similar to what the compiler does (on platforms with several versions of a shared library the most recent should be loaded), while the ctypes library loaders act like when a program is run, and call the runtime loader directly.

The ctypes.util module provides a function which can help to determine the library to load.

ctypes.util.find_library(name)
    Try to find a library and return a pathname. name is the library name without any prefix like lib, suffix like .so, .dylib or version number (this is the form used for the posix linker option -l). If no library can be found, returns None.

The exact functionality is system dependent."""

This pre-dates my experience with python, so if I inaccurate in assumptions - correct me, but please be patient with me.

What I miss is a PEP on this topic. From the limited reads of other PEPs I have read I think (rather hope) that inconsistencies in documentation could have been caught.

While (as Martin mentioned earlier in this discussion) find_library() behaves as "build" aka cc/gcc and CDLL follows "run-time" loader. imho, this is inconsistent - and the inconsistency is also in that short bit of documentation:

a) """The purpose of the find_library() function is to locate a library in a way similar to what the compiler does (on platforms with several versions of a shared library the most recent should be loaded) (i.e., always find the latest version)"""

and
b)"""The ctypes.util module provides a function which can help to determine the library to load.""" IMHO: how it is expected to be used because I (i.e., the python programmer) cannot provide additional specification to locate a specific version - find_library() is meant to find the latest always

As the Python3.5.2 documentation is nearly verbatim - this is still the documented condition.

So - I am very happy about Martin's (partial) comment:
maybe changing towards run-time would not be such a problem as I first thought.

CDLL() does a search, by definition. Maybe I do not care what it finds - but the argument to it is expected to platform dependent.
The "remaining" limitation of find_library(), even with searching *PATH* included is that it MUST also return the latest version (as a request for a specific version may not be made, or any other "extension"). An additional weakness is that what it "finds" must always be prefixed by "lib", while dlopen() has not such requirement. (FYI: I ran into these 'unusual' shared libraries while packaging sudo. While it is unlikely that python would ever load these sudo libraries it does show that there can be shared libraries that can accessed by not "found")

Hoping this helps the discussion - would be good to have at least a definition/documentation change so that decisions can be made.

I have updated the patch to apply against 3.6, and changed it to be more conservative: it only uses ld and LD_LIBRARY_PATH when trying to find a library if the existing gcc-based method fails.

Seeing that this issue has been around for so long, I would really like to get this patch committed in the short window we have before 3.6 goes into beta. Please review!

imho - it is not correct to only make a modification of this nature for a single platform.

To be realistic, if the "design" goal is to 'find' what dlopen() will find when given a argument without a pathname component - then gcc should not be used, and perhaps only your 'ld' based solution (which I have not read).

From linux man pages: both the justification to consider *PATH* for all platforms (i.e., LD_LIBRARY_PATH is a GNU/Linux construct, not the default for all platforms.)

"""
dlopen()

The function dlopen() loads the dynamic library file named by the null-terminated string filename and returns an opaque "handle" for the dynamic library. If filename is NULL, then the returned handle is for the main program. If filename contains a slash ("/"), then it is interpreted as a (relative or absolute) pathname. Otherwise, the dynamic linker searches for the library as follows (see ld.so(8) for further details):
o

(ELF only) If the executable file for the calling program contains a DT_RPATH tag, and does not contain a DT_RUNPATH tag, then the directories listed in the DT_RPATH tag are searched.

o

If, at the time that the program was started, the environment variable LD_LIBRARY_PATH was defined to contain a colon-separated list of directories, then these are searched. (As a security measure this variable is ignored for set-user-ID and set-group-ID programs.) 
""" end of excerpt

In short, this is more than just LD_LIBRARY_PATH. And my preference is that *PATH*, if predefined, should be taken into consideration by find_library. In other words, the purpose of find_library is to resolve platform naming conventions of shared libraries given a 'generic' argument, e.g., find_library("c") on Linux returns libc.so.6 while for AIX it (should return) libc.a(shr4.o).

As such, I would oppose a patch that only addresses the specifics of one platform. What is needed is a "design" clarification of the purpose of find_library. If we can agree on that "implementation" can follow. IMHO - trying to get a patch in for the convenience of one platform is not an architectural enhancement.

Again, I would like to see the documented behavior to be that find_library returns what dlopen() would open - when given an argument is the correct platform syntax. A patch for Linux does not - formally - guarantee that documented change.

In my view, the best should not be the enemy of the good, and pragmatism beats purity. I don't have the resources to test this functionality on all platforms - just Windows and Linux - and am not familiar with other platforms like AIX, Solaris or the BSDs. The documentation makes clear that the behaviour of find_library is system-dependent and makes no promises of a particular level of consistency, either with dlopen() or across platforms. Nor can we guarantee in the documentation that find_library() will exactly emulate dlopen(), since that may not hold on all platforms - and in fact, since Windows is in the mix, there is little point in trying to tie find_library() behaviour to that of dlopen() directly.

My documentation update makes clear that for Linux only, LD_LIBRARY_PATH will be searched if the existing mechanisms give no joy.

This request has been around since 2010, and in my view implementing this long-overdue patch will improve matters for Linux users and IIUC meet the goals of the issue creator and other commenters who concurred with his sentiment. This does not preclude improving the functionality on other platforms later, but I think we should implement this patch unless someone can point out that it makes things worse in some way. If anyone can improve it, that is also to be welcomed, of course.

> As such, I would oppose a patch that only addresses the specifics of one platform.

Why would improving behaviour on one platform, without any API changes or needing additional work by users, be worthy of opposition?

> What is needed is a "design" clarification of the purpose of find_library.
> If we can agree on that "implementation" can follow.

The purpose of find_library as currently documented seems adequately described, and the documentation update in my latest patch clarifies things further. IMO this is an area where the underlying platform features which find_library() relies on, as well as the run-time dynamic linking facilities available, are quite different across platforms. 

I'm not sure the agreement you seek will come any time soon, as it has not come in the last five years; it doesn't seem possible to aim for e.g. exact behaviour of dlopen(), because (a) it's not the same on Windows and (b) potentially varies too widely across POSIX platforms. Can you propose some change to the find_library() contract which you can assure will be implementable across all platforms? Certainly, fidelity with dlopen() isn't it.
 

> IMHO - trying to get a patch in for the convenience of one platform is not an architectural enhancement.

We're not trying for an architectural enhancement here, AFAIK.

I would welcome some input from others!

Added belopolsky and meador.inge to nosy as they are listed as ctypes experts in the experts index (Amaury was already there).

Vinay - I am in favor, but if the root cause of the holdup is unclear documentation then let's address the root cause.

Stronger, I am in agreement. The first patch I submitted for AIX included using LIBPATH (and could have added LD_LIBRARY_PATH) - but that was rejected.

> In my view, the best should not be the enemy of the good, and pragmatism beats purity. I don't have the resources to test this functionality on all platforms - just Windows and Linux - and am not familiar with other platforms like AIX, Solaris or the BSDs. The documentation makes clear that the behaviour of find_library is system-dependent and makes no promises of a particular level of consistency, either with dlopen() or across platforms.

* Again, basically I agree with the principles. I do not have the resources for all linux platforms, Solaris, BSD, Windows, etc..
* As far as "other platforms" and dlopen() behavior - from the man pages I have referenced I see more similarity than difference.
* behaviour of find_library is system-dependent - exactly - but unfortunately, for now AIX is whatever is in the else: box. That it works at all is a statement of the flexibility of AIX - and the sweat and tears of AIX admins to get the pieces to fit.

> Nor can we guarantee in the documentation that find_library() will exactly emulate dlopen(), since that may not hold on all platforms - and in fact, since Windows is in the mix, there is little point in trying to tie find_library() behaviour to that of dlopen() directly.

Good points. However, while it cannot be guaranteed - upfront - (although I would say that I could make them mimic each other on AIX given the leeway, better directive) - it can be stated as an objective for "posix".
:) Windows is windows - maybe someone who knows windows well enough could make something very close.

> The purpose of find_library as currently documented seems adequately described, and the documentation update in my latest patch clarifies things further.

Obviously, here is where we disagree. If it was 'adequately' described you could have used the documentation as a justification years ago. Your change touches on it, but still leaves a great deal to discuss - and especially - convince those who may actually commit.

> I'm not sure the agreement you seek will come any time soon, as it has not come in the last five years; it doesn't seem possible to aim for e.g. exact behaviour of dlopen(), because (a) it's not the same on Windows and (b) potentially varies too widely across POSIX platforms. Can you propose some change to the find_library() contract which you can assure will be implementable across all platforms? Certainly, fidelity with dlopen() isn't it.

As you mentioned earlier - the implementations are system-dependent - so yes, we could argue for find_library search behavior that emulates dlopen() (better perhaps "LoadLibrary()") - but not define any particular behavior - precisely because it may be different on different platforms. Again, the objective is to replicate that platform's LoadLibrary search behavior - whatever that may be.

Once we have established what is needed/desired as a goal, over time implementations will catch up. Linux will be leading the way because you have, presumably, already implemented it.

Michael

Updated the last patch with code for handling the case where ld is not available.

New changeset 385181e809bc by Vinay Sajip in branch 'default':
Closes #9998: Allowed find_library to search additional locations for libraries.
https://hg.python.org/cpython/rev/385181e809bc