On 02/03/2017 09:52 PM, Michael Felt wrote:
>> Anyway:
>> Unfortunately, there is no concept of embedding something like ELF's DT_SONAME tag into the Shared Object.
>> The very (PATH,BASE,MEMBER) value as (specified to and) discovered by the linker is recorded into the just-linked executable (or Shared Object).
>> This implies that the runtime loader does search for the very same filename (and member eventually) as the linker (at linktime).
> 
> I assume this is why there are many systems besides AIX that do not 
> support/use DT_SONAME.

Except for Windows, I'm not sure which "many systems besides AIX" you're talking here about, that "do not use/support DT_SONAME".

> At least I see many references to "Shared 
> Objects" libFOO.so.X.Y.Z, libFOO.so.X.Y, libFOO.so.X and libFOO.so (with 
> the latter three being symbolic links to the first).

When a system happens to find these symlinks useful, then it actually _does_ support embedding DT_SONAME (or something similar) into its binary file format.

> Another issue is support for what I believe MacOS calls "fat" objects - 
> that support both 32-bit and 64-bit applications - rather than /XXX/lib 
> for 32-bit objects and /XXX/lib/lib64 or /XXX/lib64 for 64-bit objects.

Yes, the AIX Archive Libraries supporting different bitwidths for members is quite similar to MacOS fat objects.
However - although related, the creation of "fat" AIX archives is a different topic.
But yes, Python ctypes.find_library+dlopen should know how to deal with them.

> b) One of the difficulties I faced is trying to guess what version -lFOO 
> should find when there is more than one version available.

Exactly. There is an idea below (the symbol->member map).

>> But still, how to get ctypes.find_library() working - ideally for each variant, is another story. Right now it does not work for any variant,
> Do you mean all systems, or specific to AIX - I am assuming you mean AIX.

Yes - find_library currently does not work for any variant on *AIX*.

>> but I guess that search algorithm should follow how the linker discovers the (PATH,BASE,MEMBER) values to

> I am not a tool builder. My comments are based on observations and 
> experience from when I was a developer 25+ years ago. The AIX linker is 
> not interested in the member name - it seems to go through the 
> PATH/libBASE.a looking for the first object it can find to resolve a 
> symbol. The name of the object it finds becomes the MEMBER it records in 
> it's internal table of where to look later when the application runs.

Exactly.

>>   write into just-linked executables, combined with how the runtime loader finds the Shared Object to actually load.

> I worked on a patch - to do all that - taking into consideration the way 
> libtool names .so files/members and then looking into/at "legacy" aka 
> IBM dev ways they did things before the libtool model was so prominent.
> 
> My algorithm - attempts to solve the (PATH, BASE, MEMBER) problem as 
> "dynamically" as possible. PATH and BASE are fairly straight forward - 
> but MEMBER is clearly more complex.
> 
> PATH: start by looking at the python executable -

As far as I can tell, any executable can actually link against the Python interpreter.

> and looking at it's  "blibpath" -

There also is the loadquery() subroutine in AIX, see https://www.ibm.com/support/knowledgecenter/ssw_aix_72/com.ibm.aix.basetrf1/loadquery.htm

loadquery(L_GETLIBPATH) "Returns the library path that was used at process exec time.",
which includes both the environment variable LIBPATH (or LD_LIBRARY_PATH if LIBPATH is unset) and the executable's "blibpath" value.

> and using that as the default colon separated list of PATHs

Question is if we do want to consider _current_ values of environment variable LIBPATH (or LD_LIBRARY_PATH) in addition to the "library path at process exec time"?

> to search for BASE.a archive. Once a BASE.a file is found it is examined 
> for a MEMBER. If all PATH/BASE.a do not find a potential MEMBER then the 
> PATHs are examined again for PATH/BASE.so.

Erm, nope, the AIX linker has a different algorithm (for -lNAME):
Iterating over the "library path", the first path entry containing any matching filename (either libNAME.a or libNAME.so) will be used, and no further library path iteration is performed.
This one found PATH/filename does have to provide the requested symbol in one way or another.

> When a .so file is found that 
> is returned - versioning must be accomplished via a symbolic link to a 
> versioned library.

The linker does not perform such a check, nor does it feel necessary for ctypes.find_library+dlopen as long as it does search similar to the linker.

> The program "dump -H" provides this information for both executables and 
> archive (aka BASE) members.

Eventually we might want to avoid spawning the 'dump' program, but implement reading the XCOFF Object File Format within _ctypes module instead.
At least AIX does provide the necessary headers: https://www.ibm.com/support/knowledgecenter/ssw_aix_72/com.ibm.aix.files/XCOFF.htm

> Starting from the "blibpath" values in the executable mean a cpython 
> packager can add a specific PATH by adding it to 
> LDFLAGS="-L/my/special/libdir:$LDFLAGS". Note that AIX archives also 
> have their own "blibpath" - so libraries dynamically loaded may also 
> follow additional paths that the executable is not aware of (nor need to 
> be).

There is no need for the ctypes module to search libpaths from other Shared Objects than the main executable (and current env vars).

> So - once the PATHS are determined the system is examined looking for 
> ${PATH}/BASE.a. If a target BASE.a is found, it is examined for a MEMBER 
> is set to BASE.so (now used a MEMBER.so) . If MEMBER.so is not found 
> then look for the "highest X[.Y[.Z]] aka MEMBER.so.X[.Y[.Z]] name. If 
> that is not found check AIX legacy names (mainly shr.o or shr_64.o, 
> although there are also certain key libraries that have additional 
> variations (boring)).

When ctypes.dlopen is asked to load an Archive Library (either .a or .so) without a specific member, it probably should not immediately dlopen a specific member, but fetch the list of symbols provided by useable members (either Shared Objects without the F_LOADONLY flag, as well as specified in Import Files), and return the handle to some internal symbol->member map instead.

Then, really loading a shared archive member is done by subsequent ctypes.dlsym - where it becomes clear which archive member to load.

> Again, if PATH, BASE, MEMBER is not located as a .a archive - look for 
> libFOO.so in all the PATH directories known to the executable.

Nope, see above - please iterate over a libpath list only _once_, and search for each filename while at one path list entry.

However, I'm not sure yet how to identify if we should search for .a or .so first (before .so and .a, respectively):
This depends on whether the current executable actually does use runtime linking or not, but I have no idea yet how to figure that out.
But probably we may want to use how the Python interpreter (libpython.a or libpython.so) was built.

Uhm, ultimative solution feels complex already, while still some things to decide...