See below between points.

On 12-May-16 17:45, Martin Panter wrote:
> Martin Panter added the comment:
>
> Thanks for trying to move the directory in the new patch, but it does not seem to have been picked up by Rietveld. I have only seen patches generated with Mercurial and Git work, but I don’t exactly know _why_ they work :)
>
> Here is a summary of what I would be comfortable adding to Python:
>
> 1. Add ctypes.RTLD_MEMBER constant (new API, so 3.6+ only)
Understood. Should also add RTLD_NOW as it is referenced in comments, 
and used in the _global.c (imho).
> 2. Basic ctypes.util.find_library("crypto") implementation for AIX. But cases like find_library("libintl.so") should fail. I think your earlier patches were closer to this implementation than the later ones.
>
> I am a bit hesitant about the automatic behaviour of CDLL("libcrypto.a(libcrypto.so.1.0.0)") using RTLD_MEMBER. IMO it may be better to let the caller specify RTLD_MEMBER explicitly. If a shared library file literally called “/usr/lib/libcrypto.a(libcrypto.so.1.0.0)” existed, i.e. not inside an archive, would dlopen("libcrypto.a(libcrypto.so.1.0.0)", RTLD_NOW) succeed? I admit this is an unlikely scenario, but it seems bad to reduce the domain of a low-level API.
>
> I understand it would be good to have the return value of find_library() consistent with the name accepted by CDLL(). Perhaps a new parameter format would help, such as a tuple (archive, member).
I would expect a new tuple would be seen as new API (like 
_ctypes.RTLD_MEMBER), so it would never help in Python-2.7, which is 
still where I see most people asking for support with AIX.
My guidance was: find_library("m") is legal, as is 
find_library("libm.so"), as is CDLL("libcrypt.so").

These are all dependencies, at a low-level of how the system is 
configured, i.e., it is not uncommon for a GNU/Linux installation to 
have many symbolic links to resolve multiple names, and programmers use 
one, or more of those conventions. The find_library() and CDLL() calls 
work because of these conventions.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Thu May 19 02:53:48 2016 from xxx.yyy.zzz.net 
<http://76-217-205-93.lightspeed.sndgca.sbcglobal.net>
root@x066:~# python -m ctypes.util
libm.so.6
libc.so.6
libbz2.so.1.0
<CDLL 'libm.so', handle f7c0f5e0 at 104493d0>
<CDLL 'libcrypt.so', handle 1045b2d8 at 104493d0>
libcrypt.so.1

Looking at filesystem level:
root@x066:~# cd /lib64
root@x066:/lib64# ls -l libcrypt.*
lrwxrwxrwx 1 root root 16 Feb 22  2015 libcrypt.so.1 -> libcrypt-2.13.so 
<http://libcrypt-2.13.so>
root@x066:/lib64# cd /usr/lib64
root@x066:/usr/lib64# ls -l libcrypt.*
-rw-r--r-- 1 root root 62108 Feb 22  2015 libcrypt.a
lrwxrwxrwx 1 root root    20 Feb 22  2015 libcrypt.so -> 
/lib64/libcrypt.so.1

So, CDLL.dlopen() finds /usr/lib64/libcrypt.so that is a symbolic link 
to /lib64/libcrypt.so.1 that is a symbolic link to 
/lib64/libcrypt-2.13.so <http://libcrypt-2.13.so>. I see libcrypt.so as 
a "shorthand" for whatever is installed. As a porgrammer/user of 
CDLL.dlopen() I do not care what it finally resolves to.

Linux has it's conventions: one important one I observe when porting OSS 
packages is that "installation" and updates of libraries (or API's) 
provide "shortnames" in the form of libFOO.so  that are symbolic names 
to the final object. Unfortunately, there are exceptions - especially 
look at "libc.so"

root@x066:/usr/lib64# ls -l libc.so*
-rw-r--r-- 1 root root 243 Feb 22  2015 libc.so

root@x066:/usr/lib64# cd /lib64
root@x066:/lib64# ls -l libc.so*
lrwxrwxrwx 1 root root 12 Feb 22  2015 libc.so.6 -> libc-2.13.so
root@x066:/lib64# ls -l libc-2.13.so
-rwxr-xr-x 1 root root 1758016 Feb 22  2015 libc-2.13.so

Obviously, using shorthand for libc.so on Debian is not returning 
anything equivalent to libc.so.6 (look at the different sizes). So, a 
program that looks specifically for libc.so.6 will succeed on Debian, 
and fail on AIX (as it does not exist in that form) - and my expectation 
is that existing programs that code libc.so.6 will fail on AIX. However, 
I have extreme doubts that anyone actually codes the test - as (at least 
on AIX) libc is already loaded.

AIX:
root@x071:[/root]ldd /opt/bin/python
/opt/bin/python needs:
          /usr/lib/libc.a(shr.o)
          /usr/lib/libpthreads.a(shr_xpg5.o)
          /usr/lib/libpthreads.a(shr_comm.o)
          /usr/lib/libdl.a(shr.o)
          /unix
          /usr/lib/libcrypt.a(shr.o)

And, it seems, libc is also already loaded on Debian:
root@x066:/lib64# ldd /usr/bin/python
         linux-vdso32.so.1 =>  (0x00100000)
         libpthread.so.0 => /lib/powerpc-linux-gnu/libpthread.so.0 
(0x0ffc5000)
         libdl.so.2 => /lib/powerpc-linux-gnu/libdl.so.2 (0x0ffa1000)
         libutil.so.1 => /lib/powerpc-linux-gnu/libutil.so.1 (0x0ff7e000)
         libz.so.1 => /lib/powerpc-linux-gnu/libz.so.1 (0x0ff47000)
         libm.so.6 => /lib/powerpc-linux-gnu/libm.so.6 (0x0fe79000)
         libc.so.6 => /lib/powerpc-linux-gnu/libc.so.6 (0x0fce0000)
         /lib/ld.so.1 (0x20231000)

So back to my thoughts as I developed this "behavior" patch for 
ctypes/util aka ctypes/cdll:
a) to be transparent - mask (but not hide) AIX "internals" that are 
(probably) a level of detail programmers are not interested in. That is 
what I imagined the original design goal of the Lib addition.
b) to support existing code ASIS (the first two python packages I looked 
at being cloud-init and salt-stack)
c) as a bonus: potentially resolve any performance "bugs" caused by the 
lack of /bin/ldconfig

AIX has it's conventions - the main one being to look in archives (.a 
files) for members. To signify that it is an archive member an 
additional dlopen() mode is used. Without this flag, dlopen() looks for 
a file - just like other "posix" platforms - period.

re: your objections/concerns
a) I am shooting myself for being so visible - as I have done a lot of 
porting and, in particular, "intl" aka libintl.so support is important 
for programmers that are expecting GNU gettext support for multiple 
languages.
These seems pointless on a generic Debian install - because while 
gettext is available, there is no "libintl" to be found (using either 
gcc or ldconfig -p to search), e.g.:

root@x066:/lib64# ldconfig -p | grep intl
root@x066:/lib64# gettext --version
gettext (GNU gettext-runtime) 0.18.1

The tests re: libcrypto I added because behavior re: libssl and 
libcrypto is important. Many applications need that. (libcrypt is not 
"SSL" support).

So, back to what I think are your objections:
a) seeing a name ending in ")" as being the AIX "native" format and 
adding RTLD_MEMBER without any further test of it being a file.
b) expanding a "generic" name ending in .so to a latest version IF it 
does not exist as .so in an archive, or as a file.

re: a) a test could be added to verify that a file with a name ending 
with ")") does not exist before adding the RTLD_MEMBER mode - this is 
only being considered on AIX (that condition must be determined first 
anyway) and, as you say - it is unlikely to be the case. So, a test 
could be added - that would probably always fail - slowing calls down.

re: b) I consider it better practice (and maybe this is my arrogance of 
35 years + of supporting multiple platforms in libraries (now API) 
transparently. I am the weirdo who likes to dive into low-level, 
bit-level "idiosyncrasies" of different os systems and/or levels. 
"Normal" people do not care a #$%! - they just want it to work.

Again, imho a key issue that needs to be addressed is to get as much 
code to work ASIS - as much as possible - rather than the current 
situation (without my proposed patch) the requires many changes must be 
made to AIX, as root, and these must also be maintained (performed 
again) after any update to the OS - at least controlled. This is not 
facilitate using python, better python acceptance (on AIX) - and seems 
conducive to security breaches.

Sincerely,
Michael
> I am not comfortable with other aspects. I think I would have to see more discussion with other people to change my opinion:
>
> 1. CDLL("libintl.so") should not load “libintl.a(libintl.so.1)”. I understand you want this to help code written for Gnu or Linux automatically work on AIX, but it doesn’t feel correct and robust to me. Perhaps moving this sort of thing to a separate function or package would be better.
>
> 2. find_library("libintl.so") -> "libintl.a(libintl.so.1)". I would expect it to look for a shared library installed in something like "/usr/lib/liblibintl.so.a", unless I have misunderstood how compile-time linking (cc -llibintl.so) works.
See above - again, shooting myself for having included these additional 
"behavior" examples.

> 3. find_library() should not set the LIBPATH environment variable.
This was a convenience. It may be better to have the default search path 
hard-coded using a -Wl,blibpath link flag. As the discussion re: Linux 
support was to not include it (or was it remove it after it had been 
added) . shall follow and "Will remove" to be compareable.
> ----------
>
> _______________________________________
> Python tracker<report@bugs.python.org> <mailto:report@bugs.python.org>
> <http://bugs.python.org/issue26439> <http://bugs.python.org/issue26439>
> _______________________________________