I have successful enough with python 2.7.10 (for building cloud-init), including it finding openssl libraries during the installation od setuptools (before installing pip).

I have also been able to assemble saltstack - BUT - salt-master and salt-minion fail to start because ctypes.util.find_library() always returns 'None'.
======== EXCERPT ====
  File "/opt/lib/python2.7/site-packages/salt/crypt.py", line 37, in <module>
    import salt.utils.rsax931
  File "/opt/lib/python2.7/site-packages/salt/utils/rsax931.py", line 69, in <module>
    libcrypto = _init_libcrypto()
  File "/opt/lib/python2.7/site-packages/salt/utils/rsax931.py", line 47, in _init_libcrypto
    libcrypto = _load_libcrypto()
  File "/opt/lib/python2.7/site-packages/salt/utils/rsax931.py", line 40, in _load_libcrypto
    raise OSError('Cannot locate OpenSSL libcrypto')
OSError: Cannot locate OpenSSL libcrypto
=======
I built python using the IBM compiler, and my images do not have /sbin/ldconfig installed so the assumption that /sbin/ldconfig is always installed is "a bug".

in the util.py file the code reached is:

  +219      else:
  +220
  +221          def _findSoname_ldconfig(name):
  +222              import struct
  +223              if struct.calcsize('l') == 4:
  +224                  machine = os.uname()[4] + '-32'
  +225              else:
  +226                  machine = os.uname()[4] + '-64'
  +227              mach_map = {
  +228                  'x86_64-64': 'libc6,x86-64',
  +229                  'ppc64-64': 'libc6,64bit',
  +230                  'sparc64-64': 'libc6,64bit',
  +231                  's390x-64': 'libc6,64bit',
  +232                  'ia64-64': 'libc6,IA-64',
  +233                  }
  +234              abi_type = mach_map.get(machine, 'libc6')
  +235
  +236              # XXX assuming GLIBC's ldconfig (with option -p)
  +237              expr = r'\s+(lib%s\.[^\s]+)\s+\(%s' % (re.escape(name), abi_type)
  +238              f = os.popen('/sbin/ldconfig -p 2>/dev/null')
  +239              try:
  +240                  data = f.read()
  +241              finally:
  +242                  f.close()
  +243              res = re.search(expr, data)
  +244              if not res:
  +245                  return None
  +246              return res.group(1)
  +247
  +248          def find_library(name):
  +249              return _findSoname_ldconfig(name) or _get_soname(_findLib_gcc(name))
  +250

(I have not researched _get_soname or _findLib_gcc but neither of these "feel right" as AIX, by default, does not end library (archives) with .so (archives end with .a and archive members frequently end with .so)

That this is/has not been reported more frequently may be because python programmers are avoiding ctypes - when portability is essential.

I hope that, just as for Solaris - where an alternate program is used - that AIX can have a block:
 # if os.name == "posix" and sys.platform.startswith("aix"):

so that if ldconfig is not available the command /usr/bin/dump could be used instead, and/or search LIBPATH and/or (when not empty) and/or ldd (for programs). Ideally, /sbin/ldconfig would not be need at all!

dump output:

root@x064:[/data/prj/gnu/bashRC1-4.4]dump -Xany -H /opt/bin/python

/opt/bin/python:

                        ***Loader Section***
                      Loader Header Information
VERSION#         #SYMtableENT     #RELOCent        LENidSTR
0x00000001       0x000005ac       0x000035e3       0x0000006e       

#IMPfilID        OFFidSTR         LENstrTBL        OFFstrTBL
0x00000005       0x00030ee4       0x00006772       0x00030f52       


                        ***Import File Strings***
INDEX  PATH                          BASE                MEMBER              
0      /usr/vac/lib:/usr/lib:/lib                                            
1                                    libc.a              shr.o               
2                                    libpthreads.a       shr_xpg5.o          
3                                    libpthreads.a       shr_comm.o          
4                                    libdl.a             shr.o        

root@x064:[/usr/bin]dump -Xany -H /usr/lib/libcrypto.a

/usr/lib/libcrypto.a[libcrypto.so]:

                        ***Loader Section***
                      Loader Header Information
VERSION#         #SYMtableENT     #RELOCent        LENidSTR
0x00000001       0x00000ff9       0x0000498a       0x00000038       

#IMPfilID        OFFidSTR         LENstrTBL        OFFstrTBL
0x00000003       0x0004f1f0       0x00014636       0x0004f228       


                        ***Import File Strings***
INDEX  PATH                          BASE                MEMBER              
0      /usr/lib:/lib                                                         
1                                    libc.a              shr.o               
2                                    libpthreads.a       shr_xpg5.o          

/usr/lib/libcrypto.a[libcrypto.so.0.9.8]:

                        ***Loader Section***
                      Loader Header Information
VERSION#         #SYMtableENT     #RELOCent        LENidSTR
0x00000001       0x00000c4e       0x00004312       0x00000038       

#IMPfilID        OFFidSTR         LENstrTBL        OFFstrTBL
0x00000003       0x00044c48       0x0000f236       0x00044c80       


                        ***Import File Strings***
INDEX  PATH                          BASE                MEMBER              
0      /usr/lib:/lib                                                         
1                                    libc.a              shr.o               
2                                    libpthreads.a       shr_xpg5.o          

/usr/lib/libcrypto.a[libcrypto.so.1.0.0]:

                        ***Loader Section***
                      Loader Header Information
VERSION#         #SYMtableENT     #RELOCent        LENidSTR
0x00000001       0x00000ff9       0x0000498a       0x00000038       

#IMPfilID        OFFidSTR         LENstrTBL        OFFstrTBL
0x00000003       0x0004f1f0       0x00014636       0x0004f228       


                        ***Import File Strings***
INDEX  PATH                          BASE                MEMBER              
0      /usr/lib:/lib                                                         
1                                    libc.a              shr.o               
2                                    libpthreads.a       shr_xpg5.o          

/usr/lib/libcrypto.a[libcrypto64.so]:

                        ***Loader Section***
                      Loader Header Information
VERSION#         #SYMtableENT     #RELOCent        LENidSTR
0x00000001       0x00000ff2       0x00004987       0x0000003e       

#IMPfilID        OFFidSTR         LENstrTBL        OFFstrTBL
0x00000003       0x00061758       0x00014e03       0x00061796       


                        ***Import File Strings***
INDEX  PATH                          BASE                MEMBER              
0      /usr/lib:/lib                                                         
1                                    libc.a              shr_64.o            
2                                    libpthreads.a       shr_xpg5_64.o       

/usr/lib/libcrypto.a[libcrypto64.so.0.9.8]:

                        ***Loader Section***
                      Loader Header Information
VERSION#         #SYMtableENT     #RELOCent        LENidSTR
0x00000001       0x00000c47       0x0000430d       0x0000003e       

#IMPfilID        OFFidSTR         LENstrTBL        OFFstrTBL
0x00000003       0x000557b0       0x0000f9e7       0x000557ee       


                        ***Import File Strings***
INDEX  PATH                          BASE                MEMBER              
0      /usr/lib:/lib                                                         
1                                    libc.a              shr_64.o            
2                                    libpthreads.a       shr_xpg5_64.o       

/usr/lib/libcrypto.a[libcrypto64.so.1.0.0]:

                        ***Loader Section***
                      Loader Header Information
VERSION#         #SYMtableENT     #RELOCent        LENidSTR
0x00000001       0x00000ff2       0x00004987       0x0000003e       

#IMPfilID        OFFidSTR         LENstrTBL        OFFstrTBL
0x00000003       0x00061758       0x00014e03       0x00061796       


                        ***Import File Strings***
INDEX  PATH                          BASE                MEMBER              
0      /usr/lib:/lib                                                         
1                                    libc.a              shr_64.o            
2                                    libpthreads.a       shr_xpg5_64.o       
root@x064:[/usr/bin]
root@x064:[/usr/bin]echo $?                           
0


ldd outpath:
root@x064:[/usr/bin]ldd /opt/bin/python
/opt/bin/python needs:
         /usr/lib/libc.a(shr.o)
         /usr/lib/libpthreads.a(shr_xpg5.o)
         /usr/lib/libpthreads.a(shr_comm.o)
         /usr/lib/libdl.a(shr.o)
         /unix
         /usr/lib/libcrypt.a(shr.o)

root@x064:[/usr/bin]ldd /usr/lib/libcrypto.a
ldd: /usr/lib/libcrypto.a: File is an archive.
root@x064:[/usr/bin]echo $?                 
2

p.s. On a debian (on POWER) system, the function is working, but the test seems a bit broken as well, i.e., cdll.LoadLibrary("libm.so") is not working even though 

    if os.name == "posix":
        # find and load_version
        print find_library("m")
        print find_library("c")
        print find_library("bz2")

has successfully printed.
root@ipv4:/home/michael# python -m ctypes.util
libm.so.6
libc.so.6
libbz2.so.1.0
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/lib/python2.7/ctypes/util.py", line 287, in <module>
    test()
  File "/usr/lib/python2.7/ctypes/util.py", line 282, in test
    print cdll.LoadLibrary("libm.so")
  File "/usr/lib/python2.7/ctypes/__init__.py", line 443, in LoadLibrary
    return self._dlltype(name)
  File "/usr/lib/python2.7/ctypes/__init__.py", line 365, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libm.so: cannot open shared object file: No such file or directory
root@ipv4:/home/michael# uname -a
Linux ipv4.rootvg.net 3.16.0-4-powerpc64 #1 SMP Debian 3.16.7-ckt9-3 (2015-04-23) ppc64 GNU/Linux
root@ipv4:/home/michael# ldconfig -p | grep libm.so
        libm.so.6 (libc6, OS ABI: Linux 2.6.32) => /lib/powerpc-linux-gnu/libm.so.6
root@ipv4:/home/michael# ls -l /lib/powerpc-linux-gnu/libm.so.6
lrwxrwxrwx 1 root root 12 Apr 15  2015 /lib/powerpc-linux-gnu/libm.so.6 -> libm-2.19.so
root@ipv4:/home/michael# ls -l /lib/powerpc-linux-gnu/libm-2.19.so
-rw-r--r-- 1 root root 743784 Apr 15  2015 /lib/powerpc-linux-gnu/libm-2.19.so

Further testing...

I added an extremely simple hack in util.py so that a archive (AIX library) name got returned.

Also in this case - the     print cdll.LoadLibrary("libc.a") fails.

   +74  if os.name == "posix" and sys.platform == "darwin":
   +75      from ctypes.macholib.dyld import dyld_find as _dyld_find
   +76      def find_library(name):
   +77          possible = ['lib%s.dylib' % name,
   +78                      '%s.dylib' % name,
   +79                      '%s.framework/%s' % (name, name)]
   +80          for name in possible:
   +81              try:
   +82                  return _dyld_find(name)
   +83              except ValueError:
   +84                  continue
   +85          return None
   +86
   +87  if os.name == "posix" and sys.platform.startswith("aix"):
   +88       def find_library(name):
   +89          expr = "lib" + name + ".a"
   +90          return expr
   +91
   +92  elif os.name == "posix":
   +93      # Andreas Degert's find functions, using gcc, /sbin/ldconfig, objdump
   +94      import re, tempfile, errno
   +95
   +96      def _findLib_gcc(name):

  +275          else:
  +276              print cdll.LoadLibrary("libc.a")
  +277              print cdll.LoadLibrary("libm.so")
  +278              print cdll.LoadLibrary("libcrypt.so")

root@x064:[/tmp]python -m ctypes.util
libm.a
libc.a
libbz2.a
Traceback (most recent call last):
  File "/opt/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/opt/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/opt/lib/python2.7/ctypes/util.py", line 282, in <module>
    test()
  File "/opt/lib/python2.7/ctypes/util.py", line 276, in test
    print cdll.LoadLibrary("libc.a")
  File "/opt/lib/python2.7/ctypes/__init__.py", line 443, in LoadLibrary
    return self._dlltype(name)
  File "/opt/lib/python2.7/ctypes/__init__.py", line 365, in __init__
    self._handle = _dlopen(self._name, mode)
OSError:        0509-022 Cannot load module /usr/lib/libc.a.
        0509-103   The module has an invalid magic number.

So, what needs to be returned so that cdll.LoadLibrary could use that result? (e.g., I know that the default libm.a has no shared members on AIX 5.3 TL7 - but libc.a does (shr.o and more, but not all!).

Last message (back to debian, and minor changes to learn expected behavior)

        if sys.platform == "darwin":
            print cdll.LoadLibrary("libm.dylib")
            print cdll.LoadLibrary("libcrypto.dylib")
            print cdll.LoadLibrary("libSystem.dylib")
            print cdll.LoadLibrary("System.framework/System")
        else:
            print cdll.LoadLibrary("libm.so.6")
#            print cdll.LoadLibrary("libcrypt.so")
            print find_library("crypt")
            x = find_library("crypt")
            print cdll.LoadLibrary(x)

returns:
root@ipv4:/home/michael# python -m ctypes.util
libm.so.6
libc.so.6
libbz2.so.1.0
<CDLL 'libm.so.6', handle f7e65528 at f7b51fb0>
libcrypt.so.1
<CDLL 'libcrypt.so.1', handle 106a1878 at f7b51fb0>

FYI: getting objdump is not too hard...

root@x064:[/data/prj/gnu/binutils-2.25.1]v/null /usr/lib/libcrypto.a         <
In archive /usr/lib/libcrypto.a:

libcrypto.so:     file format aixcoff-rs6000


libcrypto.so.0.9.8:     file format aixcoff-rs6000


libcrypto.so.1.0.0:     file format aixcoff-rs6000


libcrypto64.so:     file format aix5coff64-rs6000


libcrypto64.so.0.9.8:     file format aix5coff64-rs6000


libcrypto64.so.1.0.0:     file format aix5coff64-rs6000

But ldconfig (as a command within glibc) will be a real headache - and I would home unnecessary.

In closing, I hope the AIX command /usr/bin/dump will be adequate as an alternative of objdump.

Where I am still a bit lost is with the "open", i.e. _dlopen(). Suggestions/hints for debugging are welcome.

The _dlopen call in __init__.py I have been able to fix (hack) with the following:

root@x064:[/data/prj/aixtools/python/python-2.7.10/Lib/ctypes]diff -u __init__.py /opt/lib/python2.7/ctypes/__init__.py
--- __init__.py 2015-05-23 16:09:01 +0000
+++ /opt/lib/python2.7/ctypes/__init__.py       2016-02-26 08:40:19 +0000
@@ -11,6 +11,7 @@
 from _ctypes import _Pointer
 from _ctypes import CFuncPtr as _CFuncPtr
 from _ctypes import __version__ as _ctypes_version
+# from _ctypes import RTLD_LOCAL, RTLD_GLOBAL, RTLD_NOW ## fails
 from _ctypes import RTLD_LOCAL, RTLD_GLOBAL
 from _ctypes import ArgumentError
 
@@ -32,6 +33,11 @@
     if int(_os.uname()[2].split('.')[0]) < 8:
         DEFAULT_MODE = RTLD_GLOBAL
 
+if _os.name == "posix" and _sys.platform.startswith("aix"):
+        RTLD_NOW    = 0x00000002
+        RTLD_MEMBER = 0x00040000
+        DEFAULT_MODE |= (RTLD_NOW | RTLD_MEMBER)
+
 from _ctypes import FUNCFLAG_CDECL as _FUNCFLAG_CDECL, \
      FUNCFLAG_PYTHONAPI as _FUNCFLAG_PYTHONAPI, \
      FUNCFLAG_USE_ERRNO as _FUNCFLAG_USE_ERRNO, \

I have an additional hack in util.py so that, e.g.,
find_Library("crypto")

returns:
 '/usr/lib/libcrypto.a(libcrypto.so)'

When that is passed to dlopen (plus RTLD_MEMBER) the dlopen succeeds.

<CDLL '/usr/lib/libcrypto.a/(libcrypto.so)', handle 6 at 30146670>

With some help in util.py, to do some sensible searching, being able to establish if in 32 or 64-bit mode, etc. somehing nice can be made here (imho).

p.s. can any verify whether this is limited to python 2.7? Or should python 3.X be added as well?

ctypes util.py "simply" needs support for AIX.  There already is special support for Windows, Darwin, BSDs, Solaris.  Is the question about the technical details for equivalent functionality on AIX or about adding a stanza to Lib/ctpyes/util.py?

it was, partly, about the technical details - but I feel I have found the key bits - /full/path/libNAME.a(libNAME.so) + dlflags RTLD_NOW + RTLD_MEMBER for "native" AIX support.

Additionally, a patch should not break what might be working for some (via google I read of suggestions for other languages, e.g., java, where .so members are extracted or installed side-by-side with the .a archive) - as the IBM run-rime loader also accepts/looks for both .a and .so. And/or for when /sbin/ldconfig is available. And when available, is this used as last case, or preferred?

Lastly, we cannot assume we will know the name of the member based on the name of the library. In many cases, e.g., libiconv.a the IBM names are shr.o, shr4.o and shr4_64.o - and these are the only member names unless GNU libiconv.a has been added (and then libconv.a also contains libiconv.so.2 (so version support is also desired!)) in both 32 and 64 bit mode.

One technical detail I have not been able to discover yet: how to determine whether in 32-bit or 64-bit mode. This will be important for libraries that have shr4.o and shr4_64.o as the members that need to be dlopened.

Lastly: about adding the "stanza": python is a new language for me. Any assistance with a patch - to keep it properly 'pythonized'

In short, I shall continue my studies/learning. Assistance from anyone wiser than me is welcome!

AIX traditionally used member names like shr.o or shr<version>.o or shr<aix_release>.o insider the archive, with _64 designating a 64 bit object when there is a naming collision.

GNU libtool defaults to the SO name and version number insider the archive.

AIX objects (and shared objects) contain a bit in the header that specifies 32 bit or 64 bit.  Both 32 bit and 64 bit objects are intended to be archived together.  The linker only processes objects of the correct mode.

AIX shared objects contain a bit that specifies if the object may be used at link-edit time or only should be used for loading.  This is controlled by the AIX strip -e/-E option (yes, I know, strange place to hide that option).

This combination of features allows all of the libraries to be placed in a single /usr/lib directory and all of the objects to be collected into a single archive, avoiding /usr/lib64 and explosion of shared objects and symbolic links clutter.  Various packages have created /usr/local/lib64 anyway using Linux/Solaris/SVR4-style naming.

The following demonstrates the basics.

cdll.LoadLibrary will load a .so file if it can be located (you will have to believe me as I forgot to include the test in this last batch)

Further, when given the AIX format for loading a member of an archive
(e.g., libc.a(shr.o) or libcrypto.a(libcrypto.so), the dlload occurs as expected.

in ctypes/utils.py
This looks in the standard directories (as the example, but not complete, see below) to demonstrate how to respond.
Note: it is also looking for a .so file, and if it finds one, it returns the .so name, otherwise it looks for a .a file - and when it finds it, just appends the libNAME.so in the correct format.

What is not done, and probably different from ldconfig behavior!

1. The complete path is not returned (so cdll.LoadLibrary will still follow default search - which I hope includes, read uses, the default for the python executable

2. The member name is not verified for existence (e.g., libconv.a(libiconv.so) is not expected to find libiconv.a(libiconv.so.2)

3. Since member verification is not being performed, shr.o, shr4.o, shr*_64.o

And many many thanks for the strip -e/-E info. Hard to find!

root@x064:[/data/prj/aixtools/src]diff -u python-2.7.11/Lib/ctypes/__init__.py /opt/lib/python2.7/ctypes/__init__.py
--- python-2.7.11/Lib/ctypes/__init__.py        2015-12-05 19:46:56 +0000
+++ /opt/lib/python2.7/ctypes/__init__.py       2016-02-29 17:36:43 +0000
@@ -11,6 +11,7 @@
 from _ctypes import _Pointer
 from _ctypes import CFuncPtr as _CFuncPtr
 from _ctypes import __version__ as _ctypes_version
+# from _ctypes import RTLD_LOCAL, RTLD_GLOBAL, RTLD_NOW ## fails
 from _ctypes import RTLD_LOCAL, RTLD_GLOBAL
 from _ctypes import ArgumentError
 
@@ -356,6 +357,14 @@
         if use_last_error:
             flags |= _FUNCFLAG_USE_LASTERROR
 
+        if _sys.platform.startswith("aix"):
+            RTLD_NOW    = 0x00000002
+            RTLD_MEMBER = 0x00040000
+            mode |= RTLD_NOW
+            if name is not None:
+                if name[-1] == ')':
+                       mode |= RTLD_MEMBER
+
         class _FuncPtr(_CFuncPtr):
             _flags_ = flags
             _restype_ = self._func_restype_


root@x064:[/data/prj/aixtools/src]diff -u python-2.7.11/Lib/ctypes/util.py /opt/lib/python2.7/ctypes/util.py
--- python-2.7.11/Lib/ctypes/util.py    2015-12-05 19:46:56 +0000
+++ /opt/lib/python2.7/ctypes/util.py   2016-02-29 17:42:41 +0000
@@ -84,6 +84,23 @@
                 continue
         return None
 
+if os.name == "posix" and sys.platform.startswith("aix"):
+        def _findLib_aix(name):
+            paths='/usr/lib:/lib:/usr/lpp/xlC/lib'
+            for dir in paths.split(":"):
+                shlib = os.path.join(dir, "lib%s.so" % name)
+                if os.path.exists(shlib):
+                    return shlib
+                libfile = os.path.join(dir, "lib%s.a" % name)
+                if os.path.exists(libfile):
+                    shlib = "lib%s.a(lib%s.so)" % (name, name)
+                    return shlib
+
+            return None
+
+        def find_library(name):
+            return _findLib_aix(name)
+
 elif os.name == "posix":
     # Andreas Degert's find functions, using gcc, /sbin/ldconfig, objdump
     import re, tempfile, errno
@@ -267,7 +284,16 @@
             print cdll.LoadLibrary("libcrypto.dylib")
             print cdll.LoadLibrary("libSystem.dylib")
             print cdll.LoadLibrary("System.framework/System")
+        elif sys.platform.startswith("aix"):
+            print find_library("crypto")
+            x = find_library("crypto")
+            print cdll.LoadLibrary(x)
+            print cdll.LoadLibrary("libcrypto.a(libcrypto.so)")
+            print cdll.LoadLibrary("libc.a(shr.o)")
+            print find_library("crypt")
+            print cdll.LoadLibrary("libcrypt.a(shr.o)")
         else:
+            print cdll.LoadLibrary("libc.a")
             print cdll.LoadLibrary("libm.so")
             print cdll.LoadLibrary("libcrypt.so")
             print find_library("crypt")


p.s. Anyone know what flag to set in vi so that tabs are never inserted and/or spaces converted to tabs. Python seems to hate tab indents.

Oops: forgot the example output:

root@x064:[/data/prj/aixtools/src]python -m ctypes.util
libm.a(libm.so)
libc.a(libc.so)
libbz2.a(libbz2.so)
libcrypto.a(libcrypto.so)
<CDLL 'libcrypto.a(libcrypto.so)', handle 6 at 30147690>
<CDLL 'libcrypto.a(libcrypto.so)', handle 7 at 30147690>
<CDLL 'libc.a(shr.o)', handle 8 at 30147690>
libcrypt.a(libcrypt.so)
<CDLL 'libcrypt.a(shr.o)', handle 9 at 30147690>

The patch works when installed on top of pre-compiled version (e.g., copy the two files to /opt/lib/python2.7/ctypes/

but there seems to be a nasty side-effect when trying to build.

make completes, but make install fails during a compile step.

Please assist with howto see how/why ctypes/util is impacting this part.

And - for my first adventure into python please provide the needed (expected) feedback.

Michael

make install DESTDIR=/var/aixtools/aixtools/python/2.7.11.2 > .buildaix/install.out

Listing /var/aixtools/aixtools/python/2.7.11.2/opt/lib/python2.7/xml/sax ...
Compiling /var/aixtools/aixtools/python/2.7.11.2/opt/lib/python2.7/xml/sax/__init__.py ...
Compiling /var/aixtools/aixtools/python/2.7.11.2/opt/lib/python2.7/xml/sax/_exceptions.py ...
Compiling /var/aixtools/aixtools/python/2.7.11.2/opt/lib/python2.7/xml/sax/expatreader.py ...
Compiling /var/aixtools/aixtools/python/2.7.11.2/opt/lib/python2.7/xml/sax/handler.py ...
Compiling /var/aixtools/aixtools/python/2.7.11.2/opt/lib/python2.7/xml/sax/saxutils.py ...
Compiling /var/aixtools/aixtools/python/2.7.11.2/opt/lib/python2.7/xml/sax/xmlreader.py ...
Compiling /var/aixtools/aixtools/python/2.7.11.2/opt/lib/python2.7/xmllib.py ...
Compiling /var/aixtools/aixtools/python/2.7.11.2/opt/lib/python2.7/xmlrpclib.py ...
Compiling /var/aixtools/aixtools/python/2.7.11.2/opt/lib/python2.7/zipfile.py ...

So, zipfile.py seems to exit non-zero. :(

Ah, good news - the build is successful ONCE I open the file ./Lib/ctypes/util.py and find the (hiding) tab characters and replace them with 8 space characters. (And I had tried so hard to check for that in advance).

So, if you apply the patch - it may need some love.

I would be grateful for people (even just one!) looking at it.

Michael

fyi: just completed a test both as 32 and 64 bit build.

However, openssl-1.0.1.514 does not work in 64-bit mode (packaging error). Fortunately, openssl-1.0.1.515 (released 02-March-2016) fixes that.

In short, 64-bit build is dependent on openssl-1.0.1.515

Here is the corrected patch (tabs removed). name: python.Lib.ctypes.160317.patch

Maybe Issue 21826 is relevant (slowdown on AIX caused by missing ldconfig)

These are very different issues. However, this patch may resolve both!

ldconfig (-p if I recall) lists where (shared) libraries have been installed (imho, this is a GNU tool approach) - whereas AIX would use dump -H to find library paths embedded in a program and/or shared library.

Until this patch, to use shared libraries on AIX the members of an archive needed to be extracted from the .a archive, and for 64-bit members, a separate directory (e.g. /usr/lib64) is needed. With this patch find_library() (actually cdll.LoadLibrary() can load members from either both .so and .a libraries, as is normal for AIX.

So, in a way, this would also solve https://bugs.python.org/issue21826 as ldconfig is no longer needed (nor called) on AIX.

p.s. As it is well longer than a month - I would appreciate that someone actually look at the patch and tell me how it can be improved! :)

The most recent patch seems to follow AIX semantics correctly.

I notice the patch is against Python 2, and uses Python 2 specific syntax. I am unsure if this kind of change is acceptable for 2.7, or if it would have to only go into 3.6. Hopefully this version of the patch will notify the review system that it is against Python 2.

I don’t know anything about AIX specific stuff, but I left some general comments in the code review.

Is there any chance that AIX people would be relying on the current behaviour that I understand uses _findSoname_ldconfig() and _findLib_gcc()?

Is this new functionality covered by the test suite? E.g. in /Lib/ctypes/test/test_find.py, there are tests that call find_library() for GL, GLU, and gle. Are those libraries common on AIX?

As discussed in Issue 9998, it seems a lot of people use find_library() to help convert a build-time library name to a run-time shared library name that can be passed to CDLL() or LoadLibrary(). E.g. on Linux:

>>> find_library("python2.7")  # As used in cc . . . -lpython2.7
'libpython2.7.so.1.0'
>>> cdll.LoadLibrary("libpython2.7.so.1.0")
<CDLL 'libpython2.7.so.1.0', handle 7f58e7495000 at 7f58e573ac90>

Does your patch support this kind of use case?

I have not looked specifically, at least not that I remember, for differences in util/ctypes in python2 and python3. Will do so tomorrow.

I did just look briefly at the library, rather archive, built by default as libpython2.7.a - it is static members only, i.e., my build using xlc (i.e., not using gcc) does not build a shared object, so cdll.LoadLibrary and/or find_library will not find anything for python2.7.. Neither will m, or libm, on a default AIX system (with no other gcc based packages installed - these also install a gnu rte where the utilities and libs you mention might include.

The few python packages I have found, packaged by others, tend to reload everything yet again, not depending on anything that may already be there. And to use shared libraries they are extracting the members from the .a archives into two directories - when they support both 32 and 64-bit targets.

My intent is to examine the program to discover where libraries should be and find the member name that is most likely. Also, if LIBPATH is defined, those directories are searched first for a match.

In short, the key difference is to look at the program (probably python) for the blibpath string in the application as well as python (from memory, sys.* calls) to build a list of directories to search.

findLibrary('foo') first finds libfoo.a, then looks in libfoo.a for shr*.o members, libfoo.so, libfoo.so.X and/or libfoo.so.X.Y, etc..

I need to check that findLibrary('foo.so') continues to work. At one time it did, just have not looked at this for several weeks and I forget if it still works. That is what I shall make sure stays in the "testing" part of the patch.

Michael

The obvious (but easy to fix) problem I forsee with Python 3 is the print() calls. If you use print("") and print(arg), that will work with both 2 and 3. There may be more complications with bytes vs text stdout if we change the os.popen() calls to use subprocess.Popen.

I didn’t mean to use libpython2.7 specifically. Substitute any shared library that is widely available across platforms; maybe “crypto” is a better example. CDLL(find_library("crypto")) loads "libcrypto.so.1.0.0" on my Linux computer. It looks like you got the equivalent working for AIX; I was just checking.

FWIW it looks like your parsing of sys.executable to find library search paths is similar to searching the runpath (or RPATH) on ELF files, as proposed in Issue 19317. And it seems AIX’s LIBPATH environment variable is similar to LD_LIBRARY_PATH on Linux, proposed to be searched in Issue 9998. Also, I understand the equivalent OS X environment variables DYLD_(FALLBACK)_LIBRARY_PATH are already used.

On 4/28/2016 11:56 PM, Michael Felt wrote:
> Michael Felt added the comment:
>
> I have not looked specifically, at least not that I remember, for differences in util/ctypes in python2 and python3. Will do so tomorrow.
>
> I did just look briefly at the library, rather archive, built by default as libpython2.7.a - it is static members only, i.e., my build using xlc (i.e., not using gcc) does not build a shared object, so cdll.LoadLibrary and/or find_library will not find anything for python2.7.. Neither will m, or libm, on a default AIX system (with no other gcc based packages installed - these also install a gnu rte where the utilities and libs you mention might include.
>
> The few python packages I have found, packaged by others, tend to reload everything yet again, not depending on anything that may already be there. And to use shared libraries they are extracting the members from the .a archives into two directories - when they support both 32 and 64-bit targets.
>
> My intent is to examine the program to discover where libraries should be and find the member name that is most likely. Also, if LIBPATH is defined, those directories are searched first for a match.
>
> In short, the key difference is to look at the program (probably python) for the blibpath string in the application as well as python (from memory, sys.* calls) to build a list of directories to search.
>
> findLibrary('foo') first finds libfoo.a, then looks in libfoo.a for shr*.o members, libfoo.so, libfoo.so.X and/or libfoo.so.X.Y, etc..
>
> I need to check that findLibrary('foo.so') continues to work. At one time it did, just have not looked at this for several weeks and I forget if it still works. That is what I shall make sure stays in the "testing" part of the patch.
>
> Michael
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue26439>
> _______________________________________

I am reworking the logic - as many use cdll.LoadLibrary without ever 
calling find_library, and then __init__.py breaks.

More asap.

Question - before I submit a patch.

A. Is there a PEP I should read re: ctypes/util and/or ctypes/cdll?

B. I show two different behaviors of responding - My question is, what does the community think should be the response? My preference is to bring the request back to it's stub - (strip lib from the beginning, and .so* or .a from the suffix, so that it is in it's -lFOO form - as if being offered to a linker (with -lFOO, we do not use -lFOO.so.6 (e.g., for libc.so.6).
I have seen a lot of variance in how cdll is used (in only two python based projects I am currently interested in porting). Calls are frequently made using some variation including .so somewhere in the name and/or code is basing decesions based on a the number (###) returned by find_library (e.g., libFOO.so.###, and ### MUST be one of a list - and is rejected if not a match (even if newer)). I would think if there is a version dependancy this should be a call to the library loaded rather than an external string.
In any case, - for one example - as libc.so.* will never exist, by default, on AIX, for ease of use I believe that translating libFOO.* to FOO and then doing the 'find' comes closest to what was intended (long long ago) - again, is there a PEP with guidance on this AND my preference, for AIX, is to look in .a archives first for a .so member, and then look for an AIX legacy member (shr.*.o) and then look for an external .so.* file.

With my current working version - this is the output of the test in util.py:

root@x064:[/data/prj/aixtools/python/python-2.7.11.3/Lib/ctypes]../../python ./util.py
libm.a
libc.a
libbz2.a
<CDLL 'None', handle b at 700000000216630>
<CDLL 'libcrypt.a(shr_64.o)', handle c at 700000000216630>
libcrypt.a

Additional Tests for AIX
call find_library("foo")
c:  libc.a
c.a:  None
c.so:  None
libc:  libc.a
libc.a:  libc.a
libc.so.6:  libc.a
crypt:  libcrypt.a
crypto:  libcrypto.a
crypto.so:  None
libcrypto.so:  libcrypto.a

call cdll.LoadLibrary("foo")
m:  <CDLL 'None', handle d at 700000000216860>
libm.so:  <CDLL 'None', handle e at 700000000216860>
c:  <CDLL 'libc.a(shr_64.o)', handle f at 700000000216860>
c.a:  <CDLL 'libc.a(shr_64.o)', handle 10 at 700000000216860>
libc.a:  <CDLL 'libc.a(shr_64.o)', handle 11 at 700000000216860>
libc.so.6:  <CDLL 'libc.a(shr_64.o)', handle 12 at 700000000216860>
libc.so.9:  <CDLL 'libc.a(shr_64.o)', handle 13 at 700000000216860>
bz2:  <CDLL 'libbz2.a(libbz2.so.1)', handle 14 at 700000000216860>
libbz2:  <CDLL 'libbz2.a(libbz2.so.1)', handle 15 at 700000000216860>
crypt:  <CDLL 'libcrypt.a(shr_64.o)', handle 16 at 700000000216860>
crypto:  <CDLL 'libcrypto.a(libcrypto64.so)', handle 17 at 700000000216860>
crypto.so:  <CDLL 'libcrypto.a(libcrypto64.so)', handle 18 at 700000000216860>
libcrypto.so:  <CDLL 'libcrypto.a(libcrypto64.so)', handle 19 at 700000000216860>
iconv:  <CDLL 'libiconv.a(libiconv.so.2)', handle 1a at 700000000216860>
intl:  <CDLL 'libintl.a(libintl.so.1)', handle 1b at 700000000216860>

The test looks like:
################################################################
# test code

def test():
    from ctypes import cdll
    if os.name == "nt":
        print cdll.msvcrt
        print cdll.load("msvcrt")
        print find_library("msvcrt")

    if os.name == "posix":
        # find and load_version
        print find_library("m")
        print find_library("c")
        print find_library("bz2")

        # getattr
##        print cdll.m
##        print cdll.bz2

        # load
        if sys.platform == "darwin":
            print cdll.LoadLibrary("libm.dylib")
            print cdll.LoadLibrary("libcrypto.dylib")
            print cdll.LoadLibrary("libSystem.dylib")
            print cdll.LoadLibrary("System.framework/System")
        else:
            print cdll.LoadLibrary("libm.so")
            print cdll.LoadLibrary("libcrypt.so")
            print find_library("crypt")

        if sys.platform.startswith("aix"):
            print "\nAdditional Tests for AIX"
            print "call find_library(\"foo\")"
            print "c: ", find_library("c")
            print "c.a: ", find_library("c.a")
            print "c.so: ", find_library("c.so")
            print "libc: ", find_library("libc")
            print "libc.a: ", find_library("libc.a")
            print "libc.so.6: ", find_library("libc.so.6")
            print "crypt: ", find_library("crypt")
            print "crypto: ", find_library("crypto")
            print "crypto.so: ", find_library("crypto.so")
            print "libcrypto.so: ", find_library("libcrypto.so")
###
            print "\ncall cdll.LoadLibrary(\"foo\")"
            print "m: ", cdll.LoadLibrary("m")
            print "libm.so: ", cdll.LoadLibrary("libm.so")
            print "c: ", cdll.LoadLibrary("c")
            print "c.a: ", cdll.LoadLibrary("c.a")
            print "libc.a: ", cdll.LoadLibrary("libc.a")
            print "libc.so.6: ", cdll.LoadLibrary("libc.so.6")
            print "libc.so.9: ", cdll.LoadLibrary("libc.so.9")
            print "bz2: ", cdll.LoadLibrary("bz2")
            print "libbz2: ", cdll.LoadLibrary("libbz2")
            print "crypt: ", cdll.LoadLibrary("crypt")
            print "crypto: ", cdll.LoadLibrary("crypto")
            print "crypto.so: ", cdll.LoadLibrary("crypto.so")
            print "libcrypto.so: ", cdll.LoadLibrary("libcrypto.so")
            print "iconv: ", cdll.LoadLibrary("iconv")
            print "intl: ", cdll.LoadLibrary("intl")

if __name__ == "__main__":
    test()

Thank you for your considered comments. My goal is ease of use for (porting) existing projects to AIX, i.e., ideally without code change.

reworked patch. To assist port to Python3 that changes in __init__.py and util.py are minimal. There is a new file: aixutil.py

I have only tested on Python-2.7, so there may be issues for Python3. My goal is to have a single file for both versions.

The main change in util.py is lots of specific (behavior) tests that I have encountered (and initially failed) from projects using ctypes.cdll.

I feel confident that this will work "ASIS" with nearly all existing projects.

Note: normal return is not a full path name. A full path name is only returned when an archive + member could not be found, but a file with the name requested could be found. This "fullpath" is to be compatible with existing code 'demanding' unpacked archives.

implements ctypes.aixutil.find_library()
In a separate file as both __init__.py as util.py needs it's logic.

'''
call find_library("foo")
libc:  libc.a
libc.a:  libc.a
libc.so.6:  libc.a
libcrypto.so:  libcrypto.a
'''

The above don’t seem right to me, unless compiling with “cc -llibc.so.6” etc works on AIX.

'''
call cdll.LoadLibrary("foo")
m:  <CDLL 'None', handle d at 700000000216860>
libm.so:  <CDLL 'None', handle e at 700000000216860>
'''

These doesn’t look right. What happened to the library name?

With your new aixutil.py file, it might be good to give it an underscore (_) prefix, to indicate it is an internal module rather than part of the ctypes API. So your code would do

import ctypes._aixutil as aix

Your new patch calls find_library() internally in CDLL(); why? My understanding is CDLL() is a fairly lightweight wrapper around the dlopen() call. On Linux, you either pass a full library file name, or an SO-name. Both these strings can be discovered for compiled objects using e.g.:

$ ldd build/lib.linux-x86_64-2.7-pydebug/_ssl.so
	linux-vdso.so.1 (0x00007fff567fe000)
	libssl.so.1.0.0 => /usr/lib/libssl.so.1.0.0 (0x00007f598474c000)
	libcrypto.so.1.0.0 => /usr/lib/libcrypto.so.1.0.0 (0x00007f59842d4000)
	. . .

So in Python, the SO-name or full path can be used, but not the compile-time name, unless you first pass it through find_library():

>>> CDLL("libcrypto.so.1.0.0")  # soname
<CDLL 'libcrypto.so.1.0.0', handle 7f1665e1eb90 at 7f16658f34d0>
>>> CDLL("/usr/lib/libcrypto.so.1.0.0")  # Full path
<CDLL '/usr/lib/libcrypto.so.1.0.0', handle 7f1665e1eb90 at 7f1663cddcd0>
>>> CDLL("crypto")  # Compile-time name
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/ctypes/__init__.py", line 365, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: crypto: cannot open shared object file: No such file or directory
>>> find_library("crypto")  # Some people pass the result of this to CDLL()
'libcrypto.so.1.0.0'

New patch of the Lib/ctypes directory - BUT - this time as a delta based on 3.5.0.

++ changes from last patch ++
* OSError gets raised (as expected) by test/test_loading.py
* test in util.py modified (libm is replaced by libc when "aix"), and
added an additional test for AIX (showing it finds the latest version
of libintl that is installed (frequently, there are two versions)
* moved RTLD_NOW and RTLD_MEMBER definitions to the "aix" blocks in __init__.py
* in test/test_loading.py added two more libraries to look for

New version of aixutil.py

++ Changes ++
* more comments
* re-worked the 'searches' for matches after adding changing the way
the output from _get_dumpH was read (now to an array using readlines).
Also, p.stdout.close() and p.wait() are done within the _get_dumpH routine.

Suggestions on how to have cleaner workings with the 'arrays of text' and searching through them is welcomed! It is passing the internal tests is all I will say for now.

Spent more time today, testing - on 3.5.1, 3.4.4 and 2.7.11.

The patches would not apply on both Python2 and Python3, but the new file Lib/ctypes/aixutil.py is identical on all three versions.

History of patching (starting from version 3.5.1(.1) is my patched version (as are Python-3.4.4.1 and Python-2.7.11.4)

michael@x071:[/data/prj/aixtools/python/python-3.4.4.1]for i in ../*patches/*patch^Jdo^Jcat $i | patch -p1 ^Jdone
patching file Lib/ctypes/__init__.py
Hunk #1 succeeded at 341 (offset 4 lines).
patching file Lib/ctypes/test/test_loading.py
patching file Lib/ctypes/util.py
Hunk #1 succeeded at 70 (offset -6 lines).
Hunk #2 succeeded at 256 (offset -6 lines).
Hunk #3 succeeded at 275 (offset -6 lines).
patching file Modules/_ctypes/_ctypes.c
Hunk #1 succeeded at 5508 (offset -1 lines).
michael@x071:[/data/prj/aixtools/python/python-3.4.4.1]

But, not with 2.7.11

michael@x071:[/data/prj/aixtools/python/python-2.7.11.4]for i in ../*patches/*patch^Jdo^Jcat $i | patch -p1 --dry-run^Jdone
patching file Lib/ctypes/__init__.py
Hunk #1 succeeded at 355 (offset 18 lines).
patching file Lib/ctypes/test/test_loading.py
Hunk #1 succeeded at 38 (offset -5 lines).
patching file Lib/ctypes/util.py
Hunk #1 succeeded at 71 (offset -5 lines).
Hunk #2 FAILED at 262.
Hunk #3 FAILED at 274.
2 out of 3 hunks FAILED -- saving rejects to file Lib/ctypes/util.py.rej
patching file Modules/_ctypes/_ctypes.c
Hunk #1 FAILED at 5509.
1 out of 1 hunk FAILED -- saving rejects to file Modules/_ctypes/_ctypes.c.rej

So, I made new patches against 2.7.11 after patching manually.

After the builds:

==============================
aixtools.python:aixtools.python.man.en_US:2.7.11.4::I:T:::::N:man pages::::0::
aixtools.python:aixtools.python.rte:2.7.11.4::I:T:::::N:built 10-May-2016 1810 UTC::::0::
==============================
root@x064:[/data/prj/aixtools/python/python-2.7.11.4]cd Lib/ctypes
root@x064:[/data/prj/aixtools/python/python-2.7.11.4/Lib/ctypes]../../python util.py
None
libc.a(shr.o)
libbz2.a(libbz2.so)
('aix.find_library("libiconv.so")', 'libiconv.a(shr4.o)')
('aix.find_library("libintl.so")', 'libintl.a(libintl.so.1)')
('aix.find_library("libintl.so.1")', 'libintl.a(libintl.so.1)')
('aix.find_library("libintl.so.2")', None, ':: should be None!')
('aix.find_library("libintl.so.8")', None)
<CDLL 'libc.a(shr.o)', handle c at 300e5810>
<CDLL 'libintl.a(libintl.so.1)', handle d at 300e5810>
<CDLL 'libcrypt.a(shr.o)', handle e at 300e5810>
libcrypt.a(shr.o)
root@x064:[/data/prj/aixtools/python/python-2.7.11.4/Lib/ctypes]../../python test/test_loading.py
libc_name is libc.a(shr.o)
ss..sss
----------------------------------------------------------------------
Ran 7 tests in 0.189s

OK (skipped=5)

==============================
aixtools.python:aixtools.python.man.en_US:3.4.4.1::I:T:::::N:man pages::::0::
aixtools.python:aixtools.python.rte:3.4.4.1::I:T:::::N:built 10-May-2016 1812 UTC::::0::
==============================
root@x064:[/data/prj/aixtools/python/python-3.4.4.1]cd Lib/ctypes
root@x064:[/data/prj/aixtools/python/python-3.4.4.1/Lib/ctypes]../../python util.py
None
libc.a(shr.o)
libbz2.a(libbz2.so)
aix.find_library("libiconv.so") libiconv.a(shr4.o)
aix.find_library("libintl.so") libintl.a(libintl.so.1)
aix.find_library("libintl.so.1") libintl.a(libintl.so.1)
aix.find_library("libintl.so.2") None :: should be None!
aix.find_library("libintl.so.8") None
<CDLL 'libc.a(shr.o)', handle b at 300e7250>
<CDLL 'libintl.a(libintl.so.1)', handle c at 300e7250>
<CDLL 'libcrypt.a(shr.o)', handle d at 300e7250>
libcrypt.a(shr.o)
root@x064:[/data/prj/aixtools/python/python-3.4.4.1/Lib/ctypes]../../python test/test_loading.py
libc_name is libc.a(shr.o)
ss..sss
----------------------------------------------------------------------
Ran 7 tests in 0.323s

OK (skipped=5)

root@x064:[/data/prj/aixtools/python/python-3.5.1.1]installp -d . -L
aixtools.python:aixtools.python.man.en_US:3.5.1.0::I:T:::::N:man pages::::0::
aixtools.python:aixtools.python.man.en_US:3.5.1.1::I:T:::::N:man pages::::0::
aixtools.python:aixtools.python.rte:3.5.1.0::I:T:::::N:built 03-Mar-2016 0856 UTC::::0::
aixtools.python:aixtools.python.rte:3.5.1.1::I:T:::::N:built 10-May-2016 1620 UTC::::0::

And, with a 64-bit build

root@x064:[/data/prj/aixtools/python/python-3.5.1.1]installp -d . -L
aixtools.python:aixtools.python.man.en_US:3.5.1.0::I:T:::::N:man pages::::0::
aixtools.python:aixtools.python.man.en_US:3.5.1.1::I:T:::::N:man pages::::0::
aixtools.python:aixtools.python.rte:3.5.1.0::I:T:::::N:built 03-Mar-2016 0856 UTC::::0::
aixtools.python:aixtools.python.rte:3.5.1.1::I:T:::::N:built 10-May-2016 1620 UTC::::0::
root@x064:[/data/prj/aixtools/python/python-3.5.1.1]cd lib/ctypes
ksh: lib/ctypes:  not found.
root@x064:[/data/prj/aixtools/python/python-3.5.1.1]cd Lib/ctypes
root@x064:[/data/prj/aixtools/python/python-3.5.1.1/Lib/ctypes]../../python util.py
None
libc.a(shr_64.o)
None
aix.find_library("libiconv.so") libiconv.a(shr4_64.o)
aix.find_library("libintl.so") libintl.a(libintl.so.1)
aix.find_library("libintl.so.1") libintl.a(libintl.so.1)
aix.find_library("libintl.so.2") None :: should be None!
aix.find_library("libintl.so.8") None
<CDLL 'libc.a(shr_64.o)', handle a at 0x7000000001c79e8>
<CDLL 'libintl.a(libintl.so.1)', handle b at 0x7000000001c79e8>
<CDLL 'libcrypt.a(shr_64.o)', handle c at 0x7000000001c79e8>
libcrypt.a(shr_64.o)
root@x064:[/data/prj/aixtools/python/python-3.5.1.1/Lib/ctypes]../../python test/test_loading.py
libc_name is libc.a(shr_64.o)
ss..sss
----------------------------------------------------------------------
Ran 7 tests in 0.414s

OK (skipped=5)

Patches follow (question, should these be in a tar file in the future?)

On 5/8/2016 8:29 AM, Martin Panter wrote:
> Martin Panter added the comment:
>
> Your new patch calls find_library() internally in CDLL(); why?
Because arguments that work for GNU (aka Linux, even though internally 
it is called "posix") will not work for AIX.
> My understanding is CDLL() is a fairly lightweight wrapper around the dlopen() call.
For AIX, the dlopen() call is a front-end for the function 
initAndLoad(). I have not looked in depth at Module/_ctypes (only 
recently found it) and I do not know how much work is being done there 
for the different platforms.
> On Linux, you either pass a full library file name, or an SO-name.
Same for AIX: a full-path library name, relative-path library name, or a 
library name and the library search path (with optional additional 
directories provided via environment variables (LIBPATH or 
LD_LIBRARY_PATH - when it is a FILE, or an archive(member_name) (with 
archive part being full or relative, or stem) and RTLD_MEMBER ORed into 
the mode argument.
> Both these strings can be discovered for compiled objects using e.g.:
Likewise for AIX: - example: an executable - notice how many are named 
shr.o: how is cdll("shr.o") suppossed to know it is libc, or is it libcrypt?

root@x064:[/usr/lib]ldd /usr/sbin/sshd
/usr/sbin/sshd needs:
          /usr/lib/libc.a(shr.o)
          /usr/lib/libcrypto.a(libcrypto.so.1.0.0)
          /usr/lib/libz.a(libz.so.1)
          /usr/lib/libpam.a(shr.o)
          /usr/lib/libdl.a(shr.o)
          /unix
          /usr/lib/libcrypt.a(shr.o)
          /usr/lib/libpthreads.a(shr_xpg5.o)
          /usr/lib/libpthreads.a(shr_comm.o)

Or a file:
./perl-5.14.4/lib/5.14.4/aix-thread-multi-64int/auto/B/B.so needs:
          /usr/lib/libpthreads.a(shr_xpg5.o)
          /usr/lib/libc.a(shr.o)
          /unix
          /usr/lib/libpthreads.a(shr_comm.o)
          /usr/lib/libcrypt.a(shr.o)

Again, cdll("shr.o") is not going to find the correct archive.
> $ ldd build/lib.linux-x86_64-2.7-pydebug/_ssl.so
> 	linux-vdso.so.1 (0x00007fff567fe000)
> 	libssl.so.1.0.0 => /usr/lib/libssl.so.1.0.0 (0x00007f598474c000)
> 	libcrypto.so.1.0.0 => /usr/lib/libcrypto.so.1.0.0 (0x00007f59842d4000)
> 	. . .
>
> So in Python, the SO-name or full path can be used, but not the compile-time name, unless you first pass it through find_library():

>>>> CDLL("libcrypto.so.1.0.0")  # soname
> <CDLL 'libcrypto.so.1.0.0', handle 7f1665e1eb90 at 7f16658f34d0>
>>>> CDLL("/usr/lib/libcrypto.so.1.0.0")  # Full path
> <CDLL '/usr/lib/libcrypto.so.1.0.0', handle 7f1665e1eb90 at 7f1663cddcd0>
>>>> CDLL("crypto")  # Compile-time name
> Traceback (most recent call last):
>    File "<stdin>", line 1, in <module>
>    File "/usr/lib/python2.7/ctypes/__init__.py", line 365, in __init__
>      self._handle = _dlopen(self._name, mode)
> OSError: crypto: cannot open shared object file: No such file or directory
>>>> find_library("crypto")  # Some people pass the result of this to CDLL()
> 'libcrypto.so.1.0.0'
Thank you for the explanation of what should be used when. Maybe I 
worked too hard for the current result: (note - 64-bit this time! and 
new member names, even for libc.so)

michael@x071:[/usr/lib]python -m ctypes.util
None
libc.a(shr_64.o)
None
aix.find_library("libiconv.so") libiconv.a(shr4_64.o)
aix.find_library("libintl.so") libintl.a(libintl.so.1)
aix.find_library("libintl.so.1") libintl.a(libintl.so.1)
aix.find_library("libintl.so.2") None :: should be None!
aix.find_library("libintl.so.8") None
<CDLL 'libc.a(shr_64.o)', handle a at 0x7000000002242b0>
<CDLL 'libintl.a(libintl.so.1)', handle b at 0x7000000002242b0>
<CDLL 'libcrypt.a(shr_64.o)', handle c at 0x7000000002242b0>
libcrypt.a(shr_64.o)

Back to Why though.

Ease of use, and portability with existing convention. My experience 
with just two python packages I wanted to port is that cdll("libFOO.so") 
is what is used, and works on GNU. I wanted that to work ASIS. I also 
noticed that sometimes code was explicit about the version (sometimes 
the latest was not right, and the code complained). So, since "short" 
names are preferred, and are portable - those are what programmers tend 
to use. Full-path-names imply an assumption.

dlopen() is not going to know that libc.so is actually libc.a(shr.o) - 
so I call find_library() for portability aka Ease of Use. While it may 
be slightly less performance I am hoping (assuming) cdll("foo") is not 
the most central part of any application - and the overhead is acceptable.

And thanks for the question. It is simple enough to remove now, should 
it be considered a sin.

> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue26439>
> _______________________________________

Patches: I’m not sure, but maybe it would help the Reitveld review system if the src/ directory in them was eliminated (e.g. src/Python-3.5.1 vs just Python-3.5.1). Also, if you are using Gnu diff, maybe you can try the -N (--new-file) option to include aixutil.py in the diff, and -x (--exclude) __pycache__ to avoid that directory.

Personally, I would find patches a lot easier to handle than a tar file, especially if I cannot figure out which bits of the tar file are original code and which are changed code.

It seems you want to CDLL() to accept names using the Linux/BSD libFOO.so.N convention, but load an AIX library with a related name. This sounds like a bad design to me, for multiple reasons. I would like to hear other people’s opinions before going in that direction.

This is my understanding of the CDLL() calls currently needed to load the Open SSL library on various platforms:

openssl 1.0.2.g-3 package on Arch Linux: CDLL("libcrypto.so.1.0.0")
Windows: CDLL("libeay32.dll")
OS X (Darwin): CDLL("libcrypto.1.0.0.dylib")
FreeBSD: CDLL("libcrypto.so.8")
AIX: CDLL("libcrypto.a(libcrypto.so.1.0.0)", DEFAULT_MODE | RTLD_MEMBER)

a) https://bugs.python.org/review/26439/#msg12, but getting HTML 500 error)

The call to find_library here is for "ease of use" with existing code who use, e.g., cdll("libcrypto.so"). This format fails unless someone has previously done, e.g.: (cd /usr/lib; ar -X32 x libcrypto.a; mkdir -p /usr/lib64; cd /usr/lib64; ar -X64 x ../lib/libcrypto.a)

What I do think would be a valid addition here is to look for a '/' in the name anywhere (i.e., path info in name, and skip aix.find_library(). IMHO - if a programmer is usiong a relative path - he/she should take full responsilibilty (and this is basically what find_library will return anyway)

However, If there is a syntax issue I can rework the aixutil.py so there are multiple entry points - e.g., find_library() finds the .a file, and find_member() finds an archive containing a member (and returns a suitable base(member) argument for dlopen() and/or add an (undocumented) argument to modify find_library() return value.

But I think that only creates complexity for the user/programmer - additional complexity is an additional for an error (of omission) to occur.

Here, as it is now, if find_library() was used to find the name, find_library() is not called again. Again, if there is a behavior you want to force - then it can be made stricter - but compatibility with existing programs (I have been looking at cloud-init and salt) would be weak/non-existent. And, if ...startswith("aix") will be a very common addition to code.
Difficult to use is a reason programmers avoid a platform - and I would not like to see my favorite platform avoided because they had to add lots of "startswith("aix") - as one would always be forgotten, and then the attitude becomes - not going to bother with python on AIX. - my two bits.

p.s. Notice in the patches from yesterday - I have added the 'export' of RTLD_NOW and RTLD_MEMBER to Modules/_ctype/_ctype.c.
The "export" only occurs if the variable is defined, and the import only occurs in __init__.py behind 'startswith("aix")' blocks.

b) Yes, I can load and use GNU diff and resubmit (you can ignore them if you prefer - and I shall move the src/Python* to parallel with the build directories.

FYI: I tried to pip install Mercurial - but got an error message from an include file so the last bit did not compile. Problem for a later date - could be a user error on my part, or an AIX version dependency.

re:

AIX: CDLL("libcrypto.a(libcrypto.so.1.0.0)", DEFAULT_MODE | RTLD_MEMBER)

Officially it would be
dlopen("libcrypto.a(libcrypto.so.1.0.0)", RTLD_NOW | RTLD_MEMBER)

Further, that would only load the 32-bit version, as there is a legacy naming scheme for the 64-bit members (the same name may actually occur twice - once for each size)

Example: the ar command showing all members regardless of size:
root@x064:[/]ar -Xany tv /usr/lib/libcrypto.a
rwxr-xr-x     0/0     2967588 Jul 24 13:46 2015 libcrypto.so
rwxr-xr-x     0/0     2256131 Jul 24 13:43 2015 libcrypto.so.0.9.8
rwxr-xr-x     0/0     2967588 Jul 24 13:45 2015 libcrypto.so.1.0.0
rwxr-xr-x     0/0     3376521 Jul 24 13:46 2015 libcrypto64.so
rwxr-xr-x     0/0     2606185 Jul 24 13:44 2015 libcrypto64.so.0.9.8
rwxr-xr-x     0/0     3376521 Jul 24 13:45 2015 libcrypto64.so.1.0.0

Note: the default archives (ending in .so) are "same size, checksum, etc" as the .so.1.0.0 versioned members.

Also, this can be more complex - e.g., if LibreSSL is also included (I am re-working my packaging so that I will also have both 32 and 64-bit packaging, for now only 32-bit is available)

michael@x071:[/home/michael]ar -X32 tv /opt/lib/libcrypto.a
rwxr-xr-x     0/0     3060762 Jun 17 21:17 2015 libcrypto.so.33
rwxrwxr-x     0/0     2965597 Jul 25 16:57 2015 libcrypto.so
rwxrwxr-x     0/0     2253850 Jul 25 17:03 2015 libcrypto.so.0.9.8
rwxrwxr-x     0/0     2965597 Jul 25 17:03 2015 libcrypto.so.1.0.0

IMHO: this is a needless complication for a programmer using ctypes.cdll - but who am I?

From the dlopen man page (aka InfoCenter)

Flags: (aka mode)
Specifies variations of the behavior of dlopen. Either RTLD_NOW or RTLD_LAZY must always be specified. Other flags may be OR'ed with RTLD_NOW or RTLD_LAZY.

RTLD_NOW 	Load all dependents of the module being loaded and resolve all symbols.

RTLD_LAZY 	Specifies the same behavior as RTLD_NOW. In a future release of the operating system, the behavior of the RTLD_LAZY may change so that loading of dependent modules is deferred of resolution of some symbols is deferred.

RTLD_MEMBER 	The dlopen subroutine can be used to load a module that is a member of an archive. The L_LOADMEMBER flag is used when the load subroutine is called. The module name FilePath names the archive and archive member according to the rules outlined in the load subroutine.

To be complete...

RTLD_GLOBAL 	Allows symbols in the module being loaded to be visible when resolving symbols used by other dlopen calls. These symbols will also be visible when the main application is opened with dlopen(NULL, mode).

RTLD_LOCAL 	Prevent symbols in the module being loaded from being used when resolving symbols used by other dlopen calls. Symbols in the module being loaded can only be accessed by calling dlsym subroutine. 

If neither RTLD_GLOBAL nor RTLD_LOCAL is specified, the default is RTLD_LOCAL. If both flags are specified, RTLD_LOCAL is ignored.

used diff -rNu Python-3.5.1 Python-3.5.1.1 to generate

used diff -rNu Python-2.7.11 Python-2.7.11.4 to generate

Note: content same as those from yesterday - except 
Lib/ctype/test/test_loading.py is no longer changed. No longer needed.

Thanks for trying to move the directory in the new patch, but it does not seem to have been picked up by Rietveld. I have only seen patches generated with Mercurial and Git work, but I don’t exactly know _why_ they work :)

Here is a summary of what I would be comfortable adding to Python:

1. Add ctypes.RTLD_MEMBER constant (new API, so 3.6+ only)

2. Basic ctypes.util.find_library("crypto") implementation for AIX. But cases like find_library("libintl.so") should fail. I think your earlier patches were closer to this implementation than the later ones.

I am a bit hesitant about the automatic behaviour of CDLL("libcrypto.a(libcrypto.so.1.0.0)") using RTLD_MEMBER. IMO it may be better to let the caller specify RTLD_MEMBER explicitly. If a shared library file literally called “/usr/lib/libcrypto.a(libcrypto.so.1.0.0)” existed, i.e. not inside an archive, would dlopen("libcrypto.a(libcrypto.so.1.0.0)", RTLD_NOW) succeed? I admit this is an unlikely scenario, but it seems bad to reduce the domain of a low-level API.

I understand it would be good to have the return value of find_library() consistent with the name accepted by CDLL(). Perhaps a new parameter format would help, such as a tuple (archive, member).

I am not comfortable with other aspects. I think I would have to see more discussion with other people to change my opinion:

1. CDLL("libintl.so") should not load “libintl.a(libintl.so.1)”. I understand you want this to help code written for Gnu or Linux automatically work on AIX, but it doesn’t feel correct and robust to me. Perhaps moving this sort of thing to a separate function or package would be better.

2. find_library("libintl.so") -> "libintl.a(libintl.so.1)". I would expect it to look for a shared library installed in something like "/usr/lib/liblibintl.so.a", unless I have misunderstood how compile-time linking (cc -llibintl.so) works.

3. find_library() should not set the LIBPATH environment variable.

See below between points.

On 12-May-16 17:45, Martin Panter wrote:
> Martin Panter added the comment:
>
> Thanks for trying to move the directory in the new patch, but it does not seem to have been picked up by Rietveld. I have only seen patches generated with Mercurial and Git work, but I don’t exactly know _why_ they work :)
>
> Here is a summary of what I would be comfortable adding to Python:
>
> 1. Add ctypes.RTLD_MEMBER constant (new API, so 3.6+ only)
Understood. Should also add RTLD_NOW as it is referenced in comments, 
and used in the _global.c (imho).
> 2. Basic ctypes.util.find_library("crypto") implementation for AIX. But cases like find_library("libintl.so") should fail. I think your earlier patches were closer to this implementation than the later ones.
>
> I am a bit hesitant about the automatic behaviour of CDLL("libcrypto.a(libcrypto.so.1.0.0)") using RTLD_MEMBER. IMO it may be better to let the caller specify RTLD_MEMBER explicitly. If a shared library file literally called “/usr/lib/libcrypto.a(libcrypto.so.1.0.0)” existed, i.e. not inside an archive, would dlopen("libcrypto.a(libcrypto.so.1.0.0)", RTLD_NOW) succeed? I admit this is an unlikely scenario, but it seems bad to reduce the domain of a low-level API.
>
> I understand it would be good to have the return value of find_library() consistent with the name accepted by CDLL(). Perhaps a new parameter format would help, such as a tuple (archive, member).
I would expect a new tuple would be seen as new API (like 
_ctypes.RTLD_MEMBER), so it would never help in Python-2.7, which is 
still where I see most people asking for support with AIX.
My guidance was: find_library("m") is legal, as is 
find_library("libm.so"), as is CDLL("libcrypt.so").

These are all dependencies, at a low-level of how the system is 
configured, i.e., it is not uncommon for a GNU/Linux installation to 
have many symbolic links to resolve multiple names, and programmers use 
one, or more of those conventions. The find_library() and CDLL() calls 
work because of these conventions.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Thu May 19 02:53:48 2016 from xxx.yyy.zzz.net 
<http://76-217-205-93.lightspeed.sndgca.sbcglobal.net>
root@x066:~# python -m ctypes.util
libm.so.6
libc.so.6
libbz2.so.1.0
<CDLL 'libm.so', handle f7c0f5e0 at 104493d0>
<CDLL 'libcrypt.so', handle 1045b2d8 at 104493d0>
libcrypt.so.1

Looking at filesystem level:
root@x066:~# cd /lib64
root@x066:/lib64# ls -l libcrypt.*
lrwxrwxrwx 1 root root 16 Feb 22  2015 libcrypt.so.1 -> libcrypt-2.13.so 
<http://libcrypt-2.13.so>
root@x066:/lib64# cd /usr/lib64
root@x066:/usr/lib64# ls -l libcrypt.*
-rw-r--r-- 1 root root 62108 Feb 22  2015 libcrypt.a
lrwxrwxrwx 1 root root    20 Feb 22  2015 libcrypt.so -> 
/lib64/libcrypt.so.1

So, CDLL.dlopen() finds /usr/lib64/libcrypt.so that is a symbolic link 
to /lib64/libcrypt.so.1 that is a symbolic link to 
/lib64/libcrypt-2.13.so <http://libcrypt-2.13.so>. I see libcrypt.so as 
a "shorthand" for whatever is installed. As a porgrammer/user of 
CDLL.dlopen() I do not care what it finally resolves to.

Linux has it's conventions: one important one I observe when porting OSS 
packages is that "installation" and updates of libraries (or API's) 
provide "shortnames" in the form of libFOO.so  that are symbolic names 
to the final object. Unfortunately, there are exceptions - especially 
look at "libc.so"

root@x066:/usr/lib64# ls -l libc.so*
-rw-r--r-- 1 root root 243 Feb 22  2015 libc.so

root@x066:/usr/lib64# cd /lib64
root@x066:/lib64# ls -l libc.so*
lrwxrwxrwx 1 root root 12 Feb 22  2015 libc.so.6 -> libc-2.13.so
root@x066:/lib64# ls -l libc-2.13.so
-rwxr-xr-x 1 root root 1758016 Feb 22  2015 libc-2.13.so

Obviously, using shorthand for libc.so on Debian is not returning 
anything equivalent to libc.so.6 (look at the different sizes). So, a 
program that looks specifically for libc.so.6 will succeed on Debian, 
and fail on AIX (as it does not exist in that form) - and my expectation 
is that existing programs that code libc.so.6 will fail on AIX. However, 
I have extreme doubts that anyone actually codes the test - as (at least 
on AIX) libc is already loaded.

AIX:
root@x071:[/root]ldd /opt/bin/python
/opt/bin/python needs:
          /usr/lib/libc.a(shr.o)
          /usr/lib/libpthreads.a(shr_xpg5.o)
          /usr/lib/libpthreads.a(shr_comm.o)
          /usr/lib/libdl.a(shr.o)
          /unix
          /usr/lib/libcrypt.a(shr.o)

And, it seems, libc is also already loaded on Debian:
root@x066:/lib64# ldd /usr/bin/python
         linux-vdso32.so.1 =>  (0x00100000)
         libpthread.so.0 => /lib/powerpc-linux-gnu/libpthread.so.0 
(0x0ffc5000)
         libdl.so.2 => /lib/powerpc-linux-gnu/libdl.so.2 (0x0ffa1000)
         libutil.so.1 => /lib/powerpc-linux-gnu/libutil.so.1 (0x0ff7e000)
         libz.so.1 => /lib/powerpc-linux-gnu/libz.so.1 (0x0ff47000)
         libm.so.6 => /lib/powerpc-linux-gnu/libm.so.6 (0x0fe79000)
         libc.so.6 => /lib/powerpc-linux-gnu/libc.so.6 (0x0fce0000)
         /lib/ld.so.1 (0x20231000)

So back to my thoughts as I developed this "behavior" patch for 
ctypes/util aka ctypes/cdll:
a) to be transparent - mask (but not hide) AIX "internals" that are 
(probably) a level of detail programmers are not interested in. That is 
what I imagined the original design goal of the Lib addition.
b) to support existing code ASIS (the first two python packages I looked 
at being cloud-init and salt-stack)
c) as a bonus: potentially resolve any performance "bugs" caused by the 
lack of /bin/ldconfig

AIX has it's conventions - the main one being to look in archives (.a 
files) for members. To signify that it is an archive member an 
additional dlopen() mode is used. Without this flag, dlopen() looks for 
a file - just like other "posix" platforms - period.

re: your objections/concerns
a) I am shooting myself for being so visible - as I have done a lot of 
porting and, in particular, "intl" aka libintl.so support is important 
for programmers that are expecting GNU gettext support for multiple 
languages.
These seems pointless on a generic Debian install - because while 
gettext is available, there is no "libintl" to be found (using either 
gcc or ldconfig -p to search), e.g.:

root@x066:/lib64# ldconfig -p | grep intl
root@x066:/lib64# gettext --version
gettext (GNU gettext-runtime) 0.18.1

The tests re: libcrypto I added because behavior re: libssl and 
libcrypto is important. Many applications need that. (libcrypt is not 
"SSL" support).

So, back to what I think are your objections:
a) seeing a name ending in ")" as being the AIX "native" format and 
adding RTLD_MEMBER without any further test of it being a file.
b) expanding a "generic" name ending in .so to a latest version IF it 
does not exist as .so in an archive, or as a file.

re: a) a test could be added to verify that a file with a name ending 
with ")") does not exist before adding the RTLD_MEMBER mode - this is 
only being considered on AIX (that condition must be determined first 
anyway) and, as you say - it is unlikely to be the case. So, a test 
could be added - that would probably always fail - slowing calls down.

re: b) I consider it better practice (and maybe this is my arrogance of 
35 years + of supporting multiple platforms in libraries (now API) 
transparently. I am the weirdo who likes to dive into low-level, 
bit-level "idiosyncrasies" of different os systems and/or levels. 
"Normal" people do not care a #$%! - they just want it to work.

Again, imho a key issue that needs to be addressed is to get as much 
code to work ASIS - as much as possible - rather than the current 
situation (without my proposed patch) the requires many changes must be 
made to AIX, as root, and these must also be maintained (performed 
again) after any update to the OS - at least controlled. This is not 
facilitate using python, better python acceptance (on AIX) - and seems 
conducive to security breaches.

Sincerely,
Michael
> I am not comfortable with other aspects. I think I would have to see more discussion with other people to change my opinion:
>
> 1. CDLL("libintl.so") should not load “libintl.a(libintl.so.1)”. I understand you want this to help code written for Gnu or Linux automatically work on AIX, but it doesn’t feel correct and robust to me. Perhaps moving this sort of thing to a separate function or package would be better.
>
> 2. find_library("libintl.so") -> "libintl.a(libintl.so.1)". I would expect it to look for a shared library installed in something like "/usr/lib/liblibintl.so.a", unless I have misunderstood how compile-time linking (cc -llibintl.so) works.
See above - again, shooting myself for having included these additional 
"behavior" examples.

> 3. find_library() should not set the LIBPATH environment variable.
This was a convenience. It may be better to have the default search path 
hard-coded using a -Wl,blibpath link flag. As the discussion re: Linux 
support was to not include it (or was it remove it after it had been 
added) . shall follow and "Will remove" to be compareable.
> ----------
>
> _______________________________________
> Python tracker<report@bugs.python.org> <mailto:report@bugs.python.org>
> <http://bugs.python.org/issue26439> <http://bugs.python.org/issue26439>
> _______________________________________

A bug was found, and has been corrected - but I am unclear on how to best submit the differences.

Editing the current "review" code is not a good idea, as there are many differences.

Please recommend correct course of action.

p.s. I will have also removed the imports for _ctypes and the use of LIBPATH

Strictly speaking, Python 2.7 never had special support for RTLD_MEMBER or find_library(). That is why I am unsure about many of these changes being done in 2.7. They seem more like new features than bug fixes. Especially once you start talking about a smart CDLL() that tests for a file and then falls back to RTLD_MEMBER.

Whether you represent an archive member as an (archive, member) tuple, or as an "archive(member)" string, or something else, it is still an extension of the API of CDLL(), and arguably also of find_library().

I think I have already tried to explain that on Linux, find_library("libm.so") is far from normal usage (although may be technically legal if there was a library called liblibm.so.so).

Also, on Linux I wouldn’t recommend CDLL("libcrypt.so"), which relies on a file only intended for build time. On Debian, the glibc library <https://packages.debian.org/source/jessie/glibc> has separate packages for shared libraries (libc6) and development libraries (libc6-dev). It should be possible to run programs with libc6 installed, but without libc6-dev. Looking at the file lists for “i386” architecture, libc6-dev includes:

/usr/lib/i386-linux-gnu/libcrypt.a
/usr/lib/i386-linux-gnu/libcrypt.so

It symbolically links to /lib/i386-linux-gnu/libcrypt.so.1, which is provided by libc6:

/lib/i386-linux-gnu/libcrypt-2.19.so
/lib/i386-linux-gnu/libcrypt.so.1

So on Debian (and with Linux in general), CDLL("libcrypt.so") may work in some cases, but you should normally use CDLL("libcrypt.so.1") instead.

Usually people just upload a new version of the original patch (original changes, plus new changes, in one file). I think it would be too confusing looking at differences between two patches.

On 28-May-16 04:04, Martin Panter wrote:
> Martin Panter added the comment:
>
> Strictly speaking, Python 2.7 never had special support for RTLD_MEMBER or find_library(). That is why I am unsure about many of these changes being done in 2.7. They seem more like new features than bug fixes. Especially once you start talking about a smart CDLL() that tests for a file and then falls back to RTLD_MEMBER.
Well, actually, within AIX I was intending to "just use" RTLD_MEMBER 
when "the string" ended in a ")" as that is the correct additional mode 
bit for an established interface dlopen(). I perhaps do not understand 
(certainly did not) the goal of CDLL(), better Lib/ctypes

As far as Python-2.7 is concerned - I would like to see it "in" because 
there is a lot of Python-2(.7) code out there that will not work without 
manageling AIX at a low level (i.e., current practice is to extract .so 
members from archives so dlopen() will be able to open them.
>
> Whether you represent an archive member as an (archive, member) tuple, or as an "archive(member)" string, or something else, it is still an extension of the API of CDLL(), and arguably also of find_library().
I would want to disagree. A string is a string, regardless of the 
characters in it. And, I have verified - if a file 
/usr/lib/libFOO.a(libFoo.so.X) exists dlopen() will open it when 
RTLD_MEMBER is not ORed into the mode.
> I think I have already tried to explain that on Linux, find_library("libm.so") is far from normal usage (although may be technically legal if there was a library called liblibm.so.so).
find_library("libm.so") is far from normal usage: again, my 
misunderstanding - I was reading as much as possible from various 
sources - and my head was spinning.
In util.py there was no "libFOO.so", only "FOO" and "libFOO" - so my 
find_library() tests are not correct. Agreed.

>
> Also, on Linux I wouldn’t recommend CDLL("libcrypt.so"), which relies on a file only intended for build time. On Debian, the glibc library <https://packages.debian.org/source/jessie/glibc> has separate packages for shared libraries (libc6) and development libraries (libc6-dev). It should be possible to run programs with libc6 installed, but without libc6-dev. Looking at the file lists for “i386” architecture, libc6-dev includes:
>
> /usr/lib/i386-linux-gnu/libcrypt.a
> /usr/lib/i386-linux-gnu/libcrypt.so
My expectation (misconception perhaps) is that Linux uses a .a archive 
for static linking, and .so shared-library for dynamic-loading. The AIX 
loader/linker looks first in .a archives for shared libraries (when 
static is not specified) and if no member is found (member name is not 
relevant for the search, only the symbol and the object size (32 or 64 
bit)). If not found in libFOO.a then a search for a specific file is 
done libFOO.so.

An assumption of mine is: if I specified (not usual on AIX, based on my 
experience) -lc.so.6 AIX would look for an archive named libc.so.6.a 
first. If found, would look for a member in that archive. If not found 
as a .a archive and/or not found as a file at all then it will look for 
a file libFOO.so.6

>
> It symbolically links to /lib/i386-linux-gnu/libcrypt.so.1, which is provided by libc6:
>
> /lib/i386-linux-gnu/libcrypt-2.19.so
> /lib/i386-linux-gnu/libcrypt.so.1
>
> So on Debian (and with Linux in general), CDLL("libcrypt.so") may work in some cases, but you should normally use CDLL("libcrypt.so.1") instead.
Agreed - CDLL("libFOO.so") is not preferred. But, should it be refused? 
Or, if it is accepted on Linux - and works when the (symbolic) file 
exists - must it be refused on AIX? If refusing it on AIX is what it 
takes to make the patch acceptable - then code it to refuse it is.
>
> Usually people just upload a new version of the original patch (original changes, plus new changes, in one file). I think it would be too confusing looking at differences between two patches.
So, just provide the new delta between "2.7.11(.0) and what I have, not 
the changes between patches. Coming up tomorrow.

In short, my head was spinning and I thought I saw problems that are not 
meant to exist, in particular find_library("libFOO.so") - that should be 
coded to reject I guess. I would still like to let something like 
"libc.so" be creative, as libc.so will never exist, by default, on AIX. 
My guess is that is a legacy thing from AIX4. (and I think the default 
name is shr.o when using the ld flag -G to create a shared object aka 
shared library)
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue26439>
> _______________________________________

As a separate file in the hope it re-fits into Mercurial better

without aixutil.py - hopefully better for Mercurial import

Okay here are some more thoughts about your latest patch:

## Automatic RTLD_MEMBER ##

I was still uneasy about the automatic setting of RTLD_MEMBER. But I looked for how others handle this, and I found Libtool’s LTDL library, and Apache Portable Runtime (APR). Both have a similar (but stricter) automatic addition based on detecting the archive(member) notation:

http://git.savannah.gnu.org/cgit/libtool.git/commit/libltdl/loaders/dlopen.c?id=8fa719e
https://github.com/apache/apr/commit/c27f8d2

So I guess I can accept this way, if you think it is better than my other ideas:

# Always uses RTLD_MEMBER, never loads a plain file outside an archive
name = "libcrypto.a(libcrypto.so.1.0.0)"

# Other options, that could be returned by find_library() and would not conflict with a plain file name
name = ("libcrypto.a", "libcrypto.so.1.0.0")  # (archive, member)
name = ("libcrypto.a(libcrypto.so.1.0.0)", RTLD_MEMBER)  # (name, extra-flags)

libcrypto = CDLL(name)

## find_library() modes ##

In your find_library() function, you still have three parts. Can you confirm that each behaviour is intended:

A) If I have a file called "crypto" in the current directory, find_library("crypto") returns "crypto". This does not seem right. On Linux, “gcc [. . .] -lcrypto” does not look for a file exactly called “crypto”.

B) You are still stripping bits off the library name if it contains “lib” or a dot (.), so find_library("glib-2.0") is almost equivalent to find_library("b-2"). Isn’t this a bug?

C) find_library("crypto") will return "/usr/lib/crypto" if such a file exists. Just like in A), this does not seem right to me.

## Other things ##

* You don’t need to prefix most names with underscores, unless they could be confused with a public API. If you follow my earlier suggestion of renaming the new file to _aixutil.py (so it is obvious it is not a public module), then you can freely write “import re, os, sys”, etc.

* No need to add the internal variable names to the function signatures. Just write find_library(name), and if you need to initialize a variable, do that in the body.

* I suggest to go over all the regular expressions, and either change them to plain string searching, or make sure special characters and external variables are escaped as necessary. A comment explaining what the RE is trying to do might help too.

On 04-Jun-16 16:24, Martin Panter wrote:
> Martin Panter added the comment:
>
> Okay here are some more thoughts about your latest patch:
>
> ## Automatic RTLD_MEMBER ##
>
> I was still uneasy about the automatic setting of RTLD_MEMBER. But I looked for how others handle this, and I found Libtool’s LTDL library, and Apache Portable Runtime (APR). Both have a similar (but stricter) automatic addition based on detecting the archive(member) notation:
>
> http://git.savannah.gnu.org/cgit/libtool.git/commit/libltdl/loaders/dlopen.c?id=8fa719e
> https://github.com/apache/apr/commit/c27f8d2
The "stricter" from libtool is an excellent idea - I should done that as 
well, but I really hate string manipulation :)

I hope that the intent at least more clear, and should anyone ever 
complain about it not opening a file named: libFoo.a(libFOO.so) as a 
filename - by design, not accepted as a filename.
> So I guess I can accept this way, if you think it is better than my other ideas:
>
> # Always uses RTLD_MEMBER, never loads a plain file outside an archive
> name = "libcrypto.a(libcrypto.so.1.0.0)"
>
> # Other options, that could be returned by find_library() and would not conflict with a plain file name
> name = ("libcrypto.a", "libcrypto.so.1.0.0")  # (archive, member)
> name = ("libcrypto.a(libcrypto.so.1.0.0)", RTLD_MEMBER)  # (name, extra-flags)
>
> libcrypto = CDLL(name)
Isn't this more of an API change - name is no longer just a string?
>
> ## find_library() modes ##
>
> In your find_library() function, you still have three parts. Can you confirm that each behaviour is intended:
I have to catch a plane - will get back on these. Short - if I have a 
potential bug, then needs to be improved. More later.
>
> A) If I have a file called "crypto" in the current directory, find_library("crypto") returns "crypto". This does not seem right. On Linux, “gcc [. . .] -lcrypto” does not look for a file exactly called “crypto”.
>
> B) You are still stripping bits off the library name if it contains “lib” or a dot (.), so find_library("glib-2.0") is almost equivalent to find_library("b-2"). Isn’t this a bug?
>
> C) find_library("crypto") will return "/usr/lib/crypto" if such a file exists. Just like in A), this does not seem right to me.
>
> ## Other things ##
>
> * You don’t need to prefix most names with underscores, unless they could be confused with a public API. If you follow my earlier suggestion of renaming the new file to _aixutil.py (so it is obvious it is not a public module), then you can freely write “import re, os, sys”, etc.
>
> * No need to add the internal variable names to the function signatures. Just write find_library(name), and if you need to initialize a variable, do that in the body.
>
> * I suggest to go over all the regular expressions, and either change them to plain string searching, or make sure special characters and external variables are escaped as necessary. A comment explaining what the RE is trying to do might help too.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue26439>
> _______________________________________

Answering this again - now that the new patch is ready.

On 04-Jun-16 16:24, Martin Panter wrote:
> Martin Panter added the comment:
>
> Okay here are some more thoughts about your latest patch:
>
> ## Automatic RTLD_MEMBER ##
>
> I was still uneasy about the automatic setting of RTLD_MEMBER. But I looked for how others handle this, and I found Libtool’s LTDL library, and Apache Portable Runtime (APR). Both have a similar (but stricter) automatic addition based on detecting the archive(member) notation:
>
> http://git.savannah.gnu.org/cgit/libtool.git/commit/libltdl/loaders/dlopen.c?id=8fa719e
> https://github.com/apache/apr/commit/c27f8d2
>
> So I guess I can accept this way, if you think it is better than my other ideas:
a) I think it is more in line with what I perceive as normal practice
  libFOO = CDLL(find_library("FOO"))
As CDLL() has always had a 'simple' string as input and not a "tuple".
I have also added some lines to test/test_loading.py to test direct 
calling of CDLL() with fixed strings and a test of os.maxsize
and in util.py - but using e.g., CDLL(find_library("c") as behavior is 
dependent on 32 or 64-bit mode

depending on mode - different output:
  note: find_library("libssl64") is expected to return None - as it 
would be "abnormal" to have an archive libssl64.a or a file libssl64.so

cd Lib/ctypes
../../python util.py
# 32-bit mode:
None
libc.a(shr.o)
libbz2.a(libbz2.so)
find_library("c")        returns: libc.a(shr.o)
find_library("libc")     returns: libc.a(shr.o)
find_library("libssl")   returns: libssl.a(libssl.so)
find_library("libssl64") returns: None
find_library("ssl")      returns: libssl.a(libssl.so)
find_library("libiconv") returns: libiconv.a(libiconv.so.2)
find_library("intl")     returns: libintl.a(libintl.so.8)
libcrypt.a(shr.o)
<CDLL 'libc.a(shr.o)', handle c at 300e8330>
<CDLL 'libcrypt.a(shr.o)', handle d at 300e8830>

# 64-bit mode:
None
libc.a(shr_64.o)
libbz2.a(libbz2.so.1)
find_library("c")        returns: libc.a(shr_64.o)
find_library("libc")     returns: libc.a(shr_64.o)
find_library("libssl")   returns: libssl.a(libssl64.so.1.0.0)
find_library("libssl64") returns: None
find_library("ssl")      returns: libssl.a(libssl64.so.1.0.0)
find_library("libiconv") returns: libiconv.a(shr4_64.o)
find_library("intl")     returns: libintl.a(libintl.so.1)
libcrypt.a(shr_64.o)
<CDLL 'libc.a(shr_64.o)', handle c at 7000000001269e8>
<CDLL 'libcrypt.a(shr_64.o)', handle d at 700000000126a20>

>
> # Always uses RTLD_MEMBER, never loads a plain file outside an archive
> name = "libcrypto.a(libcrypto.so.1.0.0)"
>
> # Other options, that could be returned by find_library() and would not conflict with a plain file name
> name = ("libcrypto.a", "libcrypto.so.1.0.0")  # (archive, member)
> name = ("libcrypto.a(libcrypto.so.1.0.0)", RTLD_MEMBER)  # (name, extra-flags)
>
> libcrypto = CDLL(name)
>
> ## find_library() modes ##
>
> In your find_library() function, you still have three parts. Can you confirm that each behaviour is intended:
I was being "Q&D" here, not changing the aixutils.py (now _aixutils.py). 
My intent is to be comparable with other "posix" behaviors.
So, if you believe I am still not compatible with their behavior, 
forgive me - but also shoot me!
>
> A) If I have a file called "crypto" in the current directory, find_library("crypto") returns "crypto". This does not seem right. On Linux, “gcc [. . .] -lcrypto” does not look for a file exactly called “crypto”.
>
> B) You are still stripping bits off the library name if it contains “lib” or a dot (.), so find_library("glib-2.0") is almost equivalent to find_library("b-2"). Isn’t this a bug?
>
> C) find_library("crypto") will return "/usr/lib/crypto" if such a file exists. Just like in A), this does not seem right to me.
All should be seen as bugs, and I hope I coded it correctly to not do 
this anymore.
> ## Other things ##
>
> * You don’t need to prefix most names with underscores, unless they could be confused with a public API. If you follow my earlier suggestion of renaming the new file to _aixutil.py (so it is obvious it is not a public module), then you can freely write “import re, os, sys”, etc.
I had missed, certainly not understood the context, before. aixutil.py 
is now _aixutil.py. Originally I had done this to make the diff in 
util.py much simplier, but also because I incorrectly thought CDLL() was 
frequently called with "foo" or "libfoo". In short, trying to prevent a 
non-existent problem.
__init__.py delta is also much much simpler to grasp.
> * No need to add the internal variable names to the function signatures. Just write find_library(name), and if you need to initialize a variable, do that in the body.
Oops - not removed those yet. That was done to be sure there was no 
global scope interference. If you feel it is vital they be removed - 
will be done.
>
> * I suggest to go over all the regular expressions, and either change them to plain string searching, or make sure special characters and external variables are escaped as necessary. A comment explaining what the RE is trying to do might help too.
Ugh. I actually hate string stuff. I will need to spend more time on 
that. FYI - after a lot of testing with various expr strings I found not 
using the 'raw' format worked better (read: I was probably making 
beginner errors, and made fewer when using "normal" back-slash escapes.

However, I shall add comments in the next pass (which I now conclude is 
unavoidable)

However #2 - I hope we are really really close to an acceptable patch.

Michael
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue26439>
> _______________________________________

aixutil.py renamed as _aixutil.py and other changes in response to Martin's comments of 4 June

aixutil.py renamed as _aixutil.py and other changes in response to Martin's comments of 4 June

delta of changes to __init__.py, util.py, and test/test_loading.py

Will try to change the existing code from os.popen to subprocess (Issue 26439) to set a better example for new code like this

* You may also just modify my "attempt" in _aixutil.py

I recall you said something about the p.wait() being a potential to hang, but I also want to be sure the subprocess has exited properly - as it is not something to be running in parallel - and be sure that the "lines" I return are complete.

And, I see I have more underscores to remove :)

* I shall work on the comments you requested asap.

On 6/10/2016 6:33 AM, Martin Panter wrote:
> Martin Panter added the comment:
>
> Will try to change the existing code from os.popen to subprocess (Issue 26439) to set a better example for new code like this
>
> ----------
> dependencies: +avoid using a shell in ctypes.util: replace os.popen with subprocess
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue26439>
> _______________________________________

I have added comments to _aixutil.py - but while doing so I came up with 
some thoughts re "special cases", where

a) the member is not in a archive (i.e., looking for a file)

b) the filename does not begin with "lib"

c) CDLL.LoadLibrary(name) succeeds

So, given that /usr/lib is default search path, and file FOO.so exists 
as /usr/lib/FOO.so - should find_library(name) return name, or None.

d) name = ../some/where/Foo.so; CDLL.LoadLibrary("../some/where/Foo.so") 
works - should find_library(name) return name

e) assume find_library("../some/where/Foo") returns 
../some/where/libFoo.so, or None - again, given that 
../some/where/libFoo.a does not exist

.
Michael, how are you supposed to apply your latest patch? I have Gnu Patch 2.7.5, but even in the best case it doesn’t find the files to patch:

$ patch -p1 -n < python.Lib.ctypes.160608.patch
can't find file to patch at input line 2
The text leading up to this was:
--------------------------
|diff -r Python-2.7.11/Lib/ctypes/__init__.py python-2.7.11.5/Lib/ctypes/__init__.py
--------------------------

Perhaps can you make a “unified” diff (-u option) like you did in your previous patches? Whatever you did with Python3.issue26439.160511.patch seemed to work best.

Also, since your patches add new functionality (the automatic archive member detection), they will have to be for Python 3, not 2.

===

Tuple vs "archive(member)" string being an API change:
> name = "libcrypto.a(libcrypto.so.1.0.0)"
> name = ("libcrypto.a", "libcrypto.so.1.0.0")
> name = ("libcrypto.a(libcrypto.so.1.0.0)", RTLD_MEMBER)

IMO all three options are API changes. Even though the first one is still a string, we are changing the meaning of the string. It no longer always means a plain file or library name, and you lose the ability to indicate a plain library file when the special format is used (even though that ability is unlikely to be used much).

===

Yes, please remove the internal variables from the signatures. Local variables always override the global scope anyway, so it makes no difference. And if you use the wrong kind of object (mutable object) as a default value in the signature, it can actually lead to subtle bugs.

I am happy to try modifying your code (once we get the patch format etc figured out). That is probably the quickest way to fix up some of the problems that are obvious to me, e.g. variables in signatures, subprocess driving, etc.

===

With my understanding, find_library("FOO") should not recognize /usr/lib/FOO.so, it should only look for /usr/lib/libFOO.so, etc. So it should return None. Does linking with -lFOO recognize /usr/lib/FOO.so for you?

Similarly, find_library("../some/where/Foo.so") shouldn’t work. You wouldn’t pass that with -l to the linker, I don’t think.

On 6/12/2016 11:41 AM, Martin Panter wrote:
> .
> Michael, how are you supposed to apply your latest patch? I have Gnu Patch 2.7.5, but even in the best case it doesn’t find the files to patch:
>
> $ patch -p1 -n < python.Lib.ctypes.160608.patch
> can't find file to patch at input line 2
> The text leading up to this was:
> --------------------------
> |diff -r Python-2.7.11/Lib/ctypes/__init__.py python-2.7.11.5/Lib/ctypes/__init__.py
> --------------------------
I must have forgotten the u (-r rather than -ru) (read mistyped, as I 
"always" use the -u option)
> Perhaps can you make a “unified” diff (-u option) like you did in your previous patches? Whatever you did with Python3.issue26439.160511.patch seemed to work best.
Will redo all of them in about 24 hours.
> Also, since your patches add new functionality (the automatic archive member detection), they will have to be for Python 3, not 2.

Well, let me know what needs to be deleted - I see the member detection 
as the equivalent of reading the output of "ldconfig -p" to search for 
where an archive might be, or compareable with the gcc command (forgot 
the options) to say where it is looking.

Not going into Python2 misses the whole point from my perspective.

Unfortunate (for my perspective) is that noone else is saying anything 
one way or the other.

_aixutil.py to be paired with Python2.Lib.ctypes.16.06.11.patch

The patch for Python2 - however, only now see the change in selections for version. Will need to redo/test Lib/ctypes/*.py in newer version.

Note also, the additional tests in util.py are for my testing - I do not expect them to stay, but I do want you aware of how I am testing - see also change in Lib/ctypes/test/test_loading.py

updated patch for Python 2.7 (shall adopt and submit new patch for Python3 based on this - next week)

used gnu diff to generate diff output.

Additional tests in util.py are for demo only, would expect them to be removed, however, the additional test in test/test_loading.py could remain.

Uploading a combined patch that should work with Rietveld, and I will leave some comments.

I don’t think there is much that I am comfortable changing in Python 2.7. In general it is a bad idea to add new features that are only going to be available in e.g. 2.7.13+ and won’t work in older versions of 2.7. One rule of thumb is if a change would require additional documentation, then it is probably a new feature. In this case, find_library() is returning something that is not a plain filename or path name, so the documentation also needs changing.

If we added a version of find_library() that returned only file or path names (and not archive members!), I might consider that acceptable for 2.7, because it is filling in missing AIX functionality, and sticks to the current documentation. But it sounds like most libraries are archive members, so I can understand if that sort of change is not worth the effort.

I left a few more comments, but I see that many of my earlier comments (both in Rietveld and main bug thread) still apply.

I’m still struggling to understand all the special cases for the find_library() argument. You seem to have added even more recently (although I think you removed some as well). Can you point to any documentation, command lines, etc demonstrating why Python should support:

find_library("libNAME")
find_library("search:paths/NAME")
find_library("NAME.so*")
find_library("/NAME")  # Not sure if this case is even relevant

Perhaps it would help if you explained what a typical AIX compiler or linker command line looks like. Or if you updated the documentation <https://docs.python.org/3.5/library/ctypes.html#finding-shared-libraries> to explain the cases for AIX.

Thanks again for your help.

Note: I started a issue# is because too much of what I was submitting here was "extension" and not "correction".

Hopefully, the new issue# will be cleaner.

For the record, I presume you are referring to Issue 27435.

New patch (diff) - that I hope address the comments made in the previous submission

patch to include RTLD_MEMBER in Modules/_ctypes and also "document", or not document, RTLD_NOW

The documentation is in RST format in the Doc/ directory. For basic stuff, you can just copy the syntax from existing text, or see e.g. <https://docs.python.org/devguide/documenting.html#restructuredtext-primer> for more hints. For this change, I think we need to mention in Doc/library/ctypes.rst:

* the special AIX changes for find_library(); include “.. versionchanged::” (or maybe versionadded? I’m not sure) notice
* the change for the CDLL constructor, also with versionchanged

Perhaps also add an entry to Doc/whatsnew/3.6.rst.

Patch 160823 does not addressed many of my previous comments. Please have a closer look. I can make simple changes to simplify the code myself, but I don’t really know what to do about the questionable regular expressions, for instance.

Also, see <https://bugs.python.org/issue26439#msg268885>. What was wrong with the cases supported by your original patches?

On 27-Aug-16 09:11, Martin Panter wrote:
> Martin Panter added the comment:
>
> The documentation is in RST format in the Doc/ directory. For basic stuff, you can just copy the syntax from existing text, or see e.g. <https://docs.python.org/devguide/documenting.html#restructuredtext-primer> for more hints. For this change, I think we need to mention in Doc/library/ctypes.rst:
>
> * the special AIX changes for find_library(); include “.. versionchanged::” (or maybe versionadded? I’m not sure) notice
> * the change for the CDLL constructor, also with versionchanged
>
> Perhaps also add an entry to Doc/whatsnew/3.6.rst.
>
> Patch 160823 does not addressed many of my previous comments. Please have a closer look. I can make simple changes to simplify the code myself, but I don’t really know what to do about the questionable regular expressions, for instance.
As far as regular expressions go, that will always be difficult for me 
aka questionable for you. Every language has it's nuances and I never 
seem to get them right.

>
> Also, see <https://bugs.python.org/issue26439#msg268885>. What was wrong with the cases supported by your original patches?
As you clearly pointed out, there were no prior cases showing their 
need. I was taking an unbiased approach in that I knew no previous code 
well. I was just trying to make it work and was trying to think of cases 
were it could go wrong. In short, I was trying to solve issues that do 
not exist. People accept whatever libFOO.so points to as a symbolic link 
- and expect libSOO.so to be the name of the shared library that 
dlopen() is going to open somewhere.

Once I understood the boundaries of find_library("foo") I limited myself 
to only resolve in the way the compile-time resolution is done. As such, 
find_library("c") is the equivalent of -lc and needs to resolve to 
libc.a(shr.o) - period (32-bit) or libc.a(shr_64.o) (64-bit).

The linker is not looking for a particular member name: it only uses 
-lFOO to find libFOO.a, and it it exists it looks for a symbol. The 
member name that contains the symbol is what it stores in the executable 
(displayed via dump -H). So, getting back to this patch and packaging 
conventions.

Basically, libtool has standardised how members are named, i.e., 
versioning. In most cases with OSS packages the only member-name that 
gets stored is the same name that libFOO.so would be a symbolic link to. 
Again, the compiler-linker only needs the name of the archive. The 
member name within the archive is irrelevant - from a compiler/linker 
perspective.

My goal for find_library() is to find the most likely name based on some 
legacy "rules" from the way IBM packaged libraries starting over 20 
years ago (aka AIX 4 standards) that are alive today to support binary 
compatibility - as much as possible. Mainly, that is relevant for 
libraries/archives such as libc.a. For new libraries, especially OSS 
packages built using the GNU autotools (and finishing with libtool) 
there are other conventions.

Finally, we had different goals - my focus was on writing something that 
matches other platforms python2 behavior, not on writing a new syntax 
specific for Python3.6 and AIX. If I were to do be writing a new syntax 
I would prefer that it also work for other platforms. Something 
different for only one platform feels wrong - imho.

In closing:
a) regular expressions and me is always a headache for someone. please 
accept my apologies if I have given you a headache.
b) I had assumed abilities for find_library() (from studying the output 
of ldconfig and trying to follow the regular expressions in the current 
code) that are not used. To assume makes an ass of u and me - especially me.
c) I also apologize for not meeting your expectations. I cut code, 
rather than defend it, as you were correct that is was not based on 
meeting the needs of general aka current practice.

I am not trying to redesign find_library(). My hope is that most python 
code would work asis - if they follow conventions aka general practice 
(not equivalent to best practice as best depends (in part) on your goal).
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue26439>
> _______________________________________

Regarding the regular expressions, I (or someone else unfamiliar with AIX) may be able to adjust them if you can explain what you are trying to achieve. Take the first one I commented on <https://bugs.python.org/review/26439/diff/17689/Lib/ctypes/_aixutil.py#newcode132> for example. You added a comment:

# '\[%s_*64\.so\]' % name, -> has either _64 or 64 added to name

As written, this will match many strings including

[<NAME>___64.so]

However the annotation leads me to belive you want it to match two cases only:

[<NAME>64.so]
[<NAME>_64.so]

I do not know whether to fix the annotation (has 64 preceded by any number of underscores), or whether to fix the regular expression (_?64).

On 04/09/2016 06:11, Martin Panter wrote:
> I do not know whether to fix the annotation (has 64 preceded by any number of underscores), or whether to fix the regular expression (_?64).

The later - _?64. Working on this today. Thank you for the correction.

Not always as elegant as I would wish (do not like the idea of "while 1:" and later a break...

But, all in all, much improved by redoing the processing of the subprocess output and getting rid of more noise.

Hope this meets your approval!

p.s. This is a patch compared to previous patch. If a diff compared to Python-3.6.0a4 I can remake the patch.

Sigh - missed the feature cutoff that would have made this easier to get into python...

Anyway - have learned a few new things about python def: syntax and removed some bits that I thought were suitable for variable "initialization" - but tend to be static (not what I was thinking).

Also, "new and improved" comments about what the code is doing.

Thank you all for your patience and feedback - especially Martin!

New changeset 01885f78b299 by Martin Panter in branch '2.7':
Issue #26439: Document that RTLD_NOW is always added
https://hg.python.org/cpython/rev/01885f78b299

New changeset 0db4403e62c4 by Martin Panter in branch '3.5':
Issue #26439: Document that RTLD_NOW is always added
https://hg.python.org/cpython/rev/0db4403e62c4

New changeset 4b7e51998a90 by Martin Panter in branch '3.6':
Issue #26439: Merge ctypes doc from 3.5 into 3.6
https://hg.python.org/cpython/rev/4b7e51998a90

New changeset f496fb6bf4a0 by Martin Panter in branch 'default':
Issue #26439: Merge ctypes doc from 3.6
https://hg.python.org/cpython/rev/f496fb6bf4a0

Hi Michael, I have done some cleanup and modifications to your patch. The result is in aix-library.161001.patch, which has all the changes, i.e. it is not based on another patch. More significant changes I made:

* Change getExecLibPath_aix() and find_parts() to return a list object, rather than building a colon-separated string only to be pulled apart again
* Escape dots in get_legacy() regular expressions, so that they no longer match [shr_64xo], [shrxo], etc.
* Make get_dumpH() return the a list of (object, objectinfo) tuples, where objectinfo is a list of lines; avoids building multiline strings and then splitting them apart again
* Rewrite get_exactMatch() and get_version() without nested inline “for” loops; use RE capture group
* Reuse util._last_version() instead of copying the _num_version() function
* Use lower case B for liB in get_member(). This means e.g. libcrypto.so is now preferred over libcrypto.so.1.0.0.

I did test it a bit on Linux with faked dump -H output, but I may have made mistakes that I did not pick up.

Also, this still needs documentation, and I think some more tests for the test suite exercising various aspects of find_library() would be nice if possible.

Another thing: in the last few patches, you dropped the definition of RTLD_MEMBER from Modules/_ctypes/_ctypes.c. Is that intended, or just a temporary thing?

On 01-Oct-16 08:44, Martin Panter wrote:
> Martin Panter added the comment:
>
> Hi Michael, I have done some cleanup and modifications to your patch. The result is in aix-library.161001.patch, which has all the changes, i.e. it is not based on another patch.
Thanks.
>   More significant changes I made:
>
> * Change getExecLibPath_aix() and find_parts() to return a list object, rather than building a colon-separated string only to be pulled apart again
> * Escape dots in get_legacy() regular expressions, so that they no longer match [shr_64xo], [shrxo], etc.
> * Make get_dumpH() return the a list of (object, objectinfo) tuples, where objectinfo is a list of lines; avoids building multiline strings and then splitting them apart again
> * Rewrite get_exactMatch() and get_version() without nested inline “for” loops; use RE capture group
> * Reuse util._last_version() instead of copying the _num_version() function
> * Use lower case B for liB in get_member(). This means e.g. libcrypto.so is now preferred over libcrypto.so.1.0.0.
That was a typo - to be sure I was still finding the versioned ones (the 
previous ones had had a bug that they no longer found the "standard" 
one. I forgot to remove (rather save file before the diff command) - you 
see everything!
>
> I did test it a bit on Linux with faked dump -H output, but I may have made mistakes that I did not pick up.
Will apply the patch, build in 32 and 64 bit modes, and respond.
>
> Also, this still needs documentation, and I think some more tests for the test suite exercising various aspects of find_library() would be nice if possible.
Working on that - have been posting some questions on python-list as I 
want to build an interface with an AIX performance library (libperfstat).
Without this library one must run a command and then do some string 
manipulation, and sometimes call a second command once that has been 
found (e.g., uuid calls to get a MAC address, cloud-init to get 
boottime) - things that - with a library do not need a subprocess at all.

But I shall also write up the AIX dlopen() process to explain how both 
.so and .a(member.so) works (as the argument to CDLL).
>
> Another thing: in the last few patches, you dropped the definition of RTLD_MEMBER from Modules/_ctypes/_ctypes.c. Is that intended, or just a temporary thing?
Temporary thing - as I keep hoping for inclusion in Python2 and I recall 
you not wanting to add that into _ctypes on Python2.
>
> ----------
> versions:  -Python 3.6
> Added file: http://bugs.python.org/file44902/aix-library.161001.patch
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue26439>
> _______________________________________

I have spent the last two hours trying to run the test - however, it fails with:

root@x064:[/data/prj/python/python-3.6.0.177/Lib/ctypes]../../python util.py
Traceback (most recent call last):
  File "util.py", line 102, in <module>
    import ctypes._aix as aix
  File "/data/prj/python/python-3.6.0.177/Lib/ctypes/_aix.py", line 15, in <module>
    from . import util
  File "/data/prj/python/python-3.6.0.177/Lib/ctypes/util.py", line 102, in <module>
    import ctypes._aix as aix
AttributeError: module 'ctypes' has no attribute '_aix'

I have noticed several issues with the file that used to be named just
./build/_sysconfigdata.py

but is now: ./build/lib.aix-5.3-3.6/_sysconfigdata_m_aix5_.py

I am guessing something is wrong there - I am going to try copying only _aix.py to the Python2 branch and see if it works there -- and also dig deeper into what is going wrong with Python3.6*

Curious.

When in 32-bit mode changing line 15 of _aix.py to

   +14  import re, os, sys
   +15  # from . import util

The Lib/ctypes/util.py works.

In 64-bit mode it does not:
instead:

root@x064:[/data/prj/python/python-3.6.0.177/Lib/ctypes]../../python util.py
m       :: None
c       :: libc.a(shr_64.o)
Traceback (most recent call last):
  File "util.py", line 355, in <module>
    test()
  File "util.py", line 330, in test
    print("bz2\t:: %s" % find_library("bz2"))
  File "util.py", line 104, in find_library
    return aix.find_library(name)
  File "/data/prj/python/python-3.6.0.177/Lib/ctypes/_aix.py", line 255, in find_library
    (base, member) = find_shared(libpaths, name)
  File "/data/prj/python/python-3.6.0.177/Lib/ctypes/_aix.py", line 247, in find_shared
    member = get_member(re.escape(name), members)
  File "/data/prj/python/python-3.6.0.177/Lib/ctypes/_aix.py", line 189, in get_member
    member = get_version(name, members)
  File "/data/prj/python/python-3.6.0.177/Lib/ctypes/_aix.py", line 170, in get_version
    return util._last_version(versions, '.')
NameError: name 'util' is not defined

+++++
When the comment is removed, i.e.
from . import util

both 32 and 64-bit report:
root@x064:[/data/prj/python/python-3.6.0.177/Lib/ctypes]../../python util.py
Traceback (most recent call last):
  File "util.py", line 102, in <module>
    import ctypes._aix as aix
  File "/data/prj/python/python-3.6.0.177/Lib/ctypes/_aix.py", line 15, in <module>
    from . import util
  File "/data/prj/python/python-3.6.0.177/Lib/ctypes/util.py", line 102, in <module>
    import ctypes._aix as aix
AttributeError: module 'ctypes' has no attribute '_aix'

This last condition also occurs in Python2

Have a way to have both 64-bit and 32-bit modes working.

root@x064:[/data/prj/python/python-3.6.0.177/Lib/ctypes]../../python `pwd`/util.py
m       :: None
c       :: libc.a(shr_64.o)
bz2     :: libbz2.a(libbz2.so.1)
crypt   :: libcrypt.a(shr_64.o)
crypto  :: <CDLL 'libcrypto.a(libcrypto64.so.1.0.0)', handle c at 0x7000000001ffcf8>
c       :: <CDLL 'libc.a(shr_64.o)', handle d at 0x7000000001ffcf8>

root@x064:[/data/prj/python/python-3.6.0.177/Lib/ctypes]../../python `pwd`/util.py
m       :: None
c       :: libc.a(shr.o)
bz2     :: libbz2.a(libbz2.so)
crypt   :: libcrypt.a(shr.o)
crypto  :: <CDLL 'libcrypto.a(libcrypto.so)', handle c at 0x301f78b0>
c       :: <CDLL 'libc.a(shr.o)', handle d at 0x301f78b0>

Will post patch asap. Going to shorten some lines going well over 76-char lime-limit.

Besides correcting a small error, also my attempt to follow the
guidelines to not import *, but to actually specify all bits
that are to be imported.

This is a patch compered to Python-3.6b1

Just some minor comments on aix-library.161004.patch:

Instead of _util.py, I wonder if the new file should have a different name, like _util_common.py, to avoid being too similar to util.py.

+def get_shared(input):
+    """Internal support function: examine the get_dumpH() output and
+    return a list of all shareable objects indicated in the output the
+    character "[" is used to strip off the path information.

Needs a newline or new sentance to separate “output” and “the character”.

+def get_legacy(members):
+    [. . .]
+    # shr.o is the preffered name so we look for shr.o first

Spelling: preferred [single F, double R]

+def get_version(name, members):
+    """[. . .]
+    Before the GNU convention became the standard scheme regardless of
+    binary size AIX packagers used GNU convention "as-is" for 32-bit
+    archive members but used an "distinguishing" name for 64-bit members.

Should be: a "distinguishing" [not “an”]

I would like to say:
a) I am happy this is being considered for Python3-3.7

b) I still believe it is a "bug" plain and simple - it (find_library()) cannot work in "normal" situations. Why "IBM" or others never addressed this is beyond me - but - imho - that is a poor argument for not fixing it.

FYI: As I have documented in ... this does not mean that ctypes.CDLL is incapable of loading a library. If you know the 'magic', aka what is not being provided by this "implementation dependent" module - it is possible to load either a .so file (actually a file by any name as long as it is a shared archive) - or an AIX-style .a archive with shared libraries stored internally as "archive members" - again, any member name is acceptable.

The bug: The "default code" behind an "else:" block depends on the presence of the program "/sbin/ldconfig" to create a list of shared libraries. Normally, this program is not installed on an AIX system (certainly not production - maybe a system with gcc environment and using the gcc ld rather than the AIX ld (the latter being normal).

Using python -m trace -C /tmp --count Lib/src/util.py The following (key) counts are reported.

This is all that is being called - so when /sbin/ldonfig is not installed (which is usual) - so this code can only return 'None'.

Environment: AIX 5.3, AIX 6.1, AIX 7.1 - all identical

root@x064:[/]ls -l /sbin/ld*
ls: 0653-341 The file /sbin/ld* does not exist.

michael@x071:[/home/michael]ls /sbin/ld*
ls: cannot access '/sbin/ld*': A file or directory in the path name does not exist.
(GNU coreutils ls here)

root@x062:[/]ls /sbin/ld*
ls: 0653-341 The file /sbin/ld* does not exist.

So, obviously the sys.popen() call below will always 'fail'.


The basic trace from util.py:
    1:     if os.name == "posix":
               # find and load_version
    1:         print find_library("m")
    1:         print find_library("c")
    1:         print find_library("bz2")


And how this is called:

           else:

    1:         def _findSoname_ldconfig(name):
    3:             import struct
    3:             if struct.calcsize('l') == 4:
                       machine = os.uname()[4] + '-32'
                   else:
    3:                 machine = os.uname()[4] + '-64'
    3:             mach_map = {
    3:                 'x86_64-64': 'libc6,x86-64',
    3:                 'ppc64-64': 'libc6,64bit',
    3:                 'sparc64-64': 'libc6,64bit',
    3:                 's390x-64': 'libc6,64bit',
    3:                 'ia64-64': 'libc6,IA-64',
                       }
    3:             abi_type = mach_map.get(machine, 'libc6')

                   # XXX assuming GLIBC's ldconfig (with option -p)
    3:             expr = r'\s+(lib%s\.[^\s]+)\s+\(%s' % (re.escape(name), abi_type)
    3:             f = os.popen('LC_ALL=C LANG=C /sbin/ldconfig -p 2>/dev/null')
    3:             try:
    3:                 data = f.read()
                   finally:
    3:                 f.close()
    3:             res = re.search(expr, data)
    3:             if not res:
    3:                 return None
                   return res.group(1)

    1:         def find_library(name):
    3:             return _findSoname_ldconfig(name) or _get_soname(_findLib_gcc(name))


And with python2-2.7.13 - the counts start the same - but the result is the same - by definition 'None' because subprocess.Popen() also
does not have any output to search...

    1:     if os.name == "posix":
               # find and load_version
    1:         print find_library("m")
    1:         print find_library("c")
    1:         print find_library("bz2")

           else:

    1:         def _findSoname_ldconfig(name):
    3:             import struct
    3:             if struct.calcsize('l') == 4:
    3:                 machine = os.uname()[4] + '-32'
                   else:
                       machine = os.uname()[4] + '-64'
    3:             mach_map = {
    3:                 'x86_64-64': 'libc6,x86-64',
    3:                 'ppc64-64': 'libc6,64bit',
    3:                 'sparc64-64': 'libc6,64bit',
    3:                 's390x-64': 'libc6,64bit',
    3:                 'ia64-64': 'libc6,IA-64',
                       }
    3:             abi_type = mach_map.get(machine, 'libc6')

                   # XXX assuming GLIBC's ldconfig (with option -p)
    3:             expr = r'\s+(lib%s\.[^\s]+)\s+\(%s' % (re.escape(name), abi_type)

    3:             env = dict(os.environ)
    3:             env['LC_ALL'] = 'C'
    3:             env['LANG'] = 'C'
    3:             null = open(os.devnull, 'wb')
    3:             try:
    3:                 with null:
    3:                     p = subprocess.Popen(['/sbin/ldconfig', '-p'],
    3:                                           stderr=null,
    3:                                           stdout=subprocess.PIPE,
    3:                                           env=env)
    3:             except OSError:  # E.g. command not found
    3:                 return None
                   [data, _] = p.communicate()
                   res = re.search(expr, data)
                   if not res:
                       return None
                   return res.group(1)

========
In closing:

a) this is an "issue" aka bug, plain and simple - even it is has been ignored as such (other issues only complained about poorer performance because /sbin/ldconfig was not available). Please do not say - not fixing it because noone ever complained before. Otherwise, what is the point of ever accepting bug-reports aka issues if they can just be ignored.

b) I want to thank Martin for his help on many (PEP8 et al) improvements to my initial proposals for a patch.

c) more important to me right now is that this be recognized as a bug - that should have been reported and resolved years ago. Personally, I do not care why noone ever reported it BUT I would like to see it properly identified that the default code is a specific implementation that is no way related to a normal AIX system - and an AIX-specific implementation is needed for normal operation of python2 and/or python3.

Thank you for your time and thought!

OOPS: I have a ... above, meant to be a link to a message.

I also needed to 'patch' util.py - with 32-bit build with something like this:
--- src/python-2.7.13/Lib/ctypes/util.py        2016-12-17 20:05:05 +0000
+++ python-2.7.13.0/Lib/ctypes/util.py  2017-01-13 13:29:12 +0000
@@ -299,6 +299,10 @@
             print cdll.LoadLibrary("libcrypto.dylib")
             print cdll.LoadLibrary("libSystem.dylib")
             print cdll.LoadLibrary("System.framework/System")
+        elif sys.platform[:3] == "aix":
+            from ctypes import CDLL
+            RTLD_MEMBER =  0x00040000
+            print CDLL("libc.a(shr.o)", RTLD_MEMBER)
         else:
             print cdll.LoadLibrary("libm.so")
             print cdll.LoadLibrary("libcrypt.so")


And 64-bit with:

    1:         elif sys.platform[:3] == "aix":
    1:             from ctypes import CDLL
    1:             RTLD_MEMBER =  0x00040000
    1:             print CDLL("libc_64.a(shr.o)", RTLD_MEMBER)

The respective output is:

root@x064:[/data/prj/python/python-2.7.13.0]./python Lib/ctypes/util.py
None
None
None
<CDLL 'libc.a(shr.o)', handle 10 at 300fe0b0>

root@x064:[/data/prj/python/python-2.7.12.0]./python Lib/ctypes/util.py
None
None
None
<CDLL 'libc.a(shr_64.o)', handle d at 7000000001941d0>

Michael Felt: if you still want the code compatible with Python 2 and 3 (and others are happy with that), I suggest documenting that in a code comment.

New changeset c5ae169e1b73315672770517bf51cf8464286c76 by Victor Stinner (Michael Felt) in branch 'master':
bpo-26439 Fix ctypes.util.find_library failure on AIX (#4507)
https://github.com/python/cpython/commit/c5ae169e1b73315672770517bf51cf8464286c76

Michael Felt: I merged your PR 🎉🍰✨ ! Thank you very much for your patience and perseverance. 80 comments on this bug, 117 comments and 19 commits in the PR... wow! AIX is not dead 😄

I leave the bug open since Martin, Mariatta and me were interested to make further minor coding style changes.

Although I'm unable to double check for the moment, feels like the "SVR4" support still is incomplete: Remember that even the libNAME.so file may be an archive, or a symlink to an archive, with the real Shared Object as member having the F_LOADONLY flag set and usually named shr.o or shr_64.o, besides an Import File shr.imp or shr_64.imp, which actually is used at linktime and referring to the real Shared Object: This is necessary to emulate the "DT_SONAME" feature seen with ELF shared libraries.

For reference, please have a look at the shared libraries created by recent libtool when --with-aix-soname=svr4 configure option is set.

> Although I'm unable to double check for the moment, feels like the "SVR4" support still is incomplete: (...)

Sorry, I don't know what is SVR4. Is it a version of AIX? If it's rarely used, maybe a different issue should be opened to ask for an enhancement.

SVR4 - stands for AT&T System V Release 4 - the beginning (as I understand it) of shared libraries as (indivudual) .so files in UNIX. SVR3 (and earlier) used .a files (aka archives with members inside).

Within this context, the "svr4" label originates in the "-bsvr4" AIX linker flag, and actually is another (yet fully documented by the ld(1) man page) method for creating shared libraries on AIX to support filename based shared library versioning, which is known as the DT_SONAME tag with the ELF binary format.

For details please refer to the GCC install doc:
https://gcc.gnu.org/install/configure.html#WithAixSoname

New changeset c0919c27c6442aa147ae559a936b1b8deb4ee783 by Mariatta in branch 'master':
bpo-26439: Convert %s in Lib/ctypes/_aix.py to f-strings. (GH-4986)
https://github.com/python/cpython/commit/c0919c27c6442aa147ae559a936b1b8deb4ee783

will open a new PR for ./Doc/library/ctypes.rst

It looks like c5ae169e1b73315672770517bf51cf8464286c76 broke find_library on macOS; see Issue33281.

imho - this should have status "closed"

Thank you Michael Felt for your big "Implement find_library() support in ctypes/util for AIX" contribution, commit c5ae169e1b73315672770517bf51cf8464286c76!

I close the issue.