I have successful enough with python 2.7.10 (for building cloud-init), including it finding openssl libraries during the installation od setuptools (before installing pip).

I have also been able to assemble saltstack - BUT - salt-master and salt-minion fail to start because ctypes.util.find_library() always returns 'None'.
======== EXCERPT ====
File "/opt/lib/python2.7/site-packages/salt/crypt.py", line 37, in <module>
import salt.utils.rsax931
File "/opt/lib/python2.7/site-packages/salt/utils/rsax931.py", line 69, in <module>
libcrypto = _init_libcrypto()
File "/opt/lib/python2.7/site-packages/salt/utils/rsax931.py", line 47, in _init_libcrypto
libcrypto = _load_libcrypto()
File "/opt/lib/python2.7/site-packages/salt/utils/rsax931.py", line 40, in _load_libcrypto
raise OSError('Cannot locate OpenSSL libcrypto')
OSError: Cannot locate OpenSSL libcrypto
=======
I built python using the IBM compiler, and my images do not have /sbin/ldconfig installed so the assumption that /sbin/ldconfig is always installed is "a bug".

in the util.py file the code reached is:

+219 else:
+220
+221 def _findSoname_ldconfig(name):
+222 import struct
+223 if struct.calcsize('l') == 4:
+224 machine = os.uname()[4] + '-32'
+225 else:
+226 machine = os.uname()[4] + '-64'
+227 mach_map = {
+228 'x86_64-64': 'libc6,x86-64',
+229 'ppc64-64': 'libc6,64bit',
+230 'sparc64-64': 'libc6,64bit',
+231 's390x-64': 'libc6,64bit',
+232 'ia64-64': 'libc6,IA-64',
+233 }
+234 abi_type = mach_map.get(machine, 'libc6')
+235
+236 # XXX assuming GLIBC's ldconfig (with option -p)
+237 expr = r'\s+(lib%s\.[^\\s]+)\s+\(%s' % (re.escape(name), abi_type)
+238 f = os.popen('/sbin/ldconfig -p 2>/dev/null')
+239 try:
+240 data = f.read()
+241 finally:
+242 f.close()
+243 res = re.search(expr, data)
+244 if not res:
+245 return None
+246 return res.group(1)
+247
+248 def find_library(name):
+249 return _findSoname_ldconfig(name) or _get_soname(_findLib_gcc(name))
+250

(I have not researched _get_soname or _findLib_gcc but neither of these "feel right" as AIX, by default, does not end library (archives) with .so (archives end with .a and archive members frequently end with .so)

That this is/has not been reported more frequently may be because python programmers are avoiding ctypes - when portability is essential.

I hope that, just as for Solaris - where an alternate program is used - that AIX can have a block:
# if os.name == "posix" and sys.platform.startswith("aix"):

so that if ldconfig is not available the command /usr/bin/dump could be used instead, and/or search LIBPATH and/or (when not empty) and/or ldd (for programs). Ideally, /sbin/ldconfig would not be need at all!

dump output:

root@x064:[/data/prj/gnu/bashRC1-4.4]dump -Xany -H /opt/bin/python

/opt/bin/python:

                    \*\*\*Loader Section***
                  Loader Header Information

VERSION# #SYMtableENT #RELOCent LENidSTR
0x00000001 0x000005ac 0x000035e3 0x0000006e

#IMPfilID OFFidSTR LENstrTBL OFFstrTBL
0x00000005 0x00030ee4 0x00006772 0x00030f52

                    \*\*\*Import File Strings***

INDEX PATH BASE MEMBER
0 /usr/vac/lib:/usr/lib:/lib
1 libc.a shr.o
2 libpthreads.a shr_xpg5.o
3 libpthreads.a shr_comm.o
4 libdl.a shr.o

root@x064:[/usr/bin]dump -Xany -H /usr/lib/libcrypto.a

/usr/lib/libcrypto.a[libcrypto.so]:

                    \*\*\*Loader Section***
                  Loader Header Information

VERSION# #SYMtableENT #RELOCent LENidSTR
0x00000001 0x00000ff9 0x0000498a 0x00000038

#IMPfilID OFFidSTR LENstrTBL OFFstrTBL
0x00000003 0x0004f1f0 0x00014636 0x0004f228

                    \*\*\*Import File Strings***

INDEX PATH BASE MEMBER
0 /usr/lib:/lib
1 libc.a shr.o
2 libpthreads.a shr_xpg5.o

/usr/lib/libcrypto.a[libcrypto.so.0.9.8]:

                    \*\*\*Loader Section***
                  Loader Header Information

VERSION# #SYMtableENT #RELOCent LENidSTR
0x00000001 0x00000c4e 0x00004312 0x00000038

#IMPfilID OFFidSTR LENstrTBL OFFstrTBL
0x00000003 0x00044c48 0x0000f236 0x00044c80

                    \*\*\*Import File Strings***

INDEX PATH BASE MEMBER
0 /usr/lib:/lib
1 libc.a shr.o
2 libpthreads.a shr_xpg5.o

/usr/lib/libcrypto.a[libcrypto.so.1.0.0]:

                    \*\*\*Loader Section***
                  Loader Header Information

VERSION# #SYMtableENT #RELOCent LENidSTR
0x00000001 0x00000ff9 0x0000498a 0x00000038

#IMPfilID OFFidSTR LENstrTBL OFFstrTBL
0x00000003 0x0004f1f0 0x00014636 0x0004f228

                    \*\*\*Import File Strings***

INDEX PATH BASE MEMBER
0 /usr/lib:/lib
1 libc.a shr.o
2 libpthreads.a shr_xpg5.o

/usr/lib/libcrypto.a[libcrypto64.so]:

                    \*\*\*Loader Section***
                  Loader Header Information

VERSION# #SYMtableENT #RELOCent LENidSTR
0x00000001 0x00000ff2 0x00004987 0x0000003e

#IMPfilID OFFidSTR LENstrTBL OFFstrTBL
0x00000003 0x00061758 0x00014e03 0x00061796

                    \*\*\*Import File Strings***

INDEX PATH BASE MEMBER
0 /usr/lib:/lib
1 libc.a shr_64.o
2 libpthreads.a shr_xpg5_64.o

/usr/lib/libcrypto.a[libcrypto64.so.0.9.8]:

                    \*\*\*Loader Section***
                  Loader Header Information

VERSION# #SYMtableENT #RELOCent LENidSTR
0x00000001 0x00000c47 0x0000430d 0x0000003e

#IMPfilID OFFidSTR LENstrTBL OFFstrTBL
0x00000003 0x000557b0 0x0000f9e7 0x000557ee

                    \*\*\*Import File Strings***

INDEX PATH BASE MEMBER
0 /usr/lib:/lib
1 libc.a shr_64.o
2 libpthreads.a shr_xpg5_64.o

/usr/lib/libcrypto.a[libcrypto64.so.1.0.0]:

                    \*\*\*Loader Section***
                  Loader Header Information

VERSION# #SYMtableENT #RELOCent LENidSTR
0x00000001 0x00000ff2 0x00004987 0x0000003e

#IMPfilID OFFidSTR LENstrTBL OFFstrTBL
0x00000003 0x00061758 0x00014e03 0x00061796

                    \*\*\*Import File Strings***

INDEX PATH BASE MEMBER
0 /usr/lib:/lib
1 libc.a shr_64.o
2 libpthreads.a shr_xpg5_64.o
root@x064:[/usr/bin]
root@x064:[/usr/bin]echo $?
0

ldd outpath:
root@x064:[/usr/bin]ldd /opt/bin/python
/opt/bin/python needs:
/usr/lib/libc.a(shr.o)
/usr/lib/libpthreads.a(shr_xpg5.o)
/usr/lib/libpthreads.a(shr_comm.o)
/usr/lib/libdl.a(shr.o)
/unix
/usr/lib/libcrypt.a(shr.o)

root@x064:[/usr/bin]ldd /usr/lib/libcrypto.a
ldd: /usr/lib/libcrypto.a: File is an archive.
root@x064:[/usr/bin]echo $?
2

p.s. On a debian (on POWER) system, the function is working, but the test seems a bit broken as well, i.e., cdll.LoadLibrary("libm.so") is not working even though

    if os.name == "posix":
        # find and load_version
        print find_library("m")
        print find_library("c")
        print find_library("bz2")

has successfully printed.
root@ipv4:/home/michael# python -m ctypes.util
libm.so.6
libc.so.6
libbz2.so.1.0
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/lib/python2.7/ctypes/util.py", line 287, in <module>
    test()
  File "/usr/lib/python2.7/ctypes/util.py", line 282, in test
    print cdll.LoadLibrary("libm.so")
  File "/usr/lib/python2.7/ctypes/__init__.py", line 443, in LoadLibrary
    return self._dlltype(name)
  File "/usr/lib/python2.7/ctypes/__init__.py", line 365, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libm.so: cannot open shared object file: No such file or directory
root@ipv4:/home/michael# uname -a
Linux ipv4.rootvg.net 3.16.0-4-powerpc64 #1 SMP Debian 3.16.7-ckt9-3 (2015-04-23) ppc64 GNU/Linux
root@ipv4:/home/michael# ldconfig -p | grep libm.so
        libm.so.6 (libc6, OS ABI: Linux 2.6.32) => /lib/powerpc-linux-gnu/libm.so.6
root@ipv4:/home/michael# ls -l /lib/powerpc-linux-gnu/libm.so.6
lrwxrwxrwx 1 root root 12 Apr 15  2015 /lib/powerpc-linux-gnu/libm.so.6 -> libm-2.19.so
root@ipv4:/home/michael# ls -l /lib/powerpc-linux-gnu/libm-2.19.so
-rw-r--r-- 1 root root 743784 Apr 15  2015 /lib/powerpc-linux-gnu/libm-2.19.so

Further testing...

I added an extremely simple hack in util.py so that a archive (AIX library) name got returned.

Also in this case - the print cdll.LoadLibrary("libc.a") fails.

+74 if os.name == "posix" and sys.platform == "darwin":
+75 from ctypes.macholib.dyld import dyld_find as _dyld_find
+76 def find_library(name):
+77 possible = ['lib%s.dylib' % name,
+78 '%s.dylib' % name,
+79 '%s.framework/%s' % (name, name)]
+80 for name in possible:
+81 try:
+82 return _dyld_find(name)
+83 except ValueError:
+84 continue
+85 return None
+86
+87 if os.name == "posix" and sys.platform.startswith("aix"):
+88 def find_library(name):
+89 expr = "lib" + name + ".a"
+90 return expr
+91
+92 elif os.name == "posix":
+93 # Andreas Degert's find functions, using gcc, /sbin/ldconfig, objdump
+94 import re, tempfile, errno
+95
+96 def _findLib_gcc(name):

+275 else:
+276 print cdll.LoadLibrary("libc.a")
+277 print cdll.LoadLibrary("libm.so")
+278 print cdll.LoadLibrary("libcrypt.so")

root@x064:[/tmp]python -m ctypes.util
libm.a
libc.a
libbz2.a
Traceback (most recent call last):
  File "/opt/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/opt/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/opt/lib/python2.7/ctypes/util.py", line 282, in <module>
    test()
  File "/opt/lib/python2.7/ctypes/util.py", line 276, in test
    print cdll.LoadLibrary("libc.a")
  File "/opt/lib/python2.7/ctypes/__init__.py", line 443, in LoadLibrary
    return self._dlltype(name)
  File "/opt/lib/python2.7/ctypes/__init__.py", line 365, in __init__
    self._handle = _dlopen(self._name, mode)
OSError:        0509-022 Cannot load module /usr/lib/libc.a.
        0509-103   The module has an invalid magic number.

So, what needs to be returned so that cdll.LoadLibrary could use that result? (e.g., I know that the default libm.a has no shared members on AIX 5.3 TL7 - but libc.a does (shr.o and more, but not all!).

Last message (back to debian, and minor changes to learn expected behavior)

        if sys.platform == "darwin":
            print cdll.LoadLibrary("libm.dylib")
            print cdll.LoadLibrary("libcrypto.dylib")
            print cdll.LoadLibrary("libSystem.dylib")
            print cdll.LoadLibrary("System.framework/System")
        else:
            print cdll.LoadLibrary("libm.so.6")
#            print cdll.LoadLibrary("libcrypt.so")
            print find_library("crypt")
            x = find_library("crypt")
            print cdll.LoadLibrary(x)

returns:
root@ipv4:/home/michael# python -m ctypes.util
libm.so.6
libc.so.6
libbz2.so.1.0
<CDLL 'libm.so.6', handle f7e65528 at f7b51fb0>
libcrypt.so.1
<CDLL 'libcrypt.so.1', handle 106a1878 at f7b51fb0>

FYI: getting objdump is not too hard...

root@x064:[/data/prj/gnu/binutils-2.25.1]v/null /usr/lib/libcrypto.a <
In archive /usr/lib/libcrypto.a:

libcrypto.so: file format aixcoff-rs6000

libcrypto.so.0.9.8: file format aixcoff-rs6000

libcrypto.so.1.0.0: file format aixcoff-rs6000

libcrypto64.so: file format aix5coff64-rs6000

libcrypto64.so.0.9.8: file format aix5coff64-rs6000

libcrypto64.so.1.0.0: file format aix5coff64-rs6000

But ldconfig (as a command within glibc) will be a real headache - and I would home unnecessary.

In closing, I hope the AIX command /usr/bin/dump will be adequate as an alternative of objdump.

Where I am still a bit lost is with the "open", i.e. _dlopen(). Suggestions/hints for debugging are welcome.

The _dlopen call in __init__.py I have been able to fix (hack) with the following:

root@x064:[/data/prj/aixtools/python/python-2.7.10/Lib/ctypes]diff -u __init__.py /opt/lib/python2.7/ctypes/__init__.py
--- __init__.py 2015-05-23 16:09:01 +0000
+++ /opt/lib/python2.7/ctypes/__init__.py       2016-02-26 08:40:19 +0000
@@ -11,6 +11,7 @@
 from _ctypes import _Pointer
 from _ctypes import CFuncPtr as _CFuncPtr
 from _ctypes import __version__ as _ctypes_version
+# from _ctypes import RTLD_LOCAL, RTLD_GLOBAL, RTLD_NOW ## fails
 from _ctypes import RTLD_LOCAL, RTLD_GLOBAL
 from _ctypes import ArgumentError
 
@@ -32,6 +33,11 @@
     if int(_os.uname()[2].split('.')[0]) < 8:
         DEFAULT_MODE = RTLD_GLOBAL
 
+if _os.name == "posix" and _sys.platform.startswith("aix"):
+        RTLD_NOW    = 0x00000002
+        RTLD_MEMBER = 0x00040000
+        DEFAULT_MODE |= (RTLD_NOW | RTLD_MEMBER)
+
 from _ctypes import FUNCFLAG_CDECL as _FUNCFLAG_CDECL, \
      FUNCFLAG_PYTHONAPI as _FUNCFLAG_PYTHONAPI, \
      FUNCFLAG_USE_ERRNO as _FUNCFLAG_USE_ERRNO, \

I have an additional hack in util.py so that, e.g.,
find_Library("crypto")

returns:
'/usr/lib/libcrypto.a(libcrypto.so)'

When that is passed to dlopen (plus RTLD_MEMBER) the dlopen succeeds.

<CDLL '/usr/lib/libcrypto.a/(libcrypto.so)', handle 6 at 30146670>

With some help in util.py, to do some sensible searching, being able to establish if in 32 or 64-bit mode, etc. somehing nice can be made here (imho).

p.s. can any verify whether this is limited to python 2.7? Or should python 3.X be added as well?

ctypes util.py "simply" needs support for AIX. There already is special support for Windows, Darwin, BSDs, Solaris. Is the question about the technical details for equivalent functionality on AIX or about adding a stanza to Lib/ctpyes/util.py?

it was, partly, about the technical details - but I feel I have found the key bits - /full/path/libNAME.a(libNAME.so) + dlflags RTLD_NOW + RTLD_MEMBER for "native" AIX support.

Additionally, a patch should not break what might be working for some (via google I read of suggestions for other languages, e.g., java, where .so members are extracted or installed side-by-side with the .a archive) - as the IBM run-rime loader also accepts/looks for both .a and .so. And/or for when /sbin/ldconfig is available. And when available, is this used as last case, or preferred?

Lastly, we cannot assume we will know the name of the member based on the name of the library. In many cases, e.g., libiconv.a the IBM names are shr.o, shr4.o and shr4_64.o - and these are the only member names unless GNU libiconv.a has been added (and then libconv.a also contains libiconv.so.2 (so version support is also desired!)) in both 32 and 64 bit mode.

One technical detail I have not been able to discover yet: how to determine whether in 32-bit or 64-bit mode. This will be important for libraries that have shr4.o and shr4_64.o as the members that need to be dlopened.

Lastly: about adding the "stanza": python is a new language for me. Any assistance with a patch - to keep it properly 'pythonized'

In short, I shall continue my studies/learning. Assistance from anyone wiser than me is welcome!

AIX traditionally used member names like shr.o or shr<version>.o or shr<aix_release>.o insider the archive, with _64 designating a 64 bit object when there is a naming collision.

GNU libtool defaults to the SO name and version number insider the archive.

AIX objects (and shared objects) contain a bit in the header that specifies 32 bit or 64 bit. Both 32 bit and 64 bit objects are intended to be archived together. The linker only processes objects of the correct mode.

AIX shared objects contain a bit that specifies if the object may be used at link-edit time or only should be used for loading. This is controlled by the AIX strip -e/-E option (yes, I know, strange place to hide that option).

This combination of features allows all of the libraries to be placed in a single /usr/lib directory and all of the objects to be collected into a single archive, avoiding /usr/lib64 and explosion of shared objects and symbolic links clutter. Various packages have created /usr/local/lib64 anyway using Linux/Solaris/SVR4-style naming.

The following demonstrates the basics.

cdll.LoadLibrary will load a .so file if it can be located (you will have to believe me as I forgot to include the test in this last batch)

Further, when given the AIX format for loading a member of an archive
(e.g., libc.a(shr.o) or libcrypto.a(libcrypto.so), the dlload occurs as expected.

in ctypes/utils.py
This looks in the standard directories (as the example, but not complete, see below) to demonstrate how to respond.
Note: it is also looking for a .so file, and if it finds one, it returns the .so name, otherwise it looks for a .a file - and when it finds it, just appends the libNAME.so in the correct format.

What is not done, and probably different from ldconfig behavior!

1. The complete path is not returned (so cdll.LoadLibrary will still follow default search - which I hope includes, read uses, the default for the python executable

2. The member name is not verified for existence (e.g., libconv.a(libiconv.so) is not expected to find libiconv.a(libiconv.so.2)

3. Since member verification is not being performed, shr.o, shr4.o, shr*_64.o

And many many thanks for the strip -e/-E info. Hard to find!

root@x064:[/data/prj/aixtools/src]diff -u python-2.7.11/Lib/ctypes/__init__.py /opt/lib/python2.7/ctypes/__init__.py
--- python-2.7.11/Lib/ctypes/__init__.py        2015-12-05 19:46:56 +0000
+++ /opt/lib/python2.7/ctypes/__init__.py       2016-02-29 17:36:43 +0000
@@ -11,6 +11,7 @@
 from _ctypes import _Pointer
 from _ctypes import CFuncPtr as _CFuncPtr
 from _ctypes import __version__ as _ctypes_version
+# from _ctypes import RTLD_LOCAL, RTLD_GLOBAL, RTLD_NOW ## fails
 from _ctypes import RTLD_LOCAL, RTLD_GLOBAL
 from _ctypes import ArgumentError
 
@@ -356,6 +357,14 @@
         if use_last_error:
             flags |= _FUNCFLAG_USE_LASTERROR
 
+        if _sys.platform.startswith("aix"):
+            RTLD_NOW    = 0x00000002
+            RTLD_MEMBER = 0x00040000
+            mode |= RTLD_NOW
+            if name is not None:
+                if name[-1] == ')':
+                       mode |= RTLD_MEMBER
+
         class _FuncPtr(_CFuncPtr):
             _flags_ = flags
             _restype_ = self._func_restype_


root@x064:[/data/prj/aixtools/src]diff -u python-2.7.11/Lib/ctypes/util.py /opt/lib/python2.7/ctypes/util.py
--- python-2.7.11/Lib/ctypes/util.py    2015-12-05 19:46:56 +0000
+++ /opt/lib/python2.7/ctypes/util.py   2016-02-29 17:42:41 +0000
@@ -84,6 +84,23 @@
                 continue
         return None
 
+if os.name == "posix" and sys.platform.startswith("aix"):
+        def _findLib_aix(name):
+            paths='/usr/lib:/lib:/usr/lpp/xlC/lib'
+            for dir in paths.split(":"):
+                shlib = os.path.join(dir, "lib%s.so" % name)
+                if os.path.exists(shlib):
+                    return shlib
+                libfile = os.path.join(dir, "lib%s.a" % name)
+                if os.path.exists(libfile):
+                    shlib = "lib%s.a(lib%s.so)" % (name, name)
+                    return shlib
+
+            return None
+
+        def find_library(name):
+            return _findLib_aix(name)
+
 elif os.name == "posix":
     # Andreas Degert's find functions, using gcc, /sbin/ldconfig, objdump
     import re, tempfile, errno
@@ -267,7 +284,16 @@
             print cdll.LoadLibrary("libcrypto.dylib")
             print cdll.LoadLibrary("libSystem.dylib")
             print cdll.LoadLibrary("System.framework/System")
+        elif sys.platform.startswith("aix"):
+            print find_library("crypto")
+            x = find_library("crypto")
+            print cdll.LoadLibrary(x)
+            print cdll.LoadLibrary("libcrypto.a(libcrypto.so)")
+            print cdll.LoadLibrary("libc.a(shr.o)")
+            print find_library("crypt")
+            print cdll.LoadLibrary("libcrypt.a(shr.o)")
         else:
+            print cdll.LoadLibrary("libc.a")
             print cdll.LoadLibrary("libm.so")
             print cdll.LoadLibrary("libcrypt.so")
             print find_library("crypt")

p.s. Anyone know what flag to set in vi so that tabs are never inserted and/or spaces converted to tabs. Python seems to hate tab indents.

Oops: forgot the example output:

root@x064:[/data/prj/aixtools/src]python -m ctypes.util
libm.a(libm.so)
libc.a(libc.so)
libbz2.a(libbz2.so)
libcrypto.a(libcrypto.so)
<CDLL 'libcrypto.a(libcrypto.so)', handle 6 at 30147690>
<CDLL 'libcrypto.a(libcrypto.so)', handle 7 at 30147690>
<CDLL 'libc.a(shr.o)', handle 8 at 30147690>
libcrypt.a(libcrypt.so)
<CDLL 'libcrypt.a(shr.o)', handle 9 at 30147690>

The patch works when installed on top of pre-compiled version (e.g., copy the two files to /opt/lib/python2.7/ctypes/

but there seems to be a nasty side-effect when trying to build.

make completes, but make install fails during a compile step.

Please assist with howto see how/why ctypes/util is impacting this part.

And - for my first adventure into python please provide the needed (expected) feedback.

Michael

make install DESTDIR=/var/aixtools/aixtools/python/2.7.11.2 > .buildaix/install.out

Listing /var/aixtools/aixtools/python/2.7.11.2/opt/lib/python2.7/xml/sax ...
Compiling /var/aixtools/aixtools/python/2.7.11.2/opt/lib/python2.7/xml/sax/init.py ...
Compiling /var/aixtools/aixtools/python/2.7.11.2/opt/lib/python2.7/xml/sax/_exceptions.py ...
Compiling /var/aixtools/aixtools/python/2.7.11.2/opt/lib/python2.7/xml/sax/expatreader.py ...
Compiling /var/aixtools/aixtools/python/2.7.11.2/opt/lib/python2.7/xml/sax/handler.py ...
Compiling /var/aixtools/aixtools/python/2.7.11.2/opt/lib/python2.7/xml/sax/saxutils.py ...
Compiling /var/aixtools/aixtools/python/2.7.11.2/opt/lib/python2.7/xml/sax/xmlreader.py ...
Compiling /var/aixtools/aixtools/python/2.7.11.2/opt/lib/python2.7/xmllib.py ...
Compiling /var/aixtools/aixtools/python/2.7.11.2/opt/lib/python2.7/xmlrpclib.py ...
Compiling /var/aixtools/aixtools/python/2.7.11.2/opt/lib/python2.7/zipfile.py ...

So, zipfile.py seems to exit non-zero. :(

Ah, good news - the build is successful ONCE I open the file ./Lib/ctypes/util.py and find the (hiding) tab characters and replace them with 8 space characters. (And I had tried so hard to check for that in advance).

So, if you apply the patch - it may need some love.

I would be grateful for people (even just one!) looking at it.

Michael

fyi: just completed a test both as 32 and 64 bit build.

However, openssl-1.0.1.514 does not work in 64-bit mode (packaging error). Fortunately, openssl-1.0.1.515 (released 02-March-2016) fixes that.

In short, 64-bit build is dependent on openssl-1.0.1.515

Here is the corrected patch (tabs removed). name: python.Lib.ctypes.160317.patch

Maybe bpo-21826 is relevant (slowdown on AIX caused by missing ldconfig)

These are very different issues. However, this patch may resolve both!

ldconfig (-p if I recall) lists where (shared) libraries have been installed (imho, this is a GNU tool approach) - whereas AIX would use dump -H to find library paths embedded in a program and/or shared library.

Until this patch, to use shared libraries on AIX the members of an archive needed to be extracted from the .a archive, and for 64-bit members, a separate directory (e.g. /usr/lib64) is needed. With this patch find_library() (actually cdll.LoadLibrary() can load members from either both .so and .a libraries, as is normal for AIX.

So, in a way, this would also solve https://bugs.python.org/issue21826 as ldconfig is no longer needed (nor called) on AIX.

p.s. As it is well longer than a month - I would appreciate that someone actually look at the patch and tell me how it can be improved! :)

The most recent patch seems to follow AIX semantics correctly.

I notice the patch is against Python 2, and uses Python 2 specific syntax. I am unsure if this kind of change is acceptable for 2.7, or if it would have to only go into 3.6. Hopefully this version of the patch will notify the review system that it is against Python 2.

I don’t know anything about AIX specific stuff, but I left some general comments in the code review.

Is there any chance that AIX people would be relying on the current behaviour that I understand uses _findSoname_ldconfig() and _findLib_gcc()?

Is this new functionality covered by the test suite? E.g. in /Lib/ctypes/test/test_find.py, there are tests that call find_library() for GL, GLU, and gle. Are those libraries common on AIX?

As discussed in bpo-9998, it seems a lot of people use find_library() to help convert a build-time library name to a run-time shared library name that can be passed to CDLL() or LoadLibrary(). E.g. on Linux:

>>> find_library("python2.7")  # As used in cc . . . -lpython2.7
'libpython2.7.so.1.0'
>>> cdll.LoadLibrary("libpython2.7.so.1.0")
<CDLL 'libpython2.7.so.1.0', handle 7f58e7495000 at 7f58e573ac90>

Does your patch support this kind of use case?

I have not looked specifically, at least not that I remember, for differences in util/ctypes in python2 and python3. Will do so tomorrow.

I did just look briefly at the library, rather archive, built by default as libpython2.7.a - it is static members only, i.e., my build using xlc (i.e., not using gcc) does not build a shared object, so cdll.LoadLibrary and/or find_library will not find anything for python2.7.. Neither will m, or libm, on a default AIX system (with no other gcc based packages installed - these also install a gnu rte where the utilities and libs you mention might include.

The few python packages I have found, packaged by others, tend to reload everything yet again, not depending on anything that may already be there. And to use shared libraries they are extracting the members from the .a archives into two directories - when they support both 32 and 64-bit targets.

My intent is to examine the program to discover where libraries should be and find the member name that is most likely. Also, if LIBPATH is defined, those directories are searched first for a match.

In short, the key difference is to look at the program (probably python) for the blibpath string in the application as well as python (from memory, sys.* calls) to build a list of directories to search.

findLibrary('foo') first finds libfoo.a, then looks in libfoo.a for shr*.o members, libfoo.so, libfoo.so.X and/or libfoo.so.X.Y, etc..

I need to check that findLibrary('foo.so') continues to work. At one time it did, just have not looked at this for several weeks and I forget if it still works. That is what I shall make sure stays in the "testing" part of the patch.

Michael

The obvious (but easy to fix) problem I forsee with Python 3 is the print() calls. If you use print("") and print(arg), that will work with both 2 and 3. There may be more complications with bytes vs text stdout if we change the os.popen() calls to use subprocess.Popen.

I didn’t mean to use libpython2.7 specifically. Substitute any shared library that is widely available across platforms; maybe “crypto” is a better example. CDLL(find_library("crypto")) loads "libcrypto.so.1.0.0" on my Linux computer. It looks like you got the equivalent working for AIX; I was just checking.

FWIW it looks like your parsing of sys.executable to find library search paths is similar to searching the runpath (or RPATH) on ELF files, as proposed in bpo-19317. And it seems AIX’s LIBPATH environment variable is similar to LD_LIBRARY_PATH on Linux, proposed to be searched in bpo-9998. Also, I understand the equivalent OS X environment variables DYLD_(FALLBACK)_LIBRARY_PATH are already used.

On 4/28/2016 11:56 PM, Michael Felt wrote:

Michael Felt added the comment:

I have not looked specifically, at least not that I remember, for differences in util/ctypes in python2 and python3. Will do so tomorrow.

I did just look briefly at the library, rather archive, built by default as libpython2.7.a - it is static members only, i.e., my build using xlc (i.e., not using gcc) does not build a shared object, so cdll.LoadLibrary and/or find_library will not find anything for python2.7.. Neither will m, or libm, on a default AIX system (with no other gcc based packages installed - these also install a gnu rte where the utilities and libs you mention might include.

The few python packages I have found, packaged by others, tend to reload everything yet again, not depending on anything that may already be there. And to use shared libraries they are extracting the members from the .a archives into two directories - when they support both 32 and 64-bit targets.

My intent is to examine the program to discover where libraries should be and find the member name that is most likely. Also, if LIBPATH is defined, those directories are searched first for a match.

In short, the key difference is to look at the program (probably python) for the blibpath string in the application as well as python (from memory, sys.* calls) to build a list of directories to search.

findLibrary('foo') first finds libfoo.a, then looks in libfoo.a for shr*.o members, libfoo.so, libfoo.so.X and/or libfoo.so.X.Y, etc..

I need to check that findLibrary('foo.so') continues to work. At one time it did, just have not looked at this for several weeks and I forget if it still works. That is what I shall make sure stays in the "testing" part of the patch.

Michael

----------

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue26439\>

---

I am reworking the logic - as many use cdll.LoadLibrary without ever
calling find_library, and then __init__.py breaks.

More asap.

Question - before I submit a patch.

A. Is there a PEP I should read re: ctypes/util and/or ctypes/cdll?

B. I show two different behaviors of responding - My question is, what does the community think should be the response? My preference is to bring the request back to it's stub - (strip lib from the beginning, and .so* or .a from the suffix, so that it is in it's -lFOO form - as if being offered to a linker (with -lFOO, we do not use -lFOO.so.6 (e.g., for libc.so.6).
I have seen a lot of variance in how cdll is used (in only two python based projects I am currently interested in porting). Calls are frequently made using some variation including .so somewhere in the name and/or code is basing decesions based on a the number (###) returned by find_library (e.g., libFOO.so.###, and ### MUST be one of a list - and is rejected if not a match (even if newer)). I would think if there is a version dependancy this should be a call to the library loaded rather than an external string.
In any case, - for one example - as libc.so.* will never exist, by default, on AIX, for ease of use I believe that translating libFOO.* to FOO and then doing the 'find' comes closest to what was intended (long long ago) - again, is there a PEP with guidance on this AND my preference, for AIX, is to look in .a archives first for a .so member, and then look for an AIX legacy member (shr.*.o) and then look for an external .so.* file.

With my current working version - this is the output of the test in util.py:

root@x064:[/data/prj/aixtools/python/python-2.7.11.3/Lib/ctypes]../../python ./util.py
libm.a
libc.a
libbz2.a
<CDLL 'None', handle b at 700000000216630>
<CDLL 'libcrypt.a(shr_64.o)', handle c at 700000000216630>
libcrypt.a

Additional Tests for AIX
call find_library("foo")
c: libc.a
c.a: None
c.so: None
libc: libc.a
libc.a: libc.a
libc.so.6: libc.a
crypt: libcrypt.a
crypto: libcrypto.a
crypto.so: None
libcrypto.so: libcrypto.a

call cdll.LoadLibrary("foo")
m: <CDLL 'None', handle d at 700000000216860>
libm.so: <CDLL 'None', handle e at 700000000216860>
c: <CDLL 'libc.a(shr_64.o)', handle f at 700000000216860>
c.a: <CDLL 'libc.a(shr_64.o)', handle 10 at 700000000216860>
libc.a: <CDLL 'libc.a(shr_64.o)', handle 11 at 700000000216860>
libc.so.6: <CDLL 'libc.a(shr_64.o)', handle 12 at 700000000216860>
libc.so.9: <CDLL 'libc.a(shr_64.o)', handle 13 at 700000000216860>
bz2: <CDLL 'libbz2.a(libbz2.so.1)', handle 14 at 700000000216860>
libbz2: <CDLL 'libbz2.a(libbz2.so.1)', handle 15 at 700000000216860>
crypt: <CDLL 'libcrypt.a(shr_64.o)', handle 16 at 700000000216860>
crypto: <CDLL 'libcrypto.a(libcrypto64.so)', handle 17 at 700000000216860>
crypto.so: <CDLL 'libcrypto.a(libcrypto64.so)', handle 18 at 700000000216860>
libcrypto.so: <CDLL 'libcrypto.a(libcrypto64.so)', handle 19 at 700000000216860>
iconv: <CDLL 'libiconv.a(libiconv.so.2)', handle 1a at 700000000216860>
intl: <CDLL 'libintl.a(libintl.so.1)', handle 1b at 700000000216860>

The test looks like:
################################################################
# test code

def test():
    from ctypes import cdll
    if os.name == "nt":
        print cdll.msvcrt
        print cdll.load("msvcrt")
        print find_library("msvcrt")

    if os.name == "posix":
        # find and load_version
        print find_library("m")
        print find_library("c")
        print find_library("bz2")

    \# getattr

## print cdll.m
## print cdll.bz2

        # load
        if sys.platform == "darwin":
            print cdll.LoadLibrary("libm.dylib")
            print cdll.LoadLibrary("libcrypto.dylib")
            print cdll.LoadLibrary("libSystem.dylib")
            print cdll.LoadLibrary("System.framework/System")
        else:
            print cdll.LoadLibrary("libm.so")
            print cdll.LoadLibrary("libcrypt.so")
            print find_library("crypt")

        if sys.platform.startswith("aix"):
            print "\nAdditional Tests for AIX"
            print "call find_library(\"foo\")"
            print "c: ", find_library("c")
            print "c.a: ", find_library("c.a")
            print "c.so: ", find_library("c.so")
            print "libc: ", find_library("libc")
            print "libc.a: ", find_library("libc.a")
            print "libc.so.6: ", find_library("libc.so.6")
            print "crypt: ", find_library("crypt")
            print "crypto: ", find_library("crypto")
            print "crypto.so: ", find_library("crypto.so")
            print "libcrypto.so: ", find_library("libcrypto.so")
###
            print "\ncall cdll.LoadLibrary(\"foo\")"
            print "m: ", cdll.LoadLibrary("m")
            print "libm.so: ", cdll.LoadLibrary("libm.so")
            print "c: ", cdll.LoadLibrary("c")
            print "c.a: ", cdll.LoadLibrary("c.a")
            print "libc.a: ", cdll.LoadLibrary("libc.a")
            print "libc.so.6: ", cdll.LoadLibrary("libc.so.6")
            print "libc.so.9: ", cdll.LoadLibrary("libc.so.9")
            print "bz2: ", cdll.LoadLibrary("bz2")
            print "libbz2: ", cdll.LoadLibrary("libbz2")
            print "crypt: ", cdll.LoadLibrary("crypt")
            print "crypto: ", cdll.LoadLibrary("crypto")
            print "crypto.so: ", cdll.LoadLibrary("crypto.so")
            print "libcrypto.so: ", cdll.LoadLibrary("libcrypto.so")
            print "iconv: ", cdll.LoadLibrary("iconv")
            print "intl: ", cdll.LoadLibrary("intl")

if __name__ == "__main__":
    test()

Thank you for your considered comments. My goal is ease of use for (porting) existing projects to AIX, i.e., ideally without code change.

reworked patch. To assist port to Python3 that changes in __init__.py and util.py are minimal. There is a new file: aixutil.py

I have only tested on Python-2.7, so there may be issues for Python3. My goal is to have a single file for both versions.

The main change in util.py is lots of specific (behavior) tests that I have encountered (and initially failed) from projects using ctypes.cdll.

I feel confident that this will work "ASIS" with nearly all existing projects.

Note: normal return is not a full path name. A full path name is only returned when an archive + member could not be found, but a file with the name requested could be found. This "fullpath" is to be compatible with existing code 'demanding' unpacked archives.

implements ctypes.aixutil.find_library()
In a separate file as both __init__.py as util.py needs it's logic.

'''
call find_library("foo")
libc: libc.a
libc.a: libc.a
libc.so.6: libc.a
libcrypto.so: libcrypto.a
'''

The above don’t seem right to me, unless compiling with “cc -llibc.so.6” etc works on AIX.

'''
call cdll.LoadLibrary("foo")
m: <CDLL 'None', handle d at 700000000216860>
libm.so: <CDLL 'None', handle e at 700000000216860>
'''

These doesn’t look right. What happened to the library name?

With your new aixutil.py file, it might be good to give it an underscore (_) prefix, to indicate it is an internal module rather than part of the ctypes API. So your code would do

import ctypes._aixutil as aix

Your new patch calls find_library() internally in CDLL(); why? My understanding is CDLL() is a fairly lightweight wrapper around the dlopen() call. On Linux, you either pass a full library file name, or an SO-name. Both these strings can be discovered for compiled objects using e.g.:

$ ldd build/lib.linux-x86_64-2.7-pydebug/_ssl.so
	linux-vdso.so.1 (0x00007fff567fe000)
	libssl.so.1.0.0 => /usr/lib/libssl.so.1.0.0 (0x00007f598474c000)
	libcrypto.so.1.0.0 => /usr/lib/libcrypto.so.1.0.0 (0x00007f59842d4000)
	. . .

So in Python, the SO-name or full path can be used, but not the compile-time name, unless you first pass it through find_library():

>>> CDLL("libcrypto.so.1.0.0")  # soname
<CDLL 'libcrypto.so.1.0.0', handle 7f1665e1eb90 at 7f16658f34d0>
>>> CDLL("/usr/lib/libcrypto.so.1.0.0")  # Full path
<CDLL '/usr/lib/libcrypto.so.1.0.0', handle 7f1665e1eb90 at 7f1663cddcd0>
>>> CDLL("crypto")  # Compile-time name
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/ctypes/__init__.py", line 365, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: crypto: cannot open shared object file: No such file or directory
>>> find_library("crypto")  # Some people pass the result of this to CDLL()
'libcrypto.so.1.0.0'

On 27-Aug-16 09:11, Martin Panter wrote:

Martin Panter added the comment:

The documentation is in RST format in the Doc/ directory. For basic stuff, you can just copy the syntax from existing text, or see e.g. <https://docs.python.org/devguide/documenting.html#restructuredtext-primer\> for more hints. For this change, I think we need to mention in Doc/library/ctypes.rst:

• the special AIX changes for find_library(); include “.. versionchanged::” (or maybe versionadded? I’m not sure) notice
• the change for the CDLL constructor, also with versionchanged

Perhaps also add an entry to Doc/whatsnew/3.6.rst.

Patch 160823 does not addressed many of my previous comments. Please have a closer look. I can make simple changes to simplify the code myself, but I don’t really know what to do about the questionable regular expressions, for instance.
As far as regular expressions go, that will always be difficult for me
aka questionable for you. Every language has it's nuances and I never
seem to get them right.

Also, see <https://bugs.python.org/issue26439#msg268885\>. What was wrong with the cases supported by your original patches?
As you clearly pointed out, there were no prior cases showing their
need. I was taking an unbiased approach in that I knew no previous code
well. I was just trying to make it work and was trying to think of cases
were it could go wrong. In short, I was trying to solve issues that do
not exist. People accept whatever libFOO.so points to as a symbolic link

• and expect libSOO.so to be the name of the shared library that
  dlopen() is going to open somewhere.

Once I understood the boundaries of find_library("foo") I limited myself
to only resolve in the way the compile-time resolution is done. As such,
find_library("c") is the equivalent of -lc and needs to resolve to
libc.a(shr.o) - period (32-bit) or libc.a(shr_64.o) (64-bit).

The linker is not looking for a particular member name: it only uses
-lFOO to find libFOO.a, and it it exists it looks for a symbol. The
member name that contains the symbol is what it stores in the executable
(displayed via dump -H). So, getting back to this patch and packaging
conventions.

Basically, libtool has standardised how members are named, i.e.,
versioning. In most cases with OSS packages the only member-name that
gets stored is the same name that libFOO.so would be a symbolic link to.
Again, the compiler-linker only needs the name of the archive. The
member name within the archive is irrelevant - from a compiler/linker
perspective.

My goal for find_library() is to find the most likely name based on some
legacy "rules" from the way IBM packaged libraries starting over 20
years ago (aka AIX 4 standards) that are alive today to support binary
compatibility - as much as possible. Mainly, that is relevant for
libraries/archives such as libc.a. For new libraries, especially OSS
packages built using the GNU autotools (and finishing with libtool)
there are other conventions.

Finally, we had different goals - my focus was on writing something that
matches other platforms python2 behavior, not on writing a new syntax
specific for Python3.6 and AIX. If I were to do be writing a new syntax
I would prefer that it also work for other platforms. Something
different for only one platform feels wrong - imho.

In closing:
a) regular expressions and me is always a headache for someone. please
accept my apologies if I have given you a headache.
b) I had assumed abilities for find_library() (from studying the output
of ldconfig and trying to follow the regular expressions in the current
code) that are not used. To assume makes an ass of u and me - especially me.
c) I also apologize for not meeting your expectations. I cut code,
rather than defend it, as you were correct that is was not based on
meeting the needs of general aka current practice.

I am not trying to redesign find_library(). My hope is that most python
code would work asis - if they follow conventions aka general practice
(not equivalent to best practice as best depends (in part) on your goal).

----------

---

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue26439\>

---

Regarding the regular expressions, I (or someone else unfamiliar with AIX) may be able to adjust them if you can explain what you are trying to achieve. Take the first one I commented on <https://bugs.python.org/review/26439/diff/17689/Lib/ctypes/_aixutil.py#newcode132\> for example. You added a comment:

# '\[%s_*64\.so\]' % name, -> has either _64 or 64 added to name

As written, this will match many strings including

[<NAME>___64.so]

However the annotation leads me to belive you want it to match two cases only:

[<NAME>64.so]
[<NAME>_64.so]

I do not know whether to fix the annotation (has 64 preceded by any number of underscores), or whether to fix the regular expression (_?64).

On 04/09/2016 06:11, Martin Panter wrote:

I do not know whether to fix the annotation (has 64 preceded by any number of underscores), or whether to fix the regular expression (_?64).

The later - _?64. Working on this today. Thank you for the correction.

Not always as elegant as I would wish (do not like the idea of "while 1:" and later a break...

But, all in all, much improved by redoing the processing of the subprocess output and getting rid of more noise.

Hope this meets your approval!

p.s. This is a patch compared to previous patch. If a diff compared to Python-3.6.0a4 I can remake the patch.

Sigh - missed the feature cutoff that would have made this easier to get into python...

Anyway - have learned a few new things about python def: syntax and removed some bits that I thought were suitable for variable "initialization" - but tend to be static (not what I was thinking).

Also, "new and improved" comments about what the code is doing.

Thank you all for your patience and feedback - especially Martin!

New changeset 01885f78b299 by Martin Panter in branch '2.7':
Issue bpo-26439: Document that RTLD_NOW is always added
https://hg.python.org/cpython/rev/01885f78b299

New changeset 0db4403e62c4 by Martin Panter in branch '3.5':
Issue bpo-26439: Document that RTLD_NOW is always added
https://hg.python.org/cpython/rev/0db4403e62c4

New changeset 4b7e51998a90 by Martin Panter in branch '3.6':
Issue bpo-26439: Merge ctypes doc from 3.5 into 3.6
https://hg.python.org/cpython/rev/4b7e51998a90

New changeset f496fb6bf4a0 by Martin Panter in branch 'default':
Issue bpo-26439: Merge ctypes doc from 3.6
https://hg.python.org/cpython/rev/f496fb6bf4a0

Hi Michael, I have done some cleanup and modifications to your patch. The result is in aix-library.161001.patch, which has all the changes, i.e. it is not based on another patch. More significant changes I made:

• Change getExecLibPath_aix() and find_parts() to return a list object, rather than building a colon-separated string only to be pulled apart again
• Escape dots in get_legacy() regular expressions, so that they no longer match [shr_64xo], [shrxo], etc.
• Make get_dumpH() return the a list of (object, objectinfo) tuples, where objectinfo is a list of lines; avoids building multiline strings and then splitting them apart again
• Rewrite get_exactMatch() and get_version() without nested inline “for” loops; use RE capture group
• Reuse util._last_version() instead of copying the _num_version() function
• Use lower case B for liB in get_member(). This means e.g. libcrypto.so is now preferred over libcrypto.so.1.0.0.

I did test it a bit on Linux with faked dump -H output, but I may have made mistakes that I did not pick up.

Also, this still needs documentation, and I think some more tests for the test suite exercising various aspects of find_library() would be nice if possible.

Another thing: in the last few patches, you dropped the definition of RTLD_MEMBER from Modules/_ctypes/_ctypes.c. Is that intended, or just a temporary thing?

On 01-Oct-16 08:44, Martin Panter wrote:

Martin Panter added the comment:

Hi Michael, I have done some cleanup and modifications to your patch. The result is in aix-library.161001.patch, which has all the changes, i.e. it is not based on another patch.
Thanks.
More significant changes I made:

• Change getExecLibPath_aix() and find_parts() to return a list object, rather than building a colon-separated string only to be pulled apart again
• Escape dots in get_legacy() regular expressions, so that they no longer match [shr_64xo], [shrxo], etc.
• Make get_dumpH() return the a list of (object, objectinfo) tuples, where objectinfo is a list of lines; avoids building multiline strings and then splitting them apart again
• Rewrite get_exactMatch() and get_version() without nested inline “for” loops; use RE capture group
• Reuse util._last_version() instead of copying the _num_version() function
• Use lower case B for liB in get_member(). This means e.g. libcrypto.so is now preferred over libcrypto.so.1.0.0.
  That was a typo - to be sure I was still finding the versioned ones (the
  previous ones had had a bug that they no longer found the "standard"
  one. I forgot to remove (rather save file before the diff command) - you
  see everything!

I did test it a bit on Linux with faked dump -H output, but I may have made mistakes that I did not pick up.
Will apply the patch, build in 32 and 64 bit modes, and respond.

Also, this still needs documentation, and I think some more tests for the test suite exercising various aspects of find_library() would be nice if possible.
Working on that - have been posting some questions on python-list as I
want to build an interface with an AIX performance library (libperfstat).
Without this library one must run a command and then do some string
manipulation, and sometimes call a second command once that has been
found (e.g., uuid calls to get a MAC address, cloud-init to get
boottime) - things that - with a library do not need a subprocess at all.

But I shall also write up the AIX dlopen() process to explain how both
.so and .a(member.so) works (as the argument to CDLL).

Another thing: in the last few patches, you dropped the definition of RTLD_MEMBER from Modules/_ctypes/_ctypes.c. Is that intended, or just a temporary thing?
Temporary thing - as I keep hoping for inclusion in Python2 and I recall
you not wanting to add that into _ctypes on Python2.

----------
versions: -Python 3.6
Added file: http://bugs.python.org/file44902/aix-library.161001.patch

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue26439\>

---

I have spent the last two hours trying to run the test - however, it fails with:

root@x064:[/data/prj/python/python-3.6.0.177/Lib/ctypes]../../python util.py
Traceback (most recent call last):
  File "util.py", line 102, in <module>
    import ctypes._aix as aix
  File "/data/prj/python/python-3.6.0.177/Lib/ctypes/_aix.py", line 15, in <module>
    from . import util
  File "/data/prj/python/python-3.6.0.177/Lib/ctypes/util.py", line 102, in <module>
    import ctypes._aix as aix
AttributeError: module 'ctypes' has no attribute '_aix'

I have noticed several issues with the file that used to be named just
./build/_sysconfigdata.py

but is now: ./build/lib.aix-5.3-3.6/sysconfigdata_m_aix5.py

I am guessing something is wrong there - I am going to try copying only _aix.py to the Python2 branch and see if it works there -- and also dig deeper into what is going wrong with Python3.6*

Curious.

When in 32-bit mode changing line 15 of _aix.py to

+14 import re, os, sys
+15 # from . import util

The Lib/ctypes/util.py works.

In 64-bit mode it does not:
instead:

root@x064:[/data/prj/python/python-3.6.0.177/Lib/ctypes]../../python util.py
m       :: None
c       :: libc.a(shr_64.o)
Traceback (most recent call last):
  File "util.py", line 355, in <module>
    test()
  File "util.py", line 330, in test
    print("bz2\t:: %s" % find_library("bz2"))
  File "util.py", line 104, in find_library
    return aix.find_library(name)
  File "/data/prj/python/python-3.6.0.177/Lib/ctypes/_aix.py", line 255, in find_library
    (base, member) = find_shared(libpaths, name)
  File "/data/prj/python/python-3.6.0.177/Lib/ctypes/_aix.py", line 247, in find_shared
    member = get_member(re.escape(name), members)
  File "/data/prj/python/python-3.6.0.177/Lib/ctypes/_aix.py", line 189, in get_member
    member = get_version(name, members)
  File "/data/prj/python/python-3.6.0.177/Lib/ctypes/_aix.py", line 170, in get_version
    return util._last_version(versions, '.')
NameError: name 'util' is not defined

+++++
When the comment is removed, i.e.
from . import util

both 32 and 64-bit report:
root@x064:[/data/prj/python/python-3.6.0.177/Lib/ctypes]../../python util.py
Traceback (most recent call last):
  File "util.py", line 102, in <module>
    import ctypes._aix as aix
  File "/data/prj/python/python-3.6.0.177/Lib/ctypes/_aix.py", line 15, in <module>
    from . import util
  File "/data/prj/python/python-3.6.0.177/Lib/ctypes/util.py", line 102, in <module>
    import ctypes._aix as aix
AttributeError: module 'ctypes' has no attribute '_aix'

This last condition also occurs in Python2

Have a way to have both 64-bit and 32-bit modes working.

root@x064:[/data/prj/python/python-3.6.0.177/Lib/ctypes]../../python pwd/util.py
m :: None
c :: libc.a(shr_64.o)
bz2 :: libbz2.a(libbz2.so.1)
crypt :: libcrypt.a(shr_64.o)
crypto :: <CDLL 'libcrypto.a(libcrypto64.so.1.0.0)', handle c at 0x7000000001ffcf8>
c :: <CDLL 'libc.a(shr_64.o)', handle d at 0x7000000001ffcf8>

root@x064:[/data/prj/python/python-3.6.0.177/Lib/ctypes]../../python pwd/util.py
m :: None
c :: libc.a(shr.o)
bz2 :: libbz2.a(libbz2.so)
crypt :: libcrypt.a(shr.o)
crypto :: <CDLL 'libcrypto.a(libcrypto.so)', handle c at 0x301f78b0>
c :: <CDLL 'libc.a(shr.o)', handle d at 0x301f78b0>

Will post patch asap. Going to shorten some lines going well over 76-char lime-limit.

Besides correcting a small error, also my attempt to follow the
guidelines to not import *, but to actually specify all bits
that are to be imported.

This is a patch compered to Python-3.6b1

Just some minor comments on aix-library.161004.patch:

Instead of _util.py, I wonder if the new file should have a different name, like _util_common.py, to avoid being too similar to util.py.

+def get_shared(input):
+ """Internal support function: examine the get_dumpH() output and
+ return a list of all shareable objects indicated in the output the
+ character "[" is used to strip off the path information.

Needs a newline or new sentance to separate “output” and “the character”.

+def get_legacy(members):
+ [. . .]
+ # shr.o is the preffered name so we look for shr.o first

Spelling: preferred [single F, double R]

+def get_version(name, members):
+ """[. . .]
+ Before the GNU convention became the standard scheme regardless of
+ binary size AIX packagers used GNU convention "as-is" for 32-bit
+ archive members but used an "distinguishing" name for 64-bit members.

Should be: a "distinguishing" [not “an”]

I would like to say:
a) I am happy this is being considered for Python3-3.7

b) I still believe it is a "bug" plain and simple - it (find_library()) cannot work in "normal" situations. Why "IBM" or others never addressed this is beyond me - but - imho - that is a poor argument for not fixing it.

FYI: As I have documented in ... this does not mean that ctypes.CDLL is incapable of loading a library. If you know the 'magic', aka what is not being provided by this "implementation dependent" module - it is possible to load either a .so file (actually a file by any name as long as it is a shared archive) - or an AIX-style .a archive with shared libraries stored internally as "archive members" - again, any member name is acceptable.

The bug: The "default code" behind an "else:" block depends on the presence of the program "/sbin/ldconfig" to create a list of shared libraries. Normally, this program is not installed on an AIX system (certainly not production - maybe a system with gcc environment and using the gcc ld rather than the AIX ld (the latter being normal).

Using python -m trace -C /tmp --count Lib/src/util.py The following (key) counts are reported.

This is all that is being called - so when /sbin/ldonfig is not installed (which is usual) - so this code can only return 'None'.

Environment: AIX 5.3, AIX 6.1, AIX 7.1 - all identical

root@x064:[/]ls -l /sbin/ld*
ls: 0653-341 The file /sbin/ld* does not exist.

michael@x071:[/home/michael]ls /sbin/ld*
ls: cannot access '/sbin/ld*': A file or directory in the path name does not exist.
(GNU coreutils ls here)

root@x062:[/]ls /sbin/ld*
ls: 0653-341 The file /sbin/ld* does not exist.

So, obviously the sys.popen() call below will always 'fail'.

The basic trace from util.py:
1: if os.name == "posix":
# find and load_version
1: print find_library("m")
1: print find_library("c")
1: print find_library("bz2")

And how this is called:

       else:

1:         def _findSoname_ldconfig(name):
3:             import struct
3:             if struct.calcsize('l') == 4:
                   machine = os.uname()[4] + '-32'
               else:
3:                 machine = os.uname()[4] + '-64'
3:             mach_map = {
3:                 'x86_64-64': 'libc6,x86-64',
3:                 'ppc64-64': 'libc6,64bit',
3:                 'sparc64-64': 'libc6,64bit',
3:                 's390x-64': 'libc6,64bit',
3:                 'ia64-64': 'libc6,IA-64',
                   }
3:             abi_type = mach_map.get(machine, 'libc6')

               # XXX assuming GLIBC's ldconfig (with option -p)
3:             expr = r'\s+(lib%s\.[^\s]+)\s+\(%s' % (re.escape(name), abi_type)
3:             f = os.popen('LC_ALL=C LANG=C /sbin/ldconfig -p 2>/dev/null')
3:             try:
3:                 data = f.read()
               finally:
3:                 f.close()
3:             res = re.search(expr, data)
3:             if not res:
3:                 return None
               return res.group(1)

1:         def find_library(name):
3:             return _findSoname_ldconfig(name) or _get_soname(_findLib_gcc(name))

And with python2-2.7.13 - the counts start the same - but the result is the same - by definition 'None' because subprocess.Popen() also
does not have any output to search...

1:     if os.name == "posix":
           # find and load_version
1:         print find_library("m")
1:         print find_library("c")
1:         print find_library("bz2")

       else:

1:         def _findSoname_ldconfig(name):
3:             import struct
3:             if struct.calcsize('l') == 4:
3:                 machine = os.uname()[4] + '-32'
               else:
                   machine = os.uname()[4] + '-64'
3:             mach_map = {
3:                 'x86_64-64': 'libc6,x86-64',
3:                 'ppc64-64': 'libc6,64bit',
3:                 'sparc64-64': 'libc6,64bit',
3:                 's390x-64': 'libc6,64bit',
3:                 'ia64-64': 'libc6,IA-64',
                   }
3:             abi_type = mach_map.get(machine, 'libc6')

               # XXX assuming GLIBC's ldconfig (with option -p)
3:             expr = r'\s+(lib%s\.[^\s]+)\s+\(%s' % (re.escape(name), abi_type)

3:             env = dict(os.environ)
3:             env['LC_ALL'] = 'C'
3:             env['LANG'] = 'C'
3:             null = open(os.devnull, 'wb')
3:             try:
3:                 with null:
3:                     p = subprocess.Popen(['/sbin/ldconfig', '-p'],
3:                                           stderr=null,
3:                                           stdout=subprocess.PIPE,
3:                                           env=env)
3:             except OSError:  # E.g. command not found
3:                 return None
               [data, _] = p.communicate()
               res = re.search(expr, data)
               if not res:
                   return None
               return res.group(1)

========
In closing:

a) this is an "issue" aka bug, plain and simple - even it is has been ignored as such (other issues only complained about poorer performance because /sbin/ldconfig was not available). Please do not say - not fixing it because noone ever complained before. Otherwise, what is the point of ever accepting bug-reports aka issues if they can just be ignored.

b) I want to thank Martin for his help on many (PEP-8 et al) improvements to my initial proposals for a patch.

c) more important to me right now is that this be recognized as a bug - that should have been reported and resolved years ago. Personally, I do not care why noone ever reported it BUT I would like to see it properly identified that the default code is a specific implementation that is no way related to a normal AIX system - and an AIX-specific implementation is needed for normal operation of python2 and/or python3.

Thank you for your time and thought!

OOPS: I have a ... above, meant to be a link to a message.

I also needed to 'patch' util.py - with 32-bit build with something like this:
--- src/python-2.7.13/Lib/ctypes/util.py        2016-12-17 20:05:05 +0000
+++ python-2.7.13.0/Lib/ctypes/util.py  2017-01-13 13:29:12 +0000
@@ -299,6 +299,10 @@
             print cdll.LoadLibrary("libcrypto.dylib")
             print cdll.LoadLibrary("libSystem.dylib")
             print cdll.LoadLibrary("System.framework/System")
+        elif sys.platform[:3] == "aix":
+            from ctypes import CDLL
+            RTLD_MEMBER =  0x00040000
+            print CDLL("libc.a(shr.o)", RTLD_MEMBER)
         else:
             print cdll.LoadLibrary("libm.so")
             print cdll.LoadLibrary("libcrypt.so")

And 64-bit with:

1:         elif sys.platform[:3] == "aix":
1:             from ctypes import CDLL
1:             RTLD_MEMBER =  0x00040000
1:             print CDLL("libc_64.a(shr.o)", RTLD_MEMBER)

The respective output is:

root@x064:[/data/prj/python/python-2.7.13.0]./python Lib/ctypes/util.py
None
None
None
<CDLL 'libc.a(shr.o)', handle 10 at 300fe0b0>

root@x064:[/data/prj/python/python-2.7.12.0]./python Lib/ctypes/util.py
None
None
None
<CDLL 'libc.a(shr_64.o)', handle d at 7000000001941d0>

Michael Felt: if you still want the code compatible with Python 2 and 3 (and others are happy with that), I suggest documenting that in a code comment.

New changeset c5ae169 by Victor Stinner (Michael Felt) in branch 'master':
bpo-26439 Fix ctypes.util.find_library failure on AIX (bpo-4507)
c5ae169

Michael Felt: I merged your PR 🎉🍰✨ ! Thank you very much for your patience and perseverance. 80 comments on this bug, 117 comments and 19 commits in the PR... wow! AIX is not dead 😄

I leave the bug open since Martin, Mariatta and me were interested to make further minor coding style changes.

Although I'm unable to double check for the moment, feels like the "SVR4" support still is incomplete: Remember that even the libNAME.so file may be an archive, or a symlink to an archive, with the real Shared Object as member having the F_LOADONLY flag set and usually named shr.o or shr_64.o, besides an Import File shr.imp or shr_64.imp, which actually is used at linktime and referring to the real Shared Object: This is necessary to emulate the "DT_SONAME" feature seen with ELF shared libraries.

For reference, please have a look at the shared libraries created by recent libtool when --with-aix-soname=svr4 configure option is set.

Although I'm unable to double check for the moment, feels like the "SVR4" support still is incomplete: (...)

Sorry, I don't know what is SVR4. Is it a version of AIX? If it's rarely used, maybe a different issue should be opened to ask for an enhancement.

SVR4 - stands for AT&T System V Release 4 - the beginning (as I understand it) of shared libraries as (indivudual) .so files in UNIX. SVR3 (and earlier) used .a files (aka archives with members inside).

Within this context, the "svr4" label originates in the "-bsvr4" AIX linker flag, and actually is another (yet fully documented by the ld(1) man page) method for creating shared libraries on AIX to support filename based shared library versioning, which is known as the DT_SONAME tag with the ELF binary format.

For details please refer to the GCC install doc:
https://gcc.gnu.org/install/configure.html#WithAixSoname

New changeset c0919c2 by Mariatta in branch 'master':
bpo-26439: Convert %s in Lib/ctypes/_aix.py to f-strings. (GH-4986)
c0919c2

will open a new PR for ./Doc/library/ctypes.rst

It looks like c5ae169 broke find_library on macOS; see bpo-33281.

imho - this should have status "closed"

Thank you Michael Felt for your big "Implement find_library() support in ctypes/util for AIX" contribution, commit c5ae169!

I close the issue.