Author Michael.Felt
Recipients David.Edelsohn, Michael.Felt, aixtools@gmail.com, martin.panter, python-dev
Date 2017-01-13.13:40:26
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1484314828.47.0.352824271967.issue26439@psf.upfronthosting.co.za>
In-reply-to
Content
I would like to say:
a) I am happy this is being considered for Python3-3.7

b) I still believe it is a "bug" plain and simple - it (find_library()) cannot work in "normal" situations. Why "IBM" or others never addressed this is beyond me - but - imho - that is a poor argument for not fixing it.

FYI: As I have documented in ... this does not mean that ctypes.CDLL is incapable of loading a library. If you know the 'magic', aka what is not being provided by this "implementation dependent" module - it is possible to load either a .so file (actually a file by any name as long as it is a shared archive) - or an AIX-style .a archive with shared libraries stored internally as "archive members" - again, any member name is acceptable.

The bug: The "default code" behind an "else:" block depends on the presence of the program "/sbin/ldconfig" to create a list of shared libraries. Normally, this program is not installed on an AIX system (certainly not production - maybe a system with gcc environment and using the gcc ld rather than the AIX ld (the latter being normal).

Using python -m trace -C /tmp --count Lib/src/util.py The following (key) counts are reported.

This is all that is being called - so when /sbin/ldonfig is not installed (which is usual) - so this code can only return 'None'.

Environment: AIX 5.3, AIX 6.1, AIX 7.1 - all identical

root@x064:[/]ls -l /sbin/ld*
ls: 0653-341 The file /sbin/ld* does not exist.

michael@x071:[/home/michael]ls /sbin/ld*
ls: cannot access '/sbin/ld*': A file or directory in the path name does not exist.
(GNU coreutils ls here)

root@x062:[/]ls /sbin/ld*
ls: 0653-341 The file /sbin/ld* does not exist.

So, obviously the sys.popen() call below will always 'fail'.


The basic trace from util.py:
    1:     if os.name == "posix":
               # find and load_version
    1:         print find_library("m")
    1:         print find_library("c")
    1:         print find_library("bz2")


And how this is called:

           else:

    1:         def _findSoname_ldconfig(name):
    3:             import struct
    3:             if struct.calcsize('l') == 4:
                       machine = os.uname()[4] + '-32'
                   else:
    3:                 machine = os.uname()[4] + '-64'
    3:             mach_map = {
    3:                 'x86_64-64': 'libc6,x86-64',
    3:                 'ppc64-64': 'libc6,64bit',
    3:                 'sparc64-64': 'libc6,64bit',
    3:                 's390x-64': 'libc6,64bit',
    3:                 'ia64-64': 'libc6,IA-64',
                       }
    3:             abi_type = mach_map.get(machine, 'libc6')

                   # XXX assuming GLIBC's ldconfig (with option -p)
    3:             expr = r'\s+(lib%s\.[^\s]+)\s+\(%s' % (re.escape(name), abi_type)
    3:             f = os.popen('LC_ALL=C LANG=C /sbin/ldconfig -p 2>/dev/null')
    3:             try:
    3:                 data = f.read()
                   finally:
    3:                 f.close()
    3:             res = re.search(expr, data)
    3:             if not res:
    3:                 return None
                   return res.group(1)

    1:         def find_library(name):
    3:             return _findSoname_ldconfig(name) or _get_soname(_findLib_gcc(name))


And with python2-2.7.13 - the counts start the same - but the result is the same - by definition 'None' because subprocess.Popen() also
does not have any output to search...

    1:     if os.name == "posix":
               # find and load_version
    1:         print find_library("m")
    1:         print find_library("c")
    1:         print find_library("bz2")

           else:

    1:         def _findSoname_ldconfig(name):
    3:             import struct
    3:             if struct.calcsize('l') == 4:
    3:                 machine = os.uname()[4] + '-32'
                   else:
                       machine = os.uname()[4] + '-64'
    3:             mach_map = {
    3:                 'x86_64-64': 'libc6,x86-64',
    3:                 'ppc64-64': 'libc6,64bit',
    3:                 'sparc64-64': 'libc6,64bit',
    3:                 's390x-64': 'libc6,64bit',
    3:                 'ia64-64': 'libc6,IA-64',
                       }
    3:             abi_type = mach_map.get(machine, 'libc6')

                   # XXX assuming GLIBC's ldconfig (with option -p)
    3:             expr = r'\s+(lib%s\.[^\s]+)\s+\(%s' % (re.escape(name), abi_type)

    3:             env = dict(os.environ)
    3:             env['LC_ALL'] = 'C'
    3:             env['LANG'] = 'C'
    3:             null = open(os.devnull, 'wb')
    3:             try:
    3:                 with null:
    3:                     p = subprocess.Popen(['/sbin/ldconfig', '-p'],
    3:                                           stderr=null,
    3:                                           stdout=subprocess.PIPE,
    3:                                           env=env)
    3:             except OSError:  # E.g. command not found
    3:                 return None
                   [data, _] = p.communicate()
                   res = re.search(expr, data)
                   if not res:
                       return None
                   return res.group(1)

========
In closing:

a) this is an "issue" aka bug, plain and simple - even it is has been ignored as such (other issues only complained about poorer performance because /sbin/ldconfig was not available). Please do not say - not fixing it because noone ever complained before. Otherwise, what is the point of ever accepting bug-reports aka issues if they can just be ignored.

b) I want to thank Martin for his help on many (PEP8 et al) improvements to my initial proposals for a patch.

c) more important to me right now is that this be recognized as a bug - that should have been reported and resolved years ago. Personally, I do not care why noone ever reported it BUT I would like to see it properly identified that the default code is a specific implementation that is no way related to a normal AIX system - and an AIX-specific implementation is needed for normal operation of python2 and/or python3.

Thank you for your time and thought!
History
Date User Action Args
2017-01-13 13:40:28Michael.Feltsetrecipients: + Michael.Felt, python-dev, martin.panter, David.Edelsohn, aixtools@gmail.com
2017-01-13 13:40:28Michael.Feltsetmessageid: <1484314828.47.0.352824271967.issue26439@psf.upfronthosting.co.za>
2017-01-13 13:40:28Michael.Feltlinkissue26439 messages
2017-01-13 13:40:26Michael.Feltcreate