Author Michael.Felt
Recipients Michael.Felt, aixtools@gmail.com, martin.panter
Date 2017-01-03.13:07:41
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1483448863.5.0.268051914254.issue27435@psf.upfronthosting.co.za>
In-reply-to
Content
Again, I would like to draw attention to this issue - BECAUSE -

it is possible to use ctypes.CDLL() ASIS, on AIX - however, ctypes.util.find_library() always seems to fail (i.e., returns None even when a .so file exists)

When the library names are hard-coded, rather than found by find_library() everything seems to work fine.

In short, I hope this will finally be recognized as a bug (ctypes.util.find_library()).

Ideally, RTLD_MEMBER logic would be added to the Lib/ctypes/__init__.py so that CDLL does not have to have RTLD_MEMBER added manually as it does now.

(I have not yet figured out how to use LibraryLoader() properly, it seems to never complain, or behave the same as ctypes.CDLL(None) which returns the "python process" namespace - if I understand that correctly.

p.s. a) I have a newer patch; b) PEP 8, etc. pickyness is encouraged (i.e., expect multiple iterations before a patch is accepted - I do not mind learning PEP 8 et al.);

imho - nothing new is being added to what python2 (and python3!) already can do. However, a fix to make it easier and clearer to use (there must be a PEP for that case as well) - is warranted.

++++++++
Preparation:
a) the contents of the archive (example)
root@x064:[/data/prj/python/python-2.7.12.0]ar -Xany tv /usr/lib/libcurses.a
rwxr-xr-x   300/300   499495 Sep 26 22:35 2007 shr42.o
rwxr-xr-x   300/300   321459 Sep 26 22:04 2007 shr4.o
rwxr-xr-x   300/300   197552 Sep 26 22:04 2007 shr.o
rwxr-xr-x   300/300   591888 Sep 26 22:35 2007 shr42_64.o

b) create a empty .so file - MAYBE this will be found by the so-called "posix"
   routine in ctypes.util.find_library

root@x064:[/data/prj/python/python2-2.7.13]touch /usr/lib/libcurses.so

c) A test program
import sys
import ctypes
from ctypes import util

if sys.platform[:3] == "aix":
    print ctypes.util.find_library("curses")
    RTLD_MEMBER =  0x00040000
    if (sys.maxsize < 2**32):
        print 32
        print ctypes.CDLL("libcurses.a(shr.o)", RTLD_MEMBER)
    else:
        print 64
        print ctypes.CDLL("libcurses.a(shr42_64.o)", RTLD_MEMBER)

Testing:
a) notice "None" as value printed by find_library
b) notice that ctypes.CDLL can load the library when RTLD_MEMBER isd added
root@x064:[/data/prj/python/python2-2.7.13]./python ../python2-2.7.13/test*.py
None
64
<CDLL 'libcurses.a(shr42_64.o)', handle 10 at 7000000001280f0>

Changes:
-- since /usr/lib/libcurses.so is an empty file
   MAYBE - that is why the result was None
-- extract the 64-bit member and rename as /usr/lib/libcurses.so
-- test both find_library (still says None!) and CDLL of the extracted member
-- as it also loads, CDLL, i.e., dlopen(), works without RTLD_MEMBER
   (as it should)
-- there does not appear top be anything wrong with ctypes.CDLL,
   but only with the 'utils' included

root@x064:[/data/prj/python/python2-2.7.13]ar x /usr/lib/libcurses.a shr42_64.o
root@x064:[/data/prj/python/python2-2.7.13]mv shr42_64.o /usr/lib/libcurses.so

import sys
import ctypes
from ctypes import util

if sys.platform[:3] == "aix":
    print ctypes.util.find_library("curses")
    RTLD_MEMBER =  0x00040000
    if (sys.maxsize < 2**32):
        print 32
        print ctypes.CDLL("libcurses.a(shr.o)", RTLD_MEMBER)
    else:
        print 64
        print ctypes.CDLL("libcurses.a(shr42_64.o)", RTLD_MEMBER)
        print ctypes.CDLL("libcurses.so")

root@x064:[/data/prj/python/python2-2.7.13]./python ../python2-2.7.13/test*.py
None
64
<CDLL 'libcurses.a(shr42_64.o)', handle 10 at 700000000128128>
<CDLL 'libcurses.so', handle 11 at 700000000128128>

Verification of (expected) behavior:
a) CDLL in both formats returns the same function pointer for 'addch'
b) LibraryLoader always returns something, even when it is nonsense
c) CDLL - fails - aka Traceback - when an argument cannot be loaded

from ctypes import util

if sys.platform[:3] == "aix":
    print ctypes.util.find_library("curses")
    RTLD_MEMBER =  0x00040000
    if (sys.maxsize < 2**32):
        print 32
        print ctypes.CDLL("libcurses.a(shr.o)", RTLD_MEMBER)
    else:
        print 64
        print ctypes.CDLL("libcurses.a(shr42_64.o)", RTLD_MEMBER)
        lib = ctypes.CDLL("libcurses.a(shr42_64.o)", RTLD_MEMBER)
        if hasattr(lib, 'addch'):
            print "addch found"
            print lib.addch
        else:
            print "addch not found"
        print ctypes.CDLL(None)
        lib = ctypes.CDLL(None)
        if hasattr(lib, 'addch'):
            print "addch found"
            print lib.addch
        else:
            print "addch not found"
        print ctypes.CDLL("libcurses.so")
        lib = ctypes.CDLL("libcurses.so")
        if hasattr(lib, 'addch'):
            print "addch found"
            print lib.addch
        else:
            print "addch not found"
        print "LibraryLoader tests"
        print ctypes.LibraryLoader("libcurses.so")
        lib = ctypes.LibraryLoader("libcurses.so")
        print ctypes.LibraryLoader(None)
        print ctypes.LibraryLoader("libcurses.a(shr42_64.o)")
        print ctypes.LibraryLoader("libc.a(shr_64.o)")
        print "LibraryLoader XXX tests show LibraryLoader masks unloadable libraries"
        print ctypes.LibraryLoader("xxx.so")
        print ctypes.CDLL("xxx.so")

returns: (python2-2.7.12 and python2-2.7.13 return the same value)

root@x064:[/data/prj/python/python-2.7.12.0]./python ../python2-2.7.13/test*.py
None
64
<CDLL 'libcurses.a(shr42_64.o)', handle e at 700000000135a58>
addch found
<_FuncPtr object at 0x700000000174108>
<CDLL 'None', handle 10 at 7000000001354a8>
addch not found
<CDLL 'libcurses.so', handle 12 at 7000000001358d0>
addch found
<_FuncPtr object at 0x700000000174048>
LibraryLoader tests
<ctypes.LibraryLoader object at 0x7000000001354a8>
<ctypes.LibraryLoader object at 0x700000000135470>
<ctypes.LibraryLoader object at 0x700000000135470>
<ctypes.LibraryLoader object at 0x700000000135470>
LibraryLoader XXX tests show LibraryLoader masks unloadable libraries
<ctypes.LibraryLoader object at 0x700000000135470>
Traceback (most recent call last):
  File "../python2-2.7.13/test_ctypes.py", line 42, in <module>
    print ctypes.CDLL("xxx.so")
  File "/data/prj/python/python-2.7.12.0/Lib/ctypes/__init__.py", line 362, in __init__
    self._handle = _dlopen(self._name, mode)
OSError:        0509-022 Cannot load module .
        0509-026 System error: A file or directory in the path name does not exist.

root@x064:[/data/prj/python/python-2.7.12.0]cd ../py*13

root@x064:[/data/prj/python/python2-2.7.13]./python ../python2-2.7.13/test*.py
None
64
<CDLL 'libcurses.a(shr42_64.o)', handle 10 at 700000000127320>
addch found
<_FuncPtr object at 0x700000000158288>
<CDLL 'None', handle 12 at 700000000127710>
addch not found
<CDLL 'libcurses.so', handle 14 at 700000000127898>
addch found
<_FuncPtr object at 0x7000000001581c8>
LibraryLoader tests
<ctypes.LibraryLoader object at 0x700000000127908>
<ctypes.LibraryLoader object at 0x700000000127978>
<ctypes.LibraryLoader object at 0x700000000127978>
<ctypes.LibraryLoader object at 0x700000000127978>
LibraryLoader XXX tests show LibraryLoader masks unloadable libraries
<ctypes.LibraryLoader object at 0x700000000127978>
Traceback (most recent call last):
  File "../python2-2.7.13/test_ctypes.py", line 42, in <module>
    print ctypes.CDLL("xxx.so")
  File "/data/prj/python/src/python2-2.7.13/Lib/ctypes/__init__.py", line 362, in __init__
    self._handle = _dlopen(self._name, mode)
OSError:        0509-022 Cannot load module .
        0509-026 System error: A file or directory in the path name does not exist.
root@x064:[/data/prj/python/python2-2.7.13]
History
Date User Action Args
2017-01-03 13:07:43Michael.Feltsetrecipients: + Michael.Felt, martin.panter, aixtools@gmail.com
2017-01-03 13:07:43Michael.Feltsetmessageid: <1483448863.5.0.268051914254.issue27435@psf.upfronthosting.co.za>
2017-01-03 13:07:43Michael.Feltlinkissue27435 messages
2017-01-03 13:07:41Michael.Feltcreate