New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ctypes library loading and AIX - also for 2.7.X (and later) #71622
Comments
I am opening a new issue # - about the same problem described in 26439, but now calling it a bug However, with the feedback from code review I am beginning to understand what the ctypes functionality in __init__.py it is clear that there is one entry point for LoadLibrary. For find_library(), The pseudo_code is roughly this (although import ctypes._$_platform._xxx may be incorrect syntax) Unclear is how a regular .so file should be treated by find_library() when the actual file is not Currently it is "defined" 7 times (including the definition I added at line 79) Within the os.name() and sys.platform code block there are sufficient "liberities" to call
+49 def find_library(name): in block if os.name == "nt" +217 def find_library(name, is64 = False): inside: elif sys.platform == "sunos5": +220 else: ++++++ So, this looks like a design change, but it is not. All this code could be re-indented and inserted Note: the extra tests added to util.py are only for my enjoyment, i.e., verification of several cases I have Once I knew what CDLL(None) was suppossed to +++ Closing +++ find_library() is defined multiple times, so adding one for aix is not a design change, with, of course def _findAIXmember() and def _findSoname_dump() defined within the limits of util.py +++ Disclaimer :P +++ Going back to the earliest documentation I could find for ctypes https://docs.python.org/2.6/library/ctypes.html (new in 2.5) the key lines are: ctypes exports the cdll, and on Windows windll and oledll objects, for loading dynamic link libraries. You load libraries by accessing them as attributes of these objects. cdll loads libraries which export functions using the standard cdecl calling convention, while ... For find_library() - the documentation is quite clear - find_library is system dependent - mentioning that "linux", as one example, uses ldconfig to find libraries: 15.15.2.1. Finding shared libraries When programming in a compiled language, shared libraries are accessed when compiling/linking a program, and when the program is run. The purpose of the find_library() function is to locate a library in a way similar to what the compiler does (on platforms with several versions of a shared library the most recent should be loaded), while the ctypes library loaders act like when a program is run, and call the runtime loader directly. The ctypes.util module provides a function which can help to determine the library to load. ctypes.util.find_library(name)
Try to find a library and return a pathname. name is the library name without any prefix like lib, suffix like .so, .dylib or version number (this is the form used for the posix linker option -l). If no library can be found, returns None. The exact functionality is system dependent. On Linux, find_library() tries to run external programs (/sbin/ldconfig, gcc, and objdump) to find the library file. It returns the filename of the library file. .... In short, I believe there is sufficient clarity in the initial documentation - that it is an "error of ommission" (so maybe "shame on IBM" that noone took this up years ago) and has every right as Linux, Solaris, openBSD, etc.. to have the system dependent glue that is needed to work "natively" on the platform. -- but I am biased -- so if I am wrong - please be kind. Note: goal is to have the new patch, still with _aixutil.py as a matter of textual clarity rather than as language/module syntax. If being in a different file is the primary/only objection then, when I am again able to devote time - I will move the code into an "aix" block within util.py |
Adding "type behavior" as I have now read that that is actually the python friendly way of talking about a 'bug'. Testing my proposed patch for 2.7 with python2.7.12 - will update with patch when finished. |
""" |
I have mercurial "installed", but not (yet) any idea on how to use it to submit a patch. Again: going back to comments in bpo-26439 - and my 'new' responses as far as the patch submitted (but not yet using Mercurial)
Not included. RTLD_MEMBER is a well documented constant needed for AIX dlopen. It is hard-coded in __init__.py
Months further, I understand that is a variant behavior, and has been removed. find_library("c"), find_library("crypto") are supported, an argument suchas find_library("libintl.so") will return None (or something like /usr/lib/liblibintl.so will need to exist).
Again, my misunderstanding of how this function is intended - python newbie. arguments starting with "lib" are not likely to work - and it is not on find_library to strip "lib" to make the argument more -l like.
As before, and as is the status in bpo-9998 - no use of LIBPATH or LD_LIBRARY_PATH - even though dlopen() does use them. Although, if your objection is primarily on 'set' (i.e., a get could be accepted) - that is easy to put back in. Thanks again for your help. Note: the reason for the new issue# is because too much of what I was submitting before was "extension" and not "correction". Hopefully, this makes the discussion more clear. |
For 2.7, adding the automatic RTLD_MEMBER mode does not seem like a bug fix to me. Currently, I understand this code could load two separate libraries: file = CDLL("libcrypto.a(libcrypto.so.1.0.0)")
member = CDLL("libcrypto.a(libcrypto.so.1.0.0)", DEFAULT_MODE | 0x00040000) With your proposed change, the first line will do the same as the second line, and Python will no longer provide a way to load a file named like in the first operation. Maybe this is okay for the next version of Python 3 (because it is only breaking a rare corner case), but my view is it is not worth changing 2.7. |
If you are still figuring out Mercurial, maybe see <https://docs.python.org/devguide/setup.html#getting-the-source-code\> and <https://docs.python.org/devguide/patch.html#tool-usage\> if you haven’t already. |
On 02-Aug-16 15:34, Martin Panter wrote:
FYI - I did test this case, and as a file, without 0x000400000 (aka
I believe python2 has some years to go, and basically, you ask anyone I have been providing AIX support for over 20 years - NEVER have I seen In yet another case I have seen a case where they copied everything to a Yes, when I first started back in February and March: a) knew next to However, I do believe what I have here does what is done for other I hope saying "please" helps. Without it, the AIX implementation is so - PLEASE - pretty please! Michael p.s. And I shall look at the mercurial pages - and I hope have it working.
|
The more common TESTED "behavior" issue is simply: find_library("c") always returns None while it should be returning michael@x071:[/home/michael]grep RTLD_MEMBER /usr/include/.h I mention find_library("c") specifically as that is used as a core test test_callback.py, test_error.py and test_loading.py How can a test that is always returned None really be testing anything. So, how is the current behavior not a "bug" (that, for ages, has either On 11-Aug-16 17:10, Michael Felt wrote:
|
The ctypes tests all seem to be protected with code that checks for None, and explicitly skip the test in that case. The skip message should be visible when you run the test in verbose mode. We could avoid skipping these tests in 2.7 by adding a special case for AIX that loads the library via CDLL("libc.a(shr[64].o)", RTLD_MEMBER). Like an extension of the special cases already at <https://hg.python.org/cpython/file/v2.7.12/Lib/ctypes/test/test_loading.py#l8\>, but we have to set the RTLD flag as well as the library name. |
On 22-Aug-16 04:04, Martin Panter wrote:
Rather than "hard-code" proper strings in a test - shouldn't the test be For example, on AIX the IBM provided /usr/lib/libm.a only has static root@x064:[/data/prj/aixtools/python/python-2.7.12.0/Lib/ctypes]../../python
util.py
None
None
None
Traceback (most recent call last):
File "util.py", line 271, in <module>
test()
File "util.py", line 266, in test
print cdll.LoadLibrary("libm.so")
File
"/data/prj/aixtools/python/python-2.7.12.0/Lib/ctypes/__init__.py", line
440, in LoadLibrary
return self._dlltype(name)
File
"/data/prj/aixtools/python/python-2.7.12.0/Lib/ctypes/__init__.py", line
362, in __init__
self._handle = _dlopen(self._name, mode)
OSError: 0509-022 Cannot load module .
0509-026 System error: A file or directory in the path name
does not exist. The documentation says: The exact functionality is system dependent. On Linux, |find_library()| tries to run external programs Are you saying the documentation should be updated to read: And the closing bit: I apologize that my frustration is more than peeking through. But I am I understand - AIX has not been a major (user) platform for python. I am If the root cause is: "IBM showed no interest X years ago and we, In closing - I thought python was multi-platform and I had hoped to be Thank you for your time and patience with my rant!
|
I had understood that changing find_library() would only be useful in combination with the automatic RTDL_MEMBER detection. If you want to mention lack of support for AIX in the documentation, that is okay by me. If you want to propose an alternative find_library() implementation _without_ affecting the CDLL behaviour, that may be okay too. For the libuuid problem, for 2.7 I would suggest adding an explicit call to load libc.a(. . .) or whatever is appropriate. Personally I don’t have a specific interest in AIX, but I am happy to commit patches for AIX if I think they are sensible. I think I mentioned before in the other bug that your changes seemed okay for the next Python version. The beta deadline for 3.6 is in a few weeks (PEP-494), although after that there is still 3.7. |
On 23-Aug-16 02:01, Martin Panter wrote:
Now: Patched:
I see (now) that you have processed them, and set them for review - many I do not have any specific interest in python. My interest is supporting
If it misses 3.6, then I will have some soul searching to do.
|
The "core" changes in this patch are very simple - in parallel with the way find_library() is managed in util.py a) __init__.py:
add an additional mode bit for call to dlopen() (in __init__.py)
diff -u src/Python-2.7.12/Lib/ctypes/__init__.py python-2.7.12.2/Lib/ctypes/__init__.py
--- src/Python-2.7.12/Lib/ctypes/__init__.py 2016-06-25 21:49:30 +0000
+++ python-2.7.12.2/Lib/ctypes/__init__.py 2016-09-28 12:55:15 +0000
@@ -352,6 +352,16 @@
flags |= _FUNCFLAG_USE_ERRNO
if use_last_error:
flags |= _FUNCFLAG_USE_LASTERROR
+ if _sys.platform.startswith("aix"):
+ """When the name contains ".a(" and ends with ")",
+ asin "libFOO.a(libFOO.so)" this is taken to be an
+ archive(member) syntax for dlopen(), and the mode is adjusted.
+ Otherwise, name is presented to dlopen() as a file argument.
+ """
+ # from _ctypes import RTLD_NOW - not until Python3.7
+ if name and name.endswith(")") and ".a(" in name:
+ RTLD_MEMBER = 0x00040000
+ mode |= RTLD_MEMBER class _FuncPtr(_CFuncPtr):
_flags_ = flags b) util.py: diff -u src/Python-2.7.12/Lib/ctypes/util.py python-2.7.12.2/Lib/ctypes/util.py +if sys.platform.startswith("aix"):
+ # find .so members in .a files
+ # using dump loader header information + sys.
+ import ctypes._aix as aix
+
+ def find_library(name):
+ return aix.find_library(name)
+
elif os.name == "posix":
# Andreas Degert's find functions, using gcc, /sbin/ldconfig, objdump
import re, tempfile, errno
@@ -247,10 +255,11 @@
print find_library("msvcrt")
if os.name == "posix":
- # find and load_version
- print find_library("m")
- print find_library("c")
- print find_library("bz2")
+ # find
+ print("m\t:: %s" % find_library("m"))
+ print("c\t:: %s" % find_library("c"))
+ print("bz2\t:: %s" % find_library("bz2"))
+ print("crypt\t:: %s" % find_library("crypt"))
# getattr
## print cdll.m
@@ -262,6 +271,12 @@
print cdll.LoadLibrary("libcrypto.dylib")
print cdll.LoadLibrary("libSystem.dylib")
print cdll.LoadLibrary("System.framework/System")
+ elif sys.platform.startswith("aix"):
+ print("crypto\t:: %s" % cdll.LoadLibrary(find_library("crypto")))
+ if (sys.maxsize < 2**32):
+ print("c\t:: %s" % cdll.LoadLibrary("libc.a(shr.o)"))
+ else:
+ print("c\t:: %s" % cdll.LoadLibrary("libc.a(shr_64.o)"))
else:
print cdll.LoadLibrary("libm.so")
print cdll.LoadLibrary("libcrypt.so") d) in test_loading.py say test is failed if find_library does not find diff -u src/Python-2.7.12/Lib/ctypes/test/test_loading.py python-2.7.12.2/Lib/ctypes/test/test_loading.py
--- src/Python-2.7.12/Lib/ctypes/test/test_loading.py 2016-06-25 21:49:30 +0000
+++ python-2.7.12.2/Lib/ctypes/test/test_loading.py 2016-09-28 12:43:46 +0000
@@ -11,6 +11,11 @@
libc_name = "coredll"
elif sys.platform == "cygwin":
libc_name = "cygwin1.dll"
+elif sys.platform.startswith("aix"):
+ if (sys.maxsize < 2**32):
+ libc_name = "libc.a(shr.o)"
+ else:
+ libc_name = "libc.a(shr_64.o)"
else:
libc_name = find_library("c") @@ -38,11 +43,16 @@ def test_find(self):
+ # to track that at least one call to find_library() found something
+ found = False
for name in ("c", "m"):
lib = find_library(name)
if lib:
+ found = True
cdll.LoadLibrary(lib)
CDLL(lib)
+ # test is FAILED if nothing was found
+ self.assertTrue(found)
Finally - there is the additional file: _aix.py - submitted as an attachment - and a patch with them all |
Considering that python-2.7.13 is posed for a release - I ask again, considering that python-2.7.12 (and I expect a few earlier ones) can load "AIX" .a type shared libraries - if you know how to supply the name - that my patch be included in python-2.7.13. michael@x071:[/data/prj/python]-2.7.12.0/Lib/ctypes/util.py <
--- python-2.7.12/Lib/ctypes/util.py 2016-06-25 21:49:30 +0000
+++ python-2.7.12.0/Lib/ctypes/util.py 2016-12-02 10:12:01 +0000
@@ -262,6 +262,10 @@
print cdll.LoadLibrary("libcrypto.dylib")
print cdll.LoadLibrary("libSystem.dylib")
print cdll.LoadLibrary("System.framework/System")
+ if sys.platform == "aix5":
+ from ctypes import CDLL
+ RTLD_MEMBER = 0x00040000
+ print CDLL("libc.a(shr_64.o)", RTLD_MEMBER)
else:
print cdll.LoadLibrary("libm.so")
print cdll.LoadLibrary("libcrypt.so") And the result is: root@x064:[/data/prj/python/python-2.7.12.0/Lib/ctypes]../../python util.py In, other words, as is, find_library() will always return None for standard libraries (will only return a value if "root" has extracted and renamed a shared library member as a file) - while, if you know the magic that find_library cannot return - you can load such a library. IMVHO (in my Very humble opinion) - a bug that has been too long unnoticed - but can be repaired. There is no additional ability being added - it is already there. Just making it more accessible and legible for python programmers on AIX. If the patch-code needs more cleanup, improvement, etc. - fine. But do not lock out an easy interface to such a basic function. regards, |
Again, I would like to draw attention to this issue - BECAUSE - it is possible to use ctypes.CDLL() ASIS, on AIX - however, ctypes.util.find_library() always seems to fail (i.e., returns None even when a .so file exists) When the library names are hard-coded, rather than found by find_library() everything seems to work fine. In short, I hope this will finally be recognized as a bug (ctypes.util.find_library()). Ideally, RTLD_MEMBER logic would be added to the Lib/ctypes/init.py so that CDLL does not have to have RTLD_MEMBER added manually as it does now. (I have not yet figured out how to use LibraryLoader() properly, it seems to never complain, or behave the same as ctypes.CDLL(None) which returns the "python process" namespace - if I understand that correctly. p.s. a) I have a newer patch; b) PEP-8, etc. pickyness is encouraged (i.e., expect multiple iterations before a patch is accepted - I do not mind learning PEP-8 et al.); imho - nothing new is being added to what python2 (and python3!) already can do. However, a fix to make it easier and clearer to use (there must be a PEP for that case as well) - is warranted. ++++++++ b) create a empty .so file - MAYBE this will be found by the so-called "posix" root@x064:[/data/prj/python/python2-2.7.13]touch /usr/lib/libcurses.so c) A test program if sys.platform[:3] == "aix":
print ctypes.util.find_library("curses")
RTLD_MEMBER = 0x00040000
if (sys.maxsize < 2**32):
print 32
print ctypes.CDLL("libcurses.a(shr.o)", RTLD_MEMBER)
else:
print 64
print ctypes.CDLL("libcurses.a(shr42_64.o)", RTLD_MEMBER) Testing: Changes: root@x064:[/data/prj/python/python2-2.7.13]ar x /usr/lib/libcurses.a shr42_64.o import sys
import ctypes
from ctypes import util
if sys.platform[:3] == "aix":
print ctypes.util.find_library("curses")
RTLD_MEMBER = 0x00040000
if (sys.maxsize < 2**32):
print 32
print ctypes.CDLL("libcurses.a(shr.o)", RTLD_MEMBER)
else:
print 64
print ctypes.CDLL("libcurses.a(shr42_64.o)", RTLD_MEMBER)
print ctypes.CDLL("libcurses.so") root@x064:[/data/prj/python/python2-2.7.13]./python ../python2-2.7.13/test*.py Verification of (expected) behavior: from ctypes import util
if sys.platform[:3] == "aix":
print ctypes.util.find_library("curses")
RTLD_MEMBER = 0x00040000
if (sys.maxsize < 2**32):
print 32
print ctypes.CDLL("libcurses.a(shr.o)", RTLD_MEMBER)
else:
print 64
print ctypes.CDLL("libcurses.a(shr42_64.o)", RTLD_MEMBER)
lib = ctypes.CDLL("libcurses.a(shr42_64.o)", RTLD_MEMBER)
if hasattr(lib, 'addch'):
print "addch found"
print lib.addch
else:
print "addch not found"
print ctypes.CDLL(None)
lib = ctypes.CDLL(None)
if hasattr(lib, 'addch'):
print "addch found"
print lib.addch
else:
print "addch not found"
print ctypes.CDLL("libcurses.so")
lib = ctypes.CDLL("libcurses.so")
if hasattr(lib, 'addch'):
print "addch found"
print lib.addch
else:
print "addch not found"
print "LibraryLoader tests"
print ctypes.LibraryLoader("libcurses.so")
lib = ctypes.LibraryLoader("libcurses.so")
print ctypes.LibraryLoader(None)
print ctypes.LibraryLoader("libcurses.a(shr42_64.o)")
print ctypes.LibraryLoader("libc.a(shr_64.o)")
print "LibraryLoader XXX tests show LibraryLoader masks unloadable libraries"
print ctypes.LibraryLoader("xxx.so")
print ctypes.CDLL("xxx.so") returns: (python2-2.7.12 and python2-2.7.13 return the same value) root@x064:[/data/prj/python/python-2.7.12.0]./python ../python2-2.7.13/test*.py
None
64
<CDLL 'libcurses.a(shr42_64.o)', handle e at 700000000135a58>
addch found
<_FuncPtr object at 0x700000000174108>
<CDLL 'None', handle 10 at 7000000001354a8>
addch not found
<CDLL 'libcurses.so', handle 12 at 7000000001358d0>
addch found
<_FuncPtr object at 0x700000000174048>
LibraryLoader tests
<ctypes.LibraryLoader object at 0x7000000001354a8>
<ctypes.LibraryLoader object at 0x700000000135470>
<ctypes.LibraryLoader object at 0x700000000135470>
<ctypes.LibraryLoader object at 0x700000000135470>
LibraryLoader XXX tests show LibraryLoader masks unloadable libraries
<ctypes.LibraryLoader object at 0x700000000135470>
Traceback (most recent call last):
File "../python2-2.7.13/test_ctypes.py", line 42, in <module>
print ctypes.CDLL("xxx.so")
File "/data/prj/python/python-2.7.12.0/Lib/ctypes/__init__.py", line 362, in __init__
self._handle = _dlopen(self._name, mode)
OSError: 0509-022 Cannot load module .
0509-026 System error: A file or directory in the path name does not exist. root@x064:[/data/prj/python/python-2.7.12.0]cd ../py*13 root@x064:[/data/prj/python/python2-2.7.13]./python ../python2-2.7.13/test*.py
None
64
<CDLL 'libcurses.a(shr42_64.o)', handle 10 at 700000000127320>
addch found
<_FuncPtr object at 0x700000000158288>
<CDLL 'None', handle 12 at 700000000127710>
addch not found
<CDLL 'libcurses.so', handle 14 at 700000000127898>
addch found
<_FuncPtr object at 0x7000000001581c8>
LibraryLoader tests
<ctypes.LibraryLoader object at 0x700000000127908>
<ctypes.LibraryLoader object at 0x700000000127978>
<ctypes.LibraryLoader object at 0x700000000127978>
<ctypes.LibraryLoader object at 0x700000000127978>
LibraryLoader XXX tests show LibraryLoader masks unloadable libraries
<ctypes.LibraryLoader object at 0x700000000127978>
Traceback (most recent call last):
File "../python2-2.7.13/test_ctypes.py", line 42, in <module>
print ctypes.CDLL("xxx.so")
File "/data/prj/python/src/python2-2.7.13/Lib/ctypes/__init__.py", line 362, in __init__
self._handle = _dlopen(self._name, mode)
OSError: 0509-022 Cannot load module .
0509-026 System error: A file or directory in the path name does not exist.
root@x064:[/data/prj/python/python2-2.7.13] |
Feels like it is up to me to shed some light on various topics here: AIX defines (static) "Object" files and "Shared Object" files. Both can (but do not need to) be members of an "Archive Library". When talking about (dynamically loading) a "shared library" on non-AIX systems, the correct wording for AIX is (dynamically loading) a "Shared Object" - independent of whether it is a standalone file or an archive member. As you already agreed on, ctypes.util.find_library() indeed should return the location of the Shared Object - as either "/path/to/file" or "/path/to/file(member)". And ctypes.CDLL() should add the RTLD_MEMBER flag to _ctypes.dlopen() if there is the "(member)" part. However, that does not necessarily mean the RTLD_MEMBER value needs to be available through the ctypes API. Instead, RTLD_MEMBER feels more appropriate for the _ctypes API - and on AIX only. Anyway: However, to still get some kind of shared library versioning, multiple versions of one Shared Object are put into one single Archive Library, where the "old" versions get the LOADONLY flag set (via 'strip -e') - so executables linked against an old version still can load their "old" Shared Object, while the linker discovers the "current" version only. But this "single-filename" based shared library versioning is a huge burden for packages managers - either human or software, even when they maintain their packages in a private prefix (like /opt/freeware, /usr/local and similar). Neither can they put their "current" Shared Object's version into the single (system's) Archive Library, nor can they mark existing Shared Objects as LOADONLY. So for package managers it is crucial to have "multi-filename" based shared library versioning instead. But there is help: So yes, Import Files can be used for "multi-filename" based shared library versioning. Thus, libtool now offers the "--with-aix-soname=svr4" configure flag, where "libNAME.so.X" really is created as an Archive Library, containing the Shared Object "shr.o" with the LOADONLY flag set, and the Import File "shr.imp" referring to "libNAME.so.X(shr.o)" for runtime. Combined with the symlink "libNAME.so", for the "-lNAME" argument the linker discovers the Import File "libNAME.so(shr.imp)" now, while recording "libNAME.so.X(shr.o)" into the executable for the runtime loader. While libtool's "--with-aix-soname=svr4" variant creates "libNAME.a" from static Objects only, both libtool's "--with-aix-soname=aix" and "--with-aix-soname=both" - for backwards compatibility - create "libNAME.a" for traditional "single-filename" based versioning: Without any static Object - as even Shared Objects can be linked statically(!). But to statically link the Shared Object (as an archive member), neither the LOADONLY flag must be set, nor an Import File must be found (as Import Files cannot serve for static linking). And "--with-aix-soname=aix", libtool still creates standalone "libNAME.so.X.Y.Z" along the (versioning-wise useless) symlinks. So it is up to the package manager which "aix-soname" variant to choose within its prefix: For example, in Gentoo Prefix I'm using "--with-aix-soname=svr4" only. But still, how to get ctypes.find_library() working - ideally for each variant, is another story. Right now it does not work for any variant, but I guess that search algorithm should follow how the linker discovers the (PATH,BASE,MEMBER) values to write into just-linked executables, combined with how the runtime loader finds the Shared Object to actually load. |
On 31/01/2017 20:22, Michael Haubenwallner wrote:
This is why (on AIX) when a new archive is installed - even though there
My algorithm - attempts to solve the (PATH, BASE, MEMBER) problem as PATH: start by looking at the python executable - and looking at it's The program "dump -H" provides this information for both executables and Starting from the "blibpath" values in the executable mean a cpython So - once the PATHS are determined the system is examined looking for Again, if PATH, BASE, MEMBER is not located as a .a archive - look for I hope above is clear enough to continue the discussion. Thanks again Michael
|
On 02/03/2017 09:52 PM, Michael Felt wrote:
Except for Windows, I'm not sure which "many systems besides AIX" you're talking here about, that "do not use/support DT_SONAME".
When a system happens to find these symlinks useful, then it actually _does_ support embedding DT_SONAME (or something similar) into its binary file format.
Yes, the AIX Archive Libraries supporting different bitwidths for members is quite similar to MacOS fat objects.
Exactly. There is an idea below (the symbol->member map).
Yes - find_library currently does not work for any variant on *AIX*.
Exactly.
As far as I can tell, any executable can actually link against the Python interpreter.
There also is the loadquery() subroutine in AIX, see https://www.ibm.com/support/knowledgecenter/ssw_aix_72/com.ibm.aix.basetrf1/loadquery.htm loadquery(L_GETLIBPATH) "Returns the library path that was used at process exec time.",
Question is if we do want to consider _current_ values of environment variable LIBPATH (or LD_LIBRARY_PATH) in addition to the "library path at process exec time"?
Erm, nope, the AIX linker has a different algorithm (for -lNAME):
The linker does not perform such a check, nor does it feel necessary for ctypes.find_library+dlopen as long as it does search similar to the linker.
Eventually we might want to avoid spawning the 'dump' program, but implement reading the XCOFF Object File Format within _ctypes module instead.
There is no need for the ctypes module to search libpaths from other Shared Objects than the main executable (and current env vars).
When ctypes.dlopen is asked to load an Archive Library (either .a or .so) without a specific member, it probably should not immediately dlopen a specific member, but fetch the list of symbols provided by useable members (either Shared Objects without the F_LOADONLY flag, as well as specified in Import Files), and return the handle to some internal symbol->member map instead. Then, really loading a shared archive member is done by subsequent ctypes.dlsym - where it becomes clear which archive member to load.
Nope, see above - please iterate over a libpath list only _once_, and search for each filename while at one path list entry. However, I'm not sure yet how to identify if we should search for .a or .so first (before .so and .a, respectively): Uhm, ultimative solution feels complex already, while still some things to decide... |
Let's switch to github-based patches to discuss about: For the library search path I prefer to use loadquery(L_GETLIBPATH), available via new _ctypes_aix module now, but also used with L_GETINFO in Python/dynload_aix.c already. |
First, my apology that I have not responded earlier. I had other things Clearly, the detail of your comments proves me wrong on that final Jumping to your last comment:
So, rather than try for perfection in one go - set some priorities. IMHO (emphasis on the H) - having find_library return something useful, Your "find" of the routine "loadquery()" may be a big improvement over More comments (a few) below. On 20/02/2017 14:37, Michael Haubenwallner wrote:
> Michael Haubenwallner added the comment:
>
> On 02/03/2017 09:52 PM, Michael Felt wrote:
>>> Anyway:
>>> Unfortunately, there is no concept of embedding something like ELF's DT_SONAME tag into the Shared Object.
>>> The very (PATH,BASE,MEMBER) value as (specified to and) discovered by the linker is recorded into the just-linked executable (or Shared Object).
>>> This implies that the runtime loader does search for the very same filename (and member eventually) as the linker (at linktime).
>> I assume this is why there are many systems besides AIX that do not
>> support/use DT_SONAME.
> Except for Windows, I'm not sure which "many systems besides AIX" you're talking here about, that "do not use/support DT_SONAME".
Clearly, my assumption is wrong. I "grew up" on BSD Unix, not System V,
and AIX seems (imho) favor BSD in some aspects. I assume (emphasis
assume!) that soname is not part of the POSIX standard (at least not as
early as 2005 - standards change).
>> At least I see many references to "Shared
>> Objects" libFOO.so.X.Y.Z, libFOO.so.X.Y, libFOO.so.X and libFOO.so (with
>> the latter three being symbolic links to the first).
> When a system happens to find these symlinks useful, then it actually _does_ support embedding DT_SONAME (or something similar) into its binary file format.
What I have seen on AIX - packaging by others, and from memory what
libtool is doing, is the following: the file: libfoo.so.X.Y.Z gets
created, the following symbolic links are created - each pointing at
libfoo.so.X.Y.Z: libfoo.so.X.Y, libfoo.so.X and libfoo.so. I also see
the same "logic" in IBM provided archives. I use libssl.a as my guide
(for 'versioning'), but most of my 'issues' has been with supporting
backwards compatibility with libintl.a - execpt here they are not
symbolic links. The same "file" is added to the archive, but with a
different name - so if a "soname extension" was used (better, found
during linking), it is used. The order in the archive is important. If
the generic name is first, then that is the name that will be used
during linking (and remembered for execution). root@x064:[/usr/lib]lslpp -L | grep openssl.base root@x064:[/usr/lib]lslpp -L aixtools.gnu.gettext.rte root@x064:[/usr/lib]ar -Xany tv libintl.a Here, to keep ancient legacy programs - I have copied (and put at the And another example - with libz.a - where, historical moments can be root@x064:[/usr/lib]ar -Xany -tv libz.a So, getting back to your comment about it being "complex". Maybe I root@x064:[/usr/lib]ar -Xany -tv libc.a | grep shr
>>> But still, how to get ctypes.find_library() working - ideally for each variant, is another story. Right now it does not work for any variant,
>> Do you mean all systems, or specific to AIX - I am assuming you mean AIX.
> Yes - find_library currently does not work for any variant on *AIX*.
So, it is always up to the programmer to find a solution. And, my
observation is that they create an unmaintable/unmanageable situation by
extracting a member from an archive into /usr/lib and hard-coding the
library-name in a call to LoadLibrary(). Of course if the archive they
extracted from gets updated - their extracted member is not updated. etc. If they have done enough research (which I have not seen) they could use
So, just want to point out - if find_library() was working, there is In short, for AIX find_library() is broken. period. It could be fixed so >>> but I guess that search algorithm should follow how the linker discovers the (PATH,BASE,MEMBER) values to
>> I am not a tool builder. My comments are based on observations and
>> experience from when I was a developer 25+ years ago. The AIX linker is
>> not interested in the member name - it seems to go through the
>> PATH/libBASE.a looking for the first object it can find to resolve a
>> symbol. The name of the object it finds becomes the MEMBER it records in
>> it's internal table of where to look later when the application runs.
> Exactly.
See above re: my observations re: current practice/behavior I could go through all of PEP-20 - but one of my goals was to have AIX # leaving out the 'include statements' to show the 'core' objective
solib = None
solibname = find_library("foo")
if solibname:
solib = cdll.LoadLibrary(solibname)
# optionally, there might also be an else block dealing with the
'unexpected?' lack of solibname resolution. would work. There should be one-- and preferably only one --obvious way to do it. I would like to add - I may be American born, I carry a Dutch passport And, while *right* now may be too soon - but remember 'never' is - I >
>>> write into just-linked executables, combined with how the runtime loader finds the Shared Object to actually load.
>> I worked on a patch - to do all that - taking into consideration the way
>> libtool names .so files/members and then looking into/at "legacy" aka
>> IBM dev ways they did things before the libtool model was so prominent.
>>
>> My algorithm - attempts to solve the (PATH, BASE, MEMBER) problem as
>> "dynamically" as possible. PATH and BASE are fairly straight forward -
>> but MEMBER is clearly more complex.
>>
>> PATH: start by looking at the python executable -
> As far as I can tell, any executable can actually link against the Python interpreter.
I am talking about packaging python as "stand-alone"
>> and looking at it's "blibpath" -
> There also is the loadquery() subroutine in AIX, see https://www.ibm.com/support/knowledgecenter/ssw_aix_72/com.ibm.aix.basetrf1/loadquery.htm
>
> loadquery(L_GETLIBPATH) "Returns the library path that was used at process exec time.",
> which includes both the environment variable LIBPATH (or LD_LIBRARY_PATH if LIBPATH is unset) and the executable's "blibpath" value.
Not having used loadquery() I do not know if it also, by default,
returns the default library path that the executable uses for it's own
(initial) libraries. E.g, when packaging I try to use /opt/lib, rather
than /usr/lib as I do not want to conflict or overwrite existing
libraries. So, when I add -L/opt/lib the XCOFF headers of python, the
executable, include /opt/lib.
>> and using that as the default colon separated list of PATHs
> Question is if we do want to consider _current_ values of environment variable LIBPATH (or LD_LIBRARY_PATH) in addition to the "library path at process exec time"?
In Python2.X, and prior to Python 3.6 - for Linux it was ignored. I
cannot comment on other platforms. On AIX I know that CDLL("libfoo.so")
and CDLL("libfoo.a(member.o), RTLD_MEMBER) follow LIBPATH or, if LIBPATH
is not defined LD_LIBRARY_PATH if it is defined). In short,
LoadLibrary() may find something different compared to what find_library
returns.
Linux x066 3.2.0-4-powerpc64 #1 SMP Debian 3.2.78-1 ppc64 GNU/Linux root@x066:~# find / -name libc.so\* root@x066:~# python -m ctypes/util michael@x067:~$ uname -a And, as you look at the details - perhaps 'libm' shows more clearly how michael@x067:
Python2.7: |ctypes.util.||find_library|(/name/)
The exact functionality is system dependent. On Linux, |find_library()| tries to run external programs Python3.6.1 |ctypes.util.||find_library|(/name/)
The exact functionality is system dependent. On Linux, |find_library()| tries to run external programs Changed in version 3.6: On Linux, the value of the environment variable Please note: "The exact functionality is system dependent.", so I have But somewhere, perhaps in my 'noobish python attempts that needed
I agree that an executable should not need to care about the LIBPATH
Our difference is that I put .a archives - at the premium - whereas you Maybe your way is how rtld (aka InitandLoad() searches), and then that
executable: (never released though, am hung on whether to add my root@x064:[/data/prj/python/python-2.7.13.0]dump -H python python:
VERSION# #SYMtableENT #RELOCent LENidSTR #IMPfilID OFFidSTR LENstrTBL OFFstrTBL
INDEX PATH BASE MEMBER executable Uses rtld for 'other stuff'. FYI: shareable does work (i.e., root@x064:[/data/prj/python/python-2.7.13.0]ls -l lib* No shared members, only static .o files (112 of them)
Again, my apologies for the long delay since your extensive reply. I Michael
|
On 24/02/2017 09:44, Michael Haubenwallner wrote:
Back again - I could not find the _ctypes_aix you mentioned. github was Clearly, you understand that path - I (still) do not. Obviously - making I may be far enough along with git that I might be able to figure out This may also be better, i.e., more efficient, ultimately, than what was Now that I have found your work - thank you for all the time you put Michael
|
as this is not (likely) to be backported to Python2 (by python, fyi: I do include full ctypes-load_library() support in my "independent" packaging) and it is "resolved" for Python3-3.7 bpo-26439 this issue may, imho, be closed. |
Thanks Michael, finally getting it closed :) |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: