This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: ctypes' find_library breaks with ARM ABIs
Type: behavior Stage: patch review
Components: ctypes Versions: Python 3.10, Python 3.9, Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Yegor Yefremov, barry, belopolsky, eric.araujo, lool, meador.inge, stefanor, trevor.newton
Priority: normal Keywords: patch

Created on 2011-11-30 14:54 by lool, last changed 2022-04-11 14:57 by admin.

Files
File name Uploaded Description Edit
ctypes-arm.diff lool, 2011-11-30 14:55 Patch to support arm* in uname and allow for an optional hard-float ABI string in "ldconfig -p" output
Messages (4)
msg148656 - (view) Author: Loïc Minier (lool) Date: 2011-11-30 14:54
Hi,

This bug was originally reported at 
https://bugs.launchpad.net/bugs/898172

ctypes/utils.py provides a find_library function which amongst other things will scan the ldconfig -p output on linux to find libraries by name.  It applies some logic to filter out incompatible libraries, however the logic is mainly based on uname output which is incorrect.

We noticed because the new Debian/Ubuntu armhf ports have a slightly different ldconfig -p output than the armel ports; one gets ",hard-float" in the output, e.g.:
        ld-linux.so.3 (libc6,hard-float) => /lib/arm-linux-gnueabihf/ld-linux.so.3

there's provision in find_library to allow for certain strings when uname returns certain names:
             mach_map = {
                 'x86_64-64': 'libc6,x86-64',
                 'ppc64-64': 'libc6,64bit',
                 'sparc64-64': 'libc6,64bit',
                 's390x-64': 'libc6,64bit',
                 'ia64-64': 'libc6,IA-64',

but this is incorrect for multiple reasons:
a) this requires setting utsname properly before running a 32-bits python on a 64-bits kernel (e.g. "linux32 ./foo.py" instead of just "./foo.py"); this shouldn't be needed and breaks 32-bits userspace installations with a 64-bits kernel
b) uname output can be anything really, e.g. i486, i586, i686 etc. on 32-bits x86, or armv5l, armv6l, armv7l etc. on ARM
c) uname output doesn't indicate userspace ABI, a single kernel can support multiple ABIs; for instance ARM kernels can support EABI and OABI (old ABI) syscall ABIs at the same time, and even with the same syscall ABI like EABI the userspace calling conventions might allow for multiple ABIs to be present on the filesystem -- for instance soft-float and hard-float userspace calling conventions

I've attached a patch to ctypes/utils.py in the Launchpad bug which I'll also attach here.  It will work for either soft-float or hard-float, but not if "ldconfig -p" lists both types of libraries (as will be the case with biarch or multiarch systems).

It is extremely hard to reproduce correct glibc semantics in find_library, and a linux implementation would necessarily become extremely glibc and linux specific.  One possible way is to look at /proc/$pid/maps output to find information about the ABI of the currently running program, and then ask the runtime linker (ld.so) to check whether a given library is compatible or not (--verify).  Another way would be to run ldd on sys.executable to find the runtime linker or libc.  This is all extremely fragile and linux andglibc specific, and will likely fail in special cases.

Finally, one needs to wonder whether offering "find_library" as an API isn't calling for trouble; dlopen() requires one to state which SOVER should be used, e.g. dlopen("libmagic.so.1"), not dlopen("magic").  Allowing the first SOVER to be used means that the behavior is not determinstic and also means that people wont think of binary compatibility when implementing ctypes-based bindings.  I would personally prefer if this API was deprecated and if we recommended for upstreams to use ctypes.cdll.LoadLibrary("libmagic.so.1") constructs.

Cheers,
msg148657 - (view) Author: Loïc Minier (lool) Date: 2011-11-30 14:58
While I'm at it, find_library also tries creating temp files when running gcc and other issues mention trouble running gcc or propose running ld:
http://bugs.python.org/issue9998
http://bugs.python.org/issue5289

IMHO, calling binutils/gcc is troublesome, it's not necessarily installed on production systems and creating tempfiles when running a program just to locate a library is fragile at best.
msg250768 - (view) Author: Yegor Yefremov (Yegor Yefremov) Date: 2015-09-15 13:30
This issue is still up to date. Most impacted are distos like for example Buldroot (http://buildroot.org/). Usual production rootfs provide neither gcc nor /sbin/ldconfig nor objdump. So find_library() is doomed from the very beginning.

Such packages like pyusb, pyudev etc. rely on find_library() and are not usable in such environment. Pyusb developers had to create an extra backend to overcome this issue (https://github.com/walac/pyusb/pull/29).

So common solution would be very useful for such environments.
msg386671 - (view) Author: Trevor Newton (trevor.newton) Date: 2021-02-09 00:08
I am still encountering this issue when using pyudev on Python 3.8.5.
History
Date User Action Args
2022-04-11 14:57:24adminsetgithub: 57717
2021-02-09 10:44:13vstinnersetnosy: - vstinner
2021-02-09 00:08:52trevor.newtonsetnosy: + trevor.newton
messages: + msg386671
2020-11-08 18:17:42iritkatrielsetversions: + Python 3.8, Python 3.9, Python 3.10, - Python 2.7, Python 3.2, Python 3.3
2015-09-15 13:30:05Yegor Yefremovsetnosy: + Yegor Yefremov
messages: + msg250768
2011-12-30 20:55:57eric.araujosetnosy: + belopolsky
2011-12-30 20:55:46eric.araujosetnosy: + eric.araujo
2011-12-29 00:37:10pitrousetnosy: + vstinner
stage: needs patch -> patch review

versions: + Python 3.3, - Python 2.6
2011-12-26 15:46:40barrysetnosy: + barry
2011-12-26 14:55:25stefanorsetnosy: + stefanor
2011-11-30 14:58:27loolsetmessages: + msg148657
2011-11-30 14:57:51meador.ingesetnosy: + meador.inge

stage: needs patch
2011-11-30 14:55:38loolsetfiles: + ctypes-arm.diff
keywords: + patch
2011-11-30 14:54:33loolcreate