classification
Title: ctypes.util.find_library("libc") fails
Type: Stage: resolved
Components: ctypes Versions: Python 3.9
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: Alex Shpilkin, RedEyed, doko, eryksun, vstinner
Priority: normal Keywords: 3.9regression

Created on 2020-12-06 10:50 by doko, last changed 2021-02-16 15:29 by eryksun. This issue is now closed.

Messages (14)
msg382592 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2020-12-06 10:50
regression compared to 3.8:

$ python3.9
Python 3.9.1rc1 (default, Nov 27 2020, 19:38:39) 
[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> ctypes.util.find_library("libc")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'ctypes' is not defined
>>> import ctypes.util
>>> ctypes.util.find_library("libc")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.9/ctypes/util.py", line 341, in find_library
    _get_soname(_findLib_gcc(name)) or _get_soname(_findLib_ld(name))
  File "/usr/lib/python3.9/ctypes/util.py", line 147, in _findLib_gcc
    if not _is_elf(file):
  File "/usr/lib/python3.9/ctypes/util.py", line 99, in _is_elf
    with open(filename, 'br') as thefile:
FileNotFoundError: [Errno 2] No such file or directory: b'liblibc.a'

also note that the shared library is installed.
msg382593 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2020-12-06 10:57
ctypes.util.find_library("c") works in both 3.8 and 3.9
msg387102 - (view) Author: Vadym Stupakov (RedEyed) Date: 2021-02-16 11:17
can't reproduce

(py39) ➜  ~ ipython                                                 
Python 3.9.1 (default, Dec 11 2020, 14:32:07) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.20.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import ctypes.util
In [2]: a = ctypes.util.find_library("libc")
In [3]: print(a)
None
msg387103 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-02-16 11:29
This function is quite complicated on Linux:

        def find_library(name):
            # See issue #9998
            return _findSoname_ldconfig(name) or \
                   _get_soname(_findLib_gcc(name)) or _get_soname(_findLib_ld(name))

_findSoname_ldconfig() uses "/sbin/ldconfig -p" and searchs for 'libc6,x86-64' in the output (on x86-64).

_findLib_gcc() uses "gcc -Wl,-t -o tmp -lc" command and search for "lib" in the created "tmp" file.

_findLib_ld() uses "ld -t -lc" command and searchs for "lib" pattern.

The exact code is more complicated :-) You should debug this issue by running these commands manually on Debian.

Note: python3.9 -c 'import ctypes.util; print(ctypes.util.find_library("c"))' commands displays libc.so.6 with Python 3.8, 3.9 and 3.10 on Fedora. I cannot reproduce the issue on Fedora 33 (x86-64).
msg387104 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-02-16 11:30
> _findLib_gcc() uses "gcc -Wl,-t -o tmp -lc" command and search for "lib" in the created "tmp" file.

Oh, it looks for the "lib" in the process stdout (not in the created "tmp" file).
msg387106 - (view) Author: Vadym Stupakov (RedEyed) Date: 2021-02-16 12:30
Note, that adding "lib" prefix to any library is wrong, that's why it returns "None" or raises the exception.

And it works fine when you call find_library("c")

I guess the issue, that was reported here, is about raising and exception instead of returning "None", but not about finding library, isn't it?
msg387107 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-02-16 12:43
Oh right, I always use find_library("c"), not find_library("libc").
msg387108 - (view) Author: Vadym Stupakov (RedEyed) Date: 2021-02-16 12:55
So, can we close it?
msg387109 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2021-02-16 13:27
ctypes.util.find_library("libc") used to work in 3.8, not working in 3.9. As I said before, ctypes.util.find_library("c") works in both 3.8 and 3.9.
msg387110 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-02-16 13:35
Issue 41976 added ctypes.util._is_elf() to filter out linker scripts such as "libc.so". The PR was backported to 3.7. _is_elf() assumes the trace result has absolute paths that can be opened, but Matthias is getting the relative filename "liblibc.a" in the result. Whatever the reason, I think if the file can't be opened for reading, then _is_elf() should just return False. For example:

    def _is_elf(filename):
        "Return True if the given file is an ELF file"
        elf_header = b'\x7fELF'
        try:
            with open(filename, 'br') as thefile:
                return thefile.read(4) == elf_header
        except OSError:
            return False
msg387111 - (view) Author: Vadym Stupakov (RedEyed) Date: 2021-02-16 13:50
> ctypes.util.find_library("libc") used to work in 3.8, not working in 3.9. As I said before, ctypes.util.find_library("c") works in both 3.8 and 3.9.

no, it doesn't work (and it shouldn't) neither in python 3.8 nor 3.7

    Python 3.8.5 (default, Sep  4 2020, 07:30:14) 
    Type 'copyright', 'credits' or 'license' for more information
    IPython 7.19.0 -- An enhanced Interactive Python. Type '?' for help.

    In [1]: import ctypes.util
    In [2]: a = ctypes.util.find_library("libc")
    In [3]: print(a)
    None

    Python 3.7.6 (default, Jan  8 2020, 19:59:22) 
    Type 'copyright', 'credits' or 'license' for more information
    IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.
    Python 3.7.6 (default, Jan  8 2020, 19:59:22) 

    import ctypes.util
    a = ctypes.util.find_library("libc")
    print(a)
    None

as I said, adding prefix "lib" is wrong
msg387112 - (view) Author: Vadym Stupakov (RedEyed) Date: 2021-02-16 14:04
I mean, find_library relies on gcc linker, when you pass library name to the linker, it automatically ads "lib" prefix to the library
so, when you pass "libc", linker ads "lib" so u have an error with "liblibc" name.

just man ld and see "-l" option:

"
-l namespec
       --library=namespec
           Add the archive or object file specified by namespec to the
           list of files to link.  This option may be used any number of
           times.  If namespec is of the form :filename, ld will search
           the library path for a file called filename, otherwise it
           will search the library path for a file called libnamespec.a.
"

as you can see, you pass not a library name, but a "namespec"
which then transforms to "libnamespec"
msg387116 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-02-16 15:18
The documentation is explicit: you must not include the "lib" prefix.
https://docs.python.org/dev/library/ctypes.html#ctypes.util.find_library

See also examples:
https://docs.python.org/dev/library/ctypes.html#finding-shared-libraries

If it worked with "lib" prefix in Python 3.8, it wasn't on purpose.

You should fix your code, but Python works as expected. I close the issue.
msg387117 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-02-16 15:29
Can't there be a library named "libc" with the shared module "liblibc.so"? It seems to me that the _is_elf() function added in issue 41976 is breaking the contract that "[i]f no library can be found, returns None". It's not supposed to leak arbitrary exceptions from the implementation if the library isn't found.
History
Date User Action Args
2021-02-16 15:29:24eryksunsetmessages: + msg387117
2021-02-16 15:18:01vstinnersetstatus: open -> closed
resolution: not a bug
messages: + msg387116

stage: resolved
2021-02-16 14:04:28RedEyedsetmessages: + msg387112
2021-02-16 13:50:25RedEyedsetmessages: + msg387111
2021-02-16 13:35:22eryksunsetnosy: + eryksun
messages: + msg387110
2021-02-16 13:27:13dokosetmessages: + msg387109
2021-02-16 12:55:15RedEyedsetmessages: + msg387108
2021-02-16 12:43:58vstinnersetmessages: + msg387107
2021-02-16 12:30:18RedEyedsetmessages: + msg387106
2021-02-16 11:30:49vstinnersetmessages: + msg387104
2021-02-16 11:29:30vstinnersetnosy: + vstinner
messages: + msg387103
2021-02-16 11:17:36RedEyedsetnosy: + RedEyed
messages: + msg387102
2021-02-13 12:34:58Alex Shpilkinsetnosy: + Alex Shpilkin
2020-12-06 10:57:14dokosetmessages: + msg382593
2020-12-06 10:50:33dokocreate