classification
Title: ctypes.util.find_library() should return full pathname instead of filename in linux
Type: enhancement Stage: needs patch
Components: ctypes Versions: Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Hernan.Grecco, beng94, berker.peksag, martin.panter, python-dev
Priority: normal Keywords: patch

Created on 2014-03-23 22:47 by Hernan.Grecco, last changed 2016-03-16 20:32 by martin.panter.

Files
File name Uploaded Description Edit
find_lib.patch beng94, 2016-02-03 10:39 review
find_lib_v1.patch beng94, 2016-02-06 12:06 review
find_lib_v2.patch beng94, 2016-02-09 10:43 review
find_lib_v3.patch beng94, 2016-02-21 20:12 review
find_lib_v4.patch beng94, 2016-02-22 12:25 review
Messages (20)
msg214647 - (view) Author: Hernan Grecco (Hernan.Grecco) Date: 2014-03-23 22:47
In Windows and OSX, `find_library` returns the full pathname of the library file. But on Linux, it returns just the filename. Is there a reason for this difference? 

For consistency, it would be better to return the full pathname in all cases. It is easy to get the filename from the full pathname, but not the other way around.
msg259470 - (view) Author: Tamás Bence Gedai (beng94) * Date: 2016-02-03 10:39
Added a small patch that solves this issue on Ubuntu 15.10.

Produces output like:
/lib/x86_64-linux-gnu/libm.so.6
/lib/x86_64-linux-gnu/libc.so.6
/lib/x86_64-linux-gnu/libbz2.so.1.0

I'd be glad to add some test cases if someone can give me some tips on how to do that.
msg259528 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-02-04 03:40
IMO this should be treated as a new feature for the next release. But consistently returning the path sounds good to me if there is no good reason not to.

Left a question on the review. I think you also need to update the documentation, and since this is changing documented behaviour it probably needs a What’s New entry.

For tests, I would try doing something like find_library("c"), and ensuring that the result is an absolute path. We may have to end up skipping the test for platforms like Windows where this is not expected to work. Look through the files in /Lib/ctypes/test/ for a good place for it to live (if there isn’t already a test there you can modify).
msg259725 - (view) Author: Tamás Bence Gedai (beng94) * Date: 2016-02-06 12:06
Added a new patch, as Martin pointed out, I put back the ABI matching. The regex looks quite ugly, because it has to match \n\t. To be exact, it has to match something like this: "/lib/x86_64-linux-gnu/libc.so.6\n\tlibbz2.so.1.0 (libc6,x86-64)".

I updated the docs, although I don't know what should I write for the version, please someone help me with that.

For testing, there's a test function in this module, I updated that.
msg259776 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-02-07 09:42
The ABI matching looks wrong to me. If I am looking for a 32-bit library, won’t it incorrectly catch the wrong path in the following “ldconfig -p” output:

'\tlibm.so.6 (libc6,x86-64, OS ABI: Linux 2.6.32) => /usr/lib/libm.so.6\n'
'\tlibm.so.6 (libc6, OS ABI: Linux 2.6.32) => /usr/lib32/libm.so.6\n'

Perhaps the abi_type check needs to be moved in front of the path name extraction.

For the version, I would put 3.6. Since this changes documented behaviour and has the potential to break compatibilty, it is best not to change it in a bug fix release. (3.5 has already been released.)

The problem with the test() function in ctypes.util is that it is not run by the main Python regression test suite. The tests under ctypes/test/ are run by the test suite.
msg259924 - (view) Author: Tamás Bence Gedai (beng94) * Date: 2016-02-09 10:43
I fixed the ABI matching, it was a stupid mistake, thanks for pointing it out :) I think now it works as expected.

I really don't find a place for testing. Maybe a new test file could be added, but I think the testing code for find_library wouldn't be more than 10 lines. Do you have any suggestions?

Thanks Martin for all your patience :)
msg260258 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-02-14 02:55
I think the new regular expression will still find the wrong library in my libm example above. In 32-bit mode, it will be only looking to match \(libc6.*\). Since my example has the 64-bit line first, that one will match first. (I haven’t actually tested this, but I think I compiled 32-bit Python once before just by specifying CC="gcc -m32". Sorry to keep poking holes in your regular expression :)

Do you know if there is documentation for the “ldconfig -p” output format, or do we just have to go on what we see? If so, I would change it to ensure the ABI type string is either followed by a comma and space ", " or a closing bracket ")". A comma on its own, or other letters, is not a match.

I did a search for “find_library”, and the most likely place is /Lib/ctypes/test/test_find.py. You could probably get away with just adding a new method like Test_OpenGL_libs.test_path(). On my computer I have the GL and GLU libraries (but not gle), so I guess that these libraries are fairly common (plus it already has Windows and OS X versions to test).
msg260579 - (view) Author: Tamás Bence Gedai (beng94) * Date: 2016-02-20 22:47
What do you think about this regex?

'(lib%s\.[^\s]+\s\(%s(?:\)|,\s.*\))\s=>\s.*)' % (re.escape(name), abi_type))

It works on 64 bit, just like before, but I could not test it on 32 bit. I'll add tests soon.

I looked for documentation on ldconfig, but could not find anything useful.
msg260595 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-02-21 07:05
Tamás, it might be a good idea for you to sign a contributor agreement <https://www.python.org/psf/contrib/contrib-form/>.

I compiled Python in 32-bit mode and tried your v2 patch out, which found the wrong library as I predicted. Then I tried your new regex and it picked out the correct line. I had to edit it to get it to extract just the filename from the line:

r'lib%s\.[^\s]+\s\(%s(?:,\s.*)?\)\s=>\s(.*)' % (re.escape(name), abi_type)

I factored out the closing bracket from the comma + space bit, moved the group brackets to the end to extract the filename, and made it a raw string.

Without the patch, in 32-bit mode it will find 64-bit-only libraries:

>>> find_library("m")  # 32- and 64-bit available
'libm.so.6'
>>> find_library("tcl8.6")  # Only 64-bit version available!
'libtcl8.6.so'

With my edited regex:

>>> find_library("m")
'/usr/lib32/libm.so.6'
>>> find_library("tcl8.6") is None  # No 32-bit version found
True
msg260636 - (view) Author: Tamás Bence Gedai (beng94) * Date: 2016-02-21 20:12
I've added a new method to Test_OpenGL_libs as you suggested. I check whether find_library returns an absolute path. Note that I didn't distinguish different systems, as according to the docs, only Linux systems return the file name, other systems return the absolute path. (https://docs.python.org/3.5/library/ctypes.html#ctypes-reference) An other thing to note, that I introduced some code duplication as I use the same code snippet from setUpClass method to figure out the correct parameters to find_library.

The patch uses the same regex as you gave.

By the way, what do I have to do to compile CPython on a 64 bit system in 32 bit mode? I tried ./configure CC="gcc -m32" but it gave me an error. Is it the correct way?

Also, I signed contributor agreement.
msg260650 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-02-22 00:18
I left a suggestion about the duplication in the code review.

I set CC with “configure” like you said. I also had to run “make clean” to get rid of the old 64-bit stuff. But it might depend on the GCC that you have installed. On Arch Linux, I have gcc-multilib, which supports 64-bit and 32-bit targets.
msg260672 - (view) Author: Tamás Bence Gedai (beng94) * Date: 2016-02-22 12:25
Updated the patch to remove the code duplication, now it stores the values that are calculated in the setUpClass method. It was a good and simple idea, I should have come up with it... :)

I'm pretty sure I got the errors during configuration because of the gcc version I have.
msg260948 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-02-27 22:43
Thanks, this looks pretty good to me. I just need to remember to write a What’s New entry.
msg260951 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2016-02-28 00:21
I don't think we need an entry in whatsnew/3.6.rst for this (we already have documented the change in a versionchanged directive and a Misc/NEWS item)
msg261107 - (view) Author: Tamás Bence Gedai (beng94) * Date: 2016-03-02 11:47
Is there anything else that I can do for this issue?
msg261228 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-03-06 02:11
No I think this is ready Tamás. I have been away, but it is on my list of things to catch up on. I won’t add any What’s New entry.
msg261393 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-03-09 01:10
New changeset 3092cf163eb4 by Martin Panter in branch 'default':
Issue #21042: Return full path in ctypes.util.find_library() on Linux
https://hg.python.org/cpython/rev/3092cf163eb4
msg261400 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-03-09 03:20
It looks like the ldconfig parsing isn’t working for some ABIs. See the following buildbot failures:

* Cortex A15 armv7l: http://buildbot.python.org/all/builders/ARMv7%20Ubuntu%203.x/builds/3721/steps/test/logs/stdio
* ppc64le POWER8: http://buildbot.python.org/all/builders/PPC64LE%20Fedora%203.x/builds/769/steps/test/logs/stdio

I presume there are other flags in the ABI string, perhaps like (libc6,hard-float) on ARM. The code that produces these strings seems to be here: <https://sourceware.org/git/gitweb.cgi?p=glibc.git;a=blob;f=elf/cache.c;h=fbee172#l72>.

Looking closer at the find_library() implementation, I also realize it is not correct to say an absolute path is always returned. If the ldconfig check fails, it falls back to _get_soname(). I think we have the following options:

* Adjust the documentation to say an absolute path is only returned if the ldconfig call works
* Figure out how to get the right ldconfig flags for ARM and PPC
* Use the old parsing code on ARM and PPC platforms, and only return a full path on x86 or other platforms
* Revert the whole change
msg261844 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-03-16 08:41
New changeset 811ec2860dc4 by Martin Panter in branch 'default':
Issue #21042: Revert Linux find_library() to return just filename
https://hg.python.org/cpython/rev/811ec2860dc4
msg261858 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-03-16 20:32
I reverted the change until we can come up with something more consistent.
History
Date User Action Args
2016-03-16 20:32:35martin.pantersetmessages: + msg261858
stage: commit review -> needs patch
2016-03-16 08:41:45python-devsetmessages: + msg261844
2016-03-09 03:20:59martin.pantersetmessages: + msg261400
2016-03-09 01:10:26python-devsetnosy: + python-dev
messages: + msg261393
2016-03-06 02:11:12martin.pantersetmessages: + msg261228
stage: patch review -> commit review
2016-03-02 11:47:15beng94setmessages: + msg261107
2016-02-28 00:21:57berker.peksagsetnosy: + berker.peksag
messages: + msg260951
2016-02-27 22:43:41martin.pantersetmessages: + msg260948
2016-02-22 12:25:22beng94setfiles: + find_lib_v4.patch

messages: + msg260672
2016-02-22 00:18:41martin.pantersetmessages: + msg260650
2016-02-21 20:12:58beng94setfiles: + find_lib_v3.patch

messages: + msg260636
2016-02-21 07:05:44martin.pantersetmessages: + msg260595
2016-02-20 22:47:49beng94setmessages: + msg260579
2016-02-14 02:55:06martin.pantersetmessages: + msg260258
2016-02-09 10:43:46beng94setfiles: + find_lib_v2.patch

messages: + msg259924
2016-02-07 09:42:55martin.pantersetmessages: + msg259776
2016-02-06 12:06:09beng94setfiles: + find_lib_v1.patch

messages: + msg259725
2016-02-04 03:40:04martin.pantersetversions: + Python 3.6, - Python 3.5
nosy: + martin.panter

messages: + msg259528

type: enhancement
stage: patch review
2016-02-03 10:39:32beng94setfiles: + find_lib.patch

nosy: + beng94
messages: + msg259470

keywords: + patch
2014-03-23 22:47:15Hernan.Greccocreate