New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ctypes: Speed up find_library() on Linux by 500% #55467
Comments
(This applies to all versions of Python I investigated, although the attached patch is for Python 2.7) I wondered why It turns out that A quick look at the ldconfig source (namely, the print_cache routine which is invoked when you call I attached two patches that fix this problem. Choose one! ;-) The ctypes tests pass with my fixes, and here comes some benchmarking: $ cat benchmark_ctypes.py
from ctypes.util import find_library
for i in xrange(10):
for lib in ['mm', 'c', 'bz2', 'uuid']:
find_library(lib) # Current implementation # With my patch applied |
(might also be related to http://bugs.python.org/issue11063) |
Here is the first patch adapted for py3k. |
Actually, re.escape is probably still needed around name. |
Thanks for the new patch. Looking again, I wonder if there's a reason the original regexp was so complicated. ldconfig output here has lines such as:
Ideally we would factor out the parsing to a separate private function, and have tests for it. |
As far as I can tell, it doesn't matter. We're looking for the part after the => in any case - ignoring the ABI/architecture information - so the regex would chose the first of those entries. |
Ok, I think you're right. |
Reopening and reverted the commit in r88640. The patch changes behaviour by turning the previous unrooted filename ('libc.so.6') into a full path ('/lib64/libc.so.6'). This breaks builds where multiple versions of a library are available and only one is loadable, e.g. 32-bit builds on 64-bit machines: ====================================================================== Traceback (most recent call last):
File "/home2/buildbot2/slave/hg-3.x.loewis-parallel/build/Lib/ctypes/test/test_loading.py", line 26, in test_load
CDLL(libc_name)
File "/home2/buildbot2/slave/hg-3.x.loewis-parallel/build/Lib/ctypes/__init__.py", line 340, in __init__
self._handle = _dlopen(self._name, mode)
OSError: /lib64/libc.so.6: wrong ELF class: ELFCLASS64 |
Humm. Would be great to have the |
Here is an excerpt:
The "OS ABI" thing is not always there:
As you see, there are two of them with the same name but in a different path. If you return the absolute path, there is a 50% possibility that you are returning the wrong one ;) There seem to be two key differences between the original implementation and yours:
I guess it should be doable to retain the speed benefit while implementing a matching algorithm closer to the original one. |
Same here. :) The version I attached seems to work for me. It's some kind of compromise -- basically it's the original regex but with the unneccessary, performance-decreasing cruft stripped away. btw, "Jonas H." is perfectly fine - I don't care about being honored, I just want to |
*push* Any way to get this into the codebase? |
New changeset 19d9f0a177de by Antoine Pitrou in branch 'default': |
I committed a modified patch. Hopefully the buildbots won't break this time :) |
19d9f0a177de causes that test_ctypes hangs when test suite is run in Gentoo sandbox. Please reopen this issue. $ sandbox python3.3 -B -m test.regrtest --timeout=10 -v test_ctypes
== CPython 3.3a0 (default:020ebe0be33e+, Apr 24 2011, 17:52:58) [GCC 4.5.2]
== Linux-${version}
== /tmp/test_python_23902
Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=1, no_user_site=0, no_site=0, ignore_environment=0, verbose=0, bytes_warning=0, quiet=0)
[1/1] test_ctypes
Timeout (0:00:10)!
Thread 0x00007fc205f54700:
File "/usr/lib64/python3.3/subprocess.py", line 466 in _eintr_retry_call
File "/usr/lib64/python3.3/subprocess.py", line 1412 in _execute_child
File "/usr/lib64/python3.3/subprocess.py", line 766 in __init__
File "/usr/lib64/python3.3/ctypes/util.py", line 198 in _findSoname_ldconfig
File "/usr/lib64/python3.3/ctypes/util.py", line 206 in find_library
File "/usr/lib64/python3.3/ctypes/test/test_find.py", line 15 in <module>
File "/usr/lib64/python3.3/ctypes/test/__init__.py", line 64 in get_tests
File "/usr/lib64/python3.3/test/test_ctypes.py", line 11 in test_main
File "/usr/lib64/python3.3/test/regrtest.py", line 1094 in runtest_inner
File "/usr/lib64/python3.3/test/regrtest.py", line 887 in runtest
File "/usr/lib64/python3.3/test/regrtest.py", line 587 in _runtest
File "/usr/lib64/python3.3/test/regrtest.py", line 711 in main
File "/usr/lib64/python3.3/test/regrtest.py", line 1672 in <module>
File "/usr/lib64/python3.3/runpy.py", line 73 in _run_code
File "/usr/lib64/python3.3/runpy.py", line 160 in _run_module_as_main |
I'd prefer having a separate issue (which you already opened :-)). |
OK. We will use issue bpo-11915. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: