classification
Title: python wastes linux users time by checking for dylib on each dynamic library load
Type: resource usage Stage:
Components: Versions: Python 3.4
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: anal.phabet, brett.cannon, eric.snow, fzvqedi, ncoghlan, pitrou, thomaslee, vstinner
Priority: normal Keywords:

Created on 2011-09-22 18:56 by fzvqedi, last changed 2013-01-16 18:43 by brett.cannon. This issue is now closed.

Messages (6)
msg144413 - (view) Author: Roger (fzvqedi) Date: 2011-09-22 18:56
instead it should check for linux once, store that linux is used and not even once check for dylib

strace extract

[pid 23025] stat64("//lib/libreadline.dylib", 0xbfd27c1c) = -1 ENOENT (No such file or directory)
[pid 23025] stat64("//lib/libreadline.so", 0xbfd27c1c) = -1 ENOENT (No such file or directory)
[pid 23025] stat64("//lib/libreadline.a", 0xbfd27c1c) = -1 ENOENT (No such file or directory)
[pid 23025] stat64("/lib64/libreadline.dylib", 0xbfd27c1c) = -1 ENOENT (No such file or directory)
[pid 23025] stat64("/lib64/libreadline.so", 0xbfd27c1c) = -1 ENOENT (No such file or directory)
[pid 23025] stat64("/lib64/libreadline.a", 0xbfd27c1c) = -1 ENOENT (No such file or directory)
[pid 23025] stat64("/usr/lib64/libreadline.dylib", 0xbfd27c1c) = -1 ENOENT (No such file or directory)
[pid 23025] stat64("/usr/lib64/libreadline.so", 0xbfd27c1c) = -1 ENOENT (No such file or directory)
[pid 23025] stat64("/usr/lib64/libreadline.a", 0xbfd27c1c) = -1 ENOENT (No such file or directory)
[pid 23025] stat64("/lib/libreadline.dylib", 0xbfd27c1c) = -1 ENOENT (No such file or directory)
[pid 23025] stat64("/lib/libreadline.so", 0xbfd27c1c) = -1 ENOENT (No such file or directory)
[pid 23025] stat64("/lib/libreadline.a", 0xbfd27c1c) = -1 ENOENT (No such file or directory)
[pid 23025] stat64("/usr/lib/libreadline.dylib", 0xbfd27c1c) = -1 ENOENT (No such file or directory)
[pid 23025] stat64("/usr/lib/libreadline.so", 0xbfd27c1c) = -1 ENOENT (No such file or directory)
[pid 23025] stat64("/usr/lib/libreadline.a", 0xbfd27c1c) = -1 ENOENT (No such file or directory)
[pid 23025] stat64("//lib/libncursesw.dylib", 0xbfd27c1c) = -1 ENOENT (No such file or directory)
[pid 23025] stat64("//lib/libncursesw.so", 0xbfd27c1c) = -1 ENOENT (No such file or directory)
[pid 23025] stat64("//lib/libncursesw.a", 0xbfd27c1c) = -1 ENOENT (No such file or directory)
[pid 23025] stat64("/lib64/libncursesw.dylib", 0xbfd27c1c) = -1 ENOENT (No such file or directory)
[pid 23025] stat64("/lib64/libncursesw.so", 0xbfd27c1c) = -1 ENOENT (No such file or directory)
[pid 23025] stat64("/lib64/libncursesw.a", 0xbfd27c1c) = -1 ENOENT (No such file or directory)
[pid 23025] stat64("/usr/lib64/libncursesw.dylib", 0xbfd27c1c) = -1 ENOENT (No such file or directory)
[pid 23025] stat64("/usr/lib64/libncursesw.so", 0xbfd27c1c) = -1 ENOENT (No such file or directory)
[pid 23025] stat64("/usr/lib64/libncursesw.a", 0xbfd27c1c) = -1 ENOENT (No such file or directory)
[pid 23025] stat64("/lib/libncursesw.dylib", 0xbfd27c1c) = -1 ENOENT (No such file or directory)
[pid 23025] stat64("/lib/libncursesw.so", 0xbfd27c1c) = -1 ENOENT (No such file or directory)
[pid 23025] stat64("/lib/libncursesw.a", 0xbfd27c1c) = -1 ENOENT (No such file or directory)
[pid 23025] stat64("/usr/lib/libncursesw.dylib", 0xbfd27c1c) = -1 ENOENT (No such file or directory)
[pid 23025] stat64("/usr/lib/libncursesw.so", 0xbfd27c1c) = -1 ENOENT (No such file or directory)
[pid 23025] stat64("/usr/lib/libncursesw.a", 0xbfd27c1c) = -1 ENOENT (No such file or directory)
[pid 23025] stat64("//lib/libncurses.dylib", 0xbfd27c1c) = -1 ENOENT (No such file or directory)
[pid 23025] stat64("//lib/libncurses.so", 0xbfd27c1c) = -1 ENOENT (No such file or directory)
[pid 23025] stat64("//lib/libncurses.a", {st_mode=S_IFREG|0644, st_size=431218, ...}) = 0
[pid 23025] stat64("//lib/libcrypt.dylib", 0xbfd27c1c) = -1 ENOENT (No such file or directory)
[pid 23025] stat64("//lib/libcrypt.so", 0xbfd27c1c) = -1 ENOENT (No such file or directory)
[pid 23025] stat64("//lib/libcrypt.a", {st_mode=S_IFREG|0644, st_size=8, ...}) = 0
[pid 23025] stat64("//include/openssl/ssl.h", {st_mode=S_IFREG|0644, st_size=93660, ...}) = 0
[pid 23025] stat64("//include/krb5.h", 0xbfd27cac) = -1 ENOENT (No such file or directory)
[pid 23025] stat64("./krb5.h", 0xbfd27cac) = -1 ENOENT (No such file or directory)
[pid 23025] stat64("Include/krb5.h", 0xbfd27cac) = -1 ENOENT (No such file or directory)
[pid 23025] stat64("./Include/krb5.h", 0xbfd27cac) = -1 ENOENT (No such file or directory)
[pid 23025] stat64("/src/Python-2.7.2/Include/krb5.h", 0xbfd27cac) = -1 ENOENT (No such file or directory)
[pid 23025] stat64("/src/Python-2.7.2/krb5.h", 0xbfd27cac) = -1 ENOENT (No such file or directory)
[pid 23025] stat64("/usr/include/krb5.h", 0xbfd27cac) = -1 ENOENT (No such file or directory)
[pid 23025] stat64("/usr/kerberos/include/krb5.h", 0xbfd27cac) = -1 ENOENT (No such file or directory)
[pid 23025] stat64("//lib/libssl.dylib", 0xbfd27b3c) = -1 ENOENT (No such file or directory)
[pid 23025] stat64("//lib/libssl.so", 0xbfd27b3c) = -1 ENOENT (No such file or directory)
[pid 23025] stat64("//lib/libssl.a", {st_mode=S_IFREG|0644, st_size=448496, ...}) = 0
msg144415 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-09-22 19:36
The list of file extensions is written in _PyImport_DynLoadFiletab constant. This list depends on the OS. For Linux, the list is defined in: Python/dynload_shlib.c.

I don't understand because in Python 2.7, Python/dynload_shlib.c doesn't contain ".dylib".
msg171406 - (view) Author: Thomas Lee (thomaslee) (Python committer) Date: 2012-09-28 06:19
I know this is an old-ish issue, but I can't reproduce anything like this on Debian Wheezy with either Python 2.7 or tip (3.x). I think we need more details of what you're trying to do here Roger.

1. What exactly did you do to reproduce the strace output below?
2. What flavor of Linux did you attempt this on?

Further, a more detailed description of what you expected to happen vs. what actually happened would be a big help here.

As Victor says, .dylib shouldn't appear in strace output on Linux at all (and my testing seems to confirm this).
msg171572 - (view) Author: Ron Hubbard (anal.phabet) Date: 2012-09-29 13:37
i'll try to reproduce this later.

looking at the below strace output, python's behaviour seems pretty stupid tho,

for example if /lib is the only existing path out of /lib, /lib64, /usr/lib/, /usr/local/lib, etc etc, it should check for all these directories once, and then not try to open .so in unexisting directories over and over again, inflicting multiple syscalls for each dso.
msg171656 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-09-30 18:30
Which was the command run under strace exactly?
msg180097 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-01-16 18:43
Since this has been pending for over two months I'm claiming it's outdated.
History
Date User Action Args
2013-01-16 18:43:18brett.cannonsetstatus: pending -> closed
resolution: out of date
messages: + msg180097
2012-11-13 07:05:49eric.snowsetstatus: open -> pending
2012-11-13 06:35:01eric.snowsetstatus: pending -> open
nosy: + eric.snow
2012-09-30 20:15:14brett.cannonsetstatus: open -> pending
2012-09-30 18:30:11pitrousetnosy: + pitrou, brett.cannon, ncoghlan

messages: + msg171656
versions: + Python 3.4, - Python 2.7
2012-09-29 13:37:37anal.phabetsetnosy: + anal.phabet
messages: + msg171572
2012-09-28 06:19:36thomasleesetnosy: + thomaslee
messages: + msg171406
2012-09-27 19:38:47anal.phabetsettype: resource usage
2011-09-22 19:36:53vstinnersetnosy: + vstinner
messages: + msg144415
2011-09-22 18:56:46fzvqedicreate