Title: Reduce number of open calls on startup
Type: Stage:
Components: Interpreter Core Versions:
Status: closed Resolution: accepted
Dependencies: Superseder:
Assigned To: georg.brandl Nosy List: georg.brandl, georg.brandl, loewis, nnorwitz, rhettinger
Priority: normal Keywords: patch

Created on 2004-03-23 00:10 by loewis, last changed 2006-05-27 09:23 by georg.brandl. This issue is now closed.

File name Uploaded Description Edit
import.diff georg.brandl, 2006-05-25 18:47 new patch
Messages (11)
msg45624 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2004-03-23 00:10
This patch uses sys.path_importer_cache to reduce the
number of open calls, in the following way:
- if the value in path_importer_cache is None, it stats
the path to find out whether the file exists
- it then puts True/False into path_importer_cache
- if the value in path_importer_cache is False, the
path entry is skipped on all imports
- if the value is True, the stat call is skipped, and
open calls for files in the directory are made.

On Linux, this reduces the number of open calls for an
empty script from 343 to 263. The startup-time (for 100
interpreter invocations) goes down by one percent (from
0.0819s to 0.08113s per invocation).
msg45625 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2004-03-23 07:30
Logged In: YES 

I am surprised that making 25% fewer open calls doesn't save
more than 1% in startup time.

One other thought, I wonder if the timing of these changes
is affected by the OS keeping recently loaded files in
buffers so that disk access time not included.
msg45626 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2004-03-23 15:43
Logged In: YES 

It's certainly the case that the system has cached all files needed for 
startup in memory, including the directory contents of all directories 

OTOH, I assume that is the scenario in which people worry about startup 
time: high-frequency invocations of python. For a single invocation, it 
shouldn't matter much whether it takes 0.04s or 0.08s.
msg45627 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2006-02-20 10:42
Logged In: YES 

Can this go into 2.5?
msg45628 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2006-02-20 21:51
Logged In: YES 

Not sure. Anybody speaking in favour? against?
msg45629 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2006-02-25 19:55
Logged In: YES 

I'm very much for it. I haven't got too much RAM, and
whenever I start a Python program (emerge being the most
prominent example) after having worked heavily with e.g.
graphics or VMware, I'm hit by the files Python's opening
not being in the file cache anymore.
msg45630 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2006-05-25 18:47
Logged In: YES 

I reviewed this patch, in in consequence discovered a
problem with the sys.path_hooks machinery, described in

This patch fixes the problem and corrects the original patch
to not set any sys.path_importer_cache entry to True or
False when no import hooks are enabled (the p_loader
argument to find_module is NULL then).
msg45631 - (view) Author: Neal Norwitz (nnorwitz) * (Python committer) Date: 2006-05-26 06:38
Logged In: YES 

Without looking at the patch impl, I'm +1 on the idea of
reducing stat/open calls.  On NFS this is a huge time sync.
msg45632 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2006-05-26 17:35
Logged In: YES 

Your revised patch looks fine to me, so please apply.
msg45633 - (view) Author: Neal Norwitz (nnorwitz) * (Python committer) Date: 2006-05-27 03:48
Logged In: YES 

Georg, didn't you check this in or was that a diff patch?
msg45634 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2006-05-27 09:23
Logged In: YES 

Yes, I indeed checked this in, in rev. 46372. Thanks for
reminding me, Neal.
Date User Action Args
2004-03-23 00:10:00loewiscreate