This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Avoid more stat() calls in importlib
Type: performance Stage:
Components: Library (Lib) Versions: Python 3.3
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, Jim.Jewett, eric.snow, nanjekyejoannah, ncoghlan, pitrou
Priority: low Keywords: patch

Created on 2012-02-20 21:18 by pitrou, last changed 2022-04-11 14:57 by admin.

Files
File name Uploaded Description Edit
imptime.patch pitrou, 2012-02-20 21:40 review
Messages (10)
msg153809 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-02-20 21:18
This is an experimental patch that limits the frequency of stat() calls in _FileFinder.find_module(). It speeds up finding modules by 2x here, but unfortunately breaks some tests (which expect modules to appear immediately when created).
msg153814 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-02-20 23:04
The patch uses the time module which is not a built-in, so it breaks bootstrapping by directly importing a module, and an extension at that. At best you could switch to a modified _FileFinder after importlib is initially running and able to import extension modules.
msg153815 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-02-20 23:06
> The patch uses the time module which is not a built-in, so it breaks
> bootstrapping by directly importing a module, and an extension at
> that. At best you could switch to a modified _FileFinder after
> importlib is initially running and able to import extension modules.

Yup, as I said it's more of a proof-of-concept.
(I was actually surprised that time is not a built-in module)
msg153816 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-02-20 23:20
On Mon, Feb 20, 2012 at 18:06, Antoine Pitrou <report@bugs.python.org>wrote:

>
> Antoine Pitrou <pitrou@free.fr> added the comment:
>
> > The patch uses the time module which is not a built-in, so it breaks
> > bootstrapping by directly importing a module, and an extension at
> > that. At best you could switch to a modified _FileFinder after
> > importlib is initially running and able to import extension modules.
>
> Yup, as I said it's more of a proof-of-concept.
> (I was actually surprised that time is not a built-in module)

Yeah, I had to double-check myself.
msg153840 - (view) Author: Alyssa Coghlan (ncoghlan) * (Python committer) Date: 2012-02-21 03:06
FWIW, I doubt you'd get many objections if you ended up wanting to make time a builtin module and inject it into the bootstrapping namespace.

While I don't think the delay in noticing filesystem changes is reasonable as the default behaviour, it might be interesting to see how tricky it would be to create a custom importer (for either meta_path or path_hooks) that did this.
msg153891 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-02-21 18:33
> While I don't think the delay in noticing filesystem changes is
> reasonable as the default behaviour, it might be interesting to see how 
> tricky it would be to create a custom importer (for either meta_path or 
> path_hooks) that did this.

Mmmh. Then I think it would be simpler to expose a global option to disable freshness checks. That said, it has to be activated as early as possible to be effective, which doesn't make it very practical as an opt-in setting.
msg154424 - (view) Author: Jim Jewett (Jim.Jewett) * (Python triager) Date: 2012-02-27 00:46
As long as the interpreter knows about about files that *it* wrote, no repeat checks during startup seems utterly reasonable; sneaking in a new or changed file is inherently a race condition.

I think it would also be reasonable for general use, so long as there was also a way to say "for this particular directory, always check".
msg154486 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-02-27 16:18
Well, we already have a delay in detecting new files in importlib as the directory mtime check already adds a delay of the granularity of the mtime value for a directory. So if you manage to mutate a directory, import, write a file, and then import again all within the mtime of a directory, you will miss the new file (which has already triggered sporadic failures in various tests because they were not blowing out importlib's cache). IOW there is already a freshness problem that if we don't have importlib will never get bootstrapped because people will complain about performance too much.
msg404878 - (view) Author: Joannah Nanjekye (nanjekyejoannah) * (Python committer) Date: 2021-10-23 15:12
I wonder if the circumstances changed since this is an old issue? the method looks to have a deprecation warning and changed a bit too, I stand to be corrected.
msg404879 - (view) Author: Joannah Nanjekye (nanjekyejoannah) * (Python committer) Date: 2021-10-23 15:13
Related : https://bugs.python.org/issue14067
History
Date User Action Args
2022-04-11 14:57:26adminsetgithub: 58275
2021-10-23 15:13:02nanjekyejoannahsetmessages: + msg404879
2021-10-23 15:12:05nanjekyejoannahsetnosy: + nanjekyejoannah
messages: + msg404878
2020-01-29 00:56:39brett.cannonsetnosy: - brett.cannon
2013-10-19 21:02:45Arfreversetnosy: + Arfrever
2012-02-27 16:18:24brett.cannonsetmessages: + msg154486
2012-02-27 00:46:13Jim.Jewettsetnosy: + Jim.Jewett
messages: + msg154424
2012-02-21 18:33:46pitrousetmessages: + msg153891
2012-02-21 03:06:14ncoghlansetmessages: + msg153840
2012-02-20 23:20:14brett.cannonsetmessages: + msg153816
2012-02-20 23:06:06pitrousetmessages: + msg153815
2012-02-20 23:04:39brett.cannonsetmessages: + msg153814
2012-02-20 21:44:10eric.snowsetnosy: + eric.snow
2012-02-20 21:40:09pitrousetfiles: - imptime.patch
2012-02-20 21:40:01pitrousetfiles: + imptime.patch
2012-02-20 21:18:06pitroucreate