➜

This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Python calls newfstatat for "" in inspect
Type: performance Stage:
Components: Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Ivan Babrou
Priority: normal Keywords:

Created on 2020-02-16 05:41 by Ivan Babrou, last changed 2022-04-11 14:59 by admin.

Messages (2)
msg362044 - (view) Author: Ivan Babrou (Ivan Babrou) Date: 2020-02-16 05:41 
I noticed that a program (SaltStack) is a lot slower under Python 3. After some stracing I was able to find that inspect module is to blame.

In strace output I can see a ton of calls like this:

05:31:56.698829 newfstatat(AT_FDCWD, "<frozen importlib._bootstrap>", 0x7ffff6bc4cf0, 0) = -1 ENOENT (No such file or directory) <0.000033>
05:31:56.699743 newfstatat(AT_FDCWD, "<frozen importlib._bootstrap>", 0x7ffff6bc4b70, 0) = -1 ENOENT (No such file or directory) <0.000061>
05:31:56.701328 newfstatat(AT_FDCWD, "<frozen importlib._bootstrap_external>", 0x7ffff6bc4cf0, 0) = -1 ENOENT (No such file or directory) <0.000037>
05:31:56.702171 newfstatat(AT_FDCWD, "<frozen importlib._bootstrap_external>", 0x7ffff6bc4b70, 0) = -1 ENOENT (No such file or directory) <0.000031>
05:31:56.703614 newfstatat(AT_FDCWD, "<frozen importlib._bootstrap>", 0x7ffff6bc4cf0, 0) = -1 ENOENT (No such file or directory) <0.000031>
05:31:56.704421 newfstatat(AT_FDCWD, "<frozen importlib._bootstrap>", 0x7ffff6bc4b70, 0) = -1 ENOENT (No such file or directory) <0.000028>
05:31:56.705751 newfstatat(AT_FDCWD, "<frozen importlib._bootstrap>", 0x7ffff6bc4cf0, 0) = -1 ENOENT (No such file or directory) <0.000039>
05:31:56.706691 newfstatat(AT_FDCWD, "<frozen importlib._bootstrap>", 0x7ffff6bc4b70, 0) = -1 ENOENT (No such file or directory) <0.000028>
05:31:56.708148 newfstatat(AT_FDCWD, "<frozen importlib._bootstrap>", 0x7ffff6bc4cf0, 0) = -1 ENOENT (No such file or directory) <0.000032>

This is the entrypoint from Salt:

* https://github.com/saltstack/salt/blob/9adc2214c3bb/salt/utils/decorators/__init__.py#L102

Execution with stock code:

$ time sudo salt-call --local test.ping
local:
    True

real	0m23.481s
user	0m22.845s
sys	0m0.649s

Speedup after not calling into inspect.stack():

$ time sudo salt-call --local test.ping
local:
    True

real	0m3.661s
user	0m3.253s
sys	0m0.423s

Stackoverflow suggests that frames with virtual importlib should be skipped:

* https://stackoverflow.com/questions/40945752/inspect-who-imported-me
msg367991 - (view) Author: Ivan Babrou (Ivan Babrou) Date: 2020-05-03 20:03 
The issue in Salt should be fixed by the following PR:

* https://github.com/saltstack/salt/pull/57062

Still, I think Python itself could do better.
History
Date User Action Args
2022-04-11 14:59:26adminsetgithub: 83824
2020-05-03 20:03:58Ivan Babrousetmessages: + msg367991
2020-02-16 05:41:42Ivan Babroucreate