classification
Title: Reduce number of modules imported by runpy
Type: Stage: resolved
Components: Library (Lib) Versions: Python 3.10
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: shihai1991, vstinner
Priority: normal Keywords: patch

Created on 2020-06-17 13:58 by vstinner, last changed 2020-06-17 23:20 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
mod.py vstinner, 2020-06-17 13:58
Pull Requests
URL Status Linked Edit
PR 20938 merged vstinner, 2020-06-17 14:35
PR 20939 merged vstinner, 2020-06-17 14:52
PR 20940 merged vstinner, 2020-06-17 14:58
PR 20953 merged vstinner, 2020-06-17 21:46
PR 20954 merged vstinner, 2020-06-17 22:09
PR 20955 merged vstinner, 2020-06-17 23:10
Messages (12)
msg371741 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-17 13:58
Currently, the runpy module imports many modules. runpy is used by "python3 -m module". I propose to attempt to reduce the number of imports to reduce Python startup time.


With my local changes, I reduce Python startup time from 24 ms to 18 ms:

Mean +- std dev: [ref] 24.3 ms +- 0.2 ms -> [patch] 18.0 ms +- 0.3 ms: 1.35x faster (-26%)

Timing measured by:

  ./python -m venv env
  python -m pyperf command -v -o patch.json -- env/bin/python -m empty


Currently, runpy imports +21 modules:

* ./python mod.py: Total 33
* ./python -m mod: Total 54 (+21)


Example with attached mod.py:

$ ./python -m mod
['__main__',
 '_abc',
 '_codecs',
 '_collections',
 '_collections_abc',
 '_frozen_importlib',
 '_frozen_importlib_external',
 '_functools',
 '_heapq',
 '_imp',
 '_io',
 '_locale',
 '_operator',
 '_signal',
 '_sitebuiltins',
 '_sre',
 '_stat',
 '_thread',
 '_warnings',
 '_weakref',
 '_weakrefset',
 'abc',
 'builtins',
 'codecs',
 'collections',
 'collections.abc',
 'contextlib',
 'copyreg',
 'encodings',
 'encodings.aliases',
 'encodings.ascii',
 'encodings.latin_1',
 'encodings.utf_8',
 'enum',
 'functools',
 'genericpath',
 'heapq',
 'importlib',
 'importlib._bootstrap',
 'importlib._bootstrap_external',
 'importlib.abc',
 'importlib.machinery',
 'importlib.util',
 'io',
 'itertools',
 'keyword',
 'marshal',
 'operator',
 'os',
 'os.path',
 'pkgutil',
 'posix',
 'posixpath',
 're',
 'reprlib',
 'runpy',
 'site',
 'sre_compile',
 'sre_constants',
 'sre_parse',
 'stat',
 'sys',
 'time',
 'types',
 'typing',
 'typing.io',
 'typing.re',
 'warnings',
 'weakref',
 'zipimport']
Total 70
msg371742 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-17 14:00
My local changes removed the following imports from runpy:

- '_heapq',
- '_locale',
- '_sre',
- 'collections.abc',
- 'copyreg',
- 'enum',
- 'heapq',
- 'importlib.abc',
- 'itertools',
- 're',
- 'sre_compile',
- 'sre_constants',
- 'sre_parse',
- 'typing',
- 'typing.io',
- 'typing.re',
msg371746 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-17 14:47
> Currently, runpy imports +21 modules:
> * ./python mod.py: Total 33
> * ./python -m mod: Total 54 (+21)

Oops sorry, that's with my local changes!

Currently, runpy imports not less than 37 modules:

* ./python mod.py: Total 33
* ./python -m mod: Total 70 (+37)
msg371750 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-17 15:11
See also bpo-40275: "test.support has way too many imports".
msg371761 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-17 16:44
I created 3 PRs. I have a few more local branches to avoid types and itertools imports. I may create PRs for these ones as well.
msg371765 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-17 17:10
New changeset 7824cc05bfe7f8181b21848a52007ddaf5612b9b by Victor Stinner in branch 'master':
bpo-41006: collections imports lazily heap (GH-20940)
https://github.com/python/cpython/commit/7824cc05bfe7f8181b21848a52007ddaf5612b9b
msg371766 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-17 17:11
New changeset 98ce7b107e6611d04dc35a4f5b02ea215ef122cf by Victor Stinner in branch 'master':
bpo-41006: pkgutil imports lazily re (GH-20939)
https://github.com/python/cpython/commit/98ce7b107e6611d04dc35a4f5b02ea215ef122cf
msg371774 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-17 21:16
New changeset 9e09849d20987c131b28bcdd252e53440d4cd1b3 by Victor Stinner in branch 'master':
bpo-41006: importlib.util no longer imports typing (GH-20938)
https://github.com/python/cpython/commit/9e09849d20987c131b28bcdd252e53440d4cd1b3
msg371776 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-17 21:36
I close the issue. Making more imports lazy doesn't bring much benefit.

--

With the 3 changes, runpy now imports 55 modules, instead of 70.

The startup time is 1.3x faster, 18 ms instead of 24 ms:

Mean +- std dev: [before] 23.7 ms +- 0.4 ms -> [after] 17.8 ms +- 0.6 ms: 1.33x faster (-25%)

--

Avoiding itertools and types doesn't bring much benefit:

Mean +- std dev: [after] 17.8 ms +- 0.6 ms -> [WIP] 17.2 ms +- 0.4 ms: 1.03x faster (-3%)
msg371777 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-17 21:59
New changeset 4c18fc8f1def7030e5ec5d1ffb9355d7453dc408 by Victor Stinner in branch 'master':
bpo-41006: Document the runpy optimization (GH-20953)
https://github.com/python/cpython/commit/4c18fc8f1def7030e5ec5d1ffb9355d7453dc408
msg371781 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-17 23:11
New changeset 1bf7959dce0597e312c6f35476a7cc957fd0323c by Victor Stinner in branch 'master':
bpo-41006: Remove init_sys_streams() hack (GH-20954)
https://github.com/python/cpython/commit/1bf7959dce0597e312c6f35476a7cc957fd0323c
msg371782 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-17 23:20
New changeset 2c2a4f3d8545784c6e4ca8128bfc706916080712 by Victor Stinner in branch 'master':
bpo-41006: What's New: less => fewer modules (GH-20955)
https://github.com/python/cpython/commit/2c2a4f3d8545784c6e4ca8128bfc706916080712
History
Date User Action Args
2020-06-17 23:20:57vstinnersetmessages: + msg371782
2020-06-17 23:11:54vstinnersetmessages: + msg371781
2020-06-17 23:10:28vstinnersetpull_requests: + pull_request20134
2020-06-17 22:09:02vstinnersetpull_requests: + pull_request20133
2020-06-17 21:59:05vstinnersetmessages: + msg371777
2020-06-17 21:46:38vstinnersetpull_requests: + pull_request20132
2020-06-17 21:36:12vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg371776

stage: patch review -> resolved
2020-06-17 21:16:07vstinnersetmessages: + msg371774
2020-06-17 17:11:58vstinnersetmessages: + msg371766
2020-06-17 17:10:55vstinnersetmessages: + msg371765
2020-06-17 16:44:49vstinnersetmessages: + msg371761
2020-06-17 15:17:22shihai1991setnosy: + shihai1991
2020-06-17 15:11:09vstinnersetmessages: + msg371750
2020-06-17 14:58:26vstinnersetpull_requests: + pull_request20121
2020-06-17 14:52:44vstinnersetpull_requests: + pull_request20119
2020-06-17 14:47:58vstinnersetmessages: + msg371746
2020-06-17 14:35:32vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request20117
2020-06-17 14:00:50vstinnersetmessages: + msg371742
2020-06-17 13:58:21vstinnercreate