New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Most of Python's startup time is sysconfig #57359
Comments
sysconfig is imported and used by site.py. $ time ./python -S -c '' real 0m0.019s $ time ./python -S -c 'import sysconfig' real 0m0.047s $ time ./python -S -c 'import sysconfig; sysconfig.get_path("purelib")' real 0m0.053s $ time ./python -c '' real 0m0.058s |
Actually, a big part of that is compiling some regexes in the tokenize module. Just relying on the re module's internal caching shaves off 20% of total startup time. Before: $ time ./python -S -c 'import tokenize' real 0m0.034s real 0m0.055s After: $ time ./python -S -c 'import tokenize' real 0m0.021s real 0m0.044s |
New changeset df950158dc33 by Antoine Pitrou in branch 'default': |
I am curious: wouldn't be a way of keeping the compiled expressions in a static cache somewhere, so we would compile them just once and have both import time and runtime fast ? |
New changeset ed0bc92fed68 by Antoine Pitrou in branch 'default': |
Runtime shouldn't be affected. The re module has its own LRU caching. That said, it seems regular expressions are pickleable: b'\x80\x03cre\n_compile\nq\x00X\x00\x00\x00\x00q\x01K \x86q\x02Rq\x03.' |
Arg damn roundup e-mail gateway. >>> pickle.dumps(re.compile(''))
b'\x80\x03cre\n_compile\nq\x00X\x00\x00\x00\x00q\x01K \x86q\x02Rq\x03.' |
Pre-parsing and building a cached module of built-time variables (from Makefile and pyconfig.h) under POSIX also removes more than 15% of startup time. Patch attached. |
bpo-11454 is another case where pre-parsing and pickling the regular expressions in the email module may improve import time considerably. |
bpo-9878 should also help with start-up time. |
Actually, bpo-9878 should supersede this bug: it proposes to generate a C module to avoid parsing Makefile and pyconfig.h, and your patch proposes to generate a Python module with the same goal. |
Well, bpo-9878 doesn't have a patch, but perhaps Barry is willing to work on one. Also, if we have a pure Python solution, perhaps a C module isn't needed. The main advantage of the C solution, though, would be to avoid dubious parsing altogher, even at build time. |
Since bpo-9878 proposes an *alternate* solution to *part* of the sysconfig problem, I disagree with 'supersede'. A Python solution would be more useful for other implementations if enough of the sysconfig info is not CPython specific. A CPython design feature is that it parses and compiles Python code just once per run, and imported modules just once until the code changes (or might have). For functions, everything possible is put into a behind-the-scenes code object. So even inner functions are parsed and compiled just once. The problem with sysconfig, it appears, is that lacks the equivalent design feature but instead does the equivalent of re-parsing and re-compiling inner functions with each outer function call. |
|
A module doesn't have to be written in C to be CPython-specific. |
New changeset 70160b53117f by Antoine Pitrou in branch 'default': |
Done! If someone wants to give life to the C approach, they are welcome :) |
BTW, distutils2 backports the sysconfig module and cfg file from 3.3, so now the two versions will diverge. |
10x for solution, 10x for commit . |
New changeset 677e625e2ef1 by Victor Stinner in branch 'default': |
the current ability to cross-build python now relies on being able to run the build python with the host library, using the _sysconfigdata.py from the host. if somebody decides to implement _sysconfigdata as a C extension, please ensure that this information still can be passed to the build python. |
New changeset 66e30c4870bb by doko in branch '2.7':
|
New changeset d174cb3f5b9e by Benjamin Peterson in branch '2.7': |
New changeset be3b4aa2ad28 by doko in branch '2.7':
|
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: