classification
Title: Don't import re and sysconfig in site.py
Type: performance Stage: committed/rejected
Components: Library (Lib), Macintosh Versions: Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: barry, christian.heimes, haypo, pitrou, python-dev
Priority: low Keywords: patch

Created on 2013-10-09 13:15 by christian.heimes, last changed 2013-10-21 09:09 by christian.heimes. This issue is now closed.

Files
File name Uploaded Description Edit
startup_no_re.patch christian.heimes, 2013-10-11 20:11 review
Messages (15)
msg199295 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-10-09 13:20
The site module is loaded at every startup. Nowadays it imports the re and the sysconfig modules. The re module is used for venv config parsing and inside sysconfig. sysconfig is loaded to find the location of the user's site-packages directory.

Suggestions:
- Only import re and compile CONFIG_LINE when a venv config file is found

- Don't rely on sysconfig for user's site-packages directory. Instead the sysconfig module should rely on site.py to get its location.

Without re and sysconfig Python would import only 45 instead of 56 modules at startup (tested on Linux).
msg199330 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-10-09 18:52
The site and sysconfig module are too intermingled to remove the import of sysconfig from the site module. However the sysconfig module doesn't use the re module in most cases anymore. The parsing functions for Makefile and pyconfig.h are only used by distutils.

The patch moves "import re" inside three functions.
msg199502 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-10-11 20:11
Here is a new patch with unit test and patch for the locale module. The locale modules is loaded by the _io module when any standard stream is not connected to a terminal (or so).
msg199503 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-10-11 20:26
Review posted on Rietveld.
msg199506 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-10-11 21:19
I accept hacks to speedup Python is the site module, but it becomes more surprising in the locale module. The issue #9548 proposes to a more generic solution for the locale module at startup.


-CONFIG_LINE = re.compile(r'^(?P<key>(\w|[-_])+)\s*=\s*(?P<value>.*)\s*$')
+CONFIG_LINE = None

If you set the constant to None, it's better to remove it completly (or make it private). It's a public variable, someone may try to read it. I don't know why it is public.
msg199514 - (view) Author: Roundup Robot (python-dev) Date: 2013-10-11 22:25
New changeset 406529adf156 by Christian Heimes in branch 'default':
Issue #19205: Don't import the 're' module in site and sysconfig module to
http://hg.python.org/cpython/rev/406529adf156
msg199515 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-10-11 22:25
Thanks for your input!
msg199516 - (view) Author: Roundup Robot (python-dev) Date: 2013-10-11 22:28
New changeset 2cd1b28d1666 by Christian Heimes in branch 'default':
Issue #19205 fix 406529adf156
http://hg.python.org/cpython/rev/2cd1b28d1666
msg199543 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-10-12 10:13
Christian, the test is failing on Snow Leopard:


======================================================================
FAIL: test_startup_imports (test.test_site.StartupImportTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/test/test_site.py", line 435, in test_startup_imports
    self.assertFalse(modules.intersection(re_mods))
AssertionError: {'re', 'sre_compile', 'sre_constants', 'sre_parse', '_sre'} is not false

http://buildbot.python.org/all/builders/AMD64%20Snow%20Leop%203.x/builds/106/steps/test/logs/stdio
msg199545 - (view) Author: Roundup Robot (python-dev) Date: 2013-10-12 10:32
New changeset a57dfbba91f9 by Christian Heimes in branch 'default':
Issue #19205: add debugging output for failing test on Snow Leopard
http://hg.python.org/cpython/rev/a57dfbba91f9
msg199563 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-10-12 13:01
> New changeset a57dfbba91f9 by Christian Heimes in branch 'default':
> Issue #19205: add debugging output for failing test on Snow Leopard
> http://hg.python.org/cpython/rev/a57dfbba91f9

So the "import re" comes from _osx_support, _osx_support is imported by sysconfig.
msg199565 - (view) Author: Roundup Robot (python-dev) Date: 2013-10-12 13:08
New changeset 9f6ef09f6492 by Christian Heimes in branch 'default':
Issue #19205: _osx_support uses the re module all over the place. Omit the test for nw.
http://hg.python.org/cpython/rev/9f6ef09f6492
msg200721 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-10-21 08:48
The imports in _osx_support counteract the performance boost. I neither have an OS X machine and nor do I understand the internals of  _osx_support. Perhaps somebody else likes to work on the module.
msg200731 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-10-21 09:03
> The imports in _osx_support counteract the performance boost. I neither have an OS X machine and nor do I understand the internals of  _osx_support. Perhaps somebody else likes to work on the module.

Please open a separated issue for OS X, this issue can be closed IMO.
msg200734 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-10-21 09:09
Good point, #19325
History
Date User Action Args
2013-10-21 09:09:25christian.heimessetstatus: open -> closed

messages: + msg200734
stage: commit review -> committed/rejected
2013-10-21 09:03:49hayposetmessages: + msg200731
2013-10-21 08:48:30christian.heimessetpriority: normal -> low
assignee: christian.heimes ->
messages: + msg200721

components: + Macintosh, - Interpreter Core
2013-10-12 13:08:59python-devsetmessages: + msg199565
2013-10-12 13:01:53hayposetmessages: + msg199563
2013-10-12 10:32:38python-devsetmessages: + msg199545
2013-10-12 10:13:43pitrousetstatus: closed -> open
assignee: christian.heimes
messages: + msg199543
2013-10-11 22:28:26python-devsetmessages: + msg199516
2013-10-11 22:25:31christian.heimessetstatus: open -> closed
resolution: fixed
messages: + msg199515

stage: patch review -> commit review
2013-10-11 22:25:03python-devsetnosy: + python-dev
messages: + msg199514
2013-10-11 21:19:53hayposetnosy: + haypo
messages: + msg199506
2013-10-11 20:26:21pitrousetnosy: + pitrou

messages: + msg199503
stage: needs patch -> patch review
2013-10-11 20:11:16christian.heimessetfiles: + startup_no_re.patch

messages: + msg199502
2013-10-11 20:07:51christian.heimessetfiles: - sysconfig_delay_re.patch
2013-10-11 20:07:43christian.heimessetfiles: - site_no_re.patch
2013-10-09 18:52:33christian.heimessetfiles: + sysconfig_delay_re.patch

messages: + msg199330
2013-10-09 17:17:24barrysetnosy: + barry
2013-10-09 13:26:37christian.heimessetfiles: + site_no_re.patch
keywords: + patch
2013-10-09 13:20:34christian.heimessetversions: + Python 3.4
title: Don -> Don't import re and sysconfig in site.py
messages: + msg199295

components: + Interpreter Core, Library (Lib)
type: performance
stage: needs patch
2013-10-09 13:15:01christian.heimescreate