classification
Title: _osx_support imports many modules
Type: performance Stage: resolved
Components: Library (Lib), macOS Versions: Python 3.4
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: ned.deily Nosy List: Esa.Peuha, christian.heimes, inada.naoki, ned.deily, ronaldoussoren, vstinner
Priority: normal Keywords:

Created on 2013-10-21 09:09 by christian.heimes, last changed 2017-07-03 14:25 by vstinner. This issue is now closed.

Messages (7)
msg200733 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-10-21 09:09
On Mac OS X the site module imports sysconfig which imports _osx_support which in turn imports several more modules like the re module. More modules == slower startup. See #19205 for background information on the issue.
msg200920 - (view) Author: Esa Peuha (Esa.Peuha) Date: 2013-10-22 11:43
Actually Lib/_osx_support.py directly imports only five modules: os, re and sys at top level, and contextlib and tempfile in _read_output(). These last two aren't really needed at all, and there's no point even trying to avoid importing os and sys, so the only real problem is re which imports lots of other modules. It might be possible to avoid using re (the regular expressions don't look very complicated) but I noticed that some functions are only ever called from distutils, so maybe they should be moved from _osx_support to distutils._osx_support before doing anything else. I'm willing to do that if it's considered a good idea.
msg200921 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-10-22 11:45
For distutils function you can move the import of re inside the function body. It's not elegant but it does the trick w/o much slowdown or inconveniences.
msg200924 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-10-22 11:49
Yes, there are various tricks to be done.  I have another solution in mind that should eliminate the use re altogether for most cases.  (BTW, _osx_support was implemented to centralize functions in anticipation of the introduction of distutils2/packaging in Python 3.3, along with distutils.)
msg224034 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2014-07-26 09:37
The use of context lib in _read_output should not be necessary anymore as file objects and NamedTemporaryFile objects already are context managers with the right semantics. 

I'm not sure how to avoid the import of tempfile other than adding a minimal implementation of tempfile.mkstemp to _osx_support, in particular because the fallback code in _osx_support is insecure: It uses a named temporary file in /tmp with builtin.open and because the name of the file is predictable there is a small risk of overwriting arbitrary files when an attacker has access to /tmp. I guess the fallback is there to use during bootstrap, it should really be avoided afterwards. 

BTW. A small unscientific test on my laptop didn't see any differences between the regular _osx_support and a version where "import re" was moved inside the functions that use that module.  But: that's on a fast laptop with SSD for storage, there could easily be a difference on systems with slower storage.

Ned: do you remember what your idea was w.r.t. avoid the use of re? I guess its easy enough to replace the current re-using code by code that only uses str methods, but your phrasing seems to indicate another plan.
msg297160 - (view) Author: Inada Naoki (inada.naoki) * (Python committer) Date: 2017-06-28 07:28
FYI, issue29585 removes _osx_support dependency from site.py
msg297595 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-03 14:25
> FYI, issue29585 removes _osx_support dependency from site.py

Done: Lib/site.py doesn't import _osx_support anymore. So I close this issue.
History
Date User Action Args
2017-07-03 14:25:27vstinnersetresolution: fixed -> out of date
2017-07-03 14:25:21vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg297595

stage: needs patch -> resolved
2017-06-28 07:28:18inada.naokisetnosy: + inada.naoki
messages: + msg297160
2016-09-12 04:27:58ned.deilylinkissue28095 dependencies
2014-07-26 09:37:58ronaldoussorensetmessages: + msg224034
2013-10-22 11:49:13ned.deilysetmessages: + msg200924
2013-10-22 11:45:57christian.heimessetmessages: + msg200921
2013-10-22 11:43:47Esa.Peuhasetnosy: + Esa.Peuha
messages: + msg200920
2013-10-21 09:47:17ned.deilysetassignee: ronaldoussoren -> ned.deily

nosy: + ned.deily
2013-10-21 09:25:04vstinnersetnosy: + vstinner
2013-10-21 09:09:04christian.heimescreate