Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

_osx_support imports many modules #63524

Closed
tiran opened this issue Oct 21, 2013 · 7 comments
Closed

_osx_support imports many modules #63524

tiran opened this issue Oct 21, 2013 · 7 comments
Assignees
Labels
OS-mac performance Performance or resource usage stdlib Python modules in the Lib dir

Comments

@tiran
Copy link
Member

tiran commented Oct 21, 2013

BPO 19325
Nosy @ronaldoussoren, @vstinner, @tiran, @ned-deily, @methane

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = 'https://github.com/ned-deily'
closed_at = <Date 2017-07-03.14:25:21.389>
created_at = <Date 2013-10-21.09:09:04.524>
labels = ['OS-mac', 'library', 'performance']
title = '_osx_support imports many modules'
updated_at = <Date 2017-07-03.14:25:27.744>
user = 'https://github.com/tiran'

bugs.python.org fields:

activity = <Date 2017-07-03.14:25:27.744>
actor = 'vstinner'
assignee = 'ned.deily'
closed = True
closed_date = <Date 2017-07-03.14:25:21.389>
closer = 'vstinner'
components = ['Library (Lib)', 'macOS']
creation = <Date 2013-10-21.09:09:04.524>
creator = 'christian.heimes'
dependencies = []
files = []
hgrepos = []
issue_num = 19325
keywords = []
message_count = 7.0
messages = ['200733', '200920', '200921', '200924', '224034', '297160', '297595']
nosy_count = 6.0
nosy_names = ['ronaldoussoren', 'vstinner', 'christian.heimes', 'ned.deily', 'methane', 'Esa.Peuha']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue19325'
versions = ['Python 3.4']

@tiran
Copy link
Member Author

tiran commented Oct 21, 2013

On Mac OS X the site module imports sysconfig which imports _osx_support which in turn imports several more modules like the re module. More modules == slower startup. See bpo-19205 for background information on the issue.

@tiran tiran added stdlib Python modules in the Lib dir OS-mac performance Performance or resource usage labels Oct 21, 2013
@EsaPeuha
Copy link
Mannequin

EsaPeuha mannequin commented Oct 22, 2013

Actually Lib/_osx_support.py directly imports only five modules: os, re and sys at top level, and contextlib and tempfile in _read_output(). These last two aren't really needed at all, and there's no point even trying to avoid importing os and sys, so the only real problem is re which imports lots of other modules. It might be possible to avoid using re (the regular expressions don't look very complicated) but I noticed that some functions are only ever called from distutils, so maybe they should be moved from _osx_support to distutils._osx_support before doing anything else. I'm willing to do that if it's considered a good idea.

@tiran
Copy link
Member Author

tiran commented Oct 22, 2013

For distutils function you can move the import of re inside the function body. It's not elegant but it does the trick w/o much slowdown or inconveniences.

@ned-deily
Copy link
Member

Yes, there are various tricks to be done. I have another solution in mind that should eliminate the use re altogether for most cases. (BTW, _osx_support was implemented to centralize functions in anticipation of the introduction of distutils2/packaging in Python 3.3, along with distutils.)

@ronaldoussoren
Copy link
Contributor

The use of context lib in _read_output should not be necessary anymore as file objects and NamedTemporaryFile objects already are context managers with the right semantics.

I'm not sure how to avoid the import of tempfile other than adding a minimal implementation of tempfile.mkstemp to _osx_support, in particular because the fallback code in _osx_support is insecure: It uses a named temporary file in /tmp with builtin.open and because the name of the file is predictable there is a small risk of overwriting arbitrary files when an attacker has access to /tmp. I guess the fallback is there to use during bootstrap, it should really be avoided afterwards.

BTW. A small unscientific test on my laptop didn't see any differences between the regular _osx_support and a version where "import re" was moved inside the functions that use that module. But: that's on a fast laptop with SSD for storage, there could easily be a difference on systems with slower storage.

Ned: do you remember what your idea was w.r.t. avoid the use of re? I guess its easy enough to replace the current re-using code by code that only uses str methods, but your phrasing seems to indicate another plan.

@methane
Copy link
Member

methane commented Jun 28, 2017

FYI, bpo-29585 removes _osx_support dependency from site.py

@vstinner
Copy link
Member

vstinner commented Jul 3, 2017

FYI, bpo-29585 removes _osx_support dependency from site.py

Done: Lib/site.py doesn't import _osx_support anymore. So I close this issue.

@vstinner vstinner closed this as completed Jul 3, 2017
@ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
OS-mac performance Performance or resource usage stdlib Python modules in the Lib dir
Projects
None yet
Development

No branches or pull requests

5 participants