Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use of MutableMapping in os module slows down interpreter startup #63417

Closed
ericsnowcurrently opened this issue Oct 10, 2013 · 14 comments
Closed

Use of MutableMapping in os module slows down interpreter startup #63417

ericsnowcurrently opened this issue Oct 10, 2013 · 14 comments
Assignees
@tiran
Labels
stdlib Python modules in the Lib dir type-feature A feature request or enhancement

Comments

@ericsnowcurrently
Copy link
Member

BPO 19218
Nosy @warsaw, @rhettinger, @pitrou, @vstinner, @tiran, @bitdancer, @ericsnowcurrently
Files
  • os-lazy-collections.diff
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/tiran'
closed_at = <Date 2013-10-13.00:53:34.686>
created_at = <Date 2013-10-10.18:08:01.373>
labels = ['type-feature', 'library']
title = 'Use of MutableMapping in os module slows down interpreter startup'
updated_at = <Date 2014-02-26.18:26:34.415>
user = 'https://github.com/ericsnowcurrently'

    bugs.python.org fields:

    activity = <Date 2014-02-26.18:26:34.415>
actor = 'r.david.murray'
assignee = 'christian.heimes'
closed = True
closed_date = <Date 2013-10-13.00:53:34.686>
closer = 'christian.heimes'
components = ['Library (Lib)']
creation = <Date 2013-10-10.18:08:01.373>
creator = 'eric.snow'
dependencies = []
files = ['32036']
hgrepos = []
issue_num = 19218
keywords = ['patch']
message_count = 14.0
messages = ['199402', '199404', '199405', '199406', '199449', '199450', '199520', '199521', '199642', '199644', '199647', '199654', '212282', '212285']
nosy_count = 9.0
nosy_names = ['barry', 'rhettinger', 'pitrou', 'vstinner', 'christian.heimes', 'Arfrever', 'r.david.murray', 'python-dev', 'eric.snow']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue19218'
versions = ['Python 3.4']
    @ericsnowcurrently
    Copy link
    Member Author

    There has been some discussion on python-dev about improving interpreter startup time. Christian Heimes brought up how the os module imports collections (so _Environ can inherit from it) [1], and mentioned a couple of solutions [2].

    [1]https://mail.python.org/pipermail/python-dev/2013-October/129312.html
    [2]https://mail.python.org/pipermail/python-dev/2013-October/129367.html

    @ericsnowcurrently ericsnowcurrently added stdlib Python modules in the Lib dir type-feature A feature request or enhancement labels Oct 10, 2013
    @ericsnowcurrently
    Copy link
    Member Author

    Here is a patch for an alternate approach. It does not require re-implementing MutableMapping (could get out of sync) and does not require rearranging collections.abc. Instead it uses a metaclass to resolve the import issue lazily.

    @tiran
    Copy link
    Member

    tiran commented Oct 10, 2013

    Nice trick :)

    @ericsnowcurrently
    Copy link
    Member Author

    And when the lazy base class gets resolved, the metaclass gets to say "You didn't see anything and I was never here." :)

    @rhettinger rhettinger self-assigned this Oct 11, 2013
    @pitrou
    Copy link
    Member

    pitrou commented Oct 11, 2013

    -1 on such hacks. I much prefer the _abcoll approach.

    @vstinner
    Copy link
    Member

    The collections module is loaded by the io module. You removed "from collections.abc import MutableMapping" from os.py. To be useful, you have also to rewrite the whole io module to remove all references to the collections.abc module, right?

    @tiran
    Copy link
    Member

    tiran commented Oct 11, 2013

    The io module no longer imports collections.abc through the locale module.

    Eric, I'm with Antoine. Your patch is too much of a clever hack and uses tricks that are dark magic. _abcoll is much simpler and easier to understand. Plus it can be used from other modules, too.

    @ericsnowcurrently
    Copy link
    Member Author

    Not a problem. It is most definitely a hack. :) I put in up as an alternative to rearranging the collections module, but agree that doing so is preferable. I image that Raymond will make a decision on that one.

    @rhettinger
    Copy link
    Contributor

    Christian, go ahead an rearrange the collections module.

    Move Lib/collections/abc.py to Lib/_collections_abc.py.

    Also, be sure to update source link on line 18 of Doc/library.collections.abc.rst

    @rhettinger rhettinger assigned tiran and unassigned rhettinger Oct 12, 2013
    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Oct 13, 2013

    New changeset 0b6052f2a8ee by Christian Heimes in branch 'default':
    Issue bpo-19218: Rename collections.abc to _collections_abc in order to speed up interpreter start
    http://hg.python.org/cpython/rev/0b6052f2a8ee

    New changeset 7ea831581af4 by Christian Heimes in branch 'default':
    Issue bpo-19218: Add facade collections.abc
    http://hg.python.org/cpython/rev/7ea831581af4

    New changeset 62b6ecd1e463 by Christian Heimes in branch 'default':
    Issue bpo-19218: set __name__ of _collections_abc to collections.abc in order to fix tests and keep beautiful qualified names.
    http://hg.python.org/cpython/rev/62b6ecd1e463

    @tiran
    Copy link
    Member

    tiran commented Oct 13, 2013

    Thank you very much for your input and assistance!

    tip compared to v3.4.0a3:

    ### normal_startup ###
    Min: 0.506533 -> 0.313627: 1.62x faster
    Avg: 0.582504 -> 0.441796: 1.32x faster
    Significant (t=19.98)
    Stddev: 0.02397 -> 0.04366: 1.8213x larger

    @tiran tiran closed this as completed Oct 13, 2013
    @ericsnowcurrently
    Copy link
    Member Author

    Thanks, Christian.

    @bitdancer
    Copy link
    Member

    For backward compatibility, shouldn't

      import _collections_abc

    in the __init__ file be

      import _collections_abc as abc

    ?

    @bitdancer
    Copy link
    Member

    Opened bpo-20784 to address the above.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees

    @tiran tiran

    Labels
    stdlib Python modules in the Lib dir type-feature A feature request or enhancement
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    6 participants
    @vstinner @tiran @bitdancer @ericsnowcurrently @rhettinger @pitrou