Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

compiler: merge co_code and co_lnotab #86383

Closed
methane opened this issue Oct 31, 2020 · 5 comments
Closed

compiler: merge co_code and co_lnotab #86383

methane opened this issue Oct 31, 2020 · 5 comments
Labels
3.10 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) performance Performance or resource usage

Comments

@methane
Copy link
Member

methane commented Oct 31, 2020

BPO 42217
Nosy @methane, @markshannon, @serhiy-storchaka
PRs
  • bpo-42217: compiler: merge same co_code and co_linetable objects #23056
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2021-02-10.00:20:59.942>
    created_at = <Date 2020-10-31.07:50:02.020>
    labels = ['interpreter-core', '3.10', 'performance']
    title = 'compiler: merge co_code and co_lnotab'
    updated_at = <Date 2021-02-10.00:21:17.632>
    user = 'https://github.com/methane'

    bugs.python.org fields:

    activity = <Date 2021-02-10.00:21:17.632>
    actor = 'methane'
    assignee = 'none'
    closed = True
    closed_date = <Date 2021-02-10.00:20:59.942>
    closer = 'methane'
    components = ['Interpreter Core']
    creation = <Date 2020-10-31.07:50:02.020>
    creator = 'methane'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 42217
    keywords = ['patch']
    message_count = 5.0
    messages = ['380045', '380049', '380050', '380061', '386749']
    nosy_count = 3.0
    nosy_names = ['methane', 'Mark.Shannon', 'serhiy.storchaka']
    pr_nums = ['23056']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'resource usage'
    url = 'https://bugs.python.org/issue42217'
    versions = ['Python 3.10']

    @methane
    Copy link
    Member Author

    methane commented Oct 31, 2020

    co_consts are merged already, but I forget about code attributes other than co_consts.
    Duplication ratio is vary by code style, but in case of logging, we can reduce 25% of them.

    >>> data = open('Lib/logging/__pycache__/__init__.python-310.pyc', 'rb').read()
    >>> import marshal
    >>> x = []
    >>> def walk(c):
    ...     x.append(c.co_code)
    ...     x.append(c.co_lnotab)
    ...     for cc in c.co_consts:
    ...         if hasattr(cc, 'co_code'):
    ...             walk(cc)
    ...
    >>> code = marshal.loads(data[16:])
    >>> walk(code)
    >>> len(x)
    336
    >>> len(set(x))
    249
    >>> 249/336
    0.7410714285714286
    

    @methane methane added 3.10 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) performance Performance or resource usage labels Oct 31, 2020
    @serhiy-storchaka
    Copy link
    Member

    Do you mean sharing values of co_code and co_lnotab between code objects of the same module?

    How much memory does this save (in absolute and relative value)? Which functions have the same co_code?

    @markshannon
    Copy link
    Member

    PEP-626 has just been accepted.
    Could you please leave this until I have merged in the implementation of PEP-626 which uses a different line number table format

    @methane
    Copy link
    Member Author

    methane commented Oct 31, 2020

    @serhiy

    Do you mean sharing values of co_code and co_lnotab between code objects of the same module?

    Yes.

    How much memory does this save (in absolute and relative value)?

    Maybe 1~3%, but I am not sure. I am more interested in reducing number of objects, because it will reduce import time.
    Additionally, co_code is used while executing code. Unlike other cold data (e.g. docstring, annotations), sharing co_code will improve CPU cache utilization.

    Which functions have the same co_code?

    For example,

    # logging/__init__.py
    
        @property
        def manager(self):
            return self.logger.manager
    ...
        @property
        def name(self):
            return self.logger.name
    

    Such simple functions are very common in OO-style code.

    @mark Shannon

    Sure.

    @methane
    Copy link
    Member Author

    methane commented Feb 10, 2021

    New changeset bdb941b by Inada Naoki in branch 'master':
    bpo-42217: compiler: merge same co_code and co_linetable objects (GH-23056)
    bdb941b

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.10 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) performance Performance or resource usage
    Projects
    None yet
    Development

    No branches or pull requests

    3 participants