Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

singledispatchmethod significantly slower than singledispatch #85160

Closed
CaselIT mannequin opened this issue Jun 15, 2020 · 3 comments
Closed

singledispatchmethod significantly slower than singledispatch #85160

CaselIT mannequin opened this issue Jun 15, 2020 · 3 comments
Labels
performance Performance or resource usage stdlib Python modules in the Lib dir type-feature A feature request or enhancement

Comments

@CaselIT
Copy link
Mannequin

CaselIT mannequin commented Jun 15, 2020 •
edited by bedevere-bot

BPO 40988
Nosy @mental32, @CaselIT
PRs
  • gh-85160: Optimized singledispatchmethod access (noticeable improvement). #23213
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = None
created_at = <Date 2020-06-15.20:14:13.251>
labels = ['3.8', 'library', 'performance']
title = 'singledispatchmethod significantly slower than singledispatch'
updated_at = <Date 2020-11-10.05:12:05.587>
user = 'https://github.com/CaselIT'

    bugs.python.org fields:

    activity = <Date 2020-11-10.05:12:05.587>
actor = 'mental'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2020-06-15.20:14:13.251>
creator = 'CaselIT'
dependencies = []
files = []
hgrepos = []
issue_num = 40988
keywords = ['patch']
message_count = 1.0
messages = ['371594']
nosy_count = 2.0
nosy_names = ['mental', 'CaselIT']
pr_nums = ['23213']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue40988'
versions = ['Python 3.8']

    Linked PRs
    @CaselIT
    Copy link
    Mannequin Author

    CaselIT mannequin commented Jun 15, 2020

    The implementation of singledispatchmethod is significantly slower (~4x) than the normal singledispatch version

    Using timeit to test this example case:

        from functools import singledispatch, singledispatchmethod
    import timeit

    class Test:
        @singledispatchmethod
        def go(self, item, arg):
            print('general')
        
        @go.register
        def _(self, item:int, arg):
            return item + arg

    @singledispatch
    def go(item, arg):
        print('general')

    @go.register
    def _(item:int, arg):
        return item + arg
    print(timeit.timeit('t.go(1, 1)', globals={'t': Test()}))
print(timeit.timeit('go(1, 1)', globals={'go': go}))

    Prints on my system.

    3.118346
0.713173

    Looking at the singledispatchmethod implementation I believe that most of the difference is because a new function is generated every time the method is called.

    Maybe an implementation similar to cached_property could be used if the class has __dict__ attribute?
    Trying this simple patch

    diff --git a/Lib/functools.py b/Lib/functools.py
index 5cab497..e42f485 100644
--- a/Lib/functools.py
+++ b/Lib/functools.py
@@ -900,6 +900,7 @@ class singledispatchmethod:

                self.dispatcher = singledispatch(func)
            self.func = func
    +        self.attrname = None

        def register(self, cls, method=None):
            """generic_method.register(cls, func) -> func
    @@ -908,6 +909,10 @@ class singledispatchmethod:
            """
            return self.dispatcher.register(cls, func=method)
    +    def __set_name__(self, owner, name):
+        if self.attrname is None:
+            self.attrname = name
+
    def __get__(self, obj, cls=None):
        def _method(*args, **kwargs):
            method = self.dispatcher.dispatch(args[0].__class__)
@@ -916,6 +921,7 @@ class singledispatchmethod:
        _method.__isabstractmethod__ = self.__isabstractmethod__
        _method.register = self.register
        update_wrapper(_method, self.func)
+        obj.__dict__[self.attrname] = _method
        return _method

    @property

    improves the performance noticeably

    0.9720976
0.7269078

    @CaselIT CaselIT mannequin added 3.8 only security fixes stdlib Python modules in the Lib dir performance Performance or resource usage labels Jun 15, 2020
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    @jun-he jun-he mentioned this issue Jun 14, 2022
    Merged
    @iritkatriel iritkatriel added 3.12 bugs and security fixes and removed 3.8 only security fixes labels Sep 9, 2022
    @iritkatriel
    Copy link
    Member

    I see a similar time difference on main.

    @eendebakpt eendebakpt mentioned this issue Jul 5, 2023
    Closed
    @AlexWaygood AlexWaygood added type-feature A feature request or enhancement and removed 3.12 bugs and security fixes labels Jul 5, 2023
    @bedevere-bot bedevere-bot mentioned this issue Jul 5, 2023
    Closed
    @eendebakpt eendebakpt mentioned this issue Jul 23, 2023
    Merged
    AlexWaygood added a commit that referenced this issue Aug 6, 2023
    @eendebakpt @mental32 @AlexWaygood
    gh-85160: improve performance of functools.singledispatchmethod (#1…
    3e334ae 
    …07148)

Co-authored-by: mental <m3nta1@yahoo.com>
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
    @AlexWaygood
    Copy link
    Member

    #107148 significantly optimised @singledispatchmethod for all non-slotted classes. If anybody can think of further optimisations that can be safely applied, I suggest a new issue should be opened for them.

    Thanks @casselt for the report, and thanks @mental32 and @eendebakpt for working on this!

    AlexWaygood added a commit to AlexWaygood/cpython that referenced this issue Aug 7, 2023
    @AlexWaygood
    pythongh-85160: Reduce memory usage of singledispatchmethod when in…
    bfce05c 
    …stances cannot be weakref'd
    @bedevere-bot bedevere-bot mentioned this issue Aug 7, 2023
    Merged
    AlexWaygood added a commit that referenced this issue Aug 7, 2023
    @AlexWaygood @serhiy-storchaka
    gh-85160: Reduce memory usage of singledispatchmethod (#107706)
    2ac103c 
    A small followup to #107148

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    performance Performance or resource usage stdlib Python modules in the Lib dir type-feature A feature request or enhancement
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    2 participants
    @iritkatriel @AlexWaygood