This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Could operator.methodcaller be optimized using LOAD_METHOD?
Type: performance Stage: patch review
Components: Library (Lib) Versions: Python 3.11
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Antony.Lee, Dennis Sweeney, Mark.Shannon
Priority: normal Keywords: patch

Created on 2021-08-06 10:42 by Antony.Lee, last changed 2022-04-11 14:59 by admin.

Pull Requests
URL Status Linked Edit
PR 27782 open Antony.Lee, 2021-08-16 15:59
Messages (4)
msg399066 - (view) Author: Antony Lee (Antony.Lee) * Date: 2021-08-06 10:42
Currently, methodcaller is not faster than a plain lambda:
```
In [1]: class T:
   ...:     a = 1
   ...:     def f(self): pass
   ...:     

In [2]: from operator import *

In [3]: %%timeit t = T(); mc = methodcaller("f")
   ...: mc(t)
   ...: 
   ...: 
83.1 ns ± 0.862 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

In [4]: %%timeit t = T(); mc = lambda x: x.f()
   ...: mc(t)
   ...: 
   ...: 
81.4 ns ± 0.0508 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)
```
(on some machines, I find that it is even slower).

Compare with attrgetter, which *is* faster:
```
In [5]: %%timeit t = T(); ag = attrgetter("a")
   ...: ag(t)
   ...: 
   ...: 
33.7 ns ± 0.0407 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

In [6]: %%timeit t = T(); ag = lambda x: x.a
   ...: ag(t)
   ...: 
   ...: 
50.1 ns ± 0.057 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)
```

Given that the operator module explicitly advertises itself as being "efficient"/"fast", it seems reasonable to try to optimize methodcaller.  Looking at its C implementation, methodcaller currently uses PyObject_GetAttr followed by PyObject_Call; I wonder whether this can be optimized using a LOAD_METHOD-style approach to avoid the construction of the bound method (when applicable)?
msg399178 - (view) Author: Dennis Sweeney (Dennis Sweeney) * (Python committer) Date: 2021-08-07 12:50
Using _PyObject_GetMethod similarly to the way that LOAD_METHOD/CALL_METHOD does seems like a reasonable idea to me -- do you want to make a pull request? It would also be nice to see some microbenchmarks for the change once it's ready.
msg399184 - (view) Author: Dennis Sweeney (Dennis Sweeney) * (Python committer) Date: 2021-08-07 14:16
For what it's worth, in my benchmarks on 3.11, methodcaller was already a bit faster than lambda:

#################### Builtin calls ####################

PS > .\python.bat -m pyperf timeit -s "from operator import methodcaller as mc" -s "reverse_it = mc('reverse')" -s "arr = []" "reverse_it(arr)"
Running Release|x64 interpreter...
.....................
Mean +- std dev: 84.3 ns +- 1.9 ns

PS >.\python.bat -m pyperf timeit -s "reverse_it = lambda x: x.reverse()" -s "arr = []" "reverse_it(arr)"
Running Release|x64 interpreter...
.....................
Mean +- std dev: 95.5 ns +- 2.9 ns

#################### Python calls ####################

PS > .\python.bat -m pyperf timeit -s "from operator import methodcaller as mc" -s "reverse_it = mc('reverse')" -s "class A: reverse = lambda self: None" -s "arr=A()" "reverse_it(arr)"
Running Release|x64 interpreter...
.....................
Mean +- std dev: 140 ns +- 4 ns
PS > .\python.bat -m pyperf timeit -s "reverse_it = lambda x: x.reverse()" -s "class A: reverse = lambda self: None" -s "arr=A()" "reverse_it(arr)"
Running Release|x64 interpreter...
.....................
Mean +- std dev: 159 ns +- 4 ns
msg399903 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2021-08-19 10:07
If the problem is that methodcaller is not faster than a lambda, then why not use a lambda?

Lambdas are just Python functions and calling them will get faster.

We aren't going to spend time optimizing calls to methodcaller, so you might as well use a lambda.
History
Date User Action Args
2022-04-11 14:59:48adminsetgithub: 89013
2021-08-19 10:07:47Mark.Shannonsetnosy: + Mark.Shannon
messages: + msg399903
2021-08-16 15:59:26Antony.Leesetkeywords: + patch
stage: patch review
pull_requests: + pull_request26252
2021-08-07 14:16:03Dennis Sweeneysetmessages: + msg399184
2021-08-07 12:50:43Dennis Sweeneysetversions: + Python 3.11
nosy: + Dennis Sweeney

messages: + msg399178

type: performance
2021-08-06 10:42:14Antony.Leecreate