This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: speed up searching for keywords by using a dictionary
Type: performance Stage: resolved
Components: Interpreter Core Versions: Python 3.11
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: Dennis Sweeney, eric.smith, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2021-05-17 20:43 by Dennis Sweeney, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 26200 closed Dennis Sweeney, 2021-05-17 20:43
Messages (5)
msg393828 - (view) Author: Dennis Sweeney (Dennis Sweeney) * (Python committer) Date: 2021-05-17 20:43
This patch ensures keyword-passing happens in linear time, fulfilling the 26-year-old TODO comment in ceval.c:

   XXX speed up searching for keywords by using a dictionary


from time import perf_counter
from itertools import repeat

def bench(N):
    reps = 1 + 10_000_000 // N
    kwargs = {"a"*i: None for i in range(1, N+1)}
    param_list = ",".join(kwargs)
    function = eval(f"lambda {param_list}: 17")
    t0 = perf_counter()
    for _ in repeat(None, reps):
        result = function(**kwargs)
    t1 = perf_counter()
    assert result == 17
    return (t1 - t0) / reps

for i in list(range(1, 11)) + [20, 50, 100, 200, 500, 1000]:
    print(f"{i} --> {bench(i):.2e}")


before:
1 --> 1.73e-07
2 --> 2.22e-07
3 --> 2.57e-07
4 --> 3.25e-07
5 --> 3.93e-07
6 --> 4.87e-07
7 --> 5.92e-07
8 --> 7.02e-07
9 --> 8.40e-07
10 --> 9.59e-07
20 --> 3.11e-06
50 --> 1.49e-05
100 --> 5.61e-05
200 --> 2.21e-04
500 --> 1.29e-03
1000 --> 5.09e-03

after:
1 --> 1.78e-07
2 --> 2.14e-07
3 --> 2.37e-07
4 --> 2.63e-07
5 --> 2.87e-07
6 --> 3.31e-07
7 --> 3.41e-07
8 --> 3.66e-07
9 --> 3.84e-07
10 --> 4.17e-07
20 --> 8.17e-07
50 --> 1.77e-06
100 --> 3.66e-06
200 --> 8.10e-06
500 --> 2.82e-05
1000 --> 7.95e-05
msg393853 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-05-18 06:17
Please repeat benchmarking for interned names:

    kwargs = {sys.intern("a"*i): None for i in range(1, N+1)}

and for more distinctive names:

    kwargs = {sys.intern(f"a{i}"): None for i in range(1, N+1)}

I guess the difference will be lesser.
msg393854 - (view) Author: Dennis Sweeney (Dennis Sweeney) * (Python committer) Date: 2021-05-18 06:29
Good catch -- with interning, the cutoff is more like 20-50, so it's probably not worth optimizing for.

Before:
1 --> 1.67e-07
2 --> 1.77e-07
3 --> 1.90e-07
4 --> 2.05e-07
5 --> 2.14e-07
6 --> 2.35e-07
7 --> 2.51e-07
8 --> 2.67e-07
9 --> 2.76e-07
10 --> 3.01e-07
20 --> 5.43e-07
50 --> 1.58e-06
100 --> 4.35e-06
200 --> 1.19e-05
500 --> 5.11e-05
1000 --> 1.64e-04

After:
1 --> 1.79e-07
2 --> 2.07e-07
3 --> 2.22e-07
4 --> 2.35e-07
5 --> 2.65e-07
6 --> 2.89e-07
7 --> 3.02e-07
9 --> 3.45e-07
10 --> 4.15e-07
20 --> 6.69e-07
50 --> 1.45e-06
100 --> 2.99e-06
200 --> 5.64e-06
500 --> 1.42e-05
1000 --> 2.96e-05
msg393855 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2021-05-18 06:44
Maybe the XXX comment in ceval.c should be removed? If we wouldn't change the code to use a dictionary, then that comment seems incorrect.
msg393864 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-05-18 09:35
BTW, we use different approaches for builtin and Python functions.

* For Python functions: iterate all keyword arguments and search in the list of keyword parameter names.
* For builtin functions: iterate all keyword parameters and search in the list of keyword argument names (or in a dict if var-keyword arguments are passed).
History
Date User Action Args
2022-04-11 14:59:45adminsetgithub: 88326
2021-05-18 09:35:25serhiy.storchakasetmessages: + msg393864
2021-05-18 06:44:11eric.smithsetnosy: + eric.smith
messages: + msg393855
2021-05-18 06:29:34Dennis Sweeneysetstatus: open -> closed
resolution: wont fix
messages: + msg393854

stage: patch review -> resolved
2021-05-18 06:17:23serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg393853
2021-05-17 20:43:37Dennis Sweeneysetkeywords: + patch
stage: patch review
pull_requests: + pull_request24817
2021-05-17 20:43:20Dennis Sweeneycreate