classification
Title: Speed up hasattr(o, name) and getattr(o, name, default)
Type: performance Stage: resolved
Components: Interpreter Core Versions: Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: inada.naoki, levkivskyi, serhiy.storchaka, vstinner
Priority: normal Keywords: patch

Created on 2018-01-13 18:29 by inada.naoki, last changed 2018-01-16 15:42 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 5173 merged inada.naoki, 2018-01-13 18:31
Messages (6)
msg309896 - (view) Author: INADA Naoki (inada.naoki) * (Python committer) Date: 2018-01-13 18:29
ABCMeta.__new__ calls `getattr(value, "__isabstractmethod__", False)` many times.
https://github.com/python/cpython/blob/0f31c74fcfdec8f9e6157de2c366f2273de81677/Lib/abc.py#L135-L142

Since metaclass is used by subclasses, it checks all members of all subclasses (including concrete class).

But getattr() and hasattr() is inefficient when attribution is not found,
because it raises AttributeError and clear it internally.

I tried to bypass AttributeError when tp_getattro == PyObject_GenericGetAttr.
Here is quick micro benchmark:

$ ./python-patched -m perf timeit --compare-to=`pwd`/python-master -s 'def foo(): pass' 'hasattr(foo, "__isabstractmethod__")'python-master: ..................... 385 ns +- 2 ns
python-patched: ..................... 87.6 ns +- 1.6 ns

Mean +- std dev: [python-master] 385 ns +- 2 ns -> [python-patched] 87.6 ns +- 1.6 ns: 4.40x faster (-77%)

(I added Ivan to nosy list because he may implement C version of ABC.)
msg309991 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-01-15 15:23
I considered the idea of implementing _PyObject_GetAttrWithoutError() for issue31572, but dropped it because the code simplification was too small. But if this has performance benefits I'm +1 to implementing it. I'll update my still open PRs to use this API.
msg310049 - (view) Author: INADA Naoki (inada.naoki) * (Python committer) Date: 2018-01-16 08:51
I run pyperformance.
It seems django_template is significant faster.

# Without PGO+LTO

./python -m perf compare_to -G --min-speed=2 default.json patched.json
Slower (6):
- scimark_monte_carlo: 217 ms +- 9 ms -> 237 ms +- 10 ms: 1.09x slower (+9%)
- scimark_lu: 326 ms +- 14 ms -> 351 ms +- 20 ms: 1.08x slower (+8%)
- float: 219 ms +- 4 ms -> 226 ms +- 8 ms: 1.03x slower (+3%)
- crypto_pyaes: 208 ms +- 2 ms -> 213 ms +- 2 ms: 1.02x slower (+2%)
- pickle: 20.1 us +- 0.2 us -> 20.6 us +- 0.4 us: 1.02x slower (+2%)
- xml_etree_iterparse: 184 ms +- 3 ms -> 188 ms +- 3 ms: 1.02x slower (+2%)

Faster (10):
- django_template: 315 ms +- 4 ms -> 287 ms +- 3 ms: 1.10x faster (-9%)
- logging_format: 25.9 us +- 0.3 us -> 24.7 us +- 0.3 us: 1.05x faster (-5%)
- logging_simple: 21.6 us +- 0.2 us -> 20.6 us +- 0.4 us: 1.05x faster (-5%)
- sympy_str: 399 ms +- 3 ms -> 381 ms +- 2 ms: 1.05x faster (-5%)
- unpack_sequence: 117 ns +- 6 ns -> 112 ns +- 0 ns: 1.04x faster (-4%)
- sympy_expand: 886 ms +- 7 ms -> 851 ms +- 7 ms: 1.04x faster (-4%)
- regex_effbot: 5.00 ms +- 0.09 ms -> 4.84 ms +- 0.11 ms: 1.03x faster (-3%)
- xml_etree_process: 185 ms +- 3 ms -> 180 ms +- 4 ms: 1.02x faster (-2%)
- nqueens: 215 ms +- 3 ms -> 211 ms +- 3 ms: 1.02x faster (-2%)
- telco: 16.0 ms +- 0.6 ms -> 15.6 ms +- 0.4 ms: 1.02x faster (-2%)

Benchmark hidden because not significant (44)

# With PGO+LTO

$ ./python -m perf compare_to -G --min-speed=2 default-opt.json patched-opt.json
Slower (8):
- xml_etree_process: 179 ms +- 3 ms -> 195 ms +- 3 ms: 1.09x slower (+9%)
- xml_etree_generate: 208 ms +- 5 ms -> 224 ms +- 3 ms: 1.08x slower (+8%)
- xml_etree_iterparse: 190 ms +- 5 ms -> 203 ms +- 3 ms: 1.07x slower (+7%)
- pickle_list: 6.80 us +- 0.30 us -> 7.18 us +- 0.72 us: 1.06x slower (+6%)
- genshi_text: 72.8 ms +- 1.1 ms -> 76.3 ms +- 0.8 ms: 1.05x slower (+5%)
- mako: 32.6 ms +- 0.1 ms -> 34.1 ms +- 0.6 ms: 1.05x slower (+5%)
- unpickle_pure_python: 713 us +- 12 us -> 744 us +- 16 us: 1.04x slower (+4%)
- scimark_sor: 375 ms +- 13 ms -> 383 ms +- 11 ms: 1.02x slower (+2%)

Faster (14):
- django_template: 327 ms +- 3 ms -> 296 ms +- 2 ms: 1.10x faster (-9%)
- scimark_lu: 339 ms +- 20 ms -> 309 ms +- 13 ms: 1.10x faster (-9%)
- sympy_str: 418 ms +- 4 ms -> 387 ms +- 3 ms: 1.08x faster (-7%)
- sympy_expand: 926 ms +- 5 ms -> 860 ms +- 5 ms: 1.08x faster (-7%)
- logging_simple: 22.4 us +- 0.3 us -> 21.0 us +- 0.3 us: 1.07x faster (-6%)
- scimark_fft: 741 ms +- 61 ms -> 697 ms +- 25 ms: 1.06x faster (-6%)
- telco: 17.6 ms +- 0.4 ms -> 16.6 ms +- 0.4 ms: 1.06x faster (-5%)
- logging_format: 26.5 us +- 0.3 us -> 25.3 us +- 0.3 us: 1.05x faster (-5%)
- sqlite_synth: 7.35 us +- 0.16 us -> 7.08 us +- 0.18 us: 1.04x faster (-4%)
- scimark_sparse_mat_mult: 8.83 ms +- 0.42 ms -> 8.51 ms +- 0.37 ms: 1.04x faster (-4%)
- pathlib: 43.1 ms +- 0.6 ms -> 41.7 ms +- 0.7 ms: 1.03x faster (-3%)
- html5lib: 193 ms +- 7 ms -> 188 ms +- 7 ms: 1.03x faster (-3%)
- richards: 149 ms +- 3 ms -> 145 ms +- 2 ms: 1.02x faster (-2%)
- pickle_pure_python: 1.01 ms +- 0.01 ms -> 984 us +- 20 us: 1.02x faster (-2%)

Benchmark hidden because not significant (38)
msg310084 - (view) Author: INADA Naoki (inada.naoki) * (Python committer) Date: 2018-01-16 11:52
New changeset 378edee0a3b913d60653dc17dfe61d83405a8135 by INADA Naoki in branch 'master':
bpo-32544: Speed up hasattr() and getattr() (GH-5173)
https://github.com/python/cpython/commit/378edee0a3b913d60653dc17dfe61d83405a8135
msg310085 - (view) Author: INADA Naoki (inada.naoki) * (Python committer) Date: 2018-01-16 11:58
I confirmed django_template is `hasattr`-heavy benchmark.

Function                            was called by...
                                        ncalls  tottime  cumtime
{built-in method builtins.hasattr}  <-       1    0.000    0.000  /home/inada-n/local/py37/lib/python3.7/site-packages/django/apps/registry.py:20(__init__)
...
                                         16318    0.003    0.003  /home/inada-n/local/py37/lib/python3.7/site-packages/django/utils/functional.py:81(__prepare_class__)
                                         90000    0.065    0.065  /home/inada-n/local/py37/lib/python3.7/site-packages/django/utils/html.py:79(conditional_escape)
                                         93200    0.064    0.064  /home/inada-n/local/py37/lib/python3.7/site-packages/django/utils/safestring.py:129(mark_safe)
                                         90000    0.075    0.075  /home/inada-n/local/py37/lib/python3.7/site-packages/django/utils/safestring.py:149(mark_for_escaping)
msg310090 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-01-16 15:42
Cool. I always wanted to implement something similar. Nice work Naoki!
History
Date User Action Args
2018-01-16 15:42:10vstinnersetnosy: + vstinner
messages: + msg310090
2018-01-16 11:58:04inada.naokisetmessages: + msg310085
2018-01-16 11:53:07inada.naokisetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2018-01-16 11:52:47inada.naokisetmessages: + msg310084
2018-01-16 08:51:23inada.naokisetmessages: + msg310049
2018-01-15 15:23:32serhiy.storchakasetversions: + Python 3.7
nosy: + serhiy.storchaka

messages: + msg309991

components: + Interpreter Core
type: performance
2018-01-13 18:31:06inada.naokisetkeywords: + patch
stage: patch review
pull_requests: + pull_request5026
2018-01-13 18:29:29inada.naokicreate