This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author vstinner
Recipients methane, serhiy.storchaka, vstinner
Date 2017-01-16.15:53:48
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1484582030.41.0.271943634328.issue29259@psf.upfronthosting.co.za>
In-reply-to
Content
> Strange, I'm unable to reproduce the result on a different computer: (...)

Ah, the benchmark result change after a reboot. I reran the same benchmark (LTO+PGO on speed-python) after a reboot:
---
haypo@speed-python$ python3 -m perf compare_to ~/test/2017-01-13_19-18-default-3486425ed82b.json ~/patchdir/patch.json  -G 
Slower (29):
- spectral_norm: 275 ms +- 4 ms -> 301 ms +- 16 ms: 1.09x slower (+9%)
- fannkuch: 1.10 sec +- 0.01 sec -> 1.16 sec +- 0.18 sec: 1.06x slower (+6%)
- scimark_monte_carlo: 282 ms +- 3 ms -> 298 ms +- 6 ms: 1.06x slower (+6%)
- sqlite_synth: 9.39 us +- 0.19 us -> 9.82 us +- 0.15 us: 1.05x slower (+5%)
- nqueens: 261 ms +- 2 ms -> 271 ms +- 2 ms: 1.04x slower (+4%)
- meteor_contest: 209 ms +- 1 ms -> 217 ms +- 2 ms: 1.04x slower (+4%)
- scimark_fft: 684 ms +- 22 ms -> 709 ms +- 9 ms: 1.04x slower (+4%)
- pickle_pure_python: 1.35 ms +- 0.01 ms -> 1.39 ms +- 0.02 ms: 1.03x slower (+3%)
- xml_etree_iterparse: 227 ms +- 5 ms -> 233 ms +- 6 ms: 1.03x slower (+3%)
- sympy_integrate: 49.3 ms +- 0.3 ms -> 50.5 ms +- 0.3 ms: 1.02x slower (+2%)
- genshi_text: 90.3 ms +- 1.0 ms -> 92.4 ms +- 1.1 ms: 1.02x slower (+2%)
- regex_dna: 283 ms +- 1 ms -> 289 ms +- 1 ms: 1.02x slower (+2%)
- telco: 22.3 ms +- 0.5 ms -> 22.6 ms +- 0.3 ms: 1.02x slower (+2%)
- chaos: 295 ms +- 3 ms -> 300 ms +- 3 ms: 1.02x slower (+2%)
- go: 598 ms +- 5 ms -> 607 ms +- 7 ms: 1.01x slower (+1%)
- json_dumps: 30.5 ms +- 0.4 ms -> 30.9 ms +- 0.5 ms: 1.01x slower (+1%)
- richards: 183 ms +- 4 ms -> 186 ms +- 5 ms: 1.01x slower (+1%)
- deltablue: 17.1 ms +- 0.5 ms -> 17.4 ms +- 0.3 ms: 1.01x slower (+1%)
- logging_format: 42.1 us +- 0.6 us -> 42.5 us +- 0.6 us: 1.01x slower (+1%)
- sympy_str: 511 ms +- 4 ms -> 515 ms +- 5 ms: 1.01x slower (+1%)
- 2to3: 760 ms +- 4 ms -> 766 ms +- 4 ms: 1.01x slower (+1%)
- regex_compile: 462 ms +- 4 ms -> 465 ms +- 5 ms: 1.01x slower (+1%)
- django_template: 447 ms +- 5 ms -> 450 ms +- 4 ms: 1.01x slower (+1%)
- logging_simple: 35.6 us +- 0.5 us -> 35.8 us +- 0.6 us: 1.01x slower (+1%)
- hexiom: 23.0 ms +- 0.2 ms -> 23.2 ms +- 0.1 ms: 1.01x slower (+1%)
- python_startup: 16.0 ms +- 0.1 ms -> 16.1 ms +- 0.1 ms: 1.00x slower (+0%)
- regex_effbot: 5.08 ms +- 0.03 ms -> 5.10 ms +- 0.09 ms: 1.00x slower (+0%)
- call_method: 13.8 ms +- 0.1 ms -> 13.9 ms +- 0.2 ms: 1.00x slower (+0%)
- python_startup_no_site: 9.36 ms +- 0.04 ms -> 9.37 ms +- 0.07 ms: 1.00x slower (+0%)

Faster (19):
- scimark_lu: 518 ms +- 18 ms -> 458 ms +- 25 ms: 1.13x faster (-11%)
- xml_etree_process: 242 ms +- 4 ms -> 234 ms +- 4 ms: 1.03x faster (-3%)
- unpickle: 36.0 us +- 0.5 us -> 35.0 us +- 0.5 us: 1.03x faster (-3%)
- chameleon: 30.6 ms +- 0.2 ms -> 29.8 ms +- 0.3 ms: 1.03x faster (-3%)
- nbody: 240 ms +- 2 ms -> 234 ms +- 2 ms: 1.02x faster (-2%)
- call_method_slots: 13.9 ms +- 0.2 ms -> 13.6 ms +- 0.4 ms: 1.02x faster (-2%)
- unpickle_pure_python: 924 us +- 8 us -> 904 us +- 9 us: 1.02x faster (-2%)
- xml_etree_generate: 289 ms +- 4 ms -> 283 ms +- 4 ms: 1.02x faster (-2%)
- mako: 43.2 ms +- 0.4 ms -> 42.3 ms +- 0.4 ms: 1.02x faster (-2%)
- pickle_dict: 65.5 us +- 4.1 us -> 64.4 us +- 0.9 us: 1.02x faster (-2%)
- call_simple: 12.8 ms +- 0.4 ms -> 12.6 ms +- 0.4 ms: 1.01x faster (-1%)
- xml_etree_parse: 294 ms +- 13 ms -> 291 ms +- 9 ms: 1.01x faster (-1%)
- pickle_list: 9.38 us +- 0.35 us -> 9.32 us +- 0.14 us: 1.01x faster (-1%)
- unpack_sequence: 118 ns +- 3 ns -> 118 ns +- 1 ns: 1.00x faster (-0%)
- sympy_expand: 1.14 sec +- 0.01 sec -> 1.13 sec +- 0.01 sec: 1.00x faster (-0%)
- sqlalchemy_declarative: 338 ms +- 3 ms -> 337 ms +- 3 ms: 1.00x faster (-0%)
- crypto_pyaes: 236 ms +- 2 ms -> 235 ms +- 1 ms: 1.00x faster (-0%)
- pidigits: 321 ms +- 1 ms -> 320 ms +- 1 ms: 1.00x faster (-0%)
- regex_v8: 45.6 ms +- 0.7 ms -> 45.6 ms +- 0.3 ms: 1.00x faster (-0%)

Benchmark hidden because not significant (16): call_method_unknown, dulwich_log, float, genshi_xml, html5lib, json_loads, logging_silent, pathlib, pickle, raytrace, scimark_sor, scimark_sparse_mat_mult, sqlalchemy_imperative, sympy_sum, tornado_http, unpickle_list
---

Lighter verbose with a threshold of 5%:
---
haypo@speed-python$ python3 -m perf compare_to ~/test/2017-01-13_19-18-default-3486425ed82b.json ~/patchdir/patch.json  -G --min-speed=5
Slower (3):
- spectral_norm: 275 ms +- 4 ms -> 301 ms +- 16 ms: 1.09x slower (+9%)
- fannkuch: 1.10 sec +- 0.01 sec -> 1.16 sec +- 0.18 sec: 1.06x slower (+6%)
- scimark_monte_carlo: 282 ms +- 3 ms -> 298 ms +- 6 ms: 1.06x slower (+6%)

Faster (1):
- scimark_lu: 518 ms +- 18 ms -> 458 ms +- 25 ms: 1.13x faster (-11%)

Benchmark hidden because not significant (60): 2to3, call_method, ...
---

spectral_norm and scimark_lu are two unstable benchmarks. Sadly, it seems like the current patch is not really significant on benchmarks. But again, it doesn't optimize __call__() slot yet.
History
Date User Action Args
2017-01-16 15:53:51vstinnersetrecipients: + vstinner, methane, serhiy.storchaka
2017-01-16 15:53:50vstinnersetmessageid: <1484582030.41.0.271943634328.issue29259@psf.upfronthosting.co.za>
2017-01-16 15:53:50vstinnerlinkissue29259 messages
2017-01-16 15:53:48vstinnercreate