This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: Improve benchmarks precision
Type: enhancement Stage: resolved
Components: Benchmarks Versions: Python 3.5
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: brett.cannon, georg.brandl, pitrou, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2014-10-25 19:34 by pitrou, last changed 2022-04-11 14:58 by admin. This issue is now closed.

File name Uploaded Description Edit
precision.patch pitrou, 2014-10-25 19:34
Messages (5)
msg230011 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-10-25 19:34
This patch tries to improve precision of benchmarks in the benchmark suite by two measures:
- select the best timer for the pair of interpreters (i.e. perf_counter() if possible)
- make hashing deterministic to avoid fluctuations between runs
msg230013 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2014-10-25 20:05
msg230014 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-10-25 20:22
Pushed in dc7d29be5a9e.
msg230019 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-10-25 21:48
For more precise results it may be worth to run interpreters several times with random hash seed and then select the best time. Different interpreters can have different effect on the same seed.
msg230020 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-10-25 23:13
You thereotically need 2**63 runs to select the best answer, though. You don't know how the distribution of hash seeds can influence the results, which means mere sampling may not be sufficient.
Date User Action Args
2022-04-11 14:58:09adminsetgithub: 66916
2014-10-25 23:13:56pitrousetmessages: + msg230020
2014-10-25 21:48:42serhiy.storchakasetmessages: + msg230019
2014-10-25 20:22:32pitrousetstatus: open -> closed
resolution: fixed
messages: + msg230014

stage: patch review -> resolved
2014-10-25 20:05:58georg.brandlsetnosy: + georg.brandl
messages: + msg230013
2014-10-25 19:34:32pitroucreate