classification
Title: Improve benchmarks precision
Type: enhancement Stage: resolved
Components: Benchmarks Versions: Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: brett.cannon, georg.brandl, pitrou, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2014-10-25 19:34 by pitrou, last changed 2014-10-25 23:13 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
precision.patch pitrou, 2014-10-25 19:34
Messages (5)
msg230011 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-10-25 19:34
This patch tries to improve precision of benchmarks in the benchmark suite by two measures:
- select the best timer for the pair of interpreters (i.e. perf_counter() if possible)
- make hashing deterministic to avoid fluctuations between runs
msg230013 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2014-10-25 20:05
LGTM.
msg230014 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-10-25 20:22
Pushed in dc7d29be5a9e.
msg230019 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-10-25 21:48
For more precise results it may be worth to run interpreters several times with random hash seed and then select the best time. Different interpreters can have different effect on the same seed.
msg230020 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-10-25 23:13
You thereotically need 2**63 runs to select the best answer, though. You don't know how the distribution of hash seeds can influence the results, which means mere sampling may not be sufficient.
History
Date User Action Args
2014-10-25 23:13:56pitrousetmessages: + msg230020
2014-10-25 21:48:42serhiy.storchakasetmessages: + msg230019
2014-10-25 20:22:32pitrousetstatus: open -> closed
resolution: fixed
messages: + msg230014

stage: patch review -> resolved
2014-10-25 20:05:58georg.brandlsetnosy: + georg.brandl
messages: + msg230013
2014-10-25 19:34:32pitroucreate