Message238364
Not only I'm too lazy to compute manually the number of loops and repeat, but also I don't trust myself. It's even worse when someone publishs results of a micro-benchmark. I don't trust how the benchmark was calibrated. In my experience, micro-benchmark are polluted by noise in timings, so results are not reliable.
benchmarks.py calibration is based on time, whereas timeit uses hardcoded constants (loops=1000000, repeat=3) which can be modified on the command line.
benchmarks.py has 3 main parameters:
- minimum duration of a single run (--min-time): 100 ms by default
- maximum total duration of the benchmark: benchmark.py does its best to respect this duration, but it can be longer: 1 second by default
- minimum repeat: 5 by default
The minimum duration is increased if the clock resolution is bad (1 ms or more). It's the case on Windows for time.clock() on Python 2 for example. Extract of benchmark.py:
min_time = max(self.config.min_time, timer_precision * 100)
The estimation of the number of loops is not reliable, but it's written to be "fast". Since I run a micro-benchmark many times, I don't want to wait too long. It's not a power of 10, but an arbitrary integer number. Usually, when running benchmark.py multiple times, the number of loops is different each time. It's not really a big issue, but it probably makes results more difficult to compare.
My constrain is max_time. The tested function may not have a linear duration (time = time_one_iteration * loops).
https://bitbucket.org/haypo/misc/src/348bfd6108e9985b3c2298d2745eb5ddfe7042e6/python/benchmark.py?at=default#cl-416
Repeat a test at least 5 times is a compromise between the stability of the result and the total duration of the benchmark.
Feel free to reuse my code to enhance time.py. |
|
Date |
User |
Action |
Args |
2015-03-17 23:53:27 | vstinner | set | recipients:
+ vstinner, rbcollins, serhiy.storchaka |
2015-03-17 23:53:27 | vstinner | set | messageid: <1426636407.9.0.0723239353891.issue23693@psf.upfronthosting.co.za> |
2015-03-17 23:53:27 | vstinner | link | issue23693 messages |
2015-03-17 23:53:27 | vstinner | create | |
|