This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author vstinner
Recipients rbcollins, serhiy.storchaka, vstinner
Date 2015-03-17.23:53:27
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <>
Not only I'm too lazy to compute manually the number of loops and repeat, but also I don't trust myself. It's even worse when someone publishs results of a micro-benchmark. I don't trust how the benchmark was calibrated. In my experience, micro-benchmark are polluted by noise in timings, so results are not reliable. calibration is based on time, whereas timeit uses hardcoded constants (loops=1000000, repeat=3) which can be modified on the command line. has 3 main parameters:

- minimum duration of a single run (--min-time): 100 ms by default
- maximum total duration of the benchmark: does its best to respect this duration, but it can be longer: 1 second by default
- minimum repeat: 5 by default

The minimum duration is increased if the clock resolution is bad (1 ms or more). It's the case on Windows for time.clock() on Python 2 for example. Extract of

    min_time = max(self.config.min_time, timer_precision * 100)

The estimation of the number of loops is not reliable, but it's written to be "fast". Since I run a micro-benchmark many times, I don't want to wait too long. It's not a power of 10, but an arbitrary integer number. Usually, when running multiple times, the number of loops is different each time. It's not really a big issue, but it probably makes results more difficult to compare.

My constrain is max_time. The tested function may not have a linear duration (time = time_one_iteration * loops).

Repeat a test at least 5 times is a compromise between the stability of the result and the total duration of the benchmark.

Feel free to reuse my code to enhance
Date User Action Args
2015-03-17 23:53:27vstinnersetrecipients: + vstinner, rbcollins, serhiy.storchaka
2015-03-17 23:53:27vstinnersetmessageid: <>
2015-03-17 23:53:27vstinnerlinkissue23693 messages
2015-03-17 23:53:27vstinnercreate