Author serhiy.storchaka
Recipients brett.cannon, fijall, ned.deily, pitrou, serhiy.storchaka, steven.daprano, tim.peters, vstinner
Date 2016-09-21.15:17:35
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1474471055.77.0.613588138736.issue28240@psf.upfronthosting.co.za>
In-reply-to
Content
> * Display the average, rather than the minimum, of the timings *and* display the standard deviation. It should help a little bit to get more reproductible results.

This makes hard to compare results with older Python versions.

> * Change the default repeat from 3 to 5 to have a better distribution of timings. It makes the timeit CLI 66% slower (ex: 1 second instead of 600 ms). That's the price of stable benchmarks :-)

For now default timeit run takes from 0.8 to 8 sec. Adding yet 5 sec makes a user more angry.

> * Don't disable the garbage collector anymore! Disabling the GC is not fair: real applications use it.

But this makes short microbenchmarks less stable.

> * autorange: start with 1 loop instead of 10 for slow benchmarks like time.sleep(1)

This is good if you run relatively slow benchmark, but it makes the result less reliable. You always can specify -n1, but on your own risk.

> * Display large number of loops as power of 10 for readability, ex: "10^6" instead of "1000000". Also accept "10^6" syntax for the --num parameter.

10^6 syntax doesn't look Pythonic. And this change breaks third-party scripts that run timeit.

> * Add support for "ns" unit: nanoseconds (10^-9 second)

Even "pass" takes at least 0.02 usec on my computer. What you want to measure that takes < 1 ns? I think timeit is just wrong tool for this.

The patch also makes a warning about unreliable results output to stdout and always visible. This is yet one compatibility break. Current code allows the user to control the visibility of the warning by the -W Python option, and don't mix the warning with result output.
History
Date User Action Args
2016-09-21 15:17:35serhiy.storchakasetrecipients: + serhiy.storchaka, tim.peters, brett.cannon, pitrou, vstinner, ned.deily, steven.daprano, fijall
2016-09-21 15:17:35serhiy.storchakasetmessageid: <1474471055.77.0.613588138736.issue28240@psf.upfronthosting.co.za>
2016-09-21 15:17:35serhiy.storchakalinkissue28240 messages
2016-09-21 15:17:35serhiy.storchakacreate