classification
Title: timeit called from within Python should allow autoranging
Type: enhancement Stage: patch review
Components: Library (Lib) Versions: Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: steven.daprano Nosy List: amaury.forgeotdarc, asvetlov, haypo, merwok, ncoghlan, pitrou, python-dev, rbcollins, rhettinger, scott_daniels, steven.daprano, tshepang
Priority: normal Keywords: patch

Created on 2009-07-05 15:15 by scott_daniels, last changed 2016-08-15 01:15 by steven.daprano.

Files
File name Uploaded Description Edit
timeit.patch scott_daniels, 2009-07-08 16:19 Patch including changes to timeit, test, and doc
timeit-autorange.patch steven.daprano, 2016-05-11 15:28 minimal proof-of-concept patch to add autorange method to Timer review
Messages (26)
msg90157 - (view) Author: Scott David Daniels (scott_daniels) * Date: 2009-07-05 15:15
timeit.main has a _very_ handy autoranging facility to pick an
appropriate number of repetitions when not specified.  The autoranging
code should be lifted to a method on Timer instances (so non-main code
can use it).  If number is specified as 0 or None, I would like to use
the results of that autoranging code in Timer.repeat and Timer.timeit.

Patch to follow.
msg90245 - (view) Author: Scott David Daniels (scott_daniels) * Date: 2009-07-07 21:29
I've got the code "working" on trunk2 for my tests.
Should I port to py3K before checking in, and give diffs from there, or 
what?
msg90264 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2009-07-08 12:01
You can still upload available patches to this tracker.
msg90275 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2009-07-08 17:05
I would like to look at this in context of all the other proposed build-
outs to timeit.
msg122955 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2010-11-30 23:33
This does not conflict with the other proposed changes to timeit and it is in-line with Guido's desire that to expose useful parts currently buried in the command-line logic.

Amaury, you've shown an interest.  Would you like to apply it?
msg123711 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-12-09 23:27
Not sure why you chose 0.11 here. It should probably be 0.2 as in the command-line code.
As for applying the patch, this can't be done before 3.2 is released.
msg164027 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2012-06-26 02:40
Looking at this again after more time has passes, I still think exposing autoranging is a good idea but I don't like the patch as it stands.  It "infects" the API in a number of places and makes the overall module harder to use and learn.   

Ideally, there should be a cleaner interface, or more limited API change, or a separate high level function that can autorange existing functions without changing their API.

Anyone care to propose a cleaner API?
msg164070 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-06-26 12:37
In #5442, I proposed leaving the architecture of the module alone, and simply exposing the main module functionality as a high level helper function:

def measure(stmt="pass", setup="pass", timer=default_timer,
            repeat=default_repeat, number=default_number,
            verbosity=0, precision=3)

The return value would simply be a (number, results) 2-tuple with the number of iterations per test (which may have been calculated automatically), and then a list of the results. To get "timeit" style behavior, simply set "repeat=1".
msg164071 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-06-26 12:39
Oops, that link reference should have been to #5441.
msg164216 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2012-06-28 00:46
Hi, I wrote recently a similar function because timeit is not reliable by default. Results look random and require to run the same benchmark 3 times or more on the command line.

https://bitbucket.org/haypo/misc/src/tip/python/benchmark.py

By default, the benchmark takes at least 5 measures, one measure should be greater than 100 ms, and the benchmark should not be longer than 1 second. I chose these parameters to get reliable results on microbenchmarks like "abc".encode("utf-8").

The calibration function uses also the precision of the timer. The user may define a minimum time (of one measure) smaller than the timer precision, so the calibration function tries to solve such issue. The calibration computes the number of loops and the number of repetitions.

Look at BenchmarkRunner.calibrate_timer() and BenchmarkRunner.run_benchmark().
https://bitbucket.org/haypo/misc/src/bfacfb9a1224/python/benchmark.py#cl-362
msg164217 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2012-06-28 00:55
> The calibration function uses also the precision of the timer.

Oh, I forgot to mention that it computes the precision in Python, it doesn't read the precision announced by the OS or the precision of the C structure.
https://bitbucket.org/haypo/misc/src/bfacfb9a1224/python/benchmark.py#cl-66
msg168796 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-08-21 18:47
> In #5442, I proposed leaving the architecture of the module alone, and 
> simply exposing the main module functionality as a high level helper
> function:

Agreed with Nick's approach above.

Victor, if you want to improve timeit's reliability, please open a separate issue.
msg238352 - (view) Author: Robert Collins (rbcollins) * (Python committer) Date: 2015-03-17 22:19
I'm confused by the feedback on the patch. It adds a single new function, doesn't alter the public interface for any existing functions, and seems fit for purpose. Could someone help me understand how its deficient?
msg238354 - (view) Author: Robert Collins (rbcollins) * (Python committer) Date: 2015-03-17 22:35
Filed #23693 for the accuracy thing.
msg238879 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2015-03-22 06:54
The current patch moves print operations inside timeit() and repeat(), instead of leaving the main() function as the only one with side effects.

My counter-proposal was to instead extract the current main functionality out into a side-effect free public API of its own, and change the existing main function to call that new API and print the results.
msg265319 - (view) Author: Steven D'Aprano (steven.daprano) * (Python committer) Date: 2016-05-11 15:28
This issue seems to have lost momentum, I'd like to revive it by proposing a slightly different interface for the autorange function.

Attached is a proof-of-concept patch. I've moved the code which determines the number of loops out of the main function into a new Timer method, ``autorange``. The main difference between my approach and Scott's is that my autorange method doesn't do any printing directly, it takes an optional callback function. This lets the caller take responsibility for all output.

If this approach is acceptable, I hope to:

- (in 3.6) add tests and docs for the new method;

- (in 3.6 if time permits, otherwise 3.7) modify the timeit and repeat methods and functions so that they can optionally call autorange(), e.g. if the caller passes 0 as the number.

I'm not sure that there's a good reason to add a top-level autorange() function to match the timeit() and repeat() functions. Especially not once they gain the ability to autorange themselves.

I think my approach will be compatible with cleaning up and refactoring the main() function. At the moment, main() is a mess IMO, it handles argument processing, autoranging, units of time, and unreliable timing detection all from one function.
msg265322 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2016-05-11 15:50
I would suggest making the 0.2 tunable as an optional argument. Different applications (benchmarks) may want different duration / precision tradeoffs.
I also notice the repeat functionality isn't included in the patch, is there a reason?
msg265324 - (view) Author: Steven D'Aprano (steven.daprano) * (Python committer) Date: 2016-05-11 16:08
> I would suggest making the 0.2 tunable as an optional argument.

Sounds like a good idea to me.

> I also notice the repeat functionality isn't included in the patch, is there a reason?

I don't understand which repeat functionality you're referring to.
msg265326 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2016-05-11 16:46
> I don't understand which repeat functionality you're referring to.

https://docs.python.org/3/library/timeit.html#timeit.Timer.repeat

(or, similarly, what timeit's __main__ does: report the minimum of all N runs)
msg265360 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2016-05-12 04:49
The embedded side-effects were my main concern with Scott's original patch, so Steven's callback-based approach strikes me as a definite improvement. However, the awkwardness of the revised calling code in main does make me wonder whether or not this might be better implemented as a generator rather than as a function accepting a callback:

    try:
        results = list(t.autorange())
    except:
        t.print_exc()
        return 1
    if verbose:
        for number, time_taken in results:
            msg = "{} loops -> {:.{}g} secs"
            print(msg.format(number, time_taken, precision))

(Originally I had the "if verbose: print" embedded in a direct loop over t.autorange(), but writing it that way made it immediately clear that the scope of the exception handler was too broad, so I changed it to extract all the results and only then print them)
msg265418 - (view) Author: Steven D'Aprano (steven.daprano) * (Python committer) Date: 2016-05-12 16:49
On Thu, May 12, 2016 at 04:49:59AM +0000, Nick Coghlan wrote:
> The embedded side-effects were my main concern with Scott's original 
> patch, so Steven's callback-based approach strikes me as a definite 
> improvement. However, the awkwardness of the revised calling code in 
> main does make me wonder whether or not this might be better 
> implemented as a generator rather than as a function accepting a 
> callback:

I thought about a generator too, but then I thought about the *non* 
verbose case, where you don't care about the intermediate results, only 
the final (number, time_taken) pair.

# function with callback:
number, time_taken = t.autorange()

# generator
number, time_taken = list(t.autorange())[-1]

Which hints that your code snippet is buggy, or at least incomplete:

>     try:
>         results = list(t.autorange())
>     except:
>         t.print_exc()
>         return 1
>     if verbose:
>         for number, time_taken in results:
>             msg = "{} loops -> {:.{}g} secs"
>             print(msg.format(number, time_taken, precision))

If verbose is False, you never set number and time_taken. So you need an 
else clause:

    else:
        number, time_taken = results[-1]
msg265442 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2016-05-13 03:46
Good point - given that, +1 from me for the callback based version, especially since exception chaining will still disambiguate failures in the callback from other errors.
msg272140 - (view) Author: Steven D'Aprano (steven.daprano) * (Python committer) Date: 2016-08-08 00:59
Nick gave a +1 to my auto-range patch with callback on 2016-05-13, and there's been no negative feedback since. Should I go ahead and check it in for 3.6?
msg272142 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2016-08-08 02:52
I think the patch is good to go.
msg272676 - (view) Author: Roundup Robot (python-dev) Date: 2016-08-14 15:27
New changeset 424eb46f7f3a by Steven D'Aprano in branch 'default':
Issue6422 add autorange method to timeit.Timer
https://hg.python.org/cpython/rev/424eb46f7f3a
msg272704 - (view) Author: Steven D'Aprano (steven.daprano) * (Python committer) Date: 2016-08-15 01:15
Still to do (coming soon):

- make the 0.2s time configurable;
- have `timeit` and `repeat` methods (and functions) fall back 
  on `autorange` if the number is set to 0 or None.
History
Date User Action Args
2016-08-15 01:15:43steven.dapranosetassignee: steven.daprano
messages: + msg272704
2016-08-14 15:27:22python-devsetnosy: + python-dev
messages: + msg272676
2016-08-08 02:52:42rhettingersetmessages: + msg272142
2016-08-08 00:59:44steven.dapranosetmessages: + msg272140
2016-05-13 03:46:33ncoghlansetmessages: + msg265442
2016-05-12 16:49:28steven.dapranosetmessages: + msg265418
2016-05-12 04:49:58ncoghlansetmessages: + msg265360
2016-05-11 16:46:36pitrousetmessages: + msg265326
2016-05-11 16:08:20steven.dapranosetmessages: + msg265324
2016-05-11 15:50:20pitrousetmessages: + msg265322
2016-05-11 15:28:05steven.dapranosetfiles: + timeit-autorange.patch
versions: + Python 3.6, - Python 3.5
nosy: + steven.daprano

messages: + msg265319
2015-03-22 06:54:18ncoghlansetmessages: + msg238879
2015-03-17 22:35:00rbcollinssetmessages: + msg238354
2015-03-17 22:19:10rbcollinssetmessages: + msg238352
2015-03-17 21:54:18rbcollinssetnosy: + rbcollins

stage: needs patch -> patch review
2014-08-06 13:30:19pitrousetassignee: amaury.forgeotdarc -> (no value)
versions: + Python 3.5, - Python 3.4
2012-08-21 18:47:53pitrousetmessages: + msg168796
stage: patch review -> needs patch
2012-08-21 18:43:34asvetlovsetnosy: + asvetlov
2012-06-28 00:55:25hayposetmessages: + msg164217
2012-06-28 00:46:08hayposetnosy: + haypo
messages: + msg164216
2012-06-26 12:45:07ncoghlansetresolution: accepted ->
2012-06-26 12:39:27ncoghlansetmessages: + msg164071
2012-06-26 12:37:40ncoghlansetnosy: + ncoghlan
messages: + msg164070
2012-06-26 12:28:00ncoghlanlinkissue5441 superseder
2012-06-26 02:40:21rhettingersetmessages: + msg164027
versions: + Python 3.4, - Python 3.3
2012-06-25 17:55:52tshepangsetnosy: + tshepang
2010-12-09 23:27:16pitrousetnosy: + pitrou

messages: + msg123711
stage: test needed -> patch review
2010-12-09 15:55:41merwoksetnosy: + merwok

versions: + Python 3.3, - Python 3.2
2010-11-30 23:33:22rhettingersetassignee: rhettinger -> amaury.forgeotdarc
resolution: accepted
messages: + msg122955
versions: - Python 2.7
2009-07-08 17:05:43rhettingersetassignee: rhettinger

messages: + msg90275
nosy: + rhettinger
2009-07-08 16:19:53scott_danielssetfiles: + timeit.patch
keywords: + patch
2009-07-08 12:01:49amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg90264
2009-07-07 21:29:29scott_danielssetmessages: + msg90245
2009-07-07 09:25:47ezio.melottisetpriority: normal
stage: test needed
2009-07-05 15:15:10scott_danielscreate