This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author kristjan.jonsson
Recipients kristjan.jonsson
Date 2009-10-01.17:16:39
SpamBayes Score 1.4705459e-12
Marked as misclassified No
Message-id <1254417403.92.0.577059221924.issue7029@psf.upfronthosting.co.za>
In-reply-to
Content
The attached patch contains suggested fixes to pybench.py:
1) add a processtime timer for windows
2) fix a bug in timer selection: timer wasn't put in 'self'
3) Collect 'gross' times for each round
4) show statistics for per-round sums, both for sum of "adjusted" times 
for each round, and 'gross' time per round.

The last one is important.  Traditionally, the focus would be on 
individual tests.  The series of individual timings for each test would 
be  examined, min and afverage found.  These minima and averages would 
then be summed up to get a total time.  These results are very noisy.  
In addition, the sum of individual minima is not a linear operator that 
has any meaning and is thus extremely noisy.

Looking at the minimum and average of the sum is a much stabler 
indication of total benchmark performance.

Another thing that I found when working with this, is that using 
"calibration" significantly adds to noise and can produce incorrect 
results.  It adds a level of non-linearity to the results that can 
appear very strange.  Typically the 'overhead' is so low that we should 
consider skipping the calibration.  The purpose of the benchmark must be 
to measure the performance of python in context.  The meaning of 
individual operations in isolation are pretty meaningless.  Although I 
didn't change the default number of calibration runs from 10, I would 
suggest documenting that it may be useful to turn it to 0 to increase 
predictability and get numbers indicative of real performance.
History
Date User Action Args
2009-10-01 17:16:44kristjan.jonssonsetrecipients: + kristjan.jonsson
2009-10-01 17:16:43kristjan.jonssonsetmessageid: <1254417403.92.0.577059221924.issue7029@psf.upfronthosting.co.za>
2009-10-01 17:16:42kristjan.jonssonlinkissue7029 messages
2009-10-01 17:16:41kristjan.jonssoncreate