The median tends to give a better idea about benchmark results than an average as it inherently ignores outliers.

In case of even number of samples the median value is calculated as arithmetic mean of two middle samples.

med_base = (base_times[len(base_times)//2] + base_times[(len(base_times)-1)//2]) / 2

Fair enough, patch updated.

Any more comments on the patch, or can it be applied?

for the even number case, I think you shouldn't do // 2, but / 2.

In general, wouldn't it be good to let the statistics module do all the stats calculations?

> In general, wouldn't it be good to let the statistics module do all the stats calculations?

It's not available in older Python versions, e.g. 2.6.

> It's not available in older Python versions, e.g. 2.6.

I know, I was talking about 3.5+, of course. This would not be backported to Python2 anyway, would it?

ah sorry, it's late here already and I forgot what file this change is about. So forget my last comment then.

> for the even number case, I think you shouldn't do // 2, but / 2.

Right. I updated the patch.

Have you found the median to be more stable than the minimum here?

I'm actually not sure how it relates to the minimum. The more runs you have, the higher the chance of hitting the actual minimum at least once. And if none of the runs hits the real minimum, you're simply out of luck.

However, it should tend to give a much better result than the (currently printed) average, which suffers from outliers. And outliers are almost always too high for benchmarks and never too low, due to various external influences.

Then let's just replace the average with the median? I don't think it makes sense to add more statistical information to the output (IMHO, there is already too much of it :-)).

May be just drop 5% of largest values to avoid the impact of outliers?

See also issue23552.

Well, we can apply a kludge, or apply statistics.

> May be just drop 5% of largest values to avoid the impact of outliers?

In fast mode (option "-f"), there may not be enough samples for that.

The new https://github.com/python/performance benchmark suite now displays the median rather than the arithmetic mean (average) by default (it also displays the standard deviation).

See the perf issue for a discussion about median vs mean:
https://github.com/haypo/perf/issues/1

Can we now close this issue?