classification
Title: Add fractions benchmark
Type: enhancement Stage:
Components: Benchmarks Versions:
process
Status: closed Resolution: third party
Dependencies: Superseder:
Assigned To: Nosy List: asvetlov, pitrou, scoder, serhiy.storchaka, skrah, vstinner
Priority: normal Keywords:

Created on 2014-09-22 07:10 by scoder, last changed 2016-09-13 09:04 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
telco_fractions.py scoder, 2014-09-22 07:10 telco benchmark adapted to use Fraction instead of Decimal
bm_telco_fractions.py scoder, 2014-09-22 07:56 telco benchmark adapted to use Fraction, with option to use Decimal
Messages (22)
msg227252 - (view) Author: Stefan Behnel (scoder) * (Python committer) Date: 2014-09-22 07:10
Fractions are great for all sorts of exact computations (including money/currency calculations), but are quite slow due to the need for normalisation at instantiation time.

I adapted the existing telco benchmark to use Fraction instead of Decimal to make this problem more visible. One change I made was to take the data reading out of the measured loop. I/O is not part of what the benchmark should measure.

Please consider adding it to the benchmark suite.
msg227256 - (view) Author: Stefan Behnel (scoder) * (Python committer) Date: 2014-09-22 07:56
I just thought that it might also be nice to have a direct comparison with Decimal, so here's an updated benchmark that has an option "--use-decimal" to run the same code with Decimal instead of Fraction.

Decimal is about 66x faster with Py3.4 on my side (due to the C implementation).
msg227264 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-09-22 12:32
Where are Fractions used in the real world?
msg227265 - (view) Author: Stefan Behnel (scoder) * (Python committer) Date: 2014-09-22 12:51
As I said, where ever exact calculations are needed. I use them for currency calculations, for example, as they inherently avoid rounding errors during the calculations regardless of the relative size of values. They are basically like Decimal but with arbitrary precision.
msg227266 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-09-22 13:12
Le 22/09/2014 14:51, Stefan Behnel a écrit :
> 
> I use them for currency calculations, for example,
> as they inherently avoid rounding errors during the
> calculations regardless of the relative size of values.

Do other people use them for that purpose, or are you
the only one?
msg227269 - (view) Author: Stefan Behnel (scoder) * (Python committer) Date: 2014-09-22 13:29
I admit that I keep meeting developers who are not aware of their merits, simply because many other programming languages don't have them available. Specifically, many developers with a Java background firmly believe that BigDecimal is the only way to do money calculations. That's clearly wrong, and the heap of (Big-)Decimal bugs in their software that I've been fixing over the years shows just how wrong it is. Using Fractions would have avoided most of them. Decimal is surprisingly difficult to use right, Fractions are actually difficult to get wrong. They are a major feature of the standard library.

Your question is another good indication that the general awareness for them still needs to be raised. I'm not entirely sure how this relates to this ticket, though. I doubt that you are seriously suggesting that I am the only person using a module in the stdlib, so what exactly are you trying to say?
msg227276 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-09-22 15:03
My point is that if fractions are little used right now, there's little point in adding a benchmark about them in the "official" benchmark suite. The benchmark suite does not aim at measuring every possible aspect of Python performance, but at showcasing representative workloads.
msg227280 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2014-09-22 17:09
I've always viewed the `Fraction` type as more of a teaching tool than something intended for real-world calculations.  If that's not how others see it, then it might be worth investing some effort in reimplementing `Fractions.gcd` in C: the `Fraction` type is almost unusably slow for serious work, and the gcd computations are a clear bottleneck. I've experimented in the past with a C-based implementation of Lehmer's algorithm for gcd computation, getting around a ten-fold speedup.
msg227285 - (view) Author: Stefan Behnel (scoder) * (Python committer) Date: 2014-09-22 17:37
Speed improvements for fractions should have their own ticket(s). I created issue 22464 for this.
msg227291 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-09-22 18:26
> Where are Fractions used in the real world?

For example Andrew Svetlov uses them in his online game [1]. May be it will explain this in detail.

[1] http://asvetlov.blogspot.com/2012/08/numerics.html (on Russian).
msg273928 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-08-30 16:23
Can you please send a pull request to the new https://github.com/python/performance project?
msg274135 - (view) Author: Stefan Behnel (scoder) * (Python committer) Date: 2016-09-01 14:38
Done:
https://github.com/python/performance/pull/10

Note that I didn't replace the existing telco benchmark as it is more specific to Decimal. The new benchmark makes it possible to compare the decimal and fractions modules for similar operations, though.
msg274184 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2016-09-01 21:29
I think string conversion should be part of this benchmark, or it should be renamed to fraction-arith or something.

The formatting function can be a significant bottleneck, and if you use Fractions for financial calculations the result will still need to be printed in decimal floating point form.


That said, is the purpose of this benchmark to show that fractions are slow and generate interest in changing the situation?
msg274185 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-09-01 21:31
> I think string conversion should be part of this benchmark, or it should be renamed to fraction-arith or something.

The "telco" benchmark seems to be standard and well defined. I agree
that we should use a different name.

 > That said, is the purpose of this benchmark to show that fractions
are slow and generate interest in changing the situation?

I'm not really convinced that the benchmark is useful in the Python
benchmark suite, but I'm not strongly opposed to adding it neither :-)
msg274186 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2016-09-01 22:01
I'm also not opposed to adding it (-0.000) as long as we rename it to bm_fractions.py and change the docstring to "based on the telco benchmark".
msg274187 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-09-01 23:06
@Stefan Krah: Please review the pull request.

https://github.com/python/performance/pull/10

I suggested to only run the benchmark with the fractions module. I don't see the point of running the benchmark with the decimal module.
msg274234 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2016-09-02 11:45
I've left comments on GitHub.


[scoder] 
> As I said, where ever exact calculations are needed..

Even if the formatting comment is addressed, the main problem with
this benchmark is that *both* fraction and decimal calculations
are in fact exact here.

You can check that by printing out the values for "t" in the loop
like I suggested on GitHub.


This is of course one of the main features of Decimal: If you're
willing to take the penalty of a large precision, calculations
can often be exact; similar to taking the penalty of potentially
huge numerators and denominators in fractions.


So this benchmark cannot be used to show the superiority of exact
fractions.
msg274244 - (view) Author: Stefan Behnel (scoder) * (Python committer) Date: 2016-09-02 14:48
> So this benchmark cannot be used to show the superiority of exact fractions.

I don't see how a benchmark would be a way to show that. It's certainly not the goal of this benchmark to show that one is computationally better than the other. But if a benchmark for one part of Python allows a direct, reasonable speed comparison with another, why would you object to make use of that?


> is the purpose of this benchmark to show that fractions are slow and generate interest in changing the situation?

There have been a couple of attempts to make them faster already. The speed improvements in Py3.5 and 3.6 directly result from those changes. Thus, making these improvements visible and comparable across Python versions and different implementations is one of the goals of this benchmark.
msg274256 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2016-09-02 16:38
The reason is that the benchmark is a softball for fractions. Using the fastest fraction implementation (gmpy.mpq) and the best printing method for both types, gmpy.mpq seems to be about 2.5 to 3 times slower than decimal.

It is however easy to generate benchmarks where mpq is 3000 times slower, depending on how many prime factors accumulate in the denominator.


Fractions will shine in most benchmarks where decimal is exact anyway.
msg274264 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-09-02 17:39
Stefan Krah added the comment:
> Fractions will shine in most benchmarks where decimal is exact anyway.

According to Stefan's latest message, the purpose of the benchmark is
not the compare decimal to fractions, but measure the performance of
fractions for one Python implementation. For example, it can be used
to check that fractions is faster in Python 3.6 than Python 3.4
fractions.
msg274268 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2016-09-02 18:00
Which Stefan? :)


Anyway, the first half of this issue was centered around the
proposition that fractions are a "better decimal", and the
latest pull request still has the "decimal backend" in it. :)


Fast fractions have been around for a long time (Lisp/sbcl),
but apparently they have never been widely used.


A fraction-only benchmark is okay, but comparing fractions/decimals
in one particular "official" benchmark is a bit strange.
msg276217 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-09-13 09:04
The issue is not closed, but moved to 
https://github.com/python/performance/pull/10

Please continue the discussion there. I'm now slowly trying to move benchmark issues to GitHub.
History
Date User Action Args
2016-09-13 09:04:06vstinnersetstatus: open -> closed
resolution: third party
messages: + msg276217
2016-09-02 18:00:07skrahsetmessages: + msg274268
2016-09-02 17:39:44vstinnersetmessages: + msg274264
2016-09-02 16:38:50skrahsetmessages: + msg274256
2016-09-02 14:48:44scodersetmessages: + msg274244
2016-09-02 11:45:24skrahsetmessages: + msg274234
2016-09-01 23:06:05vstinnersetmessages: + msg274187
2016-09-01 22:01:03skrahsetmessages: + msg274186
2016-09-01 21:31:55vstinnersetmessages: + msg274185
2016-09-01 21:29:06skrahsetnosy: + skrah
messages: + msg274184
2016-09-01 14:38:40scodersetmessages: + msg274135
2016-08-30 19:19:26mark.dickinsonsetnosy: - mark.dickinson
2016-08-30 16:23:01vstinnersetnosy: + vstinner
messages: + msg273928
2014-10-14 16:05:48skrahsetnosy: - skrah
2014-09-22 18:26:32serhiy.storchakasetnosy: + serhiy.storchaka, asvetlov
messages: + msg227291
2014-09-22 17:37:41scodersetmessages: + msg227285
2014-09-22 17:09:35mark.dickinsonsetmessages: + msg227280
2014-09-22 15:03:50pitrousetmessages: + msg227276
2014-09-22 13:29:18scodersetmessages: + msg227269
2014-09-22 13:13:00pitrousetmessages: + msg227266
2014-09-22 12:51:35scodersetmessages: + msg227265
2014-09-22 12:32:18pitrousetnosy: + skrah, pitrou
messages: + msg227264
2014-09-22 07:56:07scodersetfiles: + bm_telco_fractions.py

messages: + msg227256
2014-09-22 07:44:17mark.dickinsonsetnosy: + mark.dickinson
2014-09-22 07:10:30scodercreate