Message272197
I spent the last 3 months on making the CPython benchmark suite more stable and enhance my procedure to run benchmarks to ensure that benchmarks are more stable.
See my articles:
https://haypo-notes.readthedocs.io/microbenchmark.html#my-articles
I forked and enhanced the benchmark suite to use my perf module to run benchmarks in multiple processes:
https://hg.python.org/sandbox/benchmarks_perf
I ran this better benchmark suite on fastcall-2.patch on my laptop. The result is quite good:
----------------
$ python3 -m perf compare_to ref.json fastcall.json -G --min-speed=5
Slower (4):
- fastpickle/pickle_dict: 326 us +- 15 us -> 350 us +- 29 us: 1.07x slower
- regex_effbot: 49.4 ms +- 1.3 ms -> 53.0 ms +- 1.2 ms: 1.07x slower
- fastpickle/pickle: 432 us +- 8 us -> 457 us +- 10 us: 1.06x slower
- pybench.ComplexPythonFunctionCalls: 838 ns +- 11 ns -> 884 ns +- 8 ns: 1.05x slower
Faster (13):
- spectral_norm: 289 ms +- 6 ms -> 250 ms +- 5 ms: 1.16x faster
- pybench.SimpleIntFloatArithmetic: 622 ns +- 9 ns -> 559 ns +- 10 ns: 1.11x faster
- pybench.SimpleIntegerArithmetic: 621 ns +- 10 ns -> 560 ns +- 9 ns: 1.11x faster
- pybench.SimpleLongArithmetic: 891 ns +- 12 ns -> 816 ns +- 10 ns: 1.09x faster
- pybench.DictCreation: 852 ns +- 13 ns -> 788 ns +- 16 ns: 1.08x faster
- pybench.ForLoops: 10.8 ns +- 0.3 ns -> 9.99 ns +- 0.23 ns: 1.08x faster
- pybench.NormalClassAttribute: 1.85 us +- 0.02 us -> 1.72 us +- 0.04 us: 1.08x faster
- pybench.SpecialClassAttribute: 1.86 us +- 0.02 us -> 1.73 us +- 0.03 us: 1.07x faster
- pybench.NestedForLoops: 21.9 ns +- 0.3 ns -> 20.7 ns +- 0.3 ns: 1.05x faster
- pybench.SimpleListManipulation: 501 ns +- 4 ns -> 476 ns +- 5 ns: 1.05x faster
- elementtree/process: 192 ms +- 3 ms -> 183 ms +- 2 ms: 1.05x faster
- elementtree/generate: 225 ms +- 5 ms -> 214 ms +- 4 ms: 1.05x faster
- hexiom2/level_25: 21.3 ms +- 0.3 ms -> 20.3 ms +- 0.1 ms: 1.05x faster
Benchmark hidden because not significant (84): (...)
----------------
Most benchmarks are not significant which is expected since fastcall-2.patch is really the most simple patch to start the work on "FASTCALL", it doesn't really implement any optimization, it only adds a new infrastructure to implement new optimizations.
A few benchmarks are faster (only benchmarks at least 5% faster are shown using --min-speed=5).
4 benchmarks are slower, but the slowdown should be temporarily: new optimizations should these benchmarks slower. See the issue #26814 for more a concrete implementation and a lot of benchmark results if you don't trust me :-)
I consider that benchmarks proved that there is no major slowdown, so fastcall-2.patch can be merged to be able to start working on real optimizations. |
|
Date |
User |
Action |
Args |
2016-08-08 22:01:15 | vstinner | set | recipients:
+ vstinner, serhiy.storchaka, yselivanov |
2016-08-08 22:01:15 | vstinner | set | messageid: <1470693675.67.0.863843301106.issue27128@psf.upfronthosting.co.za> |
2016-08-08 22:01:15 | vstinner | link | issue27128 messages |
2016-08-08 22:01:14 | vstinner | create | |
|