Brett:
> A better way to benchmark this is to run hg.python.org/benchmarks with a patched interpreter and see if that shows any performance impact.

Ok. Here is the output. Can you please explain me the result? :-) Especially the "significant" value.


$ ./python.orig ../benchmarks/perf.py ./python.orig ./python.patched

INFO:root:Automatically selected timer: perf_counter
INFO:root:Running `./python.patched ../benchmarks/lib3/2to3/2to3 -f all ../benchmarks/lib/2to3`
INFO:root:Running `./python.patched ../benchmarks/lib3/2to3/2to3 -f all ../benchmarks/lib/2to3` 1 time
INFO:root:Running `./python.orig ../benchmarks/lib3/2to3/2to3 -f all ../benchmarks/lib/2to3`
INFO:root:Running `./python.orig ../benchmarks/lib3/2to3/2to3 -f all ../benchmarks/lib/2to3` 1 time
INFO:root:Running `./python.patched ../benchmarks/performance/bm_chameleon_v2.py -n 50 --timer perf_counter`
INFO:root:Running `./python.orig ../benchmarks/performance/bm_chameleon_v2.py -n 50 --timer perf_counter`
INFO:root:Running `./python.patched ../benchmarks/performance/bm_django_v3.py -n 50 --timer perf_counter`
INFO:root:Running `./python.orig ../benchmarks/performance/bm_django_v3.py -n 50 --timer perf_counter`
INFO:root:Running `./python.patched ../benchmarks/performance/bm_pickle.py -n 50 --timer perf_counter --use_cpickle pickle`
INFO:root:Running `./python.orig ../benchmarks/performance/bm_pickle.py -n 50 --timer perf_counter --use_cpickle pickle`
INFO:root:Running `./python.patched ../benchmarks/performance/bm_pickle.py -n 50 --timer perf_counter --use_cpickle unpickle`
INFO:root:Running `./python.orig ../benchmarks/performance/bm_pickle.py -n 50 --timer perf_counter --use_cpickle unpickle`
INFO:root:Running `./python.patched ../benchmarks/performance/bm_json_v2.py -n 50 --timer perf_counter`
INFO:root:Running `./python.orig ../benchmarks/performance/bm_json_v2.py -n 50 --timer perf_counter`
INFO:root:Running `./python.patched ../benchmarks/performance/bm_json.py -n 50 --timer perf_counter json_load`
INFO:root:Running `./python.orig ../benchmarks/performance/bm_json.py -n 50 --timer perf_counter json_load`
INFO:root:Running `./python.patched ../benchmarks/performance/bm_nbody.py -n 50 --timer perf_counter`
INFO:root:Running `./python.orig ../benchmarks/performance/bm_nbody.py -n 50 --timer perf_counter`
INFO:root:Running `./python.patched ../benchmarks/performance/bm_regex_v8.py -n 50 --timer perf_counter`
INFO:root:Running `./python.orig ../benchmarks/performance/bm_regex_v8.py -n 50 --timer perf_counter`
INFO:root:Running `./python.patched ../benchmarks/performance/bm_tornado_http.py -n 100 --timer perf_counter`
INFO:root:Running `./python.orig ../benchmarks/performance/bm_tornado_http.py -n 100 --timer perf_counter`
[ 1/10] 2to3...
[ 2/10] chameleon_v2...
[ 3/10] django_v3...
[ 4/10] fastpickle...
[ 5/10] fastunpickle...
[ 6/10] json_dump_v2...
[ 7/10] json_load...
[ 8/10] nbody...
[ 9/10] regex_v8...
[10/10] tornado_http...

Report on Linux smithers 4.2.8-300.fc23.x86_64 #1 SMP Tue Dec 15 16:49:06 UTC 2015 x86_64 x86_64
Total CPU cores: 8

### 2to3 ###
6.825440 -> 6.996616: 1.03x slower

### fastpickle ###
Min: 0.449710 -> 0.458054: 1.02x slower
Avg: 0.451589 -> 0.583100: 1.29x slower
Significant (t=-8.04)
Stddev: 0.00076 -> 0.11568: 152.3886x larger

### fastunpickle ###
Min: 0.549672 -> 0.545734: 1.01x faster
Avg: 0.551284 -> 0.696920: 1.26x slower
Significant (t=-6.65)
Stddev: 0.00118 -> 0.15493: 130.9694x larger

### json_load ###
Min: 0.432500 -> 0.466602: 1.08x slower
Avg: 0.433771 -> 0.471205: 1.09x slower
Significant (t=-10.63)
Stddev: 0.00495 -> 0.02440: 4.9303x larger

### nbody ###
Min: 0.230055 -> 0.234925: 1.02x slower
Avg: 0.230985 -> 0.236441: 1.02x slower
Significant (t=-7.56)
Stddev: 0.00048 -> 0.00508: 10.5095x larger

The following not significant results are hidden, use -v to show them:
chameleon_v2, django_v3, json_dump_v2, regex_v8, tornado_http.