As of right now, to run the benchmark suite you need to use make_perf3.sh to generate a Python 3 compatible version of the benchmarks. That was fine when the benchmark suite didn't have much support for Python 3 and thus was just used for Python 3 very often. But now that a lot more benchmarks work it should be updated so that this extra step can be avoided.