Python 2.7 could be significantly slower (5x in some cases) when compiled with clang 3.x or 4.x, compared to clang 5.x. This is quite a problem on macOS, since the latest clang from Apple (which comes with Xcode 9.2) is based on LLVM 4.x. This issue was first noticed by Bart Skowron and reported to the Homebrew project.[1]

I ran some preliminary benchmarks (here[2] are the exact setup scripts) with just a simple loop:

    import time

    def f(n):
        while n > 0:
            n -= 1

    start = time.time()
    f(50000000)
    stop = time.time()
    print('%.6f' % (stop - start))

and here are my results:

- macOS 10.13.2 on a MacBook Pro:

    2.082144	/usr/bin/python2.7
    7.964049	/usr/local/bin/python2.7
    8.750652	dist/python27-apple-clang-900/bin/python2.7
    8.476405	dist/python27-clang-3.9/bin/python2.7
    8.625660	dist/python27-clang-4.0/bin/python2.7
    1.760096	dist/python27-clang-5.0/bin/python2.7
    3.254814	/usr/local/bin/python3.6
    2.864716	dist/python-master-apple-clang-900/bin/python3
    3.071757	dist/python-master-clang-3.9/bin/python3
    2.925192	dist/python-master-clang-4.0/bin/python3
    2.908782	dist/python-master-clang-5.0/bin/python3

- Ubuntu 17.10 in VirtualBox:

    1.475095	/usr/bin/python2.7
    8.576817	dist/python27-clang-3.9/bin/python2.7
    8.165588	dist/python27-clang-4.0/bin/python2.7
    1.779193	dist/python27-clang-5.0/bin/python2.7
    1.728321	dist/python27-gcc-5/bin/python2.7
    1.570040	dist/python27-gcc-6/bin/python2.7
    1.604617	dist/python27-gcc-7/bin/python2.7
    2.323037	/usr/bin/python3.6
    2.964338	dist/python-master-clang-3.9/bin/python3
    3.054277	dist/python-master-clang-4.0/bin/python3
    2.734908	dist/python-master-clang-5.0/bin/python3
    2.490278	dist/python-master-gcc-5/bin/python3
    2.494691	dist/python-master-gcc-6/bin/python3
    2.642277	dist/python-master-gcc-7/bin/python3

I haven't got time to run more rigorous benchmark suites (e.g., the performance[3] package). I did try the floating point benchmark from performance, and again saw a 2x difference in performance.

[1] https://github.com/Homebrew/homebrew-core/issues/22743
[2] https://gist.github.com/zmwangx/f8151ba8907ba8159a07fdd1528fc2b5
[3] https://pypi.python.org/pypi/performance

Has anyone done the same analysis with Python 3.6 or 3.7?

My benchmarks above do contain py37 (master) stats.

Is there anything we (the CPython developers) can do about this? 

If I read the issue correctly clang 5.x generates faster binaries than clang 3.x and 4.x.  If that is indeed the issue there's probably not much we can do about this. 

BTW. I'm -1 on building the installer with anything but the compiler included in Xcode (and it would be nice to build with a recent version of Xcode to use an up-to-date compiler and SDK)

It seems clang4 failed to efficient register assigning.
FYI, --without-computed-gotos configure option make penalty smaller.

clang 5 (wihtout CGs): 2.653426
clang 5 (with CGs): 1.997584
clang 4 (without CGs): 3.330879
clang 4 (with CGs): 8.585673

This is assembly code for FAST_DISPATCH()
https://paste.ubuntu.com/26523948/

It seems there are many redundant spills.  But I don't know how to remove them.
Are their clang expert?

Bad news: --enable-optimization doesn't solve it.
I hope same thing doesn't happen for Python 3.
Anyone tried Xcode 9.3?  What version of LLVM does Apple use?

Anyway, I think we need help of LLVM expert.

Turns out python 2.7.10 doesn't suffer from the performance issue even when compiled with stock clang 4.x, and upon further investigation, I tracked down the commit that introduced the regression:

    commit 2c992a0788536087bfd78da8f2c62b30a461d7e2
    Author: Benjamin Peterson <benjamin@python.org>
    Date:   Thu May 28 12:45:31 2015 -0500
    
        backport computed gotos (#4753)

So Naoki was right that computed gotos is (solely) to blame here.

Quick test: there doesn't seem to be a similar regression when building 3.6 with the current clang provided by Xcode 9.2, just with 2.7.  And both 2.7 and 3.6 configure HAVE_COMPUTED_GOTOS on.  Benjamin?

(FWIW, the 2.7.x binaries provided by the python.org installers do not suffer from this performance regression as they are not built with clang.)

New changeset 2942b909d9a428e6683d90b3436cfa4a81bd5d8a by INADA Naoki in branch '2.7':
bpo-32616: Disable computed gotos by default for clang < 5 (GH-5574)
https://github.com/python/cpython/commit/2942b909d9a428e6683d90b3436cfa4a81bd5d8a

I'm sorry, my patch doesn't work on Xcode (Apple LLVM).
computed-gotos is still enabled by default.

Apple doesn't expose LLVM version.  It's really annoying.

$ cat x.c
#include <stdio.h>

int main()
{
    printf("__clang__ : %d\n", __clang__);
    printf("__llvm__ : %d\n", __llvm__);
    printf("__VERSION__ : %s\n", __VERSION__);
    printf("__clang_version__ : %s\n", __clang_version__);
    printf("__clang_major__   : %d\n", __clang_major__);
    printf("__clang_minor__   : %d\n", __clang_minor__);
    printf("__clang_patchlevel__ : %d\n", __clang_patchlevel__);
}

$ cc x.c && ./a.out
__clang__ : 1
__llvm__ : 1
__VERSION__ : 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)
__clang_version__ : 9.0.0 (clang-900.0.39.2)
__clang_major__   : 9
__clang_minor__   : 0
__clang_patchlevel__ : 0

Yeah, Apple LLVM versions are a major headache. I resorted to feature detection, using C++ coroutines support as the clang 5 distinguisher[1]:

$ cat /tmp/test/stub.cc
#include <experimental/coroutine>

int main() {
    return 0;
}

$ /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang -v
Apple LLVM version 9.0.0 (clang-900.0.39.2)
Target: x86_64-apple-darwin17.4.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

$ /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang -o stub stub.cc -fcoroutines-ts -stdlib=libc++
stub.cc:1:10: fatal error: 'experimental/coroutine' file not found
#include <experimental/coroutine>
         ^~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.

$ /Applications/Xcode-beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang -v
Apple LLVM version 9.1.0 (clang-902.0.31)
Target: x86_64-apple-darwin17.4.0
Thread model: posix
InstalledDir: /Applications/Xcode-beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

$ /Applications/Xcode-beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang -o stub stub.cc -fcoroutines-ts -stdlib=libc++

Here Xcode.app is Xcode 9.2 and Xcode-beta.app is Xcode 9.3 beta 2.

The conclusion here seems to be that Apple LLVM 9.0.0 is based on LLVM 4, while Apple LLVM 9.1.0 is based on LLVM 5.

[1] http://releases.llvm.org/5.0.0/tools/clang/docs/ReleaseNotes.html#major-new-features

How can we distinguish Apple LLVM with LLVM easily?
Or should we disable computed-gotos by default on LLVM?

It's only for Python 2.
5x slowdown is too large comparing to 10% speedup.

Can anyone explain what the difference is between 2.7 and 3.6, i.e. why there is the performance regression for 2.7 but not for 3.6 using the same compiler instance?  It would be better to understand and solve that problem rather than trying to special case compiler versions.

I don't know exactly.  But as far as I saw, Python 3's eval loop has less function-wide local variables.

For example, ROT_THREE uses only block local variable.
https://github.com/python/cpython/blob/a48e78a0b7761dd74f1d03fc69e0f6caa6f02fe6/Python/ceval.c#L1109-L1111

On the other hand, there are more function-wide local variables in Python 2.  And some of them are used over `case`s actually.
https://github.com/python/cpython/blob/672fd7d8162f76aff8423fa5c7bfd2b1e91faf57/Python/ceval.c#L802-L807


I suspect that's why LLVM4 failed to optimize Python 2 but success to optimize Python 3.

I different question w.r.t. detection of the clang/llvm version on Apple's system compiler: Is it worthwhile to do so? 

If the compiler included in the Xcode 9.3 beta (and hence likely the one in Xcode 9.3 final) fixes the performance issue a very large subset of people building Python for themselves will get a fixed compiler fairly soon. It would then be enough to warn about this issue in a readme file for other users.

This should remain everyone that backporting performance improvements is not a no-brainer.

A followup - Ronald asked:
> w.r.t. detection of the clang/llvm version on Apple's system compiler: Is it worthwhile to do so?

Now that Xcode 9.3 (for macOS 10.13+) is officially released, I ran a quick series of test on it and on the most recent Xcode versions for the last several macOS OS families: 10.12, 10.11, and 10.9 (I didn't have a 10.10 system available at the moment).  Only looking at the most recent supported Xcode/compiler version for each major release is reasonable since I think most people follow Apple's strong encouragement to keep software updated and only use the most recent releases.  And I think most people follow Apple's lead in using their build tools, via Xcode or the command line utilities, rather than a third-party compiler.

By that measure, it seems clear that (1) there is only one current version that exhibits the performance degradation, that is the Xcode 9.2 version labeled Apple LLVM 9.0.0 (clang-900.0.39.2) and (2) that is now only an issue for macOS 10.12 (Sierra) where Xcode 9.2 is (and will likely remain) the most recent version.  For macOS 10.13 (High Sierra), the compiler in the newly released Xcode 9.3 does not exhibit the problem.  And the most recent versions of Xcode for the tested earlier macOS releases do not either.

BTW, the MacPorts project maintains a handy webpage listing Xcode releases and compiler versions by macOS release:
    https://trac.macports.org/wiki/XcodeVersionInfo

Here are the results.  The methodology was to download and build the just released Python 2.7.15rc1 from source using the default configure options, i.e. just ./configure, and then run the test program 3 times with it and then three times with the Apple-provided system /usr/bin/python2.7 as a baseline.

ProductName:	Mac OS X
ProductVersion:	10.13.4
BuildVersion:	17E199
2.7.15rc1 (default, Apr 15 2018, 00:22:29)
[GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)]
1.181971
1.180467
1.173380
2.7.10 (default, Oct  6 2017, 22:29:07)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.31)]
1.567072
1.587779
1.570056

ProductName:	Mac OS X
ProductVersion:	10.12.6
BuildVersion:	16G1314
2.7.15rc1 (default, Apr 15 2018, 00:31:39)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)]
6.290989
6.356802
6.295680
2.7.10 (default, Feb  7 2017, 00:08:15)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)]
1.306507
1.312231
1.302826

ProductName:	Mac OS X
ProductVersion:	10.11.6
BuildVersion:	15G20015
2.7.15rc1 (default, Apr 15 2018, 00:38:12)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)]
1.846332
1.855483
1.896600
2.7.10 (default, Oct 23 2015, 19:19:21)
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.5)]
1.453702
1.426298
1.440348

ProductName:	Mac OS X
ProductVersion:	10.9.5
BuildVersion:	13F1911
2.7.15rc1 (default, Apr 15 2018, 00:42:08)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]
1.720303
1.712045
1.710216
2.7.5 (default, Mar  9 2014, 22:15:05)
[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)]
1.681696
1.685704
1.686414

So is this now considered resolved for High Sierra users via 2.7.15rc1?

I'm using Mojave.  I don't have regression.

$ /usr/local/bin/python2 x.py  # Homebrew python@2
1.681729

$ /usr/bin/python x.py  # System python2
1.891549

Could someone using High Sierra or Sierra test it?

x.py is test script in https://bugs.python.org/issue32616#msg310395

https://github.com/Homebrew/homebrew-core/blob/b2aff6271caa04508fb1529fdd5edbb22f4e7f21/Formula/python%402.rb#L75-L82

Homebrew checks compiler version too.

I don't think it's worth enough to add more code to avoid this trouble.