In Python 2, PyErr_GivenExceptionMatches [1] calls PyObject_IsSubclass. To handle calling __subclasscheck__ in this case, 2.7 (but not 2.6) temporarily increases the recursion limit by 5. For example:

    class CMeta(type):
        def __subclasscheck__(self, other):
            import sys
            print 'recursion limit: %d' % sys.getrecursionlimit()
            frame = sys._getframe(1)
            n = 0
            while frame:
                n += 1
                frame = frame.f_back
            print 'frame: %d' % n
            return True

    class C(Exception):
        __metaclass__ = CMeta

    def f():
        try:
            f()
        except C:
            pass

    >>> sys.getrecursionlimit()
    1000
    >>> f()
    recursion limit: 1005
    frame: 1000
    >>> sys.getrecursionlimit()
    1000

If the recursion limit weren't temporarily increased, then trying to call __subclasscheck__ in the above case would raise another RuntimeError.

In Python 3, PyErr_GivenExceptionMatches [2] instead calls PyType_IsSubtype. See issue 2534. In that issue Antoine's reason for the change is that "otherwise there are some nasty issues with recursion checking". 

[1]: https://hg.python.org/cpython/file/v2.7.10/Python/errors.c#l84
[2]: https://hg.python.org/cpython/file/v3.5.0/Python/errors.c#l166