Minimum working example:

class MyMetaclass(type):
    pass

class OtherMetaclass(type):
    pass

def metaclass_callable(name, bases, namespace):
    return OtherMetaclass(name, bases, namespace)

class MyClass(metaclass=MyMetaclass):
    pass

try:
    class MyDerived(MyClass, metaclass=metaclass_callable):
        pass
except:
    print("Gotcha!")


from types import new_class
MyDerived = new_class("MyDerived", (), dict(metaclass=metaclass_callable))

print(type(MyDerived))


This is because something happened along the way and Objects/typeobject.c:type_new no longer coincides with Lib/types.py:new_class. The Python version conditionally calls _calculate_meta whereas the C version calls it unconditionally. I consider the C implementation to be the "correct" version.

I suggest that
* the Python version be made to coincide with the C version.
* the documentation be made to coincide with the C version.  Specifically, section 3.3.3.2 should read:


"The metaclass of a class definition is selected from the explicitly specified metaclass (if any) and the metaclasses (i.e. type(cls)) of all specified base classes. The selected metaclass is the one which is a subtype of all of these candidate metaclasses. If none of the candidate metaclasses meets that criterion, then the class definition will fail with TypeError. If provided, the explicit metaclass must be a callable accepting the positional arguments (name, bases, _dict) as in the three argument form of the built-in type function."

Oops, I meant:

MyDerived = new_class("MyDerived", (MyClass,), dict(metaclass=metaclass_callable))

Nevertheless, the exception line number is totally off because it's tripping in the C code rather than in the Python code of the types library.

I'm not clear on what discrepancy you're referring to, as I get the same (expected) exception for both the class statement and the dynamic type creation:

>>> class MyDerived(MyClass, metaclass=metaclass_callable):
...     pass
... 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in metaclass_callable
TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases

>>> MyDerivedDynamic = new_class("MyDerivedDynamic", (MyClass,), dict(metaclass=metaclass_callable))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.5/types.py", line 57, in new_class
    return meta(name, bases, ns, **kwds)
  File "<stdin>", line 2, in metaclass_callable
TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases

This is due to the fact that your custom metaclass function returns an instance of a subclass of type, so we end up in type_new to actually create the type, which fails the metaclass consistency check.

One of the subtle intricacies here is that, for class statements, the logic that corresponds to types.prepare_class in the Python implementation is actually in the __build_class__ builtin for the C implementation - when there's a custom metaclass that *doesn't* return a subclass of type, we don't end up running type_new at all.

As a result of this, *both* implementations include a conditional check for a more derived metaclass in their namespace preparation logic, as well as an unconditional call to that metaclass derivation logic from type_new if the calculated metaclass is either type itself, or a subclass that calls up to super().__new__.

Most relevant issues and commit history:

- last update to C implementation
  - http://bugs.python.org/issue1294232
  - https://hg.python.org/cpython/rev/c2a89b509be4
- addition of pure Python implementation
  - http://bugs.python.org/issue14588
  - https://hg.python.org/cpython/rev/befd56673c80

The test cases in those commits (particularly the first one) should help make it clear what is and isn't supported behaviour.

Note: I'd be open to suggestions for comments in the pure Python implementation that would have helped you find its CPython counterpart in bltinmodule.c - it isn't immediately obvious from the current code that the actual __build_class__ code invoked by CPython's class statement is somewhere else entirely.

(I'll also note that my final comment there is rather different from my first draft, as I almost forgot myself that the namespace preparation logic lives in __build_class__ rather than type_new. Class definitions can actually bypass type entirely, even in Python 3:

>>> def the_one_class(*args):
...     return 1
... 
>>> class TheOne(metaclass=the_one_class): pass
... 
>>> TheOne
1

The documentation suggests that you can have a metaclass that does is not the "most derived metaclass" provided you specify one that is not an instance of type.  This doesn't work in CPython, so I would suggest fixing the documentation using the text I provided.

After that, it should be clear that there's no reason for "if isinstance(meta, type):" in the code, and the Python code should be restructured.

The point is that these two functions drifted apart somewhere around Python 3, and they need to be brought back together.  I only discovered this because it was possible in Python 2 to have a non-type metaclass that is not the most derived metaclass.  That has disappeared in CPython 3, except from the documentation and Lib.

From your comment:

>>> MyDerivedDynamic = new_class("MyDerivedDynamic", (MyClass,), dict(metaclass=metaclass_callable))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.5/types.py", line 57, in new_class
    return meta(name, bases, ns, **kwds)
  File "<stdin>", line 2, in metaclass_callable
TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases

This is in the wrong place.  It should be tripping the exception you defined in Lib/types.py.  (It will do that if you replace metaclass_callable with OtherMetaclass.)

"As a result of this, *both* implementations include a conditional check for a more derived metaclass in their namespace preparation logic, as well as an unconditional call to that metaclass derivation logic from type_new if the calculated metaclass is either type itself, or a subclass that calls up to super().__new__."

I don't see why that's necessary.  Either you should have the check in one place, or else have two equivalent checks.  Right now, the Python library is confusingly checking a subset of cases (when isinstance(meta, type)).  I suggest that you have the Python library check the entire set of cases so that the raised exception shows up in the right place.

Why should it trip the PEP 3115 namespace preparation exception? We only check for __prepare__ (and hence need to do an early most-derived metaclass resolution) on instances of "type", not on arbitrary callables used as a metaclass hint

The checks are different in the two places because the rules are different in the two places. (One case that *can* be made is that we should be throwing different exceptions or chaining them to show which metaclass resolution failed, rather than re-using the same message in both places).

This means that when you define a non-type metaclass hint, you're bypassing *all* of the PEP 3115 machinery, including the early metaclass resolution.

However, if your metaclass hint still creates a type instance, you're not bypassing tp_new.

If you're asking "Where is the bug in the presented example code?", it's here, in the definition of your metaclass hint:

    def metaclass_callable(name, bases, namespace):
        return OtherMetaclass(name, bases, namespace)

Instantiating instances of "type" directly in Python 3 bypasses the PEP 3115 machinery - that's the entire reason that types.new_class was added. So if you want that metaclass hint to be PEP 3115 compliant, you need to explicitly write it that way:

    def metaclass_callable(name, bases, namespace):
        return types.new_class(name, bases, dict(metaclass=OtherMetaclass)

Thanks for taking the time to explain, but it's still not working for me:

from types import new_class


class MyMetaclass(type):
    pass

class OtherMetaclass(type):
    pass

def metaclass_callable(name, bases, namespace):
    return new_class(name, bases, dict(metaclass=OtherMetaclass))

class MyClass(metaclass=MyMetaclass):
    pass

try:
    class MyDerived(MyClass, metaclass=metaclass_callable):
        pass
except:
    print("Gotcha!")


try:
    MyDerived = new_class("MyDerived", (MyClass,), dict(metaclass=metaclass_callable))
except:
    print("Gotcha again!")

So my questions are:

1. Why shouldn't Lib/types:new_class behave in exactly the same way as declaring a class using "class…" notation?

2. What's the point of checking if the metaclass is an instance of type?  It seems to me that in Python 2, non-type metaclasses did not have to be the "most derived class" (that's what the documentation seems to suggest with the second rule).  However, we no longer accept that in CPython 3 — neither in the Lib/types, nor in a regular declaration.  In fact, the exception is:

                        "metaclass conflict: "
                        "the metaclass of a derived class "
                        "must be a (non-strict) subclass "
                        "of the metaclasses of all its bases");

So why not just check that unconditionally in Lib/types.py?

Because they're checking for different things:

- types.prepare_class is only checking "How do I call __prepare__?". It only triggers type resolution at that point if the metaclass hint is an instance of type, otherwise it skips that process entirely and queries the metaclass hint directly.

- type.__new__ is checking "Can I actually create a new instance of this metaclass with these bases?". It can only do that if either the metaclass being instantiated is a subclass of all the bases of the type being defined, or else such a metaclass exists amongst the bases.

To be clear, your example isn't failing due to the way MyDerived is defined - it's failing because OtherMetaclass is itself an instance of type, and you're declaring it (directly or indirectly) as the metaclass of MyDerived, while inheriting from MyClass, which is an instance of MyMetaclass.

Okay, I understand what you're saying, but I it says in the documentation that "if an explicit metaclass is given and it is not an instance of type(), then it is used directly as the metaclass".  My recent updated "metaclass_callable" is not an instance of type.  Why should it raise an exception?

Oops, couple of typos:

"... only triggers metaclass resolution at that point ..."

"... if either the metaclass being instantiated is a subclass of all the metaclasses of all of the bases ..."

But the only way to bypass the "most derived metaclass" rule is for the the metaclass hint to be a callable that creates something that *isn't* a subclass of type. If you look at the tracebacks you're getting, you should see that the failure *isn't* in the class statement or the outer dynamic type, it's in "metaclass_callable", where the *inner* dynamic type creation is failing.

The "used directly as the metaclass" is a reference to https://docs.python.org/3/reference/datamodel.html#creating-the-class-object further down, and specifically the "metaclass(name, bases, namespace, **kwds)" call. It's not saying Python has a way to bypass that instantiation process.

As a result, your code is consistently getting to that step just fine, but *that call* is throwing an exception.

Hence my comment earlier that there's a case to be made that we should be better indicating where we were in the type creation process when the metaclass resolution failed.