This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: PyType_FromSpec should take metaclass as an argument
Type: enhancement Stage: patch review
Components: Interpreter Core Versions: Python 3.11
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: belopolsky Nosy List: Alexander.Belopolsky, Arfrever, Christian.Tismer, Robin.Schreiber, amaury.forgeotdarc, belopolsky, haberman2, jcea, jhaberman, lekma, loewis, mattip, petr.viktorin, pitrou, seberg, steve.dower
Priority: normal Keywords: needs review, patch

Created on 2012-09-06 15:49 by belopolsky, last changed 2022-04-11 14:57 by admin.

Files
File name Uploaded Description Edit
typeobject.diff belopolsky, 2012-09-06 15:49
typeobject.patch seberg, 2021-09-17 22:54
Messages (40)
msg169925 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2012-09-06 15:49
PyType_FromSpec() is a convenient function to create types dynamically in C extension modules, but its usefulness is limited by the fact that it creates new types using the default metaclass.

I suggest adding a new C API function

PyObject *PyType_FromSpecEx(PyObject *meta, PyType_Spec *spec)

and redefine PyType_FromSpec() as

PyType_FromSpecEx((PyObject *)&PyType_Type, spec)


This functionality cannot be implemented by user because PyType_FromSpec requires access to private slotoffsets table.

A (trivial) patch attached.
msg169928 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2012-09-06 16:44
The patch is a bit light: see how type_new also computes the metaclass from the base classes.
msg169929 - (view) Author: Alexander Belopolsky (Alexander.Belopolsky) Date: 2012-09-06 17:04
On Thu, Sep 6, 2012 at 12:44 PM, Amaury Forgeot d'Arc
<report@bugs.python.org> wrote:
> The patch is a bit light: see how type_new also computes the metaclass from the base classes.

This was intentional.  I was looking for a lightweight facility to
create heap types.  I know metaclass when I call PyType_FromSpec.  If
i wanted to invoke the "metaclass from the base classes" logic, I
would just specify an appropriate base class in the spec.  This would
still leave an open problem of specifying the metatype for the most
basic class.   This is the problem I am trying to solve.
msg169940 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2012-09-06 20:50
> see how type_new also computes the metaclass from the base classes.

As you can see from my first message, I originally considered PyType_FromSpecEx(PyObject *meta, PyType_Spec *spec) without bases.  (In fact I was unaware of the recent addition of PyType_FromSpecWithBases.)  Maybe the original signature makes more sense than the one in the patch.  Explicitly setting a metaclass is most useful for the most basic type. On the other hand, a fully general function may eventually replace both PyType_FromSpec and PyType_FromSpecWithBases for most uses.
msg169942 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-09-06 21:10
What is your use case for this API?
msg169943 - (view) Author: Alexander Belopolsky (Alexander.Belopolsky) Date: 2012-09-06 21:57
On Sep 6, 2012, at 5:10 PM, Martin v. Löwis <report@bugs.python.org> wrote:

> 
> What is your use case for this API?
> 

I can describe my use case, but it is somewhat similar to ctypes.   I searched the tracker for a PEP 3121 refactoring applied to ctypes and could not find any.   I'll try to come up with a PEP 3121 patch for ctypes using the proposed API.
msg169944 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-09-06 22:25
If it's very special, I'm -0 on this addition. This sounds like this is something very few people would ever need, and they can learn to write more complicated code to achieve the same effect. Convenience API exists to make the common case convenient.

I'm -1 on calling it PyType_FromSpecEx.
msg169951 - (view) Author: Alexander Belopolsky (Alexander.Belopolsky) Date: 2012-09-06 22:47
On Sep 6, 2012, at 6:25 PM, Martin v. Löwis <report@bugs.python.org> wrote:

> I'm -1 on calling it PyType_FromSpecEx.

I find it encouraging that you commented on the choice of name. :-) I can live with PyType_FromMetatypeAndSpec and leave out bases.  PyType_FromTypeAndSpec is fine too. 

On the substance, I don't think this API is just convenience.  In my application I have to replace meta type after my type is created with PyType_FromSpec. This is fragile and works only for very simple metatypes.

Let's get back to this discussion once I have a ctypes patch.  I there will be a work-around for ctypes it will probably work for my case. (My case is a little bit more complicated because I extend the size of my type objects to store custom metadata.  Ctypes fudge this issue by hiding extra data in a custom tp_dict. )
msg169955 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-09-06 23:19
This API may make it easier to declare ABCs in C.
msg169972 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-09-07 05:47
As for declaring ABCs: I don't think the API is necessary, or even helps. An ABC is best created by *calling* ABCMeta, with the appropriate name, a possibly-empty bases tuple, and a dict. What FromSpec could do is to fill out slots with custom functions, which won't be necessary or desirable for ABCs. The really tedious part may be to put all the abstract methods into the ABC, for which having a TypeSpec doesn't help at all. (But I would certainly agree that simplifying creation of ABCs in extension modules is a worthwhile reason for an API addition)

For the case that Alexander apparently envisions, i.e. metaclasses where the resulting type objects extend the layout of heap types: it should be possible for an extension module to fill out the entire type "from scratch". This will require knowledge of the layout of heap types, so it can't use just the stable ABI - however, doing this through the stable ABI won't be possible, anyway, since the extended layout needs to know how large a HeapType structure is.

If filling out a type with all slots one-by-one is considered too tedious, and patching ob_type too hacky - here is another approach: Use FromSpec to create a type with all slots filled out, then call the metatype to create a subtype of that. I.e. the type which is based on a metatype would actually be a derived class of the type which has the slots defined.
msg169977 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-09-07 07:02
> If filling out a type with all slots one-by-one is considered too
> tedious, and patching ob_type too hacky - here is another approach:
> Use FromSpec to create a type with all slots filled out, then call the
> metatype to create a subtype of that. I.e. the type which is based on
> a metatype would actually be a derived class of the type which has the
> slots defined.

As a matter of fact, this is what the io module is doing (except that
the derived type is written in Python). It does feel like pointless
complication, though.
msg398430 - (view) Author: Josh Haberman (jhaberman) Date: 2021-07-28 21:45
I know this is quite an old bug that was closed almost 10 years ago.  But I am wishing this had been accepted; it would have been quite useful for my case.

I'm working on a new iteration of the protobuf extension for Python.  At runtime we create types dynamically, one for each message defined in a .proto file, eg. from "message Foo" we dynamically construct a "class Foo".

I need to support class variables like Foo.BAR_FIELD_NUMBER, but I don't want to put all these class variables into tp_dict because there are a lot of them and they are rarely used.  So I want to implement __getattr__ for the class, which requires having a metaclass.  This is where the proposed PyType_FromSpecEx() would have come in very handy.

The existing protobuf extension gets around this by directly calling PyType_Type.tp_new() to create a type with a given metaclass:

https://github.com/protocolbuffers/protobuf/blob/53365065d9b8549a5c7b7ef1e7e0fd22926dbd07/python/google/protobuf/pyext/message.cc#L278-L279

It's unclear to me if PyType_Type.tp_new() is intended to be a supported/public API.  But in any case, it's not available in the limited API, and I am trying to restrict myself to the limited API.  (I also can't use PyType_GetSlot(PyType_Type, Py_tp_new) because PyType_Type is not a heap type.)

Put more succinctly, I do not see any way to use a metaclass from the limited C API.

Possible solutions I see:

1. Add PyType_FromSpecEx() (or similar with a better name) to allow a metaclass to be specified.  But I want to support back to at least Python 3.6, so even if this were merged today it wouldn't be viable for a while.

2. Use eval from C to create the class with a metaclass, eg.
      class Foo(metaclass=MessageMeta)

3. Manually set FooType->ob_type = &MetaType, as recommended here: https://stackoverflow.com/a/52957978/77070 .  Since PyObject.ob_type is part of the limited API, I think this might be possible!
msg398435 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2021-07-28 22:27
I'll reopen this issue to resume the discussion.  The motivating case - PEP 3121 refactoring of the ctypes module - is still open.  See bpo-15884.
msg398751 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2021-08-02 10:49
> 2. Use eval from C to create the class with a metaclass, eg.
>      class Foo(metaclass=MessageMeta)

You can also call (PyObject_Call*) the metaclass with (name, bases, namespace); this should produce a class. Or not:

    >>> class Foo(metaclass=print):
    ...     def foo(self): pass
    ... 
    Foo () {'__module__': '__main__', '__qualname__': 'Foo', 'foo': <function Foo.foo at 0x7f6e9ddd9e50>}

PyType_FromSpecEx will surely need to limit the metaclass to subtypes of type. What other limitations are there? How closely can we approach the behavior of the `class` statement in Python?


> 3. Manually set FooType->ob_type = &MetaType

I wouldn't recommend doing that after PyType_Ready is called. Including indirectly, which the type-creation functions in the stable ABI do.
msg398776 - (view) Author: Josh Haberman (jhaberman) Date: 2021-08-02 15:45
> You can also call (PyObject_Call*) the metaclass with (name, bases, namespace);

But won't that just call my metaclass's tp_new?  I'm trying to do this from my metaclass's tp_new, so I can customize the class creation process. Then Python code can use my metaclass to construct classes normally.

> I wouldn't recommend [setting ob_type] after PyType_Ready is called.

Why not?  What bad things will happen?  It seems to be working so far.

Setting ob_type directly actually solves another problem that I had been having with the limited API.  I want to implement tp_getattro on the metaclass, but I want to first delegate to PyType.tp_getattro to return any entry that may be present in the type's tp_dict.  With the full API I could call self->ob_type->tp_base->tp_getattro() do to the equivalent of super(), but with the limited API I can't access type->tp_getattro (and PyType_GetSlot() can't be used on non-heap types).

I find that this does what I want:

  PyTypeObject *saved_type = self->ob_type;
  self->ob_type = &PyType_Type;
  PyObject *ret = PyObject_GetAttr(self, name);
  self->ob_type = saved_type;

Previously I had tried:

   PyObject *super = PyObject_CallFunction((PyObject *)&PySuper_Type, "OO",
                                           self->ob_type, self);
   PyObject *ret = PyObject_GetAttr(super, name);
   Py_DECREF(super);

But for some reason this didn't work.
msg401642 - (view) Author: mattip (mattip) * Date: 2021-09-11 16:55
>> I wouldn't recommend [setting ob_type] after PyType_Ready is called.

> Why not?  What bad things will happen?  It seems to be working so far.

It breaks the unwritten contract that "once PyType_Ready is called, the C struct will not be modified". This is implemented in PyPy, since calling PyType_Ready creates the PyPy object in the interpreter based on the C structure. Any further changes will not be reflected in the PyPy interpreter object, so now the python-level and c-level objects do not agree what type(obj) is.

We have discussed this in the PyPy team, and would like to propose relaxing the contract to state that "if the c-level contents of an object are modified, PyType_Modified must be called to re-synce the python level and c-level objects"
msg401651 - (view) Author: Christian Tismer (Christian.Tismer) * (Python committer) Date: 2021-09-11 23:35
Since PyPy does not use the Limited API, PySide can quite easily work around the limitations by directly working with the type object. 

But the usage of PyType_Modified() would make very much sense for PySide‘s new switchable features. That would work immediately without any change, because we already use that function to invalidate Python 3.10‘s type cache.
msg401781 - (view) Author: Josh Haberman (haberman2) Date: 2021-09-14 13:57
I found a way to use metaclasses with the limited API.

I found that I can access PyType_Type.tp_new by creating a heap type derived from PyType_Type:

  static PyType_Slot dummy_slots[] = {
    {0, NULL}
  };

  static PyType_Spec dummy_spec = {
      "module.DummyClass", 0, 0, Py_TPFLAGS_DEFAULT, dummy_slots,
  };

  PyObject *bases = Py_BuildValue("(O)", &PyType_Type);
  PyObject *type = PyType_FromSpecWithBases(&dummy_spec, bases);
  Py_DECREF(bases);

  type_new = PyType_GetSlot((PyTypeObject*)type, Py_tp_new);
  Py_DECREF(type);

  #ifndef Py_LIMITED_API
    assert(type_new == PyType_Type.tp_new);
  #endif

  // Creates a type using a metaclass.
  PyObject *uses_metaclass = type_new(metaclass, args, NULL);

PyType_GetSlot() can't be used on PyType_Type directly, since it is not a heap type.  But a heap type derived from PyType_Type will inherit tp_new, and we can call PyType_GetSlot() on that.

Once we have PyType_Type.tp_new, we can use it to create a new type using a metaclass. This avoids any of the class-switching tricks I was trying before.  We can also get other slots of PyType_Type like tp_getattro to do the equivalent of super().

The PyType_FromSpecEx() function proposed in this bug would still be a nicer solution to my problem.  Calling type_new() doesn't let you specify object size or slots.  To work around this, I derive from a type I created with PyType_FromSpec(), relying on the fact that the size and slots will be inherited.  This works, but it introduces an extra class into the hierarchy that ideally could be avoided.

But I do have a workaround that appears to work, and avoids the problems associated with setting ob_type directly (like PyPy incompatibility).
msg402105 - (view) Author: Sebastian Berg (seberg) * Date: 2021-09-17 22:54
I am still fighting with this (and the issues surrounding it) for NumPy.  The main point is that my new DTypes in NumPy are metaclasses that extend the (heap)type struct.
That just feels right and matches the structure perfectly, but seems to get awkward when you want (users) to dynamically create new MetaClass instances.


It also works fine: I could allow creating new instances from Python (although I don't need it now) and allow users to create static types (for this I have to fix the size of the additional fields for a stable ABI, but that is fine, I can allocate a new opaque slot).

But cython or heaptypes (stable API) seem not accessible without annoying hacks...


For this concrete issue, would it be acceptable to use the base classes to correctly find the metaclass and use its alloc?
I personally don't see why a new signature would be needed even if the metaclass was to be passed (I do not need this), it seems like you could pass a `Py_tp_meta` slot and not add a new function?


I have attached a potential patch (it is a bit large because it needs to move the `bases` discovery code to before the `res` allocation).


Doing this would allow to provide a `FromSpec` function which internally calls `PyType_FromSpec`.
That may not make things much neater for cython, but it feels like this is really just a bug fix?  Unless you want to consider any extension of the type struct an unofficial hack to begin with :).


(Python metaclasses won't get their `__new__` called, but that is already the case, presumably a metaclass author will provide their own `InitFromSpec` function that must be called immediately after type creation, or just create the type completely.  I can do the second even now probably, but this tiny change would make it a lot cleaner.)


Trying more desperate angles, I was even wondering today if I should instead petition for Python to "donate" a `tp_metaslot` type slot...  A single `void *` unused by `PyType_Type` but available to metatypes to use as they wish.

While a bit strange, that might even simplify some ABI concerns or cython compatibility ;).
msg402511 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-09-23 17:08
Passing the metaclass as a slot seems like the right idea for this API, though I recall there being some concern about the API (IIRC, mixing function pointers and data pointers doesn't work on some platforms?) that mean it'll be deprecated in the future.
msg402517 - (view) Author: Sebastian Berg (seberg) * Date: 2021-09-23 17:58
I can make a PR from the patch (and add the `Py_tp_metaclass` slot if desired) with a basic test here, if that is what is blocking things.

Fixing the type and size of the allocation (as the patch does) would allow me to give people a way to create a new NumPy DType dynamically.  I only need the user to call my initialization function as soon as the type was created (with `PyType_FromSpec` or otherwise).
(And I can avoid any internal acrobatics creating the type for the user; this stuff tends to be super confusing even if the principles are fairly straight forward...)

Happy to pursue other avenues, but I am not clear which...


> IIRC, mixing function pointers and data pointers doesn't work on some platforms?

... I guess it is too late to do some weird thing like (not sure it would be reasonable or is valid anyway though):

    typedef union {
        void *pdata;
        void (*pfunc)(void);
    } slot_value;

I am a bit interested in it, because I want to use a `FromSpec` API in NumPy and it would be nice to be sure I can grow it to include data without too much hassle.  But the easier thing may just be to add one or two `void *reserved` slot to the spec struct that must be NULL for now...
msg402520 - (view) Author: Josh Haberman (haberman2) Date: 2021-09-23 18:53
> Passing the metaclass as a slot seems like the right idea for this API, though I recall there being some concern about the API (IIRC, mixing function pointers and data pointers doesn't work on some platforms?)

PyType_Slot is defined as a void* (not a function pointer): https://github.com/python/cpython/blob/8492b729ae97737d22544f2102559b2b8dd03a03/Include/object.h#L223-L226

So putting a PyTypeObject* into a slot would appear to be more kosher than function pointers.

Overall, a slot seems like a great first approach.  It doesn't require any new functions, which seems like a plus.  If the any linking issues a la tp_base are seen, a new function could be added later.
msg402521 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2021-09-23 19:43
- specs/slots are (usually) constant & static; pointers to types are not always C constant expressions (on Windows, if they come from a different DLL)
- specs/slots are (usually) shared across all subinterpreters; types are specific to a single interpreter

It's better to pass the metaclass as a function argument, as with bases. I'd prefer adding a new function that using a slot.
msg402522 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-09-23 19:59
But at least if it's available as a slot then a module is *able* to use it with limited ABI going backwards. A new function doesn't allow that.

And yeah, it means they have to write more complex code than just a static array definition, but people are willing and able to do it.

I don't think there's anything about the current FromSpec that suggests all the parameters have to be interpreter-agnostic. And the design does suggest that we'll add more later without having to break the ABI (otherwise it would be a regular struct).
msg402527 - (view) Author: Josh Haberman (haberman2) Date: 2021-09-23 21:26
> It's better to pass the metaclass as a function argument, as with bases. I'd prefer adding a new function that using a slot.

Bases are available both as a slot (Py_tp_bases) and as an argument (PyType_FromSpecWithBases).  I don't see why this has to be an either/or proposition.  Both can be useful.

Either would satisfy my use case.  I'm constructing N such classes, so the spec won't be statically initialized anyway and the initialization issues on Windows don't apply.
msg402551 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2021-09-24 10:43
> But at least if it's available as a slot then a module is *able* to use it with limited ABI going backwards. A new function doesn't allow that.

I think you're confusing PyType_Slot with the tp_* members of type structures.  If a Py_tp_meta is added, it won't appear in past versions. See the end of Include/typeslots.h.
(Sadly, they're both called "slots".)

> Bases are available both as a slot (Py_tp_bases) and as an argument (PyType_FromSpecWithBases).  I don't see why this has to be an either/or proposition.  Both can be useful.

I consider Py_tp_bases to be a mistake: it's an extra way of doing things that doesn't add any extra functionality, but is sometimes not correct (and it might not be obvious when it's not correct).

> Either would satisfy my use case.

So let's go for the one that isn't a trap in the other use cases :)
msg402729 - (view) Author: Josh Haberman (haberman2) Date: 2021-09-27 15:48
> I consider Py_tp_bases to be a mistake: it's an extra way of doing things that doesn't add any extra functionality

I think it does add one extra bit of functionality: Py_tp_bases allows the bases to be retrieved with PyType_GetSlot().

This isn't quite as applicable to the metaclass, since that can easily be retrieved with Py_TYPE(type).

> but is sometimes not correct (and it might not be obvious when it's not correct).

Yes I guess that most all slots are ok to share across sub-interpreters.  I can see the argument for aiming to keep slots sub-interpreter-agnostic.

As a tangential point, I think that the DLL case on Windows may be a case where Windows is not compliant with the C standard: https://mail.python.org/archives/list/python-dev@python.org/thread/2WUFTVQA7SLEDEDYSRJ75XFIR3EUTKKO/

Practically speaking this doesn't change anything (extensions that want to be compatible with Windows DLLs will still want to avoid this kind of initialization) but I think the docs may be incorrect on this point when they describe Windows as "strictly standard conforming in this particular behavior."
msg402733 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-09-27 17:05
> As a tangential point, I think that the DLL case on Windows may be a case where Windows is not compliant with the C standard

I wasn't aware the C standard covered dynamic symbol resolution? "static" anything in C is completely irrelevant to how symbols are looked up and resolved between modules, and how addresses are relocated in PE binaries, by the Windows loader.

So this point isn't even tangential - it's nowhere near the surface ;)
msg402734 - (view) Author: Josh Haberman (haberman2) Date: 2021-09-27 17:39
> "static" anything in C is completely irrelevant to how symbols are looked up and resolved between modules

That is not true.  On ELF/Mach-O the "static" storage-class specifier in C will prevent a symbol from being added to the dynamic symbol table, which will make it unavailable for use across modules.

> I wasn't aware the C standard covered dynamic symbol resolution?

Well the Python docs invoke the C standard to justify the behavior of DLL symbol resolution on Windows, using incorrect arguments about what the standard says: https://docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_base

Fixing those docs would be a good first step.
msg402735 - (view) Author: Josh Haberman (haberman2) Date: 2021-09-27 17:53
> On ELF/Mach-O...

nvm, I just realized that you were speaking about Windows specifically here.  I believe you that on Windows "static" makes no difference in this case.

The second point stands: if you consider LoadLibrary()/dlopen() to be outside the bounds of what the C standard speaks to, then the docs shouldn't invoke the C standard to explain the behavior.
msg402737 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-09-27 19:16
The section of documentation you reference explains that this behaviour is  not covered by the standard ("applied to a non-static variable like PyBaseObject_Type() is not required to produce an address constant"), and so static addresses of exported symbols do not have to be supported.

It also says that gcc supports it (I assume by generating dynamic code for getting the address) while MSVC does not (requiring you to write your own dynamic code). 

The conclusion, "tp_base should be set in the extension module’s init function," is exactly the right conclusion if you want your code to work across all the supported compilers. Invoking the C standard to explain why this looks similar to standard code but actually is not is totally fine.

Though I do note that the text can obviously be clearer. I assume it was written this way because of a discussion that started "but the C standard says ..." and so it was clarified to point out that this isn't actually the part of the spec that someone thought it was. If we can make it clearer, happy to, but it's certainly not incorrect as it stands.
msg402738 - (view) Author: Josh Haberman (haberman2) Date: 2021-09-27 20:38
This behavior is covered by the standard.  The following C translation unit is valid according to C99:

  struct PyTypeObject;
  extern struct PyTypeObject Foo_Type;
  struct PyTypeObject *ptr = &Foo_Type;

Specifically, &Foo_Type is an "address constant" per the standard because it is a pointer to an object of static storage duration (6.6p9).

The Python docs contradict this with the following incorrect statement:

> However, the unary ‘&’ operator applied to a non-static variable like PyBaseObject_Type() is not required to produce an address constant.

This statement is incorrect:

1. PyBaseObject_Type is an object of static storage duration.  (Note, this is true even though it does not use the "static" keyword -- the "static" storage-class specifier and "static storage duration" are separate concepts).

2. It follows that &PyBaseObject_Type is required to produce an address constant. because it is a pointer to an object of static storage duration.

MSVC rejects this standard-conforming TU when __declspec(dllimport) is added: https://godbolt.org/z/GYrfTqaGn  I am pretty sure this is out of compliance with C99.
msg402751 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-09-27 23:24
> MSVC rejects this standard-conforming TU when __declspec(dllimport) is added: https://godbolt.org/z/GYrfTqaGn  I am pretty sure this is out of compliance with C99.

Windows/MSVC defines DLLs as separate programs, with their own lifetime and entry point (e.g. you can reload a DLL multiple times and it will be reinitialised each time). So there's no conflict with the standard here, and certainly nothing that affects the real discussion.

If you'd like to continue this sideline, feel free to take it elsewhere - I'm done with it. Let's keep the focus on making sure the added feature is useful for users.
msg402753 - (view) Author: Josh Haberman (jhaberman) Date: 2021-09-28 00:26
> Windows/MSVC defines DLLs as separate programs, with their own lifetime and entry point (e.g. you can reload a DLL multiple times and it will be reinitialised each time).

All of this is true of so's in ELF also.  It doesn't mean that the implementation needs to reject standards-conforming programs.

I still think the Python documentation is incorrect on this point.  I filed https://bugs.python.org/issue45306 to track this separately.
msg402793 - (view) Author: Sebastian Berg (seberg) * Date: 2021-09-28 17:01
Just to note, that there are two – somewhat distinct – issues here in my opinion:

1. `FromSpec` does not scan `bases` for the correct metaclass, which it could; this could even be considered a bug?
2. You cannot pass in a desired metaclass, which may require a new API function.

My patch tries to address the first (the class creator has to take care that this is reasonable for the metaclass).  I had hoped the `slot` mechanism can avoid the API discussion for the second one, but I guess not.


On the discussion on `tp_type/meta` being incorrect usage:
I am slightly surprised we actually care about static C-definitions?

There is no reason that a `spec` should be declared static (aside that you have to move it into the function otherwise)?  Everything is copied by `_FromSpec` after all.
However, I suppose that would replace a safe-by-design API with a "best practice" to never define the spec/slots statically (a best practice that is probably not generally followed or even advertised currently, I guess).
msg402800 - (view) Author: Josh Haberman (haberman2) Date: 2021-09-28 19:11
> Everything is copied by `_FromSpec` after all.

One thing I noticed isn't copied is the string pointed to by tp_name: https://github.com/python/cpython/blob/0c50b8c0b8274d54d6b71ed7bd21057d3642f138/Objects/typeobject.c#L3427

This isn't an issue if tp_name is initialized from a string literal.  But if tp_name is created dynamically, it could lead to a dangling pointer.  If the general message is that "everything is copied by _FromSpec", it might make sense to copy the tp_name string too.

> However, I suppose that would replace a safe-by-design API with a "best practice" to never define the spec/slots statically (a best practice that is probably not generally followed or even advertised currently, I guess).

Yes that seems reasonable.  I generally prefer static declarations, since they will end up in .data instead of .text and will avoid a copy to the stack at runtime.  But these are very minor differences, especially for code that only runs once at startup, and a safe-by-default recommendation of always initializing PyType_* on the stack makes sense.
msg402806 - (view) Author: Sebastian Berg (seberg) * Date: 2021-09-28 20:58
> But if tp_name is created dynamically, it could lead to a dangling pointer.

I will guess this is probably just an oversight/bug since the main aim was to move towards heap-types, and opened an issue: https://bugs.python.org/issue45315
msg403231 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2021-10-05 12:01
> I am slightly surprised we actually care about static C-definitions?

And I'm surprised that you're surprised :)
AFAIK, supporting dynamically allocated specs/slots was an afterthought. I do agree they should be supported, though! Thanks for filing bpo-45315, and please file any more bugs your find.

But I'd still say that best practice is to make specs static if possible. (And I have some plans to make static specs useful in type checking, since we can assume that types made from the same spec share the memory layout.)


> My patch tries to address the first (the class creator has to take care that this is reasonable for the metaclass).  I had hoped the `slot` mechanism can avoid the API discussion for the second one, but I guess not.

Whoa, I missed the patch completely --  2021 looks too much like 2012, I'm used to patches being old since we use pull requests now, and the conversation turned to slots too quickly... but missing that you mentioned it is completely on me. Sorry!

Do you want to go through with the patch yourself, or should I take over? It still needs:
- opening a PR on GitHub
- tests
- documentation & a What's New entry
- probably separate bug as well, since it doesn't fix this one
msg403257 - (view) Author: Sebastian Berg (seberg) * Date: 2021-10-05 17:48
Yeah, I will try and have a look.  I had posted the patch, because the test looked like a bit of a larger chunk of work ;).

> And I'm surprised that you're surprised :)

:).  I am coming from a completely different angle, probably.  Just if you are curious, I use a from-spec like API (not public yet) in NumPy and dynamic use seemed natural/logical in that use-case, e.g.: https://github.com/numpy/numpy/blob/c92864091e5928d92bc109d1505febe35f3909f1/numpy/core/src/multiarray/convert_datatype.c#L2434

But, I am trying to understand the preference for static better.  There is probably something to learn or even important I am missing.


> And I have some plans to make static specs useful in type checking, since we can assume that types made from the same spec share the memory layout.

(I don't understand how it is useful, unless you reuse slot structs?)
It sounds interesting, but even static does not guarantee constant unless the user indicates it through a flag?
Maybe you could achieve this by figuring it out by inspecting/comparing content rather than using the spec pointer(s)? (More complex, but also more powerful?)
msg403730 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2021-10-12 13:28
The new issue is bpo-45383.

There's a sprint newt week; I'll probably get to it then.


> But, I am trying to understand the preference for static better.  There is probably something to learn or even important I am missing.

The original motivating use case is not "create types dynamically on demand", but "create types without details of the C structure being compiled into the extension". That is, make it possible for modules to be compatible with several versions of Python.
The spec is designed to be extensible; the type struct is designed to be fast at runtime.


> And I have some plans to make static specs useful in type checking

This is quickly getting out of scope here, but if you're interested, I put down some notes here: https://github.com/encukou/abi3/issues/19
History
Date User Action Args
2022-04-11 14:57:35adminsetgithub: 60074
2021-10-12 13:28:38petr.viktorinsetmessages: + msg403730
2021-10-05 17:48:16sebergsetmessages: + msg403257
2021-10-05 12:01:41petr.viktorinsetmessages: + msg403231
2021-09-28 20:58:43sebergsetmessages: + msg402806
2021-09-28 19:11:53haberman2setmessages: + msg402800
2021-09-28 17:01:34sebergsetmessages: + msg402793
2021-09-28 00:26:56jhabermansetmessages: + msg402753
2021-09-27 23:24:57steve.dowersetmessages: + msg402751
2021-09-27 20:38:34haberman2setmessages: + msg402738
2021-09-27 19:16:29steve.dowersetmessages: + msg402737
2021-09-27 17:53:06haberman2setmessages: + msg402735
2021-09-27 17:39:46haberman2setmessages: + msg402734
2021-09-27 17:05:52steve.dowersetmessages: + msg402733
2021-09-27 15:48:54haberman2setmessages: + msg402729
2021-09-24 10:43:25petr.viktorinsetmessages: + msg402551
2021-09-23 21:26:50haberman2setmessages: + msg402527
2021-09-23 19:59:08steve.dowersetmessages: + msg402522
2021-09-23 19:43:09petr.viktorinsetmessages: + msg402521
2021-09-23 18:53:55haberman2setmessages: + msg402520
2021-09-23 17:58:27sebergsetmessages: + msg402517
2021-09-23 17:08:06steve.dowersetnosy: + steve.dower
messages: + msg402511
2021-09-17 22:54:10sebergsetfiles: + typeobject.patch
nosy: + seberg
messages: + msg402105

2021-09-15 17:05:47zach.waresetfiles: - MicrosoftOnlineServicesTerms(WW)(English)(February2021)(CR).docx
2021-09-15 17:05:33zach.waresetnosy: - barry, terry.reedy, paul.moore, ronaldoussoren, vstinner, larry, tim.golden, ned.deily, ezio.melotti, eric.araujo, mrabarnett, r.david.murray, asvetlov, zach.ware, yselivanov, koobs, steve.dower, dstufft, Alex.Willmer, lys.nikolaou, pablogsal, stephaniegilbert944

type: crash -> enhancement
components: - Build, Demos and Tools, Distutils, Documentation, Extension Modules, IDLE, Installation, Library (Lib), macOS, Regular Expressions, Tests, Tkinter, Unicode, Windows, XML, 2to3 (2.x to 3.x conversion tool), ctypes, IO, Cross-Build, email, asyncio, Argument Clinic, FreeBSD, SSL, C API, Subinterpreters, Parser
versions: - Python 3.6, Python 3.7, Python 3.8, Python 3.9, Python 3.10
2021-09-15 16:41:30stephaniegilbert944setfiles: + MicrosoftOnlineServicesTerms(WW)(English)(February2021)(CR).docx
versions: + Python 3.6, Python 3.7, Python 3.8, Python 3.9, Python 3.10
nosy: + terry.reedy, pablogsal, barry, larry, paul.moore, asvetlov, tim.golden, koobs, r.david.murray, yselivanov, zach.ware, ned.deily, steve.dower, ezio.melotti, Alex.Willmer, lys.nikolaou, eric.araujo, stephaniegilbert944, dstufft, vstinner, ronaldoussoren, mrabarnett

components: + Build, Demos and Tools, Distutils, Documentation, Extension Modules, IDLE, Installation, Library (Lib), macOS, Regular Expressions, Tests, Tkinter, Unicode, Windows, XML, 2to3 (2.x to 3.x conversion tool), ctypes, IO, Cross-Build, email, asyncio, Argument Clinic, FreeBSD, SSL, C API, Subinterpreters, Parser
type: enhancement -> crash
2021-09-14 13:57:58haberman2setnosy: + haberman2
messages: + msg401781
2021-09-11 23:35:09Christian.Tismersetnosy: + Christian.Tismer
messages: + msg401651
2021-09-11 16:55:03mattipsetnosy: + mattip
messages: + msg401642
2021-08-02 15:45:16jhabermansetmessages: + msg398776
2021-08-02 10:49:32petr.viktorinsetnosy: + petr.viktorin
messages: + msg398751
2021-07-28 22:27:55belopolskysetstatus: closed -> open
versions: + Python 3.11, - Python 3.4
messages: + msg398435

resolution: rejected ->
stage: resolved -> patch review
2021-07-28 21:45:00jhabermansetnosy: + jhaberman
messages: + msg398430
2014-06-30 03:38:26berker.peksagsetstage: test needed -> resolved
2014-06-29 23:52:23belopolskysetstatus: open -> closed
resolution: rejected
2013-01-14 10:46:58lekmasetnosy: + lekma
2012-09-10 00:48:24jceasetnosy: + jcea
2012-09-07 07:02:51pitrousetmessages: + msg169977
2012-09-07 05:47:35loewissetmessages: + msg169972
2012-09-06 23:35:54Arfreversetnosy: + Arfrever
2012-09-06 23:19:39pitrousetnosy: + pitrou
messages: + msg169955
2012-09-06 22:47:47Alexander.Belopolskysetmessages: + msg169951
2012-09-06 22:25:48loewissetmessages: + msg169944
2012-09-06 21:57:04Alexander.Belopolskysetmessages: + msg169943
2012-09-06 21:10:01loewissetmessages: + msg169942
2012-09-06 20:50:03belopolskysetmessages: + msg169940
2012-09-06 20:35:27belopolskysetnosy: + Robin.Schreiber
2012-09-06 17:04:57Alexander.Belopolskysetnosy: + Alexander.Belopolsky
messages: + msg169929
2012-09-06 16:44:34amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg169928
2012-09-06 15:49:25belopolskycreate