Converting static types to heap types is a work-in-progress:

* bpo-40077: "Convert static types to heap types: use PyType_FromSpec()"
* At December 29, 2020, 43% (89/206) of types are declared as heap types on a total of 206 types. For comparison, in Python 3.8, only 9% (15/172) of types were declared as heap types: 74 types have been converted in the meanwhile.
* https://vstinner.github.io/isolate-subinterpreters.html

Static types are still causing issues with Py_Initialize() / Py_Finalize() is called multiple times in the same process, which is a supported use case for embedded Python. See bpo-45691 "Partial moving of core objects to interpreter state is incorrect at best, unsafe at worse" for a recent example with sys.float_info where sys.float_info.n_unnamed_fields holds a reference to a Python object.

In june 2020, I wrote GH-20763 to clear static types at exit. But I abandoned my attempt because of bugs. Copy of my https://bugs.python.org/issue1635741#msg371119 message:

"""
I wrote PR 20763 to "finalize" static types in Py_Finalize(). It mostly works, but "./Programs/_testembed test_forced_io_encoding" crash. This program calls Py_Initialize() and Py_Finalize() multiple times in a loop.

It doesn't look to be safe to clear static types. Many functions rely on the fact that static types are "always there" and are never finalized. Also, only a few static types are cleared by my PR: many static types are left unchanged. For example, static types of the _io module.

It seems like a safer approach is to continue the work on bpo-40077: "Convert static types to PyType_FromSpec()".
"""

I propose a "best effort" approach: only clear a static type if it is no longer used. For example, check its reference count and its __subclasses__() method.

If we have static types, that means there is a mechanism to share some objects across interpreters.
And if that's the case, why can't small ints (like sys.float_info.n_unnamed_fields) be static & shared as well?

> If we have static types, that means there is a mechanism to share some objects across interpreters.

Sharing objects between interpreters is bad and is causing complex bugs. See a recent example of an object traveling from one interpreter to another and then causing a random crash on Windows related to the garbage collector:

* https://bugs.python.org/issue46070
* https://github.com/python/cpython/pull/30577#pullrequestreview-852106207


> And if that's the case, why can't small ints (like sys.float_info.n_unnamed_fields) be static & shared as well?

I would prefer to discuss that in other issues like bpo-40255 or bpo-39511, and focus this issue on fixing the static types implementation when Python is embedded in an application.

--

Hum, this issue is not really related to sub-interpreters. My proposed PR only  changes Py_Finalize(): the main interpreter.

> Sharing objects between interpreters is bad

That's your opinion, I don't necessarily share it.

> and is causing complex bugs.

But converting static things (types, small ints) to heap is also causing bugs :(

----

Anyway, for this issue: is there a way to test if the types are correctly reinitialized if there are multiple finalize/init cycles?
I worry about introducing more unexpected issues here.

If tomorrow static types are shared between sub-interpreters, it doesn't solve this problem: we still need to release memory allocated by Py_Initialize() in Py_Finalize() when Python is embedded. This issue is a sub-set of the big bpo-1635741 which explains the rationale. Again, this issue is not about sub-interpreters.

> Anyway, for this issue: is there a way to test if the types are correctly reinitialized if there are multiple finalize/init cycles? I worry about introducing more unexpected issues here.

I wasn't sure how to test it. But well, Dong-hee and you asked for tests, so I added tests for my PR ;-)

I checked with Valgrind the affect of PR 30645.

main branch:

==330902== LEAK SUMMARY:
==330902==      possibly lost: 29,128 bytes in 494 blocks
==330902==    still reachable: 353,615 bytes in 3,577 blocks

With the PR:

==332161== LEAK SUMMARY:
==332161==      possibly lost: 24,456 bytes in 417 blocks
==332161==    still reachable: 337,223 bytes in 3,423 blocks

It frees 21,064 bytes (20 kB) in Py_Finalize().

Another measure using the command:

    PYTHONHASHSEED=0 ./python -X showrefcount -c pass

I had to run the command 20 times to get a stable value, I don't know why.

main branch: [21981 refs, 5716 blocks]
PR: [21887 refs, 5667 blocks]

=> the PR removes 94 references and 49 memory blocks.

New changeset e9e3eab0b868c7d0b48e472705024240d5c39d5c by Victor Stinner in branch 'main':
bpo-46417: Finalize structseq types at exit (GH-30645)
https://github.com/python/cpython/commit/e9e3eab0b868c7d0b48e472705024240d5c39d5c

New changeset 6415e2ee4955b1a995c1e75544e2506b03780c3d by Victor Stinner in branch 'main':
bpo-46417: _testembed.c avoids Py_SetProgramName() (GH-30732)
https://github.com/python/cpython/commit/6415e2ee4955b1a995c1e75544e2506b03780c3d

New changeset f389b37fb1cebe7ed66331cdd373a014695261f6 by Victor Stinner in branch 'main':
bpo-46417: _thread uses PyStructSequence_NewType() (GH-30733)
https://github.com/python/cpython/commit/f389b37fb1cebe7ed66331cdd373a014695261f6

New changeset 17f268a4ae6190b2659c89c6f32ad2d006e0e3c8 by Victor Stinner in branch 'main':
bpo-46417: time module uses PyStructSequence_NewType() (GH-30734)
https://github.com/python/cpython/commit/17f268a4ae6190b2659c89c6f32ad2d006e0e3c8

New changeset 1781d55eb34f94029e50970232635fc5082378cb by Victor Stinner in branch 'main':
bpo-46417: _curses uses PyStructSequence_NewType() (GH-30736)
https://github.com/python/cpython/commit/1781d55eb34f94029e50970232635fc5082378cb

See also bpo-46449 "Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)".

New changeset d013b241352e902389f955f8f99d75f16c124ee2 by Victor Stinner in branch 'main':
bpo-46417: signal uses PyStructSequence_NewType() (GH-30735)
https://github.com/python/cpython/commit/d013b241352e902389f955f8f99d75f16c124ee2

New changeset ea38e436fe1e585fb8c1f0badf5482f525b7f9ff by Victor Stinner in branch 'main':
bpo-46417: Call _PyDebug_PrintTotalRefs() later (GH-30744)
https://github.com/python/cpython/commit/ea38e436fe1e585fb8c1f0badf5482f525b7f9ff

New changeset 595225e86dcc6ea520a584839925a878dce7a9b2 by Victor Stinner in branch 'main':
bpo-46417: Py_Finalize() clears static types (GH-30743)
https://github.com/python/cpython/commit/595225e86dcc6ea520a584839925a878dce7a9b2

> bpo-46417: Py_Finalize() clears static types (GH-30743)

Oh, test_unittest crashed on s390x Debian 3.x:
https://buildbot.python.org/all/#/builders/49/builds/1789

I fail to see the relationship between my change and this crash. I don't expect test_unittest to call Py_Finalize() and then Py_Initialize() again.

---
0:01:32 load avg: 4.77 [152/432/1] test_unittest crashed (Exit code -11) -- running: test_peg_generator (48.3 sec)
Fatal Python error: Segmentation fault

Current thread 0x000003ff89cf8720 (most recent call first):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/inspect.py", line 2179 in _signature_fromstr
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/inspect.py", line 2273 in _signature_from_builtin
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/inspect.py", line 2461 in _signature_from_callable
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/inspect.py", line 2465 in _signature_from_callable
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/inspect.py", line 2966 in from_callable
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/inspect.py", line 3218 in signature
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/mock.py", line 112 in _get_signature_object
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/mock.py", line 505 in _mock_add_spec
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/mock.py", line 487 in mock_add_spec
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/test/testmock/testmock.py", line 1767 in test_mock_add_spec
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/case.py", line 547 in _callTestMethod
  (...)
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/regrtest.py", line 47 in <module>
  File "<frozen runpy>", line 88 in _run_code
  File "<frozen runpy>", line 198 in _run_module_as_main

Extension modules: _testcapi (total: 1)
---

Maybe the problem is that I changed the order in which types are initialized in _PyTypes_InitTypes()?

So far, test_unittest crashed on 4 buildbot workers:

* s390x Debian 3.x: ./configure --prefix '$(PWD)/target' --with-pydebug
  https://buildbot.python.org/all/#/builders/49/builds/1789

* aarch64 Fedora Stable 3.x: ../configure --prefix '$(PWD)/target' --with-pydebug --with-platlibdir=lib64 --enable-ipv6 --enable-shared --with-computed-gotos=yes --with-dbmliborder=gdbm:ndbm:bdb --enable-loadable-sqlite-extensions --with-ssl-default-suites=openssl --without-static-libpython --with-lto
  https://buildbot.python.org/all/#/builders/125/builds/1263

* aarch64 RHEL8 3.x: ../configure --prefix '$(PWD)/target' --with-pydebug --with-platlibdir=lib64 --enable-ipv6 --enable-shared --with-computed-gotos=yes --with-dbmliborder=gdbm:ndbm:bdb --enable-loadable-sqlite-extensions --with-ssl-default-suites=openssl --without-static-libpython --with-lto
  https://buildbot.python.org/all/#/builders/529/builds/1357

* PPC64LE RHEL8 3.x: ../configure --prefix '$(PWD)/target' --with-pydebug --with-platlibdir=lib64 --enable-ipv6 --enable-shared --with-computed-gotos=yes --with-dbmliborder=gdbm:ndbm:bdb --enable-loadable-sqlite-extensions --with-ssl-default-suites=openssl --without-static-libpython --with-lto
  https://buildbot.python.org/all/#/builders/559/builds/1196

New changeset fda88864980ffce57add0ea03fb9cbda2798975e by Victor Stinner in branch 'main':
bpo-46417: Revert remove_subclass() change (GH-30750)
https://github.com/python/cpython/commit/fda88864980ffce57add0ea03fb9cbda2798975e

> bpo-46417: Revert remove_subclass() change (GH-30750)

Ok, this change fixed buildbots.

I saw code in typeobject.c which uses a borrowed reference to tp_subclasses with a loop which can modify tp_subclasses. This code should be modified to hold a strong reference to tp_subclasses while accessing it.

The test_mock_add_spec() test of test_unittest modifies the subclasses of many types and so is indirectly a stress tests for code accessing tp_subclasses. That's why the regression was only seen in this specific test.

New changeset a1bf329bca80a0259da454c936075e11e6af710f by Victor Stinner in branch 'main':
bpo-46417: Add missing types of _PyTypes_InitTypes() (GH-30749)
https://github.com/python/cpython/commit/a1bf329bca80a0259da454c936075e11e6af710f

New changeset bc67f189fdd62ed42013fa05cd0ef2df498f5967 by Victor Stinner in branch 'main':
bpo-46417: Add _PyType_CAST() macro (GH-30760)
https://github.com/python/cpython/commit/bc67f189fdd62ed42013fa05cd0ef2df498f5967

New changeset 8ee07dda139f3fa1d7c58a29532a98efc790568d by Victor Stinner in branch 'main':
bpo-46417: Add _PyType_GetSubclasses() function (GH-30761)
https://github.com/python/cpython/commit/8ee07dda139f3fa1d7c58a29532a98efc790568d

New changeset 7835cbf949c413a746324721a352cc72670a8a36 by Victor Stinner in branch 'main':
bpo-46417: Use _PyType_CAST() in Python directory (GH-30769)
https://github.com/python/cpython/commit/7835cbf949c413a746324721a352cc72670a8a36

New changeset ac1f152421fab3ac854fe4565c575b306e2bb4b5 by Victor Stinner in branch 'main':
bpo-46417: Use _PyType_CAST() in Objects directory (GH-30764)
https://github.com/python/cpython/commit/ac1f152421fab3ac854fe4565c575b306e2bb4b5

The following patch further reduces the reference but not sure if it is correct.

diff --git a/Objects/object.c b/Objects/object.c
index a5ee8eef4a..2ba6d14d5b 100644
--- a/Objects/object.c
+++ b/Objects/object.c
@@ -1853,6 +1853,9 @@ static PyTypeObject* static_types[] = {
     &PyClassMethod_Type,
     &PyCode_Type,
     &PyComplex_Type,
+    &PyContext_Type,
+    &PyContextVar_Type,
+    &PyContextToken_Type,
     &PyCoro_Type,
     &PyDictItems_Type,
     &PyDictIterItem_Type,
@@ -1911,6 +1914,13 @@ static PyTypeObject* static_types[] = {
     &_PyAsyncGenAThrow_Type,
     &_PyAsyncGenWrappedValue_Type,
     &_PyCoroWrapper_Type,
+    &_PyHamt_Type,
+    &_PyHamt_ArrayNode_Type,
+    &_PyHamt_BitmapNode_Type,
+    &_PyHamt_CollisionNode_Type,
+    &_PyHamtKeys_Type,
+    &_PyHamtValues_Type,
+    &_PyHamtItems_Type,
     &_PyInterpreterID_Type,
     &_PyManagedBuffer_Type,
     &_PyMethodWrapper_Type,


Before:
Running Debug|x64 interpreter...
[4929 refs, 1988 blocks]
After:
Running Debug|x64 interpreter...
[4541 refs, 1853 blocks]

New changeset f1c6ae3270913e095d24ae13ecf96f5a32c8c503 by Victor Stinner in branch 'main':
bpo-46417: Fix race condition on setting type __bases__ (GH-30788)
https://github.com/python/cpython/commit/f1c6ae3270913e095d24ae13ecf96f5a32c8c503

New changeset acda9f3b90c33e4020237cb9e5c676efb38f7847 by Miss Islington (bot) in branch '3.10':
bpo-46417: Fix race condition on setting type __bases__ (GH-30788) (GH-30789)
https://github.com/python/cpython/commit/acda9f3b90c33e4020237cb9e5c676efb38f7847

New changeset f1796f29478f08f34e0c30a060622c0b2d843e2c by Miss Islington (bot) in branch '3.9':
bpo-46417: Fix race condition on setting type __bases__ (GH-30788) (GH-30790)
https://github.com/python/cpython/commit/f1796f29478f08f34e0c30a060622c0b2d843e2c

New changeset 2d03b73cc9c0dada3243eab1373a46dbd98d24a0 by Victor Stinner in branch 'main':
bpo-46417: remove_subclass() clears tp_subclasses (GH-30793)
https://github.com/python/cpython/commit/2d03b73cc9c0dada3243eab1373a46dbd98d24a0

New changeset 500c146387b01ea797b52e6a54caf228384e184c by Victor Stinner in branch 'main':
bpo-46417: Clear more static types (GH-30796)
https://github.com/python/cpython/commit/500c146387b01ea797b52e6a54caf228384e184c

New changeset 3a4c15bb9815b6f4652621fe6043ae18e0d202b3 by Victor Stinner in branch 'main':
bpo-46417: Cleanup typeobject.c code (GH-30795)
https://github.com/python/cpython/commit/3a4c15bb9815b6f4652621fe6043ae18e0d202b3

New changeset 6cacdb42454264ae75cab5e32bb62876da43bf6f by Victor Stinner in branch 'main':
bpo-46417: _PyTypes_FiniTypes() clears object and type (GH-30798)
https://github.com/python/cpython/commit/6cacdb42454264ae75cab5e32bb62876da43bf6f

Kumar Aditya: "The following patch further reduces the reference but not sure if it is correct (...)"

Right! PyContext and PyHamt types were on my TODO list ;-) They are now cleared since this change:

bpo-46417: Clear more static types (GH-30796)
https://github.com/python/cpython/commit/500c146387b01ea797b52e6a54caf228384e184c

It's similar to your change, but more complete :-)

Attached cannot_deallocate.patch explains why some static types cannot be deallocated. It lists (static) types which are not cleard properly at Python exit.

Simplest example:
---
$ ./python -c pass
Cannot clear type 'object': it still has subclasses
* EncodingMap
* fieldnameiterator
* formatteriterator
* BaseException
* _io._IOBase
* _io._BytesIOBuffer
* _io.IncrementalNewlineDecoder
---

More complete example:
---
$ ./python setup.py build
running build
(...)
Cannot clear type 'types.GenericAlias': it still has subclasses
* _CallableGenericAlias
Cannot clear type 'tuple': it still has subclasses
* datetime.IsoCalendarDate
* DecimalTuple
Cannot clear type 'staticmethod': it still has subclasses
* abstractstaticmethod
Cannot clear type 'property': it still has subclasses
* abstractproperty
Cannot clear type 'list': it still has subclasses
* MyList
Cannot clear type 'dict': it still has subclasses
* collections.defaultdict
* StgDict
Cannot clear type 'classmethod': it still has subclasses
* abstractclassmethod
Cannot clear type 'type': it still has subclasses
* ABCMeta
* _ctypes.PyCStructType
* _ctypes.UnionType
* _ctypes.PyCPointerType
* _ctypes.PyCArrayType
* _ctypes.PyCSimpleType
* _ctypes.PyCFuncPtrType
Cannot clear type 'object': it still has subclasses
* type
* classmethod
* dict
* list
* property
* staticmethod
* tuple
* types.GenericAlias
* EncodingMap
* fieldnameiterator
* formatteriterator
* BaseException
* ModuleSpec
* FrozenImporter
* _io._IOBase
* _io._BytesIOBuffer
* _io.IncrementalNewlineDecoder
* _LoaderBasics
* FileLoader
* _abc._abc_data
* ABC
* Hashable
* Awaitable
* AsyncIterable
* Iterable
* Sized
* Container
* Callable
* itertools.accumulate
* itertools.combinations
* itertools.combinations_with_replacement
* itertools.cycle
* itertools.dropwhile
* itertools.takewhile
* itertools.islice
* itertools.starmap
* itertools.chain
* itertools.compress
* itertools.filterfalse
* itertools.count
* itertools.zip_longest
* itertools.pairwise
* itertools.permutations
* itertools.product
* itertools.repeat
* itertools.groupby
* itertools._grouper
* itertools._tee
* itertools._tee_dataobject
* collections.deque
* _collections._deque_iterator
* _collections._deque_reverse_iterator
* _collections._tuplegetter
* _IterationGuard
* WeakSet
* _struct.Struct
* _struct.unpack_iterator
* datetime.date
* datetime.time
* datetime.timedelta
* datetime.tzinfo
* _pickle.Pdata
* _pickle.PicklerMemoProxy
* _pickle.UnpicklerMemoProxy
* _pickle.Pickler
* _pickle.Unpickler
* _socket.socket
* _asyncio.FutureIter
* TaskStepMethWrapper
* _RunningLoopHolder
* _asyncio.Future
* _xxsubinterpreters.ChannelID
* matmulType
* ipowType
* awaitType
* MethodDescriptorBase
* GenericAlias
* Generic
* MethInstance
* MethClass
* MethStatic
* _testcapi.HeapCType
* _testcapi.ContainerNoGC
* _curses.window
* ossaudiodev.oss_audio_device
* ossaudiodev.oss_mixer_device
* _elementtree._element_iterator
* xml.etree.ElementTree.TreeBuilder
* xml.etree.ElementTree.Element
* xml.etree.ElementTree.XMLParser
* decimal.Decimal
* decimal.Context
* decimal.SignalDictMixin
* decimal.ContextManager
* Number
* _multiprocessing.SemLock
* xxlimited.Xxo
* CArgObject
* _ctypes.CThunkObject
* _ctypes._CData
* _ctypes.CField
* _ctypes.DictRemover
* _ctypes.StructParam_Type
---

New changeset f1bcdeaca6e912a2bec1fbcff76cc49e7f761d38 by Victor Stinner in branch 'main':
bpo-46417: Factorize _PyExc_InitTypes() code (GH-30804)
https://github.com/python/cpython/commit/f1bcdeaca6e912a2bec1fbcff76cc49e7f761d38

New changeset 621a45ccacd121f9ae4d8a539f040410c74b253b by Victor Stinner in branch 'main':
bpo-46417: Py_Finalize() clears static exceptioins (GH-30805)
https://github.com/python/cpython/commit/621a45ccacd121f9ae4d8a539f040410c74b253b

New changeset 1626bf4ac7aef1244e6f886e63a31f7ed65fbd10 by Victor Stinner in branch 'main':
bpo-46417: Clear Unicode static types at exit (GH-30806)
https://github.com/python/cpython/commit/1626bf4ac7aef1244e6f886e63a31f7ed65fbd10

New changeset 9c8e490b8f9e40a6fe9815be58bacaecab5369ee by Victor Stinner in branch 'main':
bpo-46417: Clear _io module static objects at exit (GH-30807)
https://github.com/python/cpython/commit/9c8e490b8f9e40a6fe9815be58bacaecab5369ee

New changeset 12f4ac3bc848244242d6b8a7ee158b985fd64744 by Victor Stinner in branch 'main':
bpo-46417: Clear symtable identifiers at exit (GH-30809)
https://github.com/python/cpython/commit/12f4ac3bc848244242d6b8a7ee158b985fd64744

New changeset a1444f43584af0f7a0af72aa06ba0a86ae5a87a2 by Victor Stinner in branch 'main':
bpo-46417: Fix _PyStaticType_Dealloc() (GH-30810)
https://github.com/python/cpython/commit/a1444f43584af0f7a0af72aa06ba0a86ae5a87a2

The _PyStaticType_Dealloc() function cannot call PyObject_ClearWeakRefs() if the refcount is not zero. I am not sure if it's a real issue or not. Maybe the weakref list must be cleared (release memory), but callbacks must not be called?

cannot_deallocate2.patch: updated patch to debug which types are not cleared at exit.

> New changeset a1444f43584af0f7a0af72aa06ba0a86ae5a87a2 by Victor Stinner in branch 'main':
> bpo-46417: Fix _PyStaticType_Dealloc() (GH-30810)
> https://github.com/python/cpython/commit/a1444f43584af0f7a0af72aa06ba0a86ae5a87a2

At this commit, Py_Finalize() no longer leaks references. It exceeded my expectations, it even makes _Py_RefTotal negative :-D

    $ ./python -I -X showrefcount -c pass
    [-4 refs, 61 blocks]

I'm not sure why it's negative, but bpo-46449 should be investigate first.

See also bpo-46476: Not all memory allocated by _Py_Quicken() is released at Python exit.

New changeset 976dec9b3b35fddbaa893c99297e0c54731451b5 by Victor Stinner in branch 'main':
bpo-46417: _PyList_Fini() clears indexerr (GH-30815)
https://github.com/python/cpython/commit/976dec9b3b35fddbaa893c99297e0c54731451b5

> See also bpo-46476: Not all memory allocated by _Py_Quicken() is released at Python exit.

If you apply my workaround for bpo-46476:
https://bugs.python.org/issue46476#msg411321

Python no longer leaks any memory block at exit for the simplest command!

$ ./python -I -X showrefcount -c pass
[-5 refs, 0 blocks]

Moreover, I modified deepfreeze to only freeze importlib._bootstrap and importlib._bootstrap_external. It confirms that bpo-46449 is causing the negative reference count, because with these additional local changes I get a positive _Py_RefTotal:

$ ./python -I -X showrefcount -c pass
[6 refs, 0 blocks]

New changeset 6c6a153dee132116611f2d5df0689a5a605f62b6 by Victor Stinner in branch 'main':
bpo-46417: signal: move siginfo_type to the module state (GH-30964)
https://github.com/python/cpython/commit/6c6a153dee132116611f2d5df0689a5a605f62b6

I close the issue. I cleared most static types at exit. Following work can be done in bpo-40077 or other issues.

New changeset 9a241271139a317597aca71d5971346b2cfe7dbd by Victor Stinner in branch 'main':
bpo-46417: _PyStructSequence_FiniType() updates _Py_RefTotal (GH-30988)
https://github.com/python/cpython/commit/9a241271139a317597aca71d5971346b2cfe7dbd