There seem to be two issues at play here:

- the atexit module (and its companion helper _Py_PyAtExit()) doesn't know about sub-interpreters.

- PyState_FindModule() doesn't know about sub-interpreters either, because the m_index field (which records the module's index in an interpreter's module list (PyInterpreterState.modules_by_index)) is recorded in the PyModuleDef structure rather than the module instance: it is therefore global to all interpreters

Having atexit work properly with sub-interpreters would require fixing these two issues AFAICT.