It would be handy to have a C API that registered an atexit function, similar to what calling atexit.register does.  This API could be used by C extension modules to register atexit functions.

I think the implementation would be fairly simple.  We need a function that calls atexit_register().  The tricky bit is that we need to make sure the atexit module has been initialized as atexit_register uses the module state.

This API is different from Py_AtExit in that atexit.register() calls the function early in the interpreter shutdown whereas Py_AtExit calls functions very late in the shutdown.

See also:
[Python-Dev] PEP 489: module m_traverse called with NULL module state
https://mail.python.org/pipermail/python-dev/2017-December/151238.html

Calling "atexit.register" using the C API isn't very difficult. The annoying part is to wrap a simple C function pointer in a callable PyObject (I don't think there is a simple C API for that).

Oh by the way, if we decide to add a function, I would prefer to start with a private function. The C API is big enough. We can decide in the next version if the function becomes useful enough to justify to be added to the public API.

Private is fine.  We want to get the design correct before making it part of the official API.  My thought is that providing a handy atexit hook would be a good thing in that it could be an alternative to 3rd party code using __del__ to do cleanup.

One downside of atexit is that there is still a question of ordering the calls during shutdown.  I had the idea that maybe register() should take some sort of argument to specify when the function should be called.  That seems like leading down the road of madness maybe.  It would be useful to look at what other languages do, e.g. Java.  My understanding is that new versions of Java encourage finalizers like weakref callbacks and discourage or disallow finalizers like __del__.  I.e. you are not allowed to see the state of the object being finalized and cannot resurrect it.

Regarding m_traverse, maybe the list of finalizers should be stored somewhere in the interpreter state, not in the atexit module state.  That would make more sense to me.  atexit could merely be a way to manage it rather than actually holding it.

> Neil Schemenauer <nas-python@arctrix.com> added the comment:
> Regarding m_traverse, maybe the list of finalizers should be stored somewhere in the interpreter
> state, not in the atexit module state.  That would make more sense to me.  atexit could merely be
> a way to manage it rather than actually holding it.

+1

This may be especially useful to make sure that extension modules that have threads that were not created by Python calling into Python (registering with the interpreter and picking up the GIL) are stopped before the interpreter starts shutting down to avoid callbacks in the middle of the tear-down procedure.

Joannah Nanjekye asked me to have a look at this issue.

The existing Py_AtExit() C function is very low-level: it's called very late in Py_Finalize(), when the Python runtime is already destroyed.

I understand that the need here is to call C code which would still like to access the Python C API.

I propose the following C API:

   void Py_AtExitRegister(PyObject* (*func) (void *data), void *data)

which would call func(data).

The return type is void since I don't see any practical way to log an useful warning if a function fails: we only gets its memory address, we cannot log that. The callback function ('func') should be responsible to log errors itself (ex: using PySys_WriteStderr which writes into sys.stderr).


Neil:
> It would be handy to have a C API that registered an atexit function, similar to what calling atexit.register does.  This API could be used by C extension modules to register atexit functions.


The PyModuleDef already has 2 slots to call code at Python exit:
  void m_clear(PyObject *module)
and:
  void m_free(void *data).

But these callbacks can be late in Py_Finalize(), while Python runtime is being destroyed: when all modules are unloaded.

atexit.register() callbacks are different: they are called "early" in Py_Finalize(), when Python is still fully working.

I guess that Py_AtExitRegister() callbacks should also be called when Python is still fully working, right?

Apart C extensions, Py_AtExitRegister() also sounds interesting when Python is embedded in an application. For example, if you call Py_Main() or Py_RunMain(), you cannot easily execute arbitrary C code just before Py_Finalize().



Antoine:
> Calling "atexit.register" using the C API isn't very difficult. The annoying part is to wrap a simple C function pointer in a callable PyObject (I don't think there is a simple C API for that).

We could store C callbacks as C function pointers. We can have 2 lists of functions in the atexit module: one for Python function, one for C functions (registered using the C API). I would suggest to call Python functions first, and then call C functions.

--

I changed my mind, and I now consider that adding a public function would be useful. Sorry, first I misunderstood the use cases.