classification
Title: allow per-thread atexit()
Type: enhancement Stage:
Components: Interpreter Core, Library (Lib) Versions: Python 3.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Glenn.Maynard, christian.heimes, grahamd, haypo, neologix, pitrou, tarek, tim.peters, zvezdan
Priority: normal Keywords:

Created on 2012-02-21 14:12 by tarek, last changed 2013-11-26 15:49 by haypo.

Messages (14)
msg153871 - (view) Author: Tarek Ziadé (tarek) * (Python committer) Date: 2012-02-21 14:12
If you try to run the code below and stop it with ctrl+C, it will lock because atexit is never reached.

Antoine proposed to add a way to have one atexit() per thread, so we can call some cleanup code when the app shuts down and there are running threads.


{{{
from wsgiref.simple_server import make_server
import threading
import time
import atexit


class Work(threading.Thread):
    def __init__(self):
        threading.Thread.__init__(self)
        self.running = False

    def run(self):
        self.running = True
        while self.running:
            time.sleep(.2)

    def stop(self):
        self.running = False
        self.join()

worker = Work()


def shutdown():
    # bye-bye
    print 'bye bye'
    worker.stop()


atexit.register(shutdown)


def hello_world_app(environ, start_response):
    status = '200 OK' # HTTP Status
    headers = [('Content-type', 'text/plain')]
    start_response(status, headers)
    return ["Hello World"]

def main():
    worker.start()
    return make_server('', 8000, hello_world_app)

if __name__ == '__main__':
    server = main()
    server.serve_forever()
}}}
msg153905 - (view) Author: Graham Dumpleton (grahamd) Date: 2012-02-21 22:11
My take on this is that if wanting to interact with a thread from an atexit callback, you are supposed to call setDaemon(True) on the thread. This is to ensure that on interpreter shutdown it doesn't try and wait on the thread completing before getting to atexit callbacks.
msg153931 - (view) Author: Tarek Ziadé (tarek) * (Python committer) Date: 2012-02-22 08:09
@grahamd : sometimes you don't own the code that contains the thread, so I think it's better to be able to shutdown properly all flavors of threads.
msg153932 - (view) Author: Graham Dumpleton (grahamd) Date: 2012-02-22 08:16
Reality is that the way Python behaviour is defined/implemented means that it will wait for non daemonised threads to complete before exiting.

Sounds like the original code is wrong in not setting it to be daemonised in the first place and should be reported as a bug in that code rather than fiddling with the interpreter.
msg153933 - (view) Author: Tarek Ziadé (tarek) * (Python committer) Date: 2012-02-22 08:24
Is there any good reason not to add this feature ? what would be the problem ?  It does seem to be for the best, I don't see any drawbacks
msg153939 - (view) Author: Graham Dumpleton (grahamd) Date: 2012-02-22 09:11
At the moment you have showed some code which is causing you problems and a vague idea. Until you show how that idea may work in practice it is a bit hard to judge whether what it does and how it does it is reasonable.
msg153940 - (view) Author: Tarek Ziadé (tarek) * (Python committer) Date: 2012-02-22 09:20
Mmm.. you did not say yet why you are against this feature, other than "the lib *should not* use non-daemonized threads"

This sounds like "the lib should not use feature X in Python because it will block everything"

And now we're proposing to remove the limitation and you are telling me I am vague and unreasonable.

Let me try differently then.

Consider this script to be a library I don't control. I need to call the .stop() function when my main application shuts down. 

I can't use signals because you forbid it in mod_wsgi

How do I do, since asking the person to daemonize his thread is not an option ?

I see several options:
1 - monkey patch the lib
2 - remove regular threads from Python, or making them always daemonized
3 - add an atexit() option in threads in Python
4 - use signals and drop the usage of mod_wsgi

I think 3- is the cleanest.
msg153943 - (view) Author: Graham Dumpleton (grahamd) Date: 2012-02-22 09:34
I haven't said I am against it. All I have done so far is explain on the WEB-SIG how mod_wsgi works and how Python currently works and how one would normally handle this situation by having the thread be daemonised.

As for the proposed solution, where is the code example showing how what you are suggesting is meant to work. Right now you are making people assume how that would work. Add an actual example here at least of how with the proposed feature your code would then look.

For the benefit of those who might even implement what you want, which will not be me anyway as I am not involved in Python core development, you might also explain where you expect these special per thread atexit callbacks to be triggered within the current steps for shutting down the interpreter. That way it will be more obvious to those who come later as to what you are actually proposing.
msg153945 - (view) Author: Tarek Ziadé (tarek) * (Python committer) Date: 2012-02-22 09:45
> Add an actual example here at least of how with the proposed feature your code would then look.

That's the part I am not sure at all about in fact. I don't know at all the internals in the shutdown process in Python and I was hoping Antoine would give us a proposal here.

I would suspect simply adding to the base thread class an .atexit() method that's called when atexit() is called, would do the trick since we'd be able to do things like:

  def atexit(self):
      ... do whatever cleanup needed...
      self.join()

but I have no real experience in these internals.
msg153947 - (view) Author: Graham Dumpleton (grahamd) Date: 2012-02-22 10:02
Except that calling it at the time of current atexit callbacks wouldn't change the current behaviour. As quoted in WEB-SIG emails the sequence is:

    wait_for_thread_shutdown();

    /* The interpreter is still entirely intact at this point, and the
     * exit funcs may be relying on that.  In particular, if some thread
     * or exit func is still waiting to do an import, the import machinery
     * expects Py_IsInitialized() to return true.  So don't say the
     * interpreter is uninitialized until after the exit funcs have run.
     * Note that Threading.py uses an exit func to do a join on all the
     * threads created thru it, so this also protects pending imports in
     * the threads created via Threading.
     */
    call_py_exitfuncs();

So would need to be done prior to wait_for_thread_shutdown() or by that function before waiting on thread.

The code in that function has:

    PyObject *threading = PyMapping_GetItemString(tstate->interp->modules,
                                                  "threading");

    ...
    result = PyObject_CallMethod(threading, "_shutdown", "");

So calls _shutdown() on the threading module.

That function is aliased to _exitfunc() method of _MainThread.

    def _exitfunc(self):
        self._stop()
        t = _pickSomeNonDaemonThread()
        if t:
            if __debug__:
                self._note("%s: waiting for other threads", self)
        while t:
            t.join()
            t = _pickSomeNonDaemonThread()
        if __debug__:
            self._note("%s: exiting", self)
        self._delete()

So can be done in here.

The decision which would need to be made is whether you call atexit() on all threads before then trying to join() on any, or call atexit() only prior to the join() of the thread.

Calling atexit() on all possibly sounds the better option but I am not sure, plus the code would need to deal with doing two passes like that which may not may not have implications.
msg204439 - (view) Author: Glenn Maynard (Glenn.Maynard) Date: 2013-11-25 23:48
This would be useful.  It shouldn't be part of atexit, since atexit.register() from a thread should register a process-exit handler; instead, something like threading.(un)register_atexit().  If called in a thread, the calls happen when run() returns; if called in the main thread, call them when regular atexits are called (perhaps interleaved with atexit, as if atexit.register had been used).

For example, this can be helpful to handle cleaning up per-thread singletons like database connections.
msg204441 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-11-25 23:56
A couple of years ago I suggested something similar. I'd like to see both thread_start and thread_stop hooks so code can track the creation and destruction of threads. It's a useful feature for e.g. PyLucene or profilers. The callback must be inside the thread and not from the main thread, though.

Perhaps somebody likes to work on a PEP for 3.5?
msg204446 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-11-26 01:47
Most logical would be an API on Thread objects (this would obviously only work with threading-created threads). A PEP also sounds unnecessary for a single new API.
msg204494 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-11-26 15:49
See also the issue #19466: the behaviour of daemon threads changed at Python exit in Python 3.4.
History
Date User Action Args
2013-11-26 15:49:14hayposetnosy: + haypo
messages: + msg204494
2013-11-26 01:47:34pitrousetnosy: + tim.peters
messages: + msg204446
2013-11-25 23:56:53christian.heimessetnosy: + christian.heimes

messages: + msg204441
versions: + Python 3.5, - Python 3.3
2013-11-25 23:48:42Glenn.Maynardsetnosy: + Glenn.Maynard
messages: + msg204439
2012-02-22 20:01:07zvezdansetnosy: + zvezdan
2012-02-22 10:02:24grahamdsetmessages: + msg153947
2012-02-22 09:45:21tareksetmessages: + msg153945
2012-02-22 09:35:00grahamdsetmessages: + msg153943
2012-02-22 09:20:27tareksetmessages: + msg153940
2012-02-22 09:11:36grahamdsetmessages: + msg153939
2012-02-22 08:24:07tareksetmessages: + msg153933
2012-02-22 08:16:59grahamdsetmessages: + msg153932
2012-02-22 08:09:55tareksetmessages: + msg153931
2012-02-21 22:11:03grahamdsetnosy: + grahamd
messages: + msg153905
2012-02-21 14:18:14pitrousetnosy: + neologix

type: behavior -> enhancement
components: + Interpreter Core, Library (Lib)
versions: - Python 3.2, Python 3.4
2012-02-21 14:12:59tareksetnosy: + pitrou
2012-02-21 14:12:07tarekcreate