classification
Title: Thread shutdown exception in Thread.notify()
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 2.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: pitrou Nosy List: Rhamphoryncus, amaury.forgeotdarc, ggenellina, gpk, hongqn, jamescooper, lobais, loewis, pitrou, r.david.murray, reacocard, sorin, thijs, thomasda, yangzhang
Priority: normal Keywords: patch

Created on 2007-05-20 22:24 by yangzhang, last changed 2010-04-13 18:41 by brett.cannon. This issue is now closed.

Files
File name Uploaded Description Edit
1722344_squelch_exception.patch jamescooper, 2008-02-05 17:57 Patch to hide these exceptions (and further ones caused by it) for the purposes of releasing a production application.
nondaemon_thread_shutdown.diff Rhamphoryncus, 2008-05-01 23:57 Move WaitForThreadShutdown call into Py_Finalize, so all exit paths use it
Messages (25)
msg32092 - (view) Author: Yang Zhang (yangzhang) Date: 2007-05-20 22:24
Hi, I sometimes see the following exceptions when shutting down my app (using Python 2.5.1):

  Unhandled exception in thread started by
  Error in sys.excepthook:
  
  Original exception was:
  Exception in thread Thread-3 (most likely raised during interpreter shutdown):
  Traceback (most recent call last):
    File "/usr/local/lib/python2.5/threading.py", line 460, in __bootstrap
    File "/usr/local/lib/python2.5/threading.py", line 440, in run
    File "/home/yang/local/armed/lib/python2.5/site-packages/afx/threads.py", line 71, in worker
    File "/usr/local/lib/python2.5/Queue.py", line 176, in get
    File "/usr/local/lib/python2.5/threading.py", line 248, in notify
  <type 'exceptions.TypeError'>: exceptions must be classes, instances, or strings (deprecated), not NoneType
  Unhandled exception in thread started by
  Error in sys.excepthook:
  
  Original exception was:
  Exception in thread Thread-6 (most likely raised during interpreter shutdown):
  Traceback (most recent call last):
    File "/usr/local/lib/python2.5/threading.py", line 460, in __bootstrap
    File "/usr/local/lib/python2.5/threading.py", line 440, in run
    File "/home/yang/local/armed/lib/python2.5/site-packages/afx/threads.py", line 71, in worker
    File "/usr/local/lib/python2.5/Queue.py", line 176, in get
    File "/usr/local/lib/python2.5/threading.py", line 248, in notify
  <type 'exceptions.TypeError'>: exceptions must be classes, instances, or strings (deprecated), not NoneType
  Unhandled exception in thread started by
  Error in sys.excepthook:
  
  Original exception was:

Here is the code from my application:

        def worker():
            debug( 'starting worker' )
            while True:
                msg = i.get() # <-- THIS IS LINE 71
                if msg is stop_msg: break
                resultbuf, func, args, kwargs = msg
                result, exc = None, None
                try:
                    result = func( *args, **kwargs )
                except:
                    t, v, tb = exc_info()
                    exc = t, v, tb.tb_next
                o.put( ( resultbuf, result, exc ) )
                s.send( 'x' ) # assuming socket.send is thread-safe
            debug( 'stopping worker' )

Here is the origin of the exception (in threading.py):

    def notify(self, n=1):
        assert self._is_owned(), "notify() of un-acquire()d lock" # <-- THIS IS LINE 248
        __waiters = self.__waiters
        waiters = __waiters[:n]
        if not waiters:
            if __debug__:
                self._note("%s.notify(): no waiters", self)
            return
        self._note("%s.notify(): notifying %d waiter%s", self, n,
                   n!=1 and "s" or "")
        for waiter in waiters:
            waiter.release()
            try:
                __waiters.remove(waiter)
            except ValueError:
                pass

I'm not sure why this is happening. The threads are not daemon threads; I terminate them cleanly. When I get a SIGINT (I usu. shut down my app with ctrl-C), I enqueue n stop_msg's to the 'i' Queue so that the n workers can all exit.

Note I usually launch 5 workers, so I'm not consistently getting an exception per worker. Also, I've been unable to reproduce this at will.
msg32093 - (view) Author: Gabriel Genellina (ggenellina) Date: 2007-05-21 12:01
Do you join() the worker threads, waiting until they finish, before exiting the main thread?
msg32094 - (view) Author: Yang Zhang (yangzhang) Date: 2007-05-21 14:47
No, as they are not daemon threads.
msg32095 - (view) Author: Thomas Dybdahl Ahle (thomasda) Date: 2007-06-06 13:32
I'm getting the same kind of errors.
I'm using a lot of threads, and one of them throws this thread nearly everytime I close my program (by gtk.main_quit)

It seems that python sets every variable to None, and then wakeup sleeping threads, which then crash, as they try to work with the None variables

Exception in thread Thread-3 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/lib/python2.4/threading.py", line 442, in __bootstrap
  File "/home/thomas/Programmering/python/skak/0.7/lib/pychess/System/ThreadPool.py", line 49, in run
  File "/usr/lib/python2.4/Queue.py", line 89, in put
  File "/usr/lib/python2.4/threading.py", line 237, in notify
exceptions.TypeError: exceptions must be classes, instances, or strings (deprecated), not NoneType
Unhandled exception in thread started by 
Error in sys.excepthook
msg32096 - (view) Author: Greg Kochanski (gpk) Date: 2007-07-30 22:13
I see the same problem.   I'm not sure my code is clean,
but I'll be darned if I can find the problem.

I've traced my code with print statements,
and I see some threads reach their return statement,
but the process hangs when you try to join() that thread.
It seems that there is something in the clean-up code
in threading.py (presumably) that can hang under some
obscure circumstances.

As a result, these threads don't terminate on time,
and various exceptions happen as python is being
dismantled and all the variables de-allocated.
msg62073 - (view) Author: James Cooper (jamescooper) Date: 2008-02-05 17:57
Though these exceptions while shutting down are mostly harmless, they
are very noisy and must be squelched in a production application.  Here
is the patch which we at Solido Design (www.solidodesign.com) are using
to hide the exceptions.  Note that this doesn't fix the problem, but it
does hide the exception for the purposes of releasing an application to
the public.
msg62078 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2008-02-05 20:01
I think the general idea of the problem has been stated, but I figured I
would state the official issue.  When Python begins to shutdown it takes
each module and sets each variable in the global namespace to None. If a
thread has not terminated before the interpreter terminates then the
thread tries to use a global variable which has been set to None.

This is not about to change since this occurs because of coding
"errors". You must make sure that either your thread is as safe as a
__del__ method (which means no global namespace access) or you can't let
the app exit until you are positive all of your threads have terminated,
not just asked them to shutdown since this is all asynchronous.
msg62079 - (view) Author: Adam Olsen (Rhamphoryncus) Date: 2008-02-05 21:24
Py_Main calls WaitForThreadShutdown before calling Py_Finalize, which
should wait for all these threads to finish shutting down before it
starts wiping their globals.

However, if SystemExit is raised (such as via sys.exit()), Py_Exit is
called, and it directly calls Py_Finalize, bypassing the
WaitForThreadShutdown.

Can someone who's experienced this bug check if they're using
SystemExit/sys.exit?
msg62080 - (view) Author: Adam Olsen (Rhamphoryncus) Date: 2008-02-05 21:32
To put it another way: SystemExit turns non-daemon threads into daemon
threads.  This is clearly wrong.  Brent, could you reopen the bug?
msg62081 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2008-02-05 22:57
Hold on, why is that wrong? What if the threads block forever,
preventing shutdown? sys.exit() is not exactly some namby-pamby function
but a forced shutdown of the interpreter that should guarantee that the
interpreter quits. Changing its semantics now would take that away.
msg62083 - (view) Author: Adam Olsen (Rhamphoryncus) Date: 2008-02-05 23:20
I disagree.  sys.exit() attempts to gracefully shutdown the interpreter,
invoking try/finally blocks and the like.  If you want to truly force
shutdown you should use os.abort() or os._exit().

Note that, as python doesn't call a main function, you have to use
sys.exit() to have an exit status.
msg62084 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2008-02-05 23:22
OK, I will re-open to see if some other core developer wants to take
this on, but personally I am passing.
msg62091 - (view) Author: Thomas Dybdahl Ahle (lobais) Date: 2008-02-06 11:36
> which means no global namespace access
Does that mean that you cannot use len and range in a Thread?
msg62108 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2008-02-06 17:55
> > which means no global namespace access
> Does that mean that you cannot use len and range in a Thread?

No, it means you have to be careful if you do. Shutting down properly
will take care of things. Otherwise you need to save a reference
locally (either on an object or as a local variable) and use that
reference instead of relying on the one defined in the global
namespace.
msg66054 - (view) Author: Adam Olsen (Rhamphoryncus) Date: 2008-05-01 23:56
This bug was introduced by r53249, which was fixing bug #1566280.

Fixed by moving the WaitForThreadShutdown call into Py_Finalize, so all
shutdown paths use it.  I also tweaked the name to follow local helper
function conventions.

Martin, since you did the previous fix, can you review this one?
msg66055 - (view) Author: Adam Olsen (Rhamphoryncus) Date: 2008-05-02 00:04
Oh, and the patch includes a testcase.  The current test_threading.py
doesn't work with older versions, but a freestanding version of this
testcase passes in 2.1 to 2.4, fails in 2.5 and trunk, and passes with
the patch.
msg94283 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-10-20 15:08
The patch looks good to me.
And since Py_Finalize() claims to "destroy all sub-interpreters" and
"free all memory allocated by the Python interpreter", I guess your
approach makes sense. Can you commit?
msg94290 - (view) Author: Adam Olsen (Rhamphoryncus) Date: 2009-10-20 16:11
Nope, no access.
msg94291 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-10-20 16:48
Ok, I'll do it then!
msg94303 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-10-20 22:09
Patch was committed in trunk, py3k and 3.1. Waiting for 2.6 to be
unfrozen before I commit it there too.
msg94558 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-10-27 13:10
Backported to 2.6 in r75749.
msg99671 - (view) Author: sorin (sorin) Date: 2010-02-21 17:11
Any idea if there is a nightly build for Python 2.6? The latest release was 2.6.4 and was 2 days before submitting the patch. 

Or the only alternative is to build it myself? Any ideas on when we could see 2.6.5? - I tried to look for a release timeline but I wasn't able to locate one.
msg99686 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2010-02-21 20:50
I have seen somewhere (ask google), that python 2.6.5 would be released mid-march.
But except for a few platforms, python.org does not provide compiled binaries.
msg99694 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-02-21 22:36
According to Barry's latest email on the subject, the dates are:

2009-03-01 Python 2.6.5 rc 1
2009-03-15 Python 2.6.5 final

And no, there are no nightly builds, you have to build it yourself, I'm afraid.
msg103025 - (view) Author: Thijs Triemstra (thijs) Date: 2010-04-13 09:18
Looks like this influenced mod_wsgi as well: http://groups.google.com/group/modwsgi/browse_thread/thread/ba82b2643564d2dd
History
Date User Action Args
2010-09-28 22:14:32twouterslinkissue4684 superseder
2010-04-13 18:41:19brett.cannonsetnosy: - brett.cannon
2010-04-13 09:18:14thijssetnosy: + thijs
messages: + msg103025
2010-02-21 22:36:01r.david.murraysetnosy: + r.david.murray
messages: + msg99694
2010-02-21 20:50:24amaury.forgeotdarcsetmessages: + msg99686
2010-02-21 17:11:54sorinsetnosy: + sorin
messages: + msg99671
2009-10-27 13:10:46pitrousetstatus: pending -> closed
resolution: accepted -> fixed
messages: + msg94558
2009-10-20 22:09:23pitrousetstatus: open -> pending

stage: commit review -> resolved
messages: + msg94303
versions: - Python 3.1, Python 2.7, Python 3.2
2009-10-20 16:48:03pitrousetassignee: pitrou
messages: + msg94291
2009-10-20 16:11:38Rhamphoryncussetmessages: + msg94290
2009-10-20 15:08:16pitrousetresolution: accepted
stage: patch review -> commit review
messages: + msg94283
versions: + Python 3.2, - Python 3.0
2009-05-19 07:56:40pitrousetnosy: + pitrou
versions: + Python 2.6, Python 3.0, Python 3.1, Python 2.7, - Python 2.5

type: behavior
stage: patch review
2009-05-19 03:15:28reacocardsetnosy: + reacocard
2009-01-26 13:50:49hongqnsetnosy: + hongqn
2008-05-02 00:04:14Rhamphoryncussetmessages: + msg66055
2008-05-01 23:57:07Rhamphoryncussetfiles: + nondaemon_thread_shutdown.diff
nosy: + loewis
messages: + msg66054
keywords: + patch
2008-02-06 17:55:08brett.cannonsetmessages: + msg62108
2008-02-06 13:00:15amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
2008-02-06 11:36:13lobaissetnosy: + lobais
messages: + msg62091
2008-02-05 23:22:47brett.cannonsetstatus: closed -> open
resolution: wont fix -> (no value)
messages: + msg62084
2008-02-05 23:20:21Rhamphoryncussetmessages: + msg62083
2008-02-05 22:57:52brett.cannonsetmessages: + msg62081
2008-02-05 21:32:13Rhamphoryncussetmessages: + msg62080
2008-02-05 21:24:48Rhamphoryncussetnosy: + Rhamphoryncus
messages: + msg62079
2008-02-05 20:01:28brett.cannonsetstatus: open -> closed
nosy: + brett.cannon
resolution: wont fix
messages: + msg62078
2008-02-05 17:57:57jamescoopersetfiles: + 1722344_squelch_exception.patch
nosy: + jamescooper
messages: + msg62073
2007-05-20 22:24:37yangzhangcreate