classification
Title: threading.Condition.wait() is not interruptible in Python 2.7
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 2.7
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: josh.r, ned.deily, neologix, pitrou, sangeeth
Priority: normal Keywords:

Created on 2014-07-04 00:40 by sangeeth, last changed 2014-07-12 04:52 by neologix. This issue is now closed.

Messages (8)
msg222250 - (view) Author: Sangeeth Saravanaraj (sangeeth) Date: 2014-07-04 00:40
Python version 2.7.7
Mac OS Darwin Kernel Version 13.2.0

I have the following code which when executed waits to be interrupted by SIGINT, SIGTERM or SIGQUIT. When an object is initialized, it creates a threading.Condition() and acquires() it! The program then registers the signal handlers where notify() and release() is called when the above mentioned signals are received. After registering the signal handlers, it calls wait() on the condition variable and block.

When I tried to stop the program with Ctrl-C, its did not respond. IOW, the _signal_handler() method did not get called.  

# start

from signal import signal, SIGINT, SIGTERM, SIGQUIT
from threading import Condition

class A:
    def __init__(self):
        self._termination_signal = Condition()
        self._termination_signal.acquire(blocking=0)

    def _signal_handler(self, signum, frame):
        print "Received terminate request - signal = {0}".format(signum)
        del frame
        self._termination_signal.notify()
        self._termination_signal.release()
        return

    def register_and_wait(self):
        signal(SIGINT, self._signal_handler)
        signal(SIGTERM, self._signal_handler)
        signal(SIGQUIT, self._signal_handler)
        print "Waiting to be interrupted!"
        self._termination_signal.wait()      # control blocks here!
        print "Notified!!"

def main():
    a = A()
    a.register_and_wait()

if __name__ == "__main__":
    main()

# end


When the same code was tried in Python 3.4, it threw a "RuntimeError: cannot notify on un-acquired lock".

More information is available in this conversation in python-list mailer - https://mail.python.org/pipermail/python-list/2014-July/674350.html
msg222256 - (view) Author: Josh Rosenberg (josh.r) * (Python triager) Date: 2014-07-04 01:32
So you want it to raise the same exception that is raised in Python 3.4? Either way, I don't think this is a deadlock (to get a deadlock, you'd have to have contested locks; this lock is being acquired only once). You're explicitly violating the rules of using Conditions.

To be precise, you:

1. Acquire the Condition's lock in the main thread (legal, if odd to do in the constructor)
2. Call the Condition's wait method (legal; you acquired the lock so you can wait)
3. (Implicit) By wait-ing, you implicitly release the lock
4. At some point, you hit Ctrl-C. If the signal handler is called as expected, it will try to call Condition.notify(), which is illegal, because the lock was released as a side-effect of calling Condition.wait() earlier.

So best case scenario, this code is wrong. The fact that it doesn't work correctly in Python 2.7 is an issue, but the possible reasons go well beyond deadlocks; you've written code that is guaranteed to be wrong (throw RuntimeExceptions) even if all goes according to plan.

Do you get more reasonable behavior if you write the class as:

class A:
    def __init__(self):
        self._termination_signal = Condition()

    def _signal_handler(self, signum, frame):
        print "Received terminate request - signal = {0}".format(signum)
        del frame
        with self._termination_signal:
            self._termination_signal.notify()

    def register_and_wait(self):
        with self._termination_signal:
            signal(SIGINT, self._signal_handler)
            signal(SIGTERM, self._signal_handler)
            signal(SIGQUIT, self._signal_handler)
            print "Waiting to be interrupted!"
            self._termination_signal.wait()      # control blocks here!
        print "Notified!!"
msg222264 - (view) Author: Sangeeth Saravanaraj (sangeeth) Date: 2014-07-04 05:23
I am convinced that the code is wrong. It was written with wrong assumptions. But Python 2.7 behaves differently compare to Python 3.4. I am not expecting the same behavior in Python 2.7 as in Python 3.4 but I am expecting that some form of exception is raised justifying why Condition.notify() is illegal (as per #4) in this scenario and not block.

In 3.4, I could see that the signal handler is being called (when hitting Ctrl-c) and then Condition.notify() is raising a RuntimeError. But in 2.7, I am not able to prove that the signal handler is called at all. That makes me think why the signal is not caught when control is in Condition.wait().  

Your updated code snippet behaves the same as my original code. It blocks at Condition.wait() and does not respond to Ctrl-C and no signal handler is called and also not raising any exceptions.

Thanks for the beautiful explanation!
msg222349 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-07-05 12:36
So the problem is mostly that 2.7 gives less diagnosis information than 3.x about an incorrect use of the API. I don't think that's very worthy of a bugfix, IMHO. Just use 3.x :-)
msg222365 - (view) Author: Josh Rosenberg (josh.r) * (Python triager) Date: 2014-07-05 14:15
Antoine: It's possible this is a legitimate failure in the signal handling code. The lack of a RuntimeError seems more likely to be due to the code never executing, not an issue with Condition.wait/Condition.notify.
msg222385 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-07-06 00:28
Josh, your analysis is right, I had forgotten that we had improved this point in 3.x (interruptibility of lock.acquire()). It is *extremely* unlikely to be backported to 2.7, though, since it is really an enhancement.
msg222651 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2014-07-10 06:19
So do we agree that the resolution for this is "wont fix"?
msg222823 - (view) Author: Charles-Fran├žois Natali (neologix) * (Python committer) Date: 2014-07-12 04:52
> So do we agree that the resolution for this is "wont fix"?

Yes.
We don't want to backport this (we've had enough regressions already, and people have been living withut loc interruptibility for a long time before Python 3).
History
Date User Action Args
2014-07-12 04:52:34neologixsetstatus: open -> closed
title: Possible deadlock in threading.Condition.wait() in Python 2.7.7 -> threading.Condition.wait() is not interruptible in Python 2.7
messages: + msg222823

resolution: wont fix
stage: resolved
2014-07-10 06:19:17ned.deilysetnosy: + ned.deily
messages: + msg222651
2014-07-06 00:28:36pitrousetmessages: + msg222385
2014-07-05 14:15:28josh.rsetmessages: + msg222365
2014-07-05 12:36:46pitrousetnosy: + neologix
messages: + msg222349
2014-07-04 05:23:51sangeethsetmessages: + msg222264
2014-07-04 01:32:58josh.rsetnosy: + josh.r
messages: + msg222256
2014-07-04 00:40:19sangeethcreate