classification
Title: Close background process if IDLE closes abnormally.
Type: behavior Stage: resolved
Components: IDLE Versions: Python 3.3, Python 3.4, Python 2.7
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: asvetlov Nosy List: asvetlov, markroseman, roger.serwy, terry.reedy
Priority: normal Keywords: patch

Created on 2012-03-29 13:28 by asvetlov, last changed 2019-05-03 19:23 by asvetlov. This issue is now closed.

Files
File name Uploaded Description Edit
issue14440.patch roger.serwy, 2012-04-03 03:47 review
issue14440_rev2.patch roger.serwy, 2012-04-04 00:01 review
Messages (12)
msg157043 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2012-03-29 13:28
Now if IDLE was ran from console and then terminated by <Ctrl-\> or kill signal — background process keep living forever.

That process have to stop itself if there are no frontend IDLE.
msg157164 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2012-03-31 03:16
You need to specify os (with version) *nix?
There have been issues like this before, for earlier Python version, at least for Windows XP, which have been fixed, at least for Windows. I think last patch (a year ago? before 3.2 I think) switched to using subprocess instead of older methods. If I kill Idle with task manager, the other, background pythonw process dies in under half a second.
(If background process dies, IDLE restarts.)
msg157170 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2012-03-31 11:32
I use Ubuntu Linux 11.10

Hmm, you are right: background process dies. But I'm pretty sure when I worked on IDLE bugs three weeks ago sometimes that process remained to live forever.

Aahh. 
At least sending SIGTERM (kill -9) to foreground IDLE process doesn't delete background. Also the same if foreground hangs by some error in code and should be killed from console. <Ctrl-C> doesn't help in this particular case, <Ctrl+\> (SIGQUIT) terminates foreground but background lives forever.
msg157209 - (view) Author: Roger Serwy (roger.serwy) * (Python committer) Date: 2012-03-31 17:16
I can confirm this problem with Ubuntu 11.04.
msg157221 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2012-03-31 18:31
Andrew: I strongly agree with the goal that IDLE should not leave zombie processes. The background process should die if either 1) IDLE restarts the shell with a new background process, as with every edit-run cycle, or 2) IDLE dies. The desired behavior seems to be both somewhat fragile and system dependent. I do not know whether 100% compliance on every system is sensibly possible.

The issue I referred to is #12540. The problem there was worse: leaving a zombie for every shell restart on Windows. Perhaps the discussion there will give you some ideas.

Ctrl-\ does not seem to do anything on Windows. I do not know whether TaskManager 'Terminate process' corresponds to *nix SIGTERM, SIGQUIT, SIGKILL, something else, or is completely Windows specific.
msg157225 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2012-03-31 20:02
Terry, sorry.

That's definitely posix-specific bug. I'll make a patch assuming Windows works well with killing IDLE.

To be polite I'll describe used signal names shortly.

SIGKILL, SIGTERM, SIGINT and SIGQUIT are used to stop process.
— SIGKILL is mimic to Windows TerminateProcess in some way. 
  It cannot be overloaded by target process and unconditionally terminates one.
— SIGTERM is default signal for kill utility. Used to stop process by some program.
— SIGINT sent if user press Ctrl+C in terminal. 
  Overridden very often.
  Python does it itself to translate this signal to KeyboardInterrupt exception.
— SIGQUIT bound to Ctrl+\ and used to stop program with memory dump. 
  The standard way to process it 
  (if need to, the most programs leaves that handler untouched) is: 
  close used resources and maybe write to stderr/logs some message. 
  Usually SIGQUIT sent to hang program to stop execution 
  if Ctrl+C doesn't help but potentially dangerous SIGKILL is not desirably 
  by risk of lacking important data or locking kernel objects.
msg157245 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2012-03-31 23:58
Just a bit more info: ^D and ^\ in Command Prompt interpreter print '^D' or '^\'. Return causes syntax error. ^Z\n in interpreter causes silent close. (Because DOS used ^Z as end-of-file.)

^D in IDLE causes silent close. IDLE ignores ^\ and ^Z both -- nothing printed or stored. Following \n gets prompt back. The difference is not nice for Windows users, but I presume IDLE behavior is same as on *nix and consider consistency across platforms good and should be kept.

If I understand, IDLE needs to install a SIGQUIT handler to terminate the background process 'resource'. That should work on Windows too as either SIGQUIT will never happens, or if it does, same should happen on Windows too. I know SIGKILL cannot be caught. What about SIGTERM?
msg157271 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2012-04-01 10:17
We can install signal handlers for everything what can stop process but I prefer to pass IDLE pid to subintepreter and periodically check for prime process existing.
msg157393 - (view) Author: Roger Serwy (roger.serwy) * (Python committer) Date: 2012-04-03 03:47
This bug is related to issue12540. The approach taken there is to have the IDLE frontend explicitly kill the subprocess. It's a band-aid to the problem that run.py doesn't exit when the socket to the IDLE frontend closes (either by shell restart or kill -9 on IDLE).

Attached is a patch to cause the subprocess to exit. I have to admit not fully understanding why it works (on Ubuntu 11.04).

It looks like the following code in _getresponse() in rpc.py is what keeps the subprocess running:

    cvar.acquire()
    while myseq not in self.responses:
        cvar.wait()

I also tried disabling the "terminate_subprocess" in PyShell.py. With that change, the subprocess does not terminate on a shell restart (unless my patch is applied). 

Andrew, Terry: What are your thoughts?
msg157420 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2012-04-03 14:14
I still prefer to check in subprocess for parent proc existing.
msg157449 - (view) Author: Roger Serwy (roger.serwy) * (Python committer) Date: 2012-04-04 00:01
Andrew, the reason the subprocess is not closing is due to a thread management problem. I found that a dummy thread handles cvar.wait() which is why the subprocess fails to terminate properly. 

If you change the tight loop in _getresponse to be:

    while myseq not in self.responses:
        print(threading.enumerate(), file=sys.__stderr__)
        sys.__stderr__.flush()
        cvar.wait(1)

Then you'll get the following dumped to the terminal once a second:
    [<_DummyThread(Dummy-1, started daemon 139991862630176)>, <Thread(SockThread, started daemon 139991766066944)>]

The MainThread already stopped, but these two daemon threads don't terminate, which is strange. Is this a bug in itself?

Polling the OS for the IDLE frontend process will give an indication to terminate the subprocess, but actually terminating the subprocess from within the subprocess is the main problem.

Attached is a patch which takes a different approach to terminating the subprocess by using .shutdown instead.
msg341353 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2019-05-03 19:23
outdated
History
Date User Action Args
2019-05-03 19:23:08asvetlovsetstatus: open -> closed
resolution: out of date
messages: + msg341353

stage: needs patch -> resolved
2015-09-18 16:32:02markrosemansetnosy: + markroseman
2013-06-15 18:55:25terry.reedysetversions: + Python 2.7, Python 3.4
2012-04-04 00:01:41roger.serwysetfiles: + issue14440_rev2.patch

messages: + msg157449
2012-04-03 14:14:02asvetlovsetmessages: + msg157420
2012-04-03 03:47:51roger.serwysetfiles: + issue14440.patch
keywords: + patch
messages: + msg157393
2012-04-01 10:17:16asvetlovsetmessages: + msg157271
2012-03-31 23:58:20terry.reedysetmessages: + msg157245
2012-03-31 20:02:03asvetlovsetmessages: + msg157225
2012-03-31 18:31:47terry.reedysetmessages: + msg157221
2012-03-31 17:16:08roger.serwysetnosy: + roger.serwy
messages: + msg157209
2012-03-31 11:37:00asvetlovsettype: behavior
2012-03-31 11:32:48asvetlovsetmessages: + msg157170
2012-03-31 03:16:25terry.reedysetnosy: + terry.reedy
messages: + msg157164
2012-03-29 13:31:09asvetlovsetcomponents: + IDLE
2012-03-29 13:28:32asvetlovcreate