classification
Title: asyncio subprocesses allow pids to be reaped, different behavior than regular subprocesses
Type: behavior Stage:
Components: asyncio Versions: Python 3.9, Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Justin.Lebar, asvetlov, c-lewin, yselivanov
Priority: normal Keywords:

Created on 2020-03-26 17:06 by Justin.Lebar, last changed 2020-04-19 03:14 by c-lewin.

Messages (2)
msg365095 - (view) Author: Justin Lebar (Justin.Lebar) Date: 2020-03-26 17:06
From https://bugs.python.org/issue1187312 about regular subprocesses:

> So as long as the application keeps a reference to the
> subprocess object, it can wait for it; auto-reaping only
> starts when the last reference was dropped [in Popen.__del__].

asyncio subprocesses seem to behave differently.  When we notice the process has exited in BaseSubprocessTransport._process_exited, we call _try_finish(), which -- if all pipes are closed -- calls _call_connection_lost and sets self._proc to None.

At this point, my understanding is that once self._proc is GC'ed, we'll run Popen.__del__ and may reap the pid.

I would expect asyncio subprocesses to behave the same way as regular Popen objects wrt pid reaping.
msg366752 - (view) Author: Carl Lewin (c-lewin) Date: 2020-04-19 03:14
Very first time engaging in such a forum. Apologies is advance if I am doing it wrong!

Observation: ps -ef shows "Defunct" process until calling script terminates

Scenario: equivalent test scripts in BASH, Python 2.7 and 3.6 that:
1. Start a ping
2. SIGTERM (kill -15) the associated PID 
3. wait for a user input (hence stopping the script terminating)

I tried P.Open and threading but behaviour is same.

BASH script does not show any "defunct" process.

Is this "Child Reaping" the cause of this observed behaviour?

Problem comes when the Parent script is required to run constantly (server type scenario) as the "defunct" processes will presumably eventually consume all system resources?
History
Date User Action Args
2020-04-19 03:14:10c-lewinsetnosy: + c-lewin
messages: + msg366752
2020-03-26 17:09:50Justin.Lebarsetversions: + Python 3.7, Python 3.8
2020-03-26 17:09:42Justin.Lebarsettype: behavior
versions: + Python 3.9
2020-03-26 17:06:33Justin.Lebarcreate