This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: subprocess.Popen should not emit a ResourceWarning for a detached process
Type: behavior Stage:
Components: Library (Lib), Windows Versions: Python 3.10, Python 3.9, Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: ShaneHarvey, behackett, eryksun, gregory.p.smith, paul.moore, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2019-11-22 00:04 by ShaneHarvey, last changed 2022-04-11 14:59 by admin.

Files
File name Uploaded Description Edit
test_daemon_win.py ShaneHarvey, 2019-11-22 00:04
Messages (6)
msg357237 - (view) Author: Shane Harvey (ShaneHarvey) * Date: 2019-11-22 00:04
In https://bugs.python.org/issue26741 Popen was changed to emit a ResourceWarning in __del__ if the process is still running. However, when running a daemon/detached process it is completely valid to delete a Popen object before the process is complete.

On Windows, a daemon process can be created by using the DETACHED_PROCESS option, eg:
import subprocess

DETACHED_PROCESS = getattr(subprocess, 'DETACHED_PROCESS', 0x00000008)
popen = subprocess.Popen(args, creationflags=DETACHED_PROCESS)
print('Running daemon process: ', popen.pid)
del popen

Unfortunately, the above will emit the warning:
C:\python\Python37\lib\subprocess.py:839: ResourceWarning: subprocess 840 is still running
  ResourceWarning, source=self)
Running daemon process:  3212

This makes it complicated to run a daemon process without emitting the resource warning. The best solution I've got is to manually set the Popen.returncode to circumvent the warning, ie:

popen = subprocess.Popen(args, creationflags=DETACHED_PROCESS)
print('Running daemon process: ', popen.pid)
# Set the returncode to avoid erroneous warning:
# "ResourceWarning: subprocess XXX is still running".
popen.returncode = 0
del popen


Running the attached script on Windows only one warning is emitted:
$ python.exe -Wall test_daemon_win.py
C:\python\Python37\lib\subprocess.py:839: ResourceWarning: subprocess 3584 is still running
  ResourceWarning, source=self)
Running daemon process:  3584
Running daemon process:  1012

I propose that Popen should not raise the resource warning when the creationFlags includes DETACHED_PROCESS.
msg357451 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2019-11-25 18:06
For a console application, normally the system connects to either the console that's inherited from the parent or, if no console is inherited, a newly allocated console. The creation flag DETACHED_PROCESS sets the initial console handle in the child to a special value that tells the system to skip connecting to a console. If the child process doesn't subsequently call AllocConsole, AttachConsole, or create a visible top-level window, then it will considered to be background process. 

Note that 'detached' in this case doesn't mean detached from the parent process. There's no such attachment to begin with because Windows does not maintain a process tree. It uses a nested job-object hierarchy as the equivalent of a Unix process tree. The creation flag CREATE_BREAKAWAY_FROM_JOB breaks away from jobs that allow it.

The Popen warning is a tool to help programmers become aware of leaked child processes. I do think there should be a documented way to disable this warning for a child process. However, this is unrelated to the DETACHED_PROCESS flag.
msg357455 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-11-25 19:03
Didn't we clean up this warning completely on Windows recently?

It apparently matters on POSIX to join() the child process, but on Windows you aren't going to leak any resources by just forgetting about it, so I thought we stopped tracking them entirely.
msg357458 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2019-11-25 19:37
> Didn't we clean up this warning completely on Windows recently? It
> apparently matters on POSIX to join() the child process, but on
> Windows you aren't going to leak any resources by just forgetting
> about it, so I thought we stopped tracking them entirely.

There's only a PID in POSIX, so the OS keeps a zombie process until the parent waits on it. Windows uses a handle for the process, which gets automatically closed when the Popen instance is finalized, so we don't have to explicitly wait on the process, and doing so has no bearing on the lifetime of the kernel process object. 

Recently, a change was made to set _active to None in Windows, so _cleanup is no longer called unnecessarily. But Victor wanted to keep the resource warning for its educational value, given some programmers might not be aware when they've left a background process running. So Popen.__del__ still issues a resource warning if returncode is None. The OP's workaround to manually set returncode works, but it's undocumented and feels like a kludge to me.
msg388906 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-03-17 04:59
I wonder if this should be handled more productively by supporting a `daemon` parameter. In POSIX, use the double fork technique for creating a daemon process. In Windows, use DETACHED_PROCESS | CREATE_NEW_PROCESS_GROUP | CREATE_BREAKAWAY_FROM_JOB. For a daemon process, the finalizer would not emit a resource warning or add it to the _active list.
msg389196 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2021-03-20 23:05
On POSIX the norm for anything daemonizing itself is to fork() and let the parent die so that its original process with the child pid has ended.  But I'm used to this being the responsibility of the daemon process.  Not the code launching the daemon.

Related, a deferred PEP on supporting any form of daemon stuff in the stdlib:  https://www.python.org/dev/peps/pep-3143/ along with https://pypi.org/project/python-daemon/ and a plethora of other daemon/daemonize related PyPI libraries.

Those tend to be about implement proper become a daemon behavior in Python processes themselves.
History
Date User Action Args
2022-04-11 14:59:23adminsetgithub: 83071
2021-03-20 23:05:48gregory.p.smithsetnosy: + gregory.p.smith
messages: + msg389196
2021-03-17 04:59:05eryksunsettitle: A subprocess.Popen created with creationFlags=DETACHED_PROCESS on Windows should not emit a ResourceWarning -> subprocess.Popen should not emit a ResourceWarning for a detached process
messages: + msg388906
versions: + Python 3.10, - Python 3.6, Python 3.7
2019-11-25 19:37:21eryksunsetmessages: + msg357458
2019-11-25 19:03:09steve.dowersetmessages: + msg357455
2019-11-25 18:08:25eryksunsetnosy: + paul.moore, tim.golden, zach.ware, steve.dower
components: + Windows
2019-11-25 18:06:38eryksunsetnosy: + eryksun
messages: + msg357451
2019-11-22 02:45:26behackettsetnosy: + behackett
2019-11-22 00:04:10ShaneHarveycreate