classification
Title: Risk of confusion in multiprocessing module - daemonic processes
Type: Stage: resolved
Components: Documentation, Library (Lib) Versions: Python 3.10
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: eryksun, jnoller, mallyvai, pakal, terry.reedy, tim.peters
Priority: normal Keywords:

Created on 2009-05-02 16:27 by pakal, last changed 2020-10-26 22:45 by terry.reedy. This issue is now closed.

Messages (15)
msg86957 - (view) Author: Pascal Chambon (pakal) * Date: 2009-05-02 16:27
"Usually, daemon processes are processes which got disconnected from
their parent process, and work in the background, often under a
different user identity.
The multiprocessing module has the concept of "daemon" too, but this
time in reference to the "threading" module, in which dameons are just
threads that wont prevent the application termination, even if they are
still running. Thus, daemonic processes launched through multiprocessing
API are normal processes that will be terminated (and not joined) if 
non-dameonic processes are all over."

I guess this difference of concepts would deserve a little paragraph of
clarification, in both multiprocessing and threading APIs (does the
paragraph above fit for that ?)
msg86959 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2009-05-02 16:52
The multiprocessing lib mimics the threading library, and the 
threading.Thread.daemon has always maintained these are not 
services/daemons/etc.

I don't see that the clarification is needed, but let me think about it.
msg86960 - (view) Author: Pascal Chambon (pakal) * Date: 2009-05-02 17:10
I agree that for someone who discovers the multiprocessing api as a
"generalization" of the threading api, there won't be problems ; 

I'm just worried about those (like me) who will see "daemonic" as coming
from unix processes, and not the threading library B-)

Cheers, 
pascal
msg87105 - (view) Author: Vaibhav Mallya (mallyvai) Date: 2009-05-04 08:36
I understand pakal's line of reasoning. The term 'daemon' in the general
Unix sense has a specific meaning that is at odds with the
multiprocessing module's usage of 'daemon'. Clarification would be
useful, I feel, especially if an outright rename of that part of the API
is out of the question.
msg87129 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2009-05-04 14:11
I'm more than familiar with the Unix meaning of Daemon - and the API will 
*not* be changed or renamed.
msg87152 - (view) Author: Pascal Chambon (pakal) * Date: 2009-05-04 19:07
I guess we all agree on the fact that a renaming of the API would be
highly disproportionate ;-)
A simple line of warning in the doc, next to the daemonic-related
methods, should be sufficient to prevent people like me from wondering
weird stuffs for hours ^^
msg89870 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2009-06-29 18:21
committed, r73693 on trunk
msg137820 - (view) Author: Pascal Chambon (pakal) * Date: 2011-06-07 13:06
I've just crossed again the doc of the daemon flag (http://docs.python.org/library/multiprocessing.html), and it's still quite confusing to newcomers.

daemon
    The process’s daemon flag, a Boolean value. This must be set before start() is called.
    The initial value is inherited from the creating process. [1]
    When a process exits, it attempts to terminate all of its daemonic child processes.

[1] this sentence is weird: since daemonic processes are not allowed to have children, isn't this flag always False in a new Process instance ?
[2] typo, it meant "all of its NON-daemonic child processes" instead, didn't it ?
msg291001 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2017-04-01 18:07
I'll take a crack at these ancient comments ;-)

"""
daemon
    The process’s daemon flag, a Boolean value. This must be set before start() is called.
    The initial value is inherited from the creating process. [1]
    When a process exits, it attempts to terminate all of its daemonic child processes. [2]

[1] this sentence is weird: since daemonic processes are not allowed to have children, isn't this flag always False in a new Process instance ?
"""

Yes.  I agree it's confusing; looks like it was pasted from the `threading` docs a long time ago, in which context it had a non-degenerate meaning.

I've personally found this `multiprocessing` restriction (daemonic processes can't have children) to be nothing but a pain in the ass, especially since `Pool` worker processes are (although this doesn't appear to be documented) created as daemonic processes.  (Note that `concurrent.futures.ProcessPoolExecutor` doesn't have this restriction.)

"""
[2] typo, it meant "all of its NON-daemonic child processes" instead, didn't it ?
"""

No, this was correct.  When a process exits, it waits to join its non-daemonic children, but brutally terminates its daemonic children midstream.  For example, run this:

import multiprocessing as mp
from time import sleep

def e(tag):
    for i in range(10):
        print(tag, i)
        sleep(1)

if __name__ == '__main__':
    p = mp.Process(target=e, args=("daemon",), daemon=True)
    q = mp.Process(target=e, args=("normal",))
    p.start()
    q.start()
    sleep(5)

You'll see that, after 5 seconds, the main process exits and terminates the daemonic process immediately, but waits for the non-daemonic process to finish:

daemon 0
normal 0
daemon 1
normal 1
daemon 2
normal 2
daemon 3
normal 3
daemon 4
normal 4
normal 5
normal 6
normal 7
normal 8
normal 9
msg291008 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2017-04-02 00:19
> I've personally found this `multiprocessing` restriction (daemonic 
> processes can't have children) to be nothing but a pain in the ass 

To make that reliable on Windows you could create a silent-breakaway, kill-on-close Job object in each process for its daemon children. Closing the job handle would cascade terminate all descendant daemon processes in that branch of the process tree, i.e. when a process is terminated it closes the Job handle, which terminates all of its daemon children, which closes their Job handles, which terminates all of their daemon children, and so on. Non-daemons would still be orphaned when the parent is terminated.
msg291009 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2017-04-02 00:50
@Eryk, not me ;-)  I find the "daemonic or not?" distinction useless for processes - it just gets in the way at times (e.g., a Pool worker dies with an assert(!) error if it tries to create its own Pool for something).

I don't care about orphans either.

It's one reason `concurrent.futures.ProcessPoolExecutor` is sometimes more attractive (it has no "daemon" concept, the processes it creates don't vanish midstream at sometimes surprising times, and neither do they refuse to create additional cf process pools).
msg291010 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2017-04-02 02:29
> e.g., a Pool worker dies with an assert(!) error if it tries to create 
> its own Pool for something

A daemon process could create a child daemon when the OS can ensure that no orphans are left behind (e.g. call Linux prctl to set PR_SET_PDEATHSIG as SIGKILL). This would allow a pool worker to create another process pool.
msg291014 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2017-04-02 03:02
Again, I don't care about orphans.  In the usual cases people are running code with no concern about what happens in case of forced termination.  The only thing stopping it now is that the code goes out of its way to prevent it (or, alternatively, goes out of its way to force Pool workers to be daemonic).  So, e.g., people publish recipes for replacing multiprocessing.Pool to remove this restriction, like here:

http://stackoverflow.com/questions/6974695/python-process-pool-non-daemonic/8963618#8963618
msg379371 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2020-10-22 22:05
Is the paragraph "Usually, daemon processes ..." in Pascal's initial post supposed to be a quote from https://docs.python.org/3/library/multiprocessing.html?  'daemon p' is nowhere in the corrent doc and I find nothing like this paragraph.  Is this issue still relevant?
msg379690 - (view) Author: Pascal Chambon (pakal) * Date: 2020-10-26 21:47
The latest doc has a quick mention about the fact that daemon is not used in the Unix sens, so it seems fine now  B-)

https://docs.python.org/3/library/multiprocessing.html?#multiprocessing.Process.daemon

"""Additionally, these are not Unix daemons or services, they are normal processes that will be terminated (and not joined) if non-daemonic processes have exited."""

My paragraph was just my one attempt at distinguishing concepts, it was never part of the official docs
History
Date User Action Args
2020-10-26 22:45:41terry.reedysetstatus: open -> closed
resolution: out of date
stage: resolved
2020-10-26 21:47:58pakalsetmessages: + msg379690
2020-10-22 22:05:16terry.reedysetversions: + Python 3.10, - Python 2.6, Python 3.0, Python 3.1, Python 2.7
nosy: + terry.reedy

messages: + msg379371

assignee: jnoller ->
components: + Documentation
2017-04-02 03:02:43tim.peterssetmessages: + msg291014
2017-04-02 02:29:45eryksunsetmessages: + msg291010
2017-04-02 00:50:21tim.peterssetmessages: + msg291009
2017-04-02 00:19:38eryksunsetnosy: + eryksun
messages: + msg291008
2017-04-01 18:07:16tim.peterssetnosy: + tim.peters
messages: + msg291001
2011-06-07 13:06:28pakalsetstatus: closed -> open
resolution: fixed -> (no value)
messages: + msg137820
2009-06-29 18:21:04jnollersetstatus: open -> closed
resolution: fixed
messages: + msg89870
2009-05-04 19:07:47pakalsetmessages: + msg87152
2009-05-04 14:11:53jnollersetmessages: + msg87129
2009-05-04 08:36:38mallyvaisetnosy: + mallyvai
messages: + msg87105
2009-05-02 17:10:23pakalsetmessages: + msg86960
2009-05-02 16:52:45jnollersetmessages: + msg86959
2009-05-02 16:29:22benjamin.petersonsetassignee: jnoller

nosy: + jnoller
2009-05-02 16:27:59pakalcreate