msg86957 - (view) |
Author: Pascal Chambon (pakal) * |
Date: 2009-05-02 16:27 |
"Usually, daemon processes are processes which got disconnected from
their parent process, and work in the background, often under a
different user identity.
The multiprocessing module has the concept of "daemon" too, but this
time in reference to the "threading" module, in which dameons are just
threads that wont prevent the application termination, even if they are
still running. Thus, daemonic processes launched through multiprocessing
API are normal processes that will be terminated (and not joined) if
non-dameonic processes are all over."
I guess this difference of concepts would deserve a little paragraph of
clarification, in both multiprocessing and threading APIs (does the
paragraph above fit for that ?)
|
msg86959 - (view) |
Author: Jesse Noller (jnoller) * |
Date: 2009-05-02 16:52 |
The multiprocessing lib mimics the threading library, and the
threading.Thread.daemon has always maintained these are not
services/daemons/etc.
I don't see that the clarification is needed, but let me think about it.
|
msg86960 - (view) |
Author: Pascal Chambon (pakal) * |
Date: 2009-05-02 17:10 |
I agree that for someone who discovers the multiprocessing api as a
"generalization" of the threading api, there won't be problems ;
I'm just worried about those (like me) who will see "daemonic" as coming
from unix processes, and not the threading library B-)
Cheers,
pascal
|
msg87105 - (view) |
Author: Vaibhav Mallya (mallyvai) |
Date: 2009-05-04 08:36 |
I understand pakal's line of reasoning. The term 'daemon' in the general
Unix sense has a specific meaning that is at odds with the
multiprocessing module's usage of 'daemon'. Clarification would be
useful, I feel, especially if an outright rename of that part of the API
is out of the question.
|
msg87129 - (view) |
Author: Jesse Noller (jnoller) * |
Date: 2009-05-04 14:11 |
I'm more than familiar with the Unix meaning of Daemon - and the API will
*not* be changed or renamed.
|
msg87152 - (view) |
Author: Pascal Chambon (pakal) * |
Date: 2009-05-04 19:07 |
I guess we all agree on the fact that a renaming of the API would be
highly disproportionate ;-)
A simple line of warning in the doc, next to the daemonic-related
methods, should be sufficient to prevent people like me from wondering
weird stuffs for hours ^^
|
msg89870 - (view) |
Author: Jesse Noller (jnoller) * |
Date: 2009-06-29 18:21 |
committed, r73693 on trunk
|
msg137820 - (view) |
Author: Pascal Chambon (pakal) * |
Date: 2011-06-07 13:06 |
I've just crossed again the doc of the daemon flag (http://docs.python.org/library/multiprocessing.html), and it's still quite confusing to newcomers.
daemon
The process’s daemon flag, a Boolean value. This must be set before start() is called.
The initial value is inherited from the creating process. [1]
When a process exits, it attempts to terminate all of its daemonic child processes.
[1] this sentence is weird: since daemonic processes are not allowed to have children, isn't this flag always False in a new Process instance ?
[2] typo, it meant "all of its NON-daemonic child processes" instead, didn't it ?
|
msg291001 - (view) |
Author: Tim Peters (tim.peters) * |
Date: 2017-04-01 18:07 |
I'll take a crack at these ancient comments ;-)
"""
daemon
The process’s daemon flag, a Boolean value. This must be set before start() is called.
The initial value is inherited from the creating process. [1]
When a process exits, it attempts to terminate all of its daemonic child processes. [2]
[1] this sentence is weird: since daemonic processes are not allowed to have children, isn't this flag always False in a new Process instance ?
"""
Yes. I agree it's confusing; looks like it was pasted from the `threading` docs a long time ago, in which context it had a non-degenerate meaning.
I've personally found this `multiprocessing` restriction (daemonic processes can't have children) to be nothing but a pain in the ass, especially since `Pool` worker processes are (although this doesn't appear to be documented) created as daemonic processes. (Note that `concurrent.futures.ProcessPoolExecutor` doesn't have this restriction.)
"""
[2] typo, it meant "all of its NON-daemonic child processes" instead, didn't it ?
"""
No, this was correct. When a process exits, it waits to join its non-daemonic children, but brutally terminates its daemonic children midstream. For example, run this:
import multiprocessing as mp
from time import sleep
def e(tag):
for i in range(10):
print(tag, i)
sleep(1)
if __name__ == '__main__':
p = mp.Process(target=e, args=("daemon",), daemon=True)
q = mp.Process(target=e, args=("normal",))
p.start()
q.start()
sleep(5)
You'll see that, after 5 seconds, the main process exits and terminates the daemonic process immediately, but waits for the non-daemonic process to finish:
daemon 0
normal 0
daemon 1
normal 1
daemon 2
normal 2
daemon 3
normal 3
daemon 4
normal 4
normal 5
normal 6
normal 7
normal 8
normal 9
|
msg291008 - (view) |
Author: Eryk Sun (eryksun) * |
Date: 2017-04-02 00:19 |
> I've personally found this `multiprocessing` restriction (daemonic
> processes can't have children) to be nothing but a pain in the ass
To make that reliable on Windows you could create a silent-breakaway, kill-on-close Job object in each process for its daemon children. Closing the job handle would cascade terminate all descendant daemon processes in that branch of the process tree, i.e. when a process is terminated it closes the Job handle, which terminates all of its daemon children, which closes their Job handles, which terminates all of their daemon children, and so on. Non-daemons would still be orphaned when the parent is terminated.
|
msg291009 - (view) |
Author: Tim Peters (tim.peters) * |
Date: 2017-04-02 00:50 |
@Eryk, not me ;-) I find the "daemonic or not?" distinction useless for processes - it just gets in the way at times (e.g., a Pool worker dies with an assert(!) error if it tries to create its own Pool for something).
I don't care about orphans either.
It's one reason `concurrent.futures.ProcessPoolExecutor` is sometimes more attractive (it has no "daemon" concept, the processes it creates don't vanish midstream at sometimes surprising times, and neither do they refuse to create additional cf process pools).
|
msg291010 - (view) |
Author: Eryk Sun (eryksun) * |
Date: 2017-04-02 02:29 |
> e.g., a Pool worker dies with an assert(!) error if it tries to create
> its own Pool for something
A daemon process could create a child daemon when the OS can ensure that no orphans are left behind (e.g. call Linux prctl to set PR_SET_PDEATHSIG as SIGKILL). This would allow a pool worker to create another process pool.
|
msg291014 - (view) |
Author: Tim Peters (tim.peters) * |
Date: 2017-04-02 03:02 |
Again, I don't care about orphans. In the usual cases people are running code with no concern about what happens in case of forced termination. The only thing stopping it now is that the code goes out of its way to prevent it (or, alternatively, goes out of its way to force Pool workers to be daemonic). So, e.g., people publish recipes for replacing multiprocessing.Pool to remove this restriction, like here:
http://stackoverflow.com/questions/6974695/python-process-pool-non-daemonic/8963618#8963618
|
msg379371 - (view) |
Author: Terry J. Reedy (terry.reedy) * |
Date: 2020-10-22 22:05 |
Is the paragraph "Usually, daemon processes ..." in Pascal's initial post supposed to be a quote from https://docs.python.org/3/library/multiprocessing.html? 'daemon p' is nowhere in the corrent doc and I find nothing like this paragraph. Is this issue still relevant?
|
msg379690 - (view) |
Author: Pascal Chambon (pakal) * |
Date: 2020-10-26 21:47 |
The latest doc has a quick mention about the fact that daemon is not used in the Unix sens, so it seems fine now B-)
https://docs.python.org/3/library/multiprocessing.html?#multiprocessing.Process.daemon
"""Additionally, these are not Unix daemons or services, they are normal processes that will be terminated (and not joined) if non-daemonic processes have exited."""
My paragraph was just my one attempt at distinguishing concepts, it was never part of the official docs
|
|
Date |
User |
Action |
Args |
2022-04-11 14:56:48 | admin | set | github: 50156 |
2020-10-26 22:45:41 | terry.reedy | set | status: open -> closed resolution: out of date stage: resolved |
2020-10-26 21:47:58 | pakal | set | messages:
+ msg379690 |
2020-10-22 22:05:16 | terry.reedy | set | versions:
+ Python 3.10, - Python 2.6, Python 3.0, Python 3.1, Python 2.7 nosy:
+ terry.reedy
messages:
+ msg379371
assignee: jnoller -> components:
+ Documentation |
2017-04-02 03:02:43 | tim.peters | set | messages:
+ msg291014 |
2017-04-02 02:29:45 | eryksun | set | messages:
+ msg291010 |
2017-04-02 00:50:21 | tim.peters | set | messages:
+ msg291009 |
2017-04-02 00:19:38 | eryksun | set | nosy:
+ eryksun messages:
+ msg291008
|
2017-04-01 18:07:16 | tim.peters | set | nosy:
+ tim.peters messages:
+ msg291001
|
2011-06-07 13:06:28 | pakal | set | status: closed -> open resolution: fixed -> (no value) messages:
+ msg137820
|
2009-06-29 18:21:04 | jnoller | set | status: open -> closed resolution: fixed messages:
+ msg89870
|
2009-05-04 19:07:47 | pakal | set | messages:
+ msg87152 |
2009-05-04 14:11:53 | jnoller | set | messages:
+ msg87129 |
2009-05-04 08:36:38 | mallyvai | set | nosy:
+ mallyvai messages:
+ msg87105
|
2009-05-02 17:10:23 | pakal | set | messages:
+ msg86960 |
2009-05-02 16:52:45 | jnoller | set | messages:
+ msg86959 |
2009-05-02 16:29:22 | benjamin.peterson | set | assignee: jnoller
nosy:
+ jnoller |
2009-05-02 16:27:59 | pakal | create | |