classification
Title: multiprocessing sentinel resource leak
Type: resource usage Stage: resolved
Components: Library (Lib) Versions: Python 3.7, Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Winterflower, davin, haypo, holdenweb, jnoller, pitrou, quick-b, sbt
Priority: normal Keywords:

Created on 2016-04-10 21:58 by quick-b, last changed 2017-07-31 00:12 by quick-b.

Pull Requests
URL Status Linked Edit
PR 2813 merged pitrou, 2017-07-22 10:04
Messages (13)
msg263153 - (view) Author: Kevin Quick (quick-b) Date: 2016-04-10 21:58
The sentinel creates a named pipe, but the parent's end of the pipe is inherited by subsequently created children.  

import multiprocessing,signal,sys
def sproc(x): signal.pause()
for each in range(int(sys.argv[1])):
  multiprocessing.Process(target=sproc, args=(each,)).start()
signal.pause()

Running the above on Linux with varying numbers of child processes (expressed as the argument to the above) and using techniques like "$ sudo ls /proc/NNNN/fd" it is possible to see an ever growing number of pipe connections for subsequent children.
msg263154 - (view) Author: Kevin Quick (quick-b) Date: 2016-04-10 21:59
(Sorry, an unnamed pipe, but a pipe nonetheless.)
msg263282 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2016-04-12 22:57
I confirm the issue with Python 3.6 on Linux.

File descriptors of the parent process:

haypo@selma$ ls -l /proc/31564/fd/
lrwx------. 1 haypo haypo 64 13 avril 00:55 0 -> /dev/pts/0
lrwx------. 1 haypo haypo 64 13 avril 00:55 1 -> /dev/pts/0
lr-x------. 1 haypo haypo 64 13 avril 00:55 10 -> pipe:[697354]
lr-x------. 1 haypo haypo 64 13 avril 00:55 11 -> pipe:[697355]
lr-x------. 1 haypo haypo 64 13 avril 00:55 12 -> pipe:[697356]
lrwx------. 1 haypo haypo 64 13 avril 00:54 2 -> /dev/pts/0
lr-x------. 1 haypo haypo 64 13 avril 00:55 3 -> pipe:[697347]
lr-x------. 1 haypo haypo 64 13 avril 00:55 4 -> pipe:[697348]
lr-x------. 1 haypo haypo 64 13 avril 00:55 5 -> pipe:[697349]
lr-x------. 1 haypo haypo 64 13 avril 00:55 6 -> pipe:[697350]
lr-x------. 1 haypo haypo 64 13 avril 00:55 7 -> pipe:[697351]
lr-x------. 1 haypo haypo 64 13 avril 00:55 8 -> pipe:[697352]
lr-x------. 1 haypo haypo 64 13 avril 00:55 9 -> pipe:[697353]

File descriptors of the first child process:

haypo@selma$ ls -l /proc/31565/fd/
lrwx------. 1 haypo haypo 64 13 avril 00:54 0 -> /dev/pts/0
lrwx------. 1 haypo haypo 64 13 avril 00:54 1 -> /dev/pts/0
lrwx------. 1 haypo haypo 64 13 avril 00:54 2 -> /dev/pts/0
lr-x------. 1 haypo haypo 64 13 avril 00:54 3 -> /dev/null
l-wx------. 1 haypo haypo 64 13 avril 00:54 4 -> pipe:[697347]

File descriptors of the second child process:

haypo@selma$ ls -l /proc/31566/fd/
lrwx------. 1 haypo haypo 64 13 avril 00:54 0 -> /dev/pts/0
lrwx------. 1 haypo haypo 64 13 avril 00:54 1 -> /dev/pts/0
lrwx------. 1 haypo haypo 64 13 avril 00:54 2 -> /dev/pts/0
lr-x------. 1 haypo haypo 64 13 avril 00:54 3 -> pipe:[697347]
lr-x------. 1 haypo haypo 64 13 avril 00:54 4 -> /dev/null
l-wx------. 1 haypo haypo 64 13 avril 00:54 5 -> pipe:[697348]

(...)

File descriptors of the last child process:

haypo@selma$ ls -l /proc/31574/fd/
lrwx------. 1 haypo haypo 64 13 avril 00:57 0 -> /dev/pts/0
lrwx------. 1 haypo haypo 64 13 avril 00:57 1 -> /dev/pts/0
lr-x------. 1 haypo haypo 64 13 avril 00:57 10 -> pipe:[697354]
lr-x------. 1 haypo haypo 64 13 avril 00:57 11 -> pipe:[697355]
lr-x------. 1 haypo haypo 64 13 avril 00:57 12 -> /dev/null
l-wx------. 1 haypo haypo 64 13 avril 00:57 13 -> pipe:[697356]
lrwx------. 1 haypo haypo 64 13 avril 00:54 2 -> /dev/pts/0
lr-x------. 1 haypo haypo 64 13 avril 00:57 3 -> pipe:[697347]
lr-x------. 1 haypo haypo 64 13 avril 00:57 4 -> pipe:[697348]
lr-x------. 1 haypo haypo 64 13 avril 00:57 5 -> pipe:[697349]
lr-x------. 1 haypo haypo 64 13 avril 00:57 6 -> pipe:[697350]
lr-x------. 1 haypo haypo 64 13 avril 00:57 7 -> pipe:[697351]
lr-x------. 1 haypo haypo 64 13 avril 00:57 8 -> pipe:[697352]
lr-x------. 1 haypo haypo 64 13 avril 00:57 9 -> pipe:[697353]
msg298844 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-07-22 09:46
This is expected with the "fork" method, but also happens with the "forkserver" method.
msg298846 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-07-22 11:22
New changeset 896145d9d266ee2758cfcd7691238cbc1f9e1ab8 by Antoine Pitrou in branch 'master':
bpo-26732: fix too many fds in processes started with the "forkserver" method (#2813)
https://github.com/python/cpython/commit/896145d9d266ee2758cfcd7691238cbc1f9e1ab8
msg299051 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-07-25 11:45
The newly added test now hangs on almost all Windows buildbots, see:
http://bugs.python.org/issue31009#msg299050
msg299052 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-07-25 11:46
By the way, 3.6 is marked as affected, but the change wasn't backported to 3.6?
msg299157 - (view) Author: Kevin Quick (quick-b) Date: 2017-07-25 22:48
Thank you for the fix, pitrou!

Regarding your comment regarding the behavior being expected with the "fork" method, I would suggest a documentation modification in the description of the fork method to add something to the effect of "All resources of the parent are inherited by the child process; this includes the ability of later children to act as the parent of earlier children via the various functions in this package."
msg299174 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-07-26 00:11
Antoine: Do you want to backport your change 896145d9d266ee2758cfcd7691238cbc1f9e1ab8 to Python 3.6? If yes, don't forget to include my fix 302bbbe9ba5c72559913e2ea006c921f698a729d for Windows (bpo-31009).

If you prefer, I can handle the backport.
msg299418 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-07-28 18:26
Victor, you can try to backport the fix, it should be interesting, though I suspect you will run into conflicts :-)
msg299419 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-07-28 18:27
Kevin:

> Regarding your comment regarding the behavior being expected with the "fork" method, I would suggest a documentation modification in the description of the fork method to add something to the effect of "All resources of the parent are inherited by the child process; this includes the ability of later children to act as the parent of earlier children via the various functions in this package."

I would rather keep the latter part ("the ability of later children to act as the parent of earlier children via the various functions in this package") an implementation detail rather than a documented feature.  Who knows, perhaps some day we'll change the implementation so that those are not inherited anymore.
msg299424 - (view) Author: Steve Holden (holdenweb) * (Python committer) Date: 2017-07-28 19:24
IIRC sockets can be marked as inheritable or not. It seems to me it would be a useful enhancement to allow the same determination for the pipes.
msg299531 - (view) Author: Kevin Quick (quick-b) Date: 2017-07-31 00:12
Hi Antoine,

> ... an implementation detail rather than a documented feature

I understand your desire, but this is a leaky abstraction and I would still suggest that it should be more clear that the fork method will inherit *all* resources from the parent, including multiprocessing internal resources related to other processes.  This is potentially important if there are (a) large numbers of processes to be created and (b) for security aspects.

Regards,
  Kevin
History
Date User Action Args
2017-07-31 00:12:15quick-bsetmessages: + msg299531
2017-07-28 19:24:23holdenwebsetnosy: + holdenweb
messages: + msg299424
2017-07-28 18:27:51pitrousetmessages: + msg299419
2017-07-28 18:26:22pitrousetmessages: + msg299418
2017-07-26 00:11:36hayposetmessages: + msg299174
2017-07-25 22:48:21quick-bsetmessages: + msg299157
2017-07-25 11:46:12hayposetmessages: + msg299052
2017-07-25 11:45:46hayposetstatus: closed -> open
resolution: fixed ->
messages: + msg299051
2017-07-22 11:23:05pitrousetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2017-07-22 11:22:57pitrousetmessages: + msg298846
2017-07-22 10:05:17pitrousetstage: needs patch -> patch review
2017-07-22 10:04:29pitrousetpull_requests: + pull_request2865
2017-07-22 09:46:34pitrousetversions: + Python 3.6, Python 3.7, - Python 3.5
nosy: + pitrou

messages: + msg298844

stage: needs patch
2016-05-08 18:51:15Winterflowersetnosy: + Winterflower
2016-04-12 22:57:46hayposetnosy: + haypo
messages: + msg263282
2016-04-11 08:40:24SilentGhostsetnosy: + jnoller, sbt
2016-04-10 22:00:01davinsetnosy: + davin
2016-04-10 21:59:12quick-bsetmessages: + msg263154
2016-04-10 21:58:21quick-bcreate