classification
Title: Create a way to check that the parent process is alive for deamonized processes
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: davin, miss-islington, pitrou, tomMoral, vstinner
Priority: normal Keywords: patch

Created on 2019-05-11 15:36 by tomMoral, last changed 2019-06-25 22:27 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 13247 merged tomMoral, 2019-05-11 15:38
PR 14286 merged pierreglaser, 2019-06-21 13:46
PR 14382 merged miss-islington, 2019-06-25 20:44
Messages (7)
msg342199 - (view) Author: Thomas Moreau (tomMoral) * Date: 2019-05-11 15:36
In the std lib, the semaphore_tracker and the Manager rely on daemonized processes that are launched with server like loops. The cleaning of such processes is made complicated by the fact that there is no cannonical way to check that the parent process is alive.

I propose to add in context a parent_process function that would give access to a Process object representing the parent process. This way, we could benefit from sentinel to improve the clean up of this process that can be left dangling in case of hard stop from the main interpreter.
msg342945 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2019-05-20 19:37
New changeset c09a9f56c08d80567454cae6f78f738a89e1ae94 by Antoine Pitrou (Thomas Moreau) in branch 'master':
bpo-36888: Add multiprocessing.parent_process() (GH-13247)
https://github.com/python/cpython/commit/c09a9f56c08d80567454cae6f78f738a89e1ae94
msg342946 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2019-05-20 19:38
Waiting for the next PR now :-)
msg345321 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-12 11:45
This new test is not reliable, it failed on x86 Gentoo Refleaks 3.8:
https://buildbot.python.org/all/#/builders/223/builds/9

Please fix the test or revert the change. The CI must remain green ;-)


running: test_multiprocessing_spawn (31 min 51 sec), test_multiprocessing_forkserver (3 min 7 ms)
3:20:42 load avg: 6.89 [415/423/1] test_multiprocessing_spawn failed (31 min 30 sec) -- running: test_multiprocessing_forkserver (3 min 7 sec)
beginning 6 repetitions
123456
....Process Process-1588:1:
Traceback (most recent call last):
  File "/buildbot/buildarea/cpython/3.8.ware-gentoo-x86.refleak/build/Lib/multiprocessing/process.py", line 313, in _bootstrap
    self.run()
  File "/buildbot/buildarea/cpython/3.8.ware-gentoo-x86.refleak/build/Lib/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/buildbot/buildarea/cpython/3.8.ware-gentoo-x86.refleak/build/Lib/test/_test_multiprocessing.py", line 326, in _test_report_parent_status
    wconn.send("alive" if parent_process().is_alive() else "not alive")
  File "/buildbot/buildarea/cpython/3.8.ware-gentoo-x86.refleak/build/Lib/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/buildbot/buildarea/cpython/3.8.ware-gentoo-x86.refleak/build/Lib/multiprocessing/connection.py", line 411, in _send_bytes
    self._send(header + buf)
  File "/buildbot/buildarea/cpython/3.8.ware-gentoo-x86.refleak/build/Lib/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Warning -- Dangling processes: {<Process name='Process-1588' pid=6665 parent=13022 started>}
Warning -- Dangling processes: {<Process name='Process-1588' pid=6665 parent=13022 started>}
Warning -- multiprocessing.process._dangling was modified by test_multiprocessing_spawn
  Before: set()
  After:  {<weakref at 0xb6d7e840; to 'Process' at 0xb4573878>} 
test test_multiprocessing_spawn failed -- Traceback (most recent call last):
  File "/buildbot/buildarea/cpython/3.8.ware-gentoo-x86.refleak/build/Lib/test/_test_multiprocessing.py", line 305, in test_parent_process
    raise AssertionError("Could not communicate with child process")
AssertionError: Could not communicate with child process

/buildbot/buildarea/cpython/3.8.ware-gentoo-x86.refleak/build/Lib/multiprocessing/resource_tracker.py:203: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/buildbot/buildarea/cpython/3.8.ware-gentoo-x86.refleak/build/Lib/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: '//psm_b32d1a2f': [Errno 2] No such file or directory: '//psm_b32d1a2f'
  warnings.warn('resource_tracker: %r: %s' % (name, e))

(...)

Re-running test_multiprocessing_spawn in verbose mode

======================================================================
FAIL: test_parent_process (test.test_multiprocessing_spawn.WithProcessesTestProcess)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/buildbot/buildarea/cpython/3.8.ware-gentoo-x86.refleak/build/Lib/test/_test_multiprocessing.py", line 305, in test_parent_process
    raise AssertionError("Could not communicate with child process")
AssertionError: Could not communicate with child process
msg346558 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-25 20:44
New changeset 594d9b9f58e1ef0a60b5ce9e590e0f45cd684f26 by Victor Stinner (Pierre Glaser) in branch 'master':
bpo-36888, test_multiprocessing: Increase test_parent_process timeout (GH-14286)
https://github.com/python/cpython/commit/594d9b9f58e1ef0a60b5ce9e590e0f45cd684f26
msg346560 - (view) Author: miss-islington (miss-islington) Date: 2019-06-25 21:12
New changeset 4adc38e79495bd06878355fd5270c3f84b77f528 by Miss Islington (bot) in branch '3.8':
bpo-36888, test_multiprocessing: Increase test_parent_process timeout (GH-14286)
https://github.com/python/cpython/commit/4adc38e79495bd06878355fd5270c3f84b77f528
msg346563 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-25 22:27
Thanks Pierre Glaser. Let's see if it's enough to make the test more reliable on buildbots ;-)
History
Date User Action Args
2019-06-25 22:27:51vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg346563

stage: patch review -> resolved
2019-06-25 21:12:50miss-islingtonsetnosy: + miss-islington
messages: + msg346560
2019-06-25 20:44:31vstinnersetmessages: + msg346558
2019-06-25 20:44:29miss-islingtonsetpull_requests: + pull_request14196
2019-06-21 13:46:36pierreglasersetstage: resolved -> patch review
pull_requests: + pull_request14108
2019-06-12 11:45:55vstinnersetstatus: closed -> open

nosy: + vstinner
messages: + msg345321

resolution: fixed -> (no value)
2019-05-20 19:38:32pitrousetstatus: open -> closed
versions: - Python 3.9
messages: + msg342946

resolution: fixed
stage: patch review -> resolved
2019-05-20 19:37:10pitrousetmessages: + msg342945
2019-05-11 17:00:14cheryl.sabellasetnosy: + pitrou, davin
2019-05-11 15:38:44tomMoralsetkeywords: + patch
stage: patch review
pull_requests: + pull_request13158
2019-05-11 15:36:32tomMoralcreate