In the std lib, the semaphore_tracker and the Manager rely on daemonized processes that are launched with server like loops. The cleaning of such processes is made complicated by the fact that there is no cannonical way to check that the parent process is alive.

I propose to add in context a parent_process function that would give access to a Process object representing the parent process. This way, we could benefit from sentinel to improve the clean up of this process that can be left dangling in case of hard stop from the main interpreter.

New changeset c09a9f56c08d80567454cae6f78f738a89e1ae94 by Antoine Pitrou (Thomas Moreau) in branch 'master':
bpo-36888: Add multiprocessing.parent_process() (GH-13247)
https://github.com/python/cpython/commit/c09a9f56c08d80567454cae6f78f738a89e1ae94

Waiting for the next PR now :-)

This new test is not reliable, it failed on x86 Gentoo Refleaks 3.8:
https://buildbot.python.org/all/#/builders/223/builds/9

Please fix the test or revert the change. The CI must remain green ;-)


running: test_multiprocessing_spawn (31 min 51 sec), test_multiprocessing_forkserver (3 min 7 ms)
3:20:42 load avg: 6.89 [415/423/1] test_multiprocessing_spawn failed (31 min 30 sec) -- running: test_multiprocessing_forkserver (3 min 7 sec)
beginning 6 repetitions
123456
....Process Process-1588:1:
Traceback (most recent call last):
  File "/buildbot/buildarea/cpython/3.8.ware-gentoo-x86.refleak/build/Lib/multiprocessing/process.py", line 313, in _bootstrap
    self.run()
  File "/buildbot/buildarea/cpython/3.8.ware-gentoo-x86.refleak/build/Lib/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/buildbot/buildarea/cpython/3.8.ware-gentoo-x86.refleak/build/Lib/test/_test_multiprocessing.py", line 326, in _test_report_parent_status
    wconn.send("alive" if parent_process().is_alive() else "not alive")
  File "/buildbot/buildarea/cpython/3.8.ware-gentoo-x86.refleak/build/Lib/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/buildbot/buildarea/cpython/3.8.ware-gentoo-x86.refleak/build/Lib/multiprocessing/connection.py", line 411, in _send_bytes
    self._send(header + buf)
  File "/buildbot/buildarea/cpython/3.8.ware-gentoo-x86.refleak/build/Lib/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Warning -- Dangling processes: {<Process name='Process-1588' pid=6665 parent=13022 started>}
Warning -- Dangling processes: {<Process name='Process-1588' pid=6665 parent=13022 started>}
Warning -- multiprocessing.process._dangling was modified by test_multiprocessing_spawn
  Before: set()
  After:  {<weakref at 0xb6d7e840; to 'Process' at 0xb4573878>} 
test test_multiprocessing_spawn failed -- Traceback (most recent call last):
  File "/buildbot/buildarea/cpython/3.8.ware-gentoo-x86.refleak/build/Lib/test/_test_multiprocessing.py", line 305, in test_parent_process
    raise AssertionError("Could not communicate with child process")
AssertionError: Could not communicate with child process

/buildbot/buildarea/cpython/3.8.ware-gentoo-x86.refleak/build/Lib/multiprocessing/resource_tracker.py:203: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/buildbot/buildarea/cpython/3.8.ware-gentoo-x86.refleak/build/Lib/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: '//psm_b32d1a2f': [Errno 2] No such file or directory: '//psm_b32d1a2f'
  warnings.warn('resource_tracker: %r: %s' % (name, e))

(...)

Re-running test_multiprocessing_spawn in verbose mode

======================================================================
FAIL: test_parent_process (test.test_multiprocessing_spawn.WithProcessesTestProcess)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/buildbot/buildarea/cpython/3.8.ware-gentoo-x86.refleak/build/Lib/test/_test_multiprocessing.py", line 305, in test_parent_process
    raise AssertionError("Could not communicate with child process")
AssertionError: Could not communicate with child process

New changeset 594d9b9f58e1ef0a60b5ce9e590e0f45cd684f26 by Victor Stinner (Pierre Glaser) in branch 'master':
bpo-36888, test_multiprocessing: Increase test_parent_process timeout (GH-14286)
https://github.com/python/cpython/commit/594d9b9f58e1ef0a60b5ce9e590e0f45cd684f26

New changeset 4adc38e79495bd06878355fd5270c3f84b77f528 by Miss Islington (bot) in branch '3.8':
bpo-36888, test_multiprocessing: Increase test_parent_process timeout (GH-14286)
https://github.com/python/cpython/commit/4adc38e79495bd06878355fd5270c3f84b77f528

Thanks Pierre Glaser. Let's see if it's enough to make the test more reliable on buildbots ;-)