classification
Title: forkserver process isn't re-launched if it died
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Matthew Rocklin, davin, pitrou
Priority: normal Keywords: patch

Created on 2017-08-30 14:27 by pitrou, last changed 2017-11-03 13:00 by pitrou. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 3246 merged pitrou, 2017-08-30 15:25
PR 4252 merged pitrou, 2017-11-03 12:41
Messages (8)
msg301027 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-08-30 14:27
It may happen that the forkserver process dies (for example if SIGINT is received because the user pressed Ctrl-C) while the parent process is still alive.  In that case, if the parent tries to create a new Process instance, an exception is received.

The exception looks like this:

  File "/xxx/lib/python3.5/multiprocessing/popen_forkserver.py", line 52, in _launch
	self.sentinel, w = forkserver.connect_to_new_process(self._fds)
  File "/xxx/lib/python3.5/multiprocessing/forkserver.py", line 66, in connect_to_new_process
	client.connect(self._forkserver_address)
ConnectionRefusedError: [Errno 111] Connection refused
msg301854 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2017-09-11 03:43
I have two concerns with this:
1) The implicit restart of the forkserver process seems in conflict with the zen of making things explicit.
2) This would seem to make forkserver's behavior inconsistent with the behavior of things like the Manager which similarly creates its own process for managing resources but does not automatically restart that process if it should die or become unreachable.  In the case of the Manager, I don't think we'd want it to automagically restart anything in these situations so it's not a simple matter of enhancing the Manager to adopt similar behavior.

I do appreciate the use cases that would be addressed by having a convenient way to detect that a forkserver has died and then restart it.  If the forkserver dies, I doubt we really want it to try to restart a potentially infinite number of times.

Maybe a better path would be if we had a way to explicitly request that the Process trigger a restart of the forkserver, if necessary, but this setting/request defaults to False?
msg301868 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-09-11 09:18
> 1) The implicit restart of the forkserver process seems in conflict with the zen of making things explicit.

Well, the forkserver process really is an implementation detail.  It already is started on-demand, so restarting it if needed sounded like a natural evolution.

Another example: in a multiprocessing Pool, if a worker process crashes, another will be restarted transparently (same in concurrent.futures.ProcessPoolExecutor, incidentally).

> 2) This would seem to make forkserver's behavior inconsistent with the behavior of things like the Manager which similarly creates its own process for managing resources but does not automatically restart that process if it should die or become unreachable.

That's possible.  I've never seen the Manager used in the wild.  Though if some fundamental difference justifies the inconsistency, the inconsistency is not really a problem.
msg303458 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-10-01 12:15
Looks like this will not make into 3.6.3.

I'd like to sum up the discussion a bit:
- I think at the very least least we must protect the forkserver against SIGINT, like the semaphore tracker already is
- I think it's beneficial to also restart it on demand if necessary
- I can't think of a use case where the user asks "no, please don't restart the forkserver on demand" as there is no way to start it manually anyway, and not restarting it breaks functionality

Also, bpo-31310 is a similar issue for the semaphore tracker.
msg304468 - (view) Author: Matthew Rocklin (Matthew Rocklin) Date: 2017-10-16 12:21
From a user's perspective I would definitely appreciate forkserver being protected from SIGINT.  This bug affects me in practice.  If the forkserver goes down I personally have no objection to it restarting automatically, though I appreciate that I have a narrow view of this topic.

Davin, at your last comment it seemed like you had reservations about this going in.  Did Antoine's recent comments resolve these concerns or no?  Do you have any suggestions on what might be done to protect users from SIGINT crashing the forkserver?
msg305471 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-11-03 12:11
Davin, I think I'm going to merge the PR for this.  If you object it, it can still be reverted later.
msg305474 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-11-03 12:34
New changeset fc6b348b12ad401cab0261b7b71a65c60a08c0a8 by Antoine Pitrou in branch 'master':
bpo-31308: If multiprocessing's forkserver dies, launch it again when necessary (#3246)
https://github.com/python/cpython/commit/fc6b348b12ad401cab0261b7b71a65c60a08c0a8
msg305477 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-11-03 12:59
New changeset 019c99f325287741d1e0eefeef2b75c8e00b884f by Antoine Pitrou in branch '3.6':
[3.6] bpo-31308: If multiprocessing's forkserver dies, launch it again when necessary (GH-3246) (#4252)
https://github.com/python/cpython/commit/019c99f325287741d1e0eefeef2b75c8e00b884f
History
Date User Action Args
2017-11-03 13:00:04pitrousetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2017-11-03 12:59:48pitrousetmessages: + msg305477
2017-11-03 12:41:35pitrousetkeywords: + patch
pull_requests: + pull_request4214
2017-11-03 12:34:25pitrousetmessages: + msg305474
2017-11-03 12:11:38pitrousetmessages: + msg305471
2017-10-16 12:21:01Matthew Rocklinsetnosy: + Matthew Rocklin
messages: + msg304468
2017-10-01 12:15:27pitrousetmessages: + msg303458
2017-09-11 09:18:26pitrousetmessages: + msg301868
2017-09-11 03:43:59davinsetmessages: + msg301854
2017-08-30 19:17:13pitrousetstage: needs patch -> patch review
2017-08-30 15:25:40pitrousetpull_requests: + pull_request3291
2017-08-30 14:27:24pitrousetnosy: + davin
2017-08-30 14:27:15pitroucreate