Title: Multiprocessing.Manager: fix behavior and doc inconsistencies
Type: behavior Stage:
Components: Documentation, Library (Lib) Versions: Python 3.4, Python 3.5, Python 2.7
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: davin, docs@python, pythonhacker, sbt
Priority: normal Keywords:

Created on 2015-03-18 06:33 by pythonhacker, last changed 2015-05-14 09:47 by pythonhacker.

File name Uploaded Description Edit pythonhacker, 2015-03-18 06:33 Test file for demonstrating the above behaviours. Requires psutil module to be installed.
Messages (5)
msg238379 - (view) Author: Anand B Pillai (pythonhacker) * Date: 2015-03-18 06:33
multiprocessing.Manager seems to have an inconsistency in behaviour/documentation or both.

The behaviour inconsistency is documented in the attached test script which should run for both Python2 and Python3. 

Briefly, multiprocessing.managers.BaseManager class starts a Python subprocess only after a call to its "start" method. Whereas its subclass, multiprocessing.managers.SyncManager does at the time of object creation. This is undocumented and against the base classe's documented behaviour.

Also, the SyncManager is more commonly advertised via the facade multiprocessing.Manager() which is again at odds with the BaseManager's interface in that,

1. It takes no arguments (Python2) and different arguments (Python 3).
2. You can't call "start" on it but you can call "start" if you initialize it via multiprocessing.managers.SyncManager directly (!)
3. Even if you can't call a start on it, apparently you can shut it down via a call to "shutdown"
4. When you try to start such a manager you get a strange AssertionError complaining the state is not INITIAL. A better error is required here.

Please find the attached file for all the sequence of tests done.

Suggested changes are,

1. Fix the inconsistency in documented behaviour between SyncManager and its base class. Either don't start the child process upon object creation or document it.
2. Fix the inconsistency in SyncManager object creation interface and behaviour via "mulitprocessing.Manager()" and directly via "multiprocessing.managers.SyncManager(...)" . One should be able to start both objects cleanly. And the former should also take the address argument. If not document it properly.
3. The AssertionError when trying to start the SyncManager object obtained via a call to "multiprocessing.Manager()" looks like a bug. It should be fixed - otherwise a bette error should be raised and documentation updated.
msg238430 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2015-03-18 13:21
This is closely related to issue23510.  The two issues are reacting to the same underlying inconsistency though in this issue the problem is more generally (and clearly) stated whereas issue23510 is more concerned with the consequences to using these as Context Managers.

The solution that gets implemented here should address both issues.

As mentioned in issue23510, I have a working fix though am looking for ways to further minimize the impact to existing code that uses Manager/SyncManager -- simply documenting the current inconsistent state is inadequate.
msg238524 - (view) Author: Anand B Pillai (pythonhacker) * Date: 2015-03-19 14:03
Is your patch uploaded anywhere ?

Anyway, before thinking whether this should be a patch or doc fix, one needs multiprocessing module author to possibly comment on the right behaviour expected here.
msg238525 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2015-03-19 14:09
No, it's premature to share the start of my patch.

Very much agreed on getting that input/insight -- sbt (multiprocessing module maintainer and original author) was added to the nosy list yesterday too.
msg243165 - (view) Author: Anand B Pillai (pythonhacker) * Date: 2015-05-14 09:47
@sbt - Any comments on this ?
Date User Action Args
2015-05-14 09:47:40pythonhackersetmessages: + msg243165
2015-03-21 00:36:48terry.reedysettitle: Fix inconsistencies in behaviour and document it correctly for multiprocessing.Manager facade -> Multiprocessing.Manager: fix behavior and doc inconsistencies
2015-03-19 14:09:54davinsetmessages: + msg238525
2015-03-19 14:04:46pythonhackersettitle: Fix documentation for multiprocessing.Manager -> Fix inconsistencies in behaviour and document it correctly for multiprocessing.Manager facade
2015-03-19 14:03:12pythonhackersetmessages: + msg238524
2015-03-18 13:21:20davinsetmessages: + msg238430
2015-03-18 09:24:24ned.deilysetnosy: + sbt, davin

versions: - Python 3.2, Python 3.3, Python 3.6
2015-03-18 06:33:05pythonhackercreate