classification
Title: Adding a SyncManager Queue proxy to a SyncManager dict or Namespace proxy raises an exception
Type: behavior Stage: patch review
Components: Library (Lib) Versions: Python 3.8, Python 3.7, Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Alexander Prokhorov, Locane, Philipp Rehs, davin, finefoot, hniksic, jjdmon, mjpieters, pitrou, xiang.zhang
Priority: normal Keywords: patch

Created on 2017-05-03 11:53 by jjdmon, last changed 2020-09-08 17:12 by mjpieters.

Pull Requests
URL Status Linked Edit
PR 4819 closed uspike, 2017-12-12 20:02
PR 16341 open finefoot, 2019-09-24 00:52
Messages (7)
msg292889 - (view) Author: Jeff DuMonthier (jjdmon) Date: 2017-05-03 11:53
In multiprocessing, attempting to add a Queue proxy to a dict or Namespace proxy (all returned by the same SyncManager) raises an exception indicating a keyword argument 'manager_owned=True' has been passed to the function AutoProxy() but is not an argument of that function.

In lib/python3.6/multiprocessing/managers.py, in function RebuildProxy(), line 873: "kwds['manager_owned'] = True" adds this argument to a keyword argument dictionary.  This function calls AutoProxy which has an argument list defined on lines 909-910 as:
def AutoProxy(token, serializer, manager=None, authkey=None,
              exposed=None, incref=True):
This raises an exception because there is no manager_owned argument defined.  I added "manager_owned=False" as a keyword argument to AutoProxy which seems to have fixed the problem.  There is no exception and I am able to pass Queue proxies through dict and Namespace proxies to other processes and use them.  I don't know the purpose of that argument though or if the AutoProxy function should actually use it for something.  My fix allows but ignores it.
msg311592 - (view) Author: Hrvoje Nikšić (hniksic) * Date: 2018-02-04 08:40
I encountered this bug while testing the code in this StackOverflow answer:

    https://stackoverflow.com/a/48565011/1600898

The code at the end of the answer runs on Python 3.5, but fails on 3.6 with the "unexpected keyword argument 'manager_owned'" error.

If someone knows of a workaround until the PR is accepted, it would be appreciated as well.
msg311594 - (view) Author: Hrvoje Nikšić (hniksic) * Date: 2018-02-04 10:16
The issue is also present in Python 3.7.0b1.
msg346334 - (view) Author: (finefoot) * Date: 2019-06-23 19:10
This is still an issue: https://stackoverflow.com/questions/56716470/python-multiprocessing-nested-shared-objects-doesnt-work-with-queue

Is there a specific reason, why https://github.com/python/cpython/pull/4819 doesn't get reviewed?
msg352960 - (view) Author: Michael Tippie (Locane) Date: 2019-09-22 05:50
I am getting this error now, too.  I'm not sure what's causing it - when I use multiprocessing with a Manager.Queue, if I am passing around an object on the queue stack, I get this error.

Would really like it to "just work".
msg361034 - (view) Author: Philipp Rehs (Philipp Rehs) Date: 2020-01-30 10:34
Are there any reasons why it does not get merged?
This issue is open since more than two years and the fix is quiet small
msg376584 - (view) Author: Martijn Pieters (mjpieters) * Date: 2020-09-08 17:12
Might it be better to just *drop* the AutoProxy object altogether?

All that it adds is a delayed call to MakeProxyType(f"AutoProxy[{typeid}]", exposed) (with exposed defaulting to public_methods(instance)), per instance per process.

It could be replaced by a direct call to `MakeProxyType()`, using `public_methods` directly on the registered type. This wouldn't work for callables that are not classes or where instances add functions to the instance dict, but for those rare cases you can pass in the `exposed` argument.

The advantage is that it would simplify the codebase; no more need to special-case the BaseProxy.__reduce__ method, removing the get_methods() method on the Server class, etc. Less surface for this class of bugs to happen in the future.
History
Date User Action Args
2020-09-08 17:12:13mjpieterssetnosy: + mjpieters
messages: + msg376584
2020-02-03 15:16:22Philipp Rehssettype: behavior
2020-01-30 10:34:01Philipp Rehssetnosy: + Philipp Rehs

messages: + msg361034
versions: + Python 3.8
2019-09-24 00:52:11finefootsetpull_requests: + pull_request15918
2019-09-22 05:50:49Locanesetnosy: + Locane
messages: + msg352960
2019-06-23 19:10:54finefootsetnosy: + finefoot
messages: + msg346334
2018-07-02 12:02:53xiang.zhangsetnosy: + pitrou, davin, xiang.zhang
2018-02-04 10:16:59hniksicsetmessages: + msg311594
versions: + Python 3.7
2018-02-04 08:40:20hniksicsetnosy: + hniksic
messages: + msg311592
2017-12-12 20:02:38uspikesetkeywords: + patch
stage: patch review
pull_requests: + pull_request4709
2017-11-27 22:11:39Alexander Prokhorovsetnosy: + Alexander Prokhorov
2017-05-03 11:53:34jjdmoncreate