classification
Title: get_lock() method is not present for Values created using multiprocessing.Manager()
Type: Stage:
Components: Documentation, Library (Lib), Windows Versions: Python 3.7, Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: Lorenzo Persichetti, docs@python, josh.r, nemeskeyd, pablogsal, paul.moore, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2019-01-19 21:50 by Lorenzo Persichetti, last changed 2019-08-07 18:54 by josh.r.

Messages (5)
msg334070 - (view) Author: Lorenzo Persichetti (Lorenzo Persichetti) Date: 2019-01-19 21:50
According to the documentation of the multiprocessing.Value() class available here https://docs.python.org/3.6/library/multiprocessing.html#multiprocessing.Value

Operations like += which involve a read and write are not atomic. So if, for instance, you want to atomically increment a shared value it is insufficient to just do

counter.value += 1
Assuming the associated lock is recursive (which it is by default) you can instead do

with counter.get_lock():
    counter.value += 1

What happens is that when running the following snippet

import multiprocessing
manager = multiprocessing.Manager()
value = manager.Value('i', 0)
value.get_lock()

the result is 
AttributeError: 'ValueProxy' object has no attribute 'get_lock'
msg334106 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-01-20 22:04
You are not using multiprocessing.Value:

>>> import multiprocessing
>>> x = multiprocessing.Value("i", 0)
>>> x.get_lock()
<RLock(None, 0)>
msg349155 - (view) Author: Dávid Nemeskey (nemeskeyd) Date: 2019-08-07 09:00
Nothing in the documentation says that multiprocessing.Value and the object returned by manager.Value() is any different. Nor is it clear why they should be.

It is perfectly understandable to expect that manager.Value() is actually of type multiprocessing.Value. It is also a valid assumption that the "canonical" way to acquire such objects are from a Manager, not directly. An example that reinforces this assumption is that of the Queue class, which HAS to be created through a Manager, lest we get "RuntimeError: Queue objects should only be shared between processes through inheritance".

In conclusion, I think this is definitely a valid issue. What I am not so sure about is if it is (just) an issue in the documentation or the API itself.
msg349156 - (view) Author: Dávid Nemeskey (nemeskeyd) Date: 2019-08-07 09:03
OK, actually: trying to create a multiprocessing.Value object and sharing it between a Pool of processes results in "RuntimeError: Synchronized objects should only be shared between processes through inheritance". So the only way seems to be through a Manager, but its Value() is of a different class?
msg349191 - (view) Author: Josh Rosenberg (josh.r) * (Python triager) Date: 2019-08-07 18:54
Reading the docs, I'd definitely expect multiprocessing.Manager().Value to obey the same interface as multiprocessing.Value. The SyncManager docs say:

> Its methods create and return Proxy Objects for a number of commonly used data types to be synchronized across processes.

It does also say (under the linked definition for Proxy Objects):

> A proxy object has methods which invoke corresponding methods of its referent (although not every method of the referent will necessarily be available through the proxy).

which implies the proxy might be more limited, but in fact the proxy is wrapping a completely different underlying class, multiprocessing.managers.Value (that has nothing to do with the class you'd get from calling multiprocessing.Value or multiprocessing.sharedctypes.Value).

It looks like there was, at some point, an attempt to make certain interfaces match; multiprocessing.managers.Value's initializer has a prototype of:

def __init__(self, typecode, value, lock=True):

(documented prototype excludes the lock argument), but AFAICT, lock is completely ignored, and typecode is stored as _typecode on the underlying Value, but otherwise ignored; the multiprocessing.managers.ValueProxy object returned by manager.Value() doesn't expose it (except insofar as you can call ._getvalue() on it to retrieve a copy of the underlying Value), and even if you get the unwrapped object, all _typecode does is change the repr; it's not used anywhere else.

It seems like SyncManager.Value is just the worst of all possible worlds; it has a prototype that (roughly) matches multiprocessing.Value/multiprocessing.sharedctypes.Value, and the limited documentation implies it serves the same function, but:

1. It effectively ignores every argument aside from value (which is the second argument, so you're stuck passing the first argument even though nothing uses it)
2. It doesn't actually use ctypes to achieve efficient storage/IPC communication (updating value involves pickling it, not just sending the raw data of a C array/struct)
3. It doesn't provide any of the interface of the other Value, so you can't use get_lock (or use a with statement to lock it, or manually call acquire or release) or get_obj.

I'm not sure what to do about it though; the manager version of Value is much more flexible, and I'm sure there is existing code that takes advantage of that, so we can't rewrite to make it ctypes backed. The multiprocessing.managers code is too new/complex for me to determine if any easy solution exists to expand multiprocessing.managers.ValueProxy to support context management/get_lock/get_obj to match the behavior of the class returned by multiprocessing.Value/multiprocessing.sharedctypes.Value, but it seems like, if we're going to let the docs imply a relationship, we ought to at least try to make the API of the two classes roughly match.
History
Date User Action Args
2019-08-07 18:54:01josh.rsetnosy: + josh.r
messages: + msg349191
2019-08-07 09:03:56nemeskeydsetmessages: + msg349156
2019-08-07 09:00:58nemeskeydsetnosy: + nemeskeyd

messages: + msg349155
versions: + Python 3.7
2019-01-20 22:04:15pablogsalsetnosy: + pablogsal
messages: + msg334106
2019-01-19 21:50:32Lorenzo Persichetticreate