classification
Title: multiprocessing bug SyncManager and 'with'
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 2.7
process
Status: closed Resolution: out of date
Dependencies: Superseder: Use context manager protocol for more multiprocessing types
View: 15064
Assigned To: Nosy List: Cezary.Wagner, davin, iritkatriel
Priority: normal Keywords:

Created on 2015-02-24 13:27 by Cezary.Wagner, last changed 2020-10-20 19:44 by iritkatriel. This issue is now closed.

Messages (7)
msg236490 - (view) Author: Cezary Wagner (Cezary.Wagner) Date: 2015-02-24 13:27
'with' code with manager will not work:
AttributeError: 'SyncManager' object has no attribute 'shutdown'

Test code:
# coding=utf-8
import multiprocessing
import multiprocessing.managers
import logging

def callback(result):
  print multiprocessing.current_process().name, 'callback', result

def worker(io_lock, value):
  # error
  raise RuntimeError()
  result = value + 1
  with io_lock:
    print multiprocessing.current_process().name, value, result
  return result

def main():
  manager = multiprocessing.managers.SyncManager()
  with manager:
    io_lock = manager.Lock()
    pool = multiprocessing.Pool(multiprocessing.cpu_count())
    for i in range(10):
      print pool.apply_async(worker, args=(io_lock, i), callback = callback)
    pool.close()
    pool.join()

if __name__ == '__main__':
  logging.basicConfig(level = logging.DEBUG)
  main()



See this code from multiprocessing.managers.SyncManager:

class SyncManager(BaseManager):
  pass

And now see 'with' methods in Basemanager:

    def __enter__(self):
        return self

    def __exit__(self, exc_type, exc_val, exc_tb):
        self.shutdown()
msg236515 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2015-02-24 16:56
Successfully reproduced on OS X 10.10 with 2.7.9.

Changing the issue type to "behavior" because "crash" is reserved for "Hard crashes of the Python interpreter – possibly with a core dump or a Windows error box."  In this issue, we do see an exception get raised but the interpreter itself is still happy and alive.
msg236521 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2015-02-24 17:20
A much simpler example of code triggering the described issue:

    import multiprocessing.managers
    with multiprocessing.managers.SyncManager() as s:
        print "here"


Running the above code in 2.7.9 results in an exception with the traceback:

    Traceback (most recent call last):
      File "<stdin>", line 2, in <module>
      File "/home/somewhere/python/lib/python2.7/multiprocessing/managers.py", line 602, in __exit__
        self.shutdown()
    AttributeError: 'SyncManager' object has no attribute 'shutdown'



This is because the implementation of context manager functionality for multiprocessing.managers.BaseManager appears to have a bug.  The above snippet of code might as well have been written to use BaseManager instead of SyncManager and the resulting behavior would have been the same.

To be fair, nothing in the documentation appears to suggest that BaseManager or its subclasses can / should be used with the 'with' statement (as proper context managers).  But it is also natural to react to the existence of '__enter__' and '__exit__' and think to try using it with a 'with' statement.

Specifically in BaseManager, it looks like self.shutdown should have been set via __enter__'s call to start() yet inside __exit__ it's not resolving.

I believe we need:
1) A patch to fix the broken behavior;
2) a test to validate that it works and specifically exercise the use of BaseManager with a 'with' statement;
3) an update to the docs to advertise context manager capability in these classes.
msg236522 - (view) Author: Cezary Wagner (Cezary.Wagner) Date: 2015-02-24 17:23
Yes it is what is wanted by me and other people I think.

1. enter has not start call.
2. exit has shutdown but it is not supported
3. 'with' is what is need since it is simple to manage

Good analysis.
msg236650 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2015-02-26 06:13
I have a working patch that addresses this behavior and tests to go along with it but am still working on updating the docs.  Updating the docs in 2.7 in a way to as closely match the docs in 3.4/3.5 as possible seems reasonable.  It remains unclear why __enter__ and __exit__ were added in 2.7 yet never advertised in the docs -- attempting to dig into that further.  My intention is to complete the patch (with mods to the docs) this weekend.
msg377103 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2020-09-18 09:44
I don't see this problem in python 3.10. Is it a python 2-only issue, and if so then can this issue be closed?


Running Release|Win32 interpreter...
Python 3.10.0a0 (heads/bpo-24792-dirty:1e183f4de5, Sep 14 2020, 15:53:51) [MSC v.1916 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import multiprocessing.managers
>>> with multiprocessing.managers.SyncManager() as s:
...     print("here")
...
here
>>>
msg379157 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2020-10-20 19:44
Yes, it was fixed in version 3.3, under issue15064:

https://github.com/python/cpython/commit/ac38571f00ccb5e1e0346ad27a8fac96d05e61c3
History
Date User Action Args
2020-10-20 19:44:59iritkatrielsetresolution: duplicate -> out of date
2020-10-20 19:44:15iritkatrielsetstatus: open -> closed
superseder: Use context manager protocol for more multiprocessing types
messages: + msg379157

resolution: duplicate
stage: needs patch -> resolved
2020-09-18 09:44:16iritkatrielsetnosy: + iritkatriel
messages: + msg377103
2015-02-26 06:13:36davinsetmessages: + msg236650
2015-02-24 17:23:41Cezary.Wagnersetmessages: + msg236522
2015-02-24 17:20:55davinsetmessages: + msg236521
stage: needs patch
2015-02-24 16:56:17davinsettype: crash -> behavior

messages: + msg236515
nosy: + davin
2015-02-24 13:28:23Cezary.Wagnersettitle: multiprocessing bug -> multiprocessing bug SyncManager and 'with'
2015-02-24 13:27:54Cezary.Wagnercreate