Uninformative error message in multiprocessing.Manager() #65400

wojtekwalczak · 2014-04-11T17:20:20Z

BPO	21201
Nosy	@berkerpeksag, @applio
Files	managers_tb_msg.patch: A patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/applio'
closed_at = <Date 2016-09-08.19:53:36.249>
created_at = <Date 2014-04-11.17:20:20.453>
labels = ['type-bug', 'library']
title = 'Uninformative error message in multiprocessing.Manager()'
updated_at = <Date 2016-09-08.19:53:36.248>
user = 'https://bugs.python.org/wojtekwalczak'

bugs.python.org fields:

activity = <Date 2016-09-08.19:53:36.248>
actor = 'davin'
assignee = 'davin'
closed = True
closed_date = <Date 2016-09-08.19:53:36.249>
closer = 'davin'
components = ['Library (Lib)']
creation = <Date 2014-04-11.17:20:20.453>
creator = 'wojtekwalczak'
dependencies = []
files = ['34785']
hgrepos = []
issue_num = 21201
keywords = ['patch']
message_count = 7.0
messages = ['215934', '236621', '259110', '275070', '275094', '275126', '275128']
nosy_count = 5.0
nosy_names = ['wojtekwalczak', 'python-dev', 'sbt', 'berker.peksag', 'davin']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue21201'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6']

wojtekwalczak · 2014-04-11T17:20:18Z

While using multiprocessing.Manager() to send data between processes I've noticed that one of the traceback messages is not very informative:

Traceback (most recent call last):
  File "age_predict.py", line 39, in <module>
    train_data, train_target, test_data = prepare_data()
  File "age_predict.py", line 30, in prepare_data
    train_data = q.get(True, 10000)
  File "<string>", line 2, in get
  File "/usr/lib/python2.7/multiprocessing/managers.py", line 777, in _callmethod
    raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError:

Unserializable message: ('#RETURN', [A WHOLE LOT OF DATA GOES HERE]

The real problem here is that my machine is running out of memory, but the traceback message shadows this information and reports "Unserializable message" instead.

After a simple path (see: attached file) the traceback message is as follows:

Traceback (most recent call last):
  File "age_predict.py", line 39, in <module>
    train_data, train_target, test_data = prepare_data()
  File "age_predict.py", line 30, in prepare_data
    train_data = q.get(True, 10000)
  File "<string>", line 2, in get
  File "/usr/lib/python2.7/multiprocessing/managers.py", line 775, in _callmethod
    raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError: 
---------------------------------------------------------------------
Unserializable message: Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/managers.py", line 288, in serve_client
    send(msg)
MemoryError: out of memory

Here's another example (see: http://bugs.python.org/issue6766):

This python3 code:

from multiprocessing import Manager
manager = Manager()
ns_proxy = manager.Namespace()
evt_proxy = manager.Event()
ns_proxy.my_event_proxy = evt_proxy
print(ns_proxy.my_event_proxy)

Produces following traceback message:

Traceback (most recent call last):
  File "test.py", line 6, in <module>
    print(ns_proxy.my_event_proxy)
  File "/cpython/Lib/multiprocessing/managers.py", line 1032, in __getattr__
    return callmethod('__getattribute__', (key,))
  File "/cpython/Lib/multiprocessing/managers.py", line 748, in _callmethod
    raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError:

Unserializable message: ('#RETURN', <threading.Event object at 0x7f9484aff278>)
----------------------------------------------------------------------

...and after applying the attached patch it becomes clearer what's going on:

Traceback (most recent call last):
  File "test.py", line 6, in <module>
    print(ns_proxy.my_event_proxy)
  File "/cpython/Lib/multiprocessing/managers.py", line 1032, in __getattr__
    return callmethod('__getattribute__', (key,))
  File "/cpython/Lib/multiprocessing/managers.py", line 748, in _callmethod
    raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError: 
----------------------------------------------------------------------
Unserializable message: Traceback (most recent call last):
  File "/cpython/Lib/multiprocessing/managers.py", line 276, in serve_client
    send(msg)
  File "/cpython/Lib/multiprocessing/connection.py", line 206, in send
    self._send_bytes(ForkingPickler.dumps(obj))
  File "/cpython/Lib/multiprocessing/reduction.py", line 50, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <class '_thread.lock'>: attribute lookup lock on _thread failed

This patch doesn't break any tests.

BreamoreBoy · 2015-02-25T21:38:15Z

Can someone review the patch with a view to commit please. It's a change to one line as explained in msg215934.

berkerpeksag · 2016-01-28T09:21:50Z

Looks good to me. Perhaps we can use utils.info() to print the exception.

applio · 2016-09-08T17:17:01Z

Looks good to me as well. I'll go ahead with applying the patch.

Berker: I'm not sure it's worth adding something like util.info("Serialization failure on: %r", msg). If serialization failed for some reason other than MemoryError, we will be communicating the output of format_exc() back to the client. If serialization failed due to a MemoryError (which has tripped up plenty of people) then our util.info may similarly fail.

Note that with the patch recently applied to issue6766, under 3.6 the second example no longer triggers an exception.  That second example can even continue:
    >>> ns_proxy.my_event_proxy.set()
    >>> ns_proxy.my_event_proxy.is_set()
    True
    >>> evt_proxy.is_set()
    True
    >>> evt_proxy.clear()
    >>> ns_proxy.my_event_proxy.is_set()
    False

python-dev · 2016-09-08T18:07:41Z

New changeset cbf81969aba4 by Davin Potts in branch '2.7':
Issue bpo-21201: Improves readability of multiprocessing error message from server to client for certain exceptions
https://hg.python.org/cpython/rev/cbf81969aba4

python-dev · 2016-09-08T19:47:49Z

New changeset 9b1f8c68de4c by Davin Potts in branch '3.5':
Issue bpo-21201: Improves readability of multiprocessing error message from server to client for certain exceptions
https://hg.python.org/cpython/rev/9b1f8c68de4c

New changeset 199aca18d9a1 by Davin Potts in branch 'default':
Issue bpo-21201: Improves readability of multiprocessing error message from server to client for certain exceptions
https://hg.python.org/cpython/rev/199aca18d9a1

applio · 2016-09-08T19:53:36Z

Thanks for your patch @wojtekwalczak!

wojtekwalczak mannequin added stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error labels Apr 11, 2014

applio self-assigned this Sep 8, 2016

applio closed this as completed Sep 8, 2016

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uninformative error message in multiprocessing.Manager() #65400

Uninformative error message in multiprocessing.Manager() #65400

wojtekwalczak mannequin commented Apr 11, 2014

wojtekwalczak mannequin commented Apr 11, 2014

BreamoreBoy mannequin commented Feb 25, 2015

berkerpeksag commented Jan 28, 2016

applio commented Sep 8, 2016

python-dev mannequin commented Sep 8, 2016

python-dev mannequin commented Sep 8, 2016

applio commented Sep 8, 2016

Uninformative error message in multiprocessing.Manager() #65400

Uninformative error message in multiprocessing.Manager() #65400

Comments

wojtekwalczak mannequin commented Apr 11, 2014

wojtekwalczak mannequin commented Apr 11, 2014

BreamoreBoy mannequin commented Feb 25, 2015

berkerpeksag commented Jan 28, 2016

applio commented Sep 8, 2016

python-dev mannequin commented Sep 8, 2016

python-dev mannequin commented Sep 8, 2016

applio commented Sep 8, 2016