classification
Title: Unknown-source assertion failure in multiprocessing/managers.py
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: berker.peksag, davin, drallensmith, pitrou
Priority: normal Keywords: patch

Created on 2017-08-10 02:17 by drallensmith, last changed 2017-08-12 15:39 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
managers-3.5.patch drallensmith, 2017-08-10 15:03 Patch to 3.5 needed to tell where AssertionError is _not_ from
managers-2.7.patch drallensmith, 2017-08-10 15:04 Patch to 2.7 needed to tell where AssertionError is _not_ coming from
Pull Requests
URL Status Linked Edit
PR 3078 merged drallensmith, 2017-08-12 12:45
Messages (12)
msg300037 - (view) Author: (drallensmith) * Date: 2017-08-10 02:17
I am seeing a triggering of one of the assertions in convert_to_error, in managers.py. (This assertion is completely without any message, incidentally, which is also problematic - I've commented on this in issue5001.) This is in common across all versions of cpython that I have access to, and possibly to pypy as well. See https://travis-ci.org/drallensmith/neat-python/builds/262839326 for a set of examples. (The assertion is that "result" is a string. It is somewhat problematic when error-reporting code has an error...)
msg300077 - (view) Author: (drallensmith) * Date: 2017-08-10 12:53
An example on 2.7.13:

Traceback (most recent call last):
  File "/opt/python/2.7.9/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/opt/python/2.7.9/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/travis/build/drallensmith/neat-python/tests/test_distributed.py", line 412, in run_secondary
    de.start(secondary_wait=3, exit_on_stop=True)
  File "/home/travis/build/drallensmith/neat-python/neat/distributed.py", line 414, in start
    self._secondary_loop(reconnect_max_time=reconnect_max_time)
  File "/home/travis/build/drallensmith/neat-python/neat/distributed.py", line 616, in _secondary_loop
    self.outqueue.put(res)
  File "<string>", line 2, in put
  File "/opt/python/2.7.9/lib/python2.7/multiprocessing/managers.py", line 774, in _callmethod
    raise convert_to_error(kind, result)
AssertionError
Process Child evaluation process (multiple workers):
Traceback (most recent call last):
  File "/opt/python/2.7.9/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/opt/python/2.7.9/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/travis/build/drallensmith/neat-python/tests/test_distributed.py", line 412, in run_secondary
    de.start(secondary_wait=3, exit_on_stop=True)
  File "/home/travis/build/drallensmith/neat-python/neat/distributed.py", line 414, in start
    self._secondary_loop(reconnect_max_time=reconnect_max_time)
  File "/home/travis/build/drallensmith/neat-python/neat/distributed.py", line 616, in _secondary_loop
    self.outqueue.put(res)
  File "<string>", line 2, in put
  File "/opt/python/2.7.9/lib/python2.7/multiprocessing/managers.py", line 774, in _callmethod
    raise convert_to_error(kind, result)
AssertionError
msg300078 - (view) Author: (drallensmith) * Date: 2017-08-10 12:55
Correction - 2.7.9 - Travis seems to be a bit behind. Here's one from 3.6.2:

Traceback (most recent call last):
  File "/opt/python/3.6.2/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/opt/python/3.6.2/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/travis/build/drallensmith/neat-python/tests/test_distributed.py", line 412, in run_secondary
    de.start(secondary_wait=3, exit_on_stop=True)
  File "/home/travis/build/drallensmith/neat-python/neat/distributed.py", line 414, in start
    self._secondary_loop(reconnect_max_time=reconnect_max_time)
  File "/home/travis/build/drallensmith/neat-python/neat/distributed.py", line 616, in _secondary_loop
    self.outqueue.put(res)
  File "<string>", line 2, in put
  File "/opt/python/3.6.2/lib/python3.6/multiprocessing/managers.py", line 772, in _callmethod
    raise convert_to_error(kind, result)
AssertionError
Process Child evaluation process (multiple workers):
Traceback (most recent call last):
  File "/opt/python/3.6.2/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/opt/python/3.6.2/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/travis/build/drallensmith/neat-python/tests/test_distributed.py", line 412, in run_secondary
    de.start(secondary_wait=3, exit_on_stop=True)
  File "/home/travis/build/drallensmith/neat-python/neat/distributed.py", line 414, in start
    self._secondary_loop(reconnect_max_time=reconnect_max_time)
  File "/home/travis/build/drallensmith/neat-python/neat/distributed.py", line 616, in _secondary_loop
    self.outqueue.put(res)
  File "<string>", line 2, in put
  File "/opt/python/3.6.2/lib/python3.6/multiprocessing/managers.py", line 772, in _callmethod
    raise convert_to_error(kind, result)
AssertionError
msg300079 - (view) Author: (drallensmith) * Date: 2017-08-10 12:58
The section in question is:

def convert_to_error(kind, result):
    if kind == '#ERROR':
        return result
    elif kind == '#TRACEBACK':
        assert type(result) is str
        return  RemoteError(result)
    elif kind == '#UNSERIALIZABLE':
        assert type(result) is str
        return RemoteError('Unserializable message: %s\n' % result)
    else:
return ValueError('Unrecognized message type')
msg300087 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2017-08-10 13:49
For those who want to triage this issue, the test case can be found at https://github.com/drallensmith/neat-python/blob/e4aeb39eccefbd73babfb61bb13fd23feef2a102/tests/test_distributed.py#L234
msg300098 - (view) Author: (drallensmith) * Date: 2017-08-10 15:03
Well, it looks like I was incorrect in where the AssertionError is coming from - the attached patches, while an improvement IMO on the current code, did not result in a change in behavior. Unfortunately, the combination of bare asserts (with no messages) with remote raising of errors makes it rather hard to debug - I will see if pdb will do it, but given that it's involving a subprocess I will be very impressed if it does. (Coverage.py frequently misses lines that are only executed in subprocesses, for instance.)
msg300099 - (view) Author: (drallensmith) * Date: 2017-08-10 15:04
Here is the patch for 2.7.
msg300101 - (view) Author: (drallensmith) * Date: 2017-08-10 15:20
pdb is not currently working for debugging a subprocess. I suspect I will need to put "import pdb" and "pdb.set_trace()" into managers.py.

BTW, a thank-you to berker.peksag; while the link is present in Travis, I should still have put it in.
msg300103 - (view) Author: (drallensmith) * Date: 2017-08-10 15:34
I've updated the title to be more accurate.

On second thought, putting in a pdb.set_trace() would require that I know where the assertion failure is taking place... sigh. BTW, is there some way to edit earlier messages? I am used to github in that regard...
msg300199 - (view) Author: (drallensmith) * Date: 2017-08-12 12:47
Pull request equivalent for the patch submitted. This will not, of course, fix this particular bug, but it should help others with debugging such errors.
msg300202 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-08-12 15:37
New changeset 48d9823a0ebde4dfab8bc154bb6df462fb2ee403 by Antoine Pitrou (Allen W. Smith, Ph.D) in branch 'master':
bpo-5001, bpo-31169: Fix two uninformative asserts in multiprocessing/managers.py (#3078)
https://github.com/python/cpython/commit/48d9823a0ebde4dfab8bc154bb6df462fb2ee403
msg300203 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-08-12 15:39
Now fixed in git master thanks to drallensmith.
History
Date User Action Args
2017-08-12 15:39:35pitrousetstatus: open -> closed
versions: - Python 2.7, Python 3.3, Python 3.4, Python 3.5, Python 3.6
messages: + msg300203

resolution: fixed
stage: resolved
2017-08-12 15:37:11pitrousetmessages: + msg300202
2017-08-12 12:47:19drallensmithsetmessages: + msg300199
2017-08-12 12:45:07drallensmithsetpull_requests: + pull_request3118
2017-08-10 15:34:40drallensmithsetmessages: + msg300103
title: convert_to_error assertion failure in multiprocessing/managers.py -> Unknown-source assertion failure in multiprocessing/managers.py
2017-08-10 15:20:58drallensmithsetmessages: + msg300101
2017-08-10 15:04:40drallensmithsetfiles: + managers-2.7.patch

messages: + msg300099
2017-08-10 15:03:53drallensmithsetfiles: + managers-3.5.patch
keywords: + patch
messages: + msg300098
2017-08-10 13:49:42berker.peksagsetnosy: + berker.peksag, pitrou, davin
messages: + msg300087
2017-08-10 12:58:15drallensmithsetmessages: + msg300079
2017-08-10 12:55:49drallensmithsetmessages: + msg300078
2017-08-10 12:53:15drallensmithsetmessages: + msg300077
2017-08-10 02:17:46drallensmithcreate