I am seeing a triggering of one of the assertions in convert_to_error, in managers.py. (This assertion is completely without any message, incidentally, which is also problematic - I've commented on this in issue5001.) This is in common across all versions of cpython that I have access to, and possibly to pypy as well. See https://travis-ci.org/drallensmith/neat-python/builds/262839326 for a set of examples. (The assertion is that "result" is a string. It is somewhat problematic when error-reporting code has an error...)

An example on 2.7.13:

Traceback (most recent call last):
  File "/opt/python/2.7.9/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/opt/python/2.7.9/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/travis/build/drallensmith/neat-python/tests/test_distributed.py", line 412, in run_secondary
    de.start(secondary_wait=3, exit_on_stop=True)
  File "/home/travis/build/drallensmith/neat-python/neat/distributed.py", line 414, in start
    self._secondary_loop(reconnect_max_time=reconnect_max_time)
  File "/home/travis/build/drallensmith/neat-python/neat/distributed.py", line 616, in _secondary_loop
    self.outqueue.put(res)
  File "<string>", line 2, in put
  File "/opt/python/2.7.9/lib/python2.7/multiprocessing/managers.py", line 774, in _callmethod
    raise convert_to_error(kind, result)
AssertionError
Process Child evaluation process (multiple workers):
Traceback (most recent call last):
  File "/opt/python/2.7.9/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/opt/python/2.7.9/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/travis/build/drallensmith/neat-python/tests/test_distributed.py", line 412, in run_secondary
    de.start(secondary_wait=3, exit_on_stop=True)
  File "/home/travis/build/drallensmith/neat-python/neat/distributed.py", line 414, in start
    self._secondary_loop(reconnect_max_time=reconnect_max_time)
  File "/home/travis/build/drallensmith/neat-python/neat/distributed.py", line 616, in _secondary_loop
    self.outqueue.put(res)
  File "<string>", line 2, in put
  File "/opt/python/2.7.9/lib/python2.7/multiprocessing/managers.py", line 774, in _callmethod
    raise convert_to_error(kind, result)
AssertionError

Correction - 2.7.9 - Travis seems to be a bit behind. Here's one from 3.6.2:

Traceback (most recent call last):
  File "/opt/python/3.6.2/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/opt/python/3.6.2/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/travis/build/drallensmith/neat-python/tests/test_distributed.py", line 412, in run_secondary
    de.start(secondary_wait=3, exit_on_stop=True)
  File "/home/travis/build/drallensmith/neat-python/neat/distributed.py", line 414, in start
    self._secondary_loop(reconnect_max_time=reconnect_max_time)
  File "/home/travis/build/drallensmith/neat-python/neat/distributed.py", line 616, in _secondary_loop
    self.outqueue.put(res)
  File "<string>", line 2, in put
  File "/opt/python/3.6.2/lib/python3.6/multiprocessing/managers.py", line 772, in _callmethod
    raise convert_to_error(kind, result)
AssertionError
Process Child evaluation process (multiple workers):
Traceback (most recent call last):
  File "/opt/python/3.6.2/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/opt/python/3.6.2/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/travis/build/drallensmith/neat-python/tests/test_distributed.py", line 412, in run_secondary
    de.start(secondary_wait=3, exit_on_stop=True)
  File "/home/travis/build/drallensmith/neat-python/neat/distributed.py", line 414, in start
    self._secondary_loop(reconnect_max_time=reconnect_max_time)
  File "/home/travis/build/drallensmith/neat-python/neat/distributed.py", line 616, in _secondary_loop
    self.outqueue.put(res)
  File "<string>", line 2, in put
  File "/opt/python/3.6.2/lib/python3.6/multiprocessing/managers.py", line 772, in _callmethod
    raise convert_to_error(kind, result)
AssertionError

The section in question is:

def convert_to_error(kind, result):
    if kind == '#ERROR':
        return result
    elif kind == '#TRACEBACK':
        assert type(result) is str
        return  RemoteError(result)
    elif kind == '#UNSERIALIZABLE':
        assert type(result) is str
        return RemoteError('Unserializable message: %s\n' % result)
    else:
return ValueError('Unrecognized message type')

For those who want to triage this issue, the test case can be found at https://github.com/drallensmith/neat-python/blob/e4aeb39eccefbd73babfb61bb13fd23feef2a102/tests/test_distributed.py#L234

Well, it looks like I was incorrect in where the AssertionError is coming from - the attached patches, while an improvement IMO on the current code, did not result in a change in behavior. Unfortunately, the combination of bare asserts (with no messages) with remote raising of errors makes it rather hard to debug - I will see if pdb will do it, but given that it's involving a subprocess I will be very impressed if it does. (Coverage.py frequently misses lines that are only executed in subprocesses, for instance.)

Here is the patch for 2.7.

pdb is not currently working for debugging a subprocess. I suspect I will need to put "import pdb" and "pdb.set_trace()" into managers.py.

BTW, a thank-you to berker.peksag; while the link is present in Travis, I should still have put it in.

I've updated the title to be more accurate.

On second thought, putting in a pdb.set_trace() would require that I know where the assertion failure is taking place... sigh. BTW, is there some way to edit earlier messages? I am used to github in that regard...

Pull request equivalent for the patch submitted. This will not, of course, fix this particular bug, but it should help others with debugging such errors.

New changeset 48d9823a0ebde4dfab8bc154bb6df462fb2ee403 by Antoine Pitrou (Allen W. Smith, Ph.D) in branch 'master':
bpo-5001, bpo-31169: Fix two uninformative asserts in multiprocessing/managers.py (#3078)
https://github.com/python/cpython/commit/48d9823a0ebde4dfab8bc154bb6df462fb2ee403

Now fixed in git master thanks to drallensmith.