Unknown-source assertion failure in multiprocessing/managers.py #75352

drallensmith · 2017-08-10T02:17:46Z

BPO	31169
Nosy	@pitrou, @berkerpeksag, @applio, @drallensmith
PRs	bpo-5001, bpo-31169: Fix two uninformative asserts in multiprocessing/managers.py #3078
Files	managers-3.5.patch: Patch to 3.5 needed to tell where AssertionError is not from managers-2.7.patch: Patch to 2.7 needed to tell where AssertionError is not coming from

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-08-12.15:39:35.561>
created_at = <Date 2017-08-10.02:17:46.266>
labels = ['3.7', 'type-bug', 'library']
title = 'Unknown-source assertion failure in multiprocessing/managers.py'
updated_at = <Date 2017-08-12.15:39:35.559>
user = 'https://github.com/drallensmith'

bugs.python.org fields:

activity = <Date 2017-08-12.15:39:35.559>
actor = 'pitrou'
assignee = 'none'
closed = True
closed_date = <Date 2017-08-12.15:39:35.561>
closer = 'pitrou'
components = ['Library (Lib)']
creation = <Date 2017-08-10.02:17:46.266>
creator = 'drallensmith'
dependencies = []
files = ['47075', '47076']
hgrepos = []
issue_num = 31169
keywords = ['patch']
message_count = 12.0
messages = ['300037', '300077', '300078', '300079', '300087', '300098', '300099', '300101', '300103', '300199', '300202', '300203']
nosy_count = 4.0
nosy_names = ['pitrou', 'berker.peksag', 'davin', 'drallensmith']
pr_nums = ['3078']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue31169'
versions = ['Python 3.7']

drallensmith · 2017-08-10T02:17:45Z

I am seeing a triggering of one of the assertions in convert_to_error, in managers.py. (This assertion is completely without any message, incidentally, which is also problematic - I've commented on this in bpo-5001.) This is in common across all versions of cpython that I have access to, and possibly to pypy as well. See https://travis-ci.org/drallensmith/neat-python/builds/262839326 for a set of examples. (The assertion is that "result" is a string. It is somewhat problematic when error-reporting code has an error...)

drallensmith · 2017-08-10T12:53:15Z

An example on 2.7.13:

Traceback (most recent call last):
  File "/opt/python/2.7.9/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/opt/python/2.7.9/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/travis/build/drallensmith/neat-python/tests/test_distributed.py", line 412, in run_secondary
    de.start(secondary_wait=3, exit_on_stop=True)
  File "/home/travis/build/drallensmith/neat-python/neat/distributed.py", line 414, in start
    self._secondary_loop(reconnect_max_time=reconnect_max_time)
  File "/home/travis/build/drallensmith/neat-python/neat/distributed.py", line 616, in _secondary_loop
    self.outqueue.put(res)
  File "<string>", line 2, in put
  File "/opt/python/2.7.9/lib/python2.7/multiprocessing/managers.py", line 774, in _callmethod
    raise convert_to_error(kind, result)
AssertionError
Process Child evaluation process (multiple workers):
Traceback (most recent call last):
  File "/opt/python/2.7.9/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/opt/python/2.7.9/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/travis/build/drallensmith/neat-python/tests/test_distributed.py", line 412, in run_secondary
    de.start(secondary_wait=3, exit_on_stop=True)
  File "/home/travis/build/drallensmith/neat-python/neat/distributed.py", line 414, in start
    self._secondary_loop(reconnect_max_time=reconnect_max_time)
  File "/home/travis/build/drallensmith/neat-python/neat/distributed.py", line 616, in _secondary_loop
    self.outqueue.put(res)
  File "<string>", line 2, in put
  File "/opt/python/2.7.9/lib/python2.7/multiprocessing/managers.py", line 774, in _callmethod
    raise convert_to_error(kind, result)
AssertionError

drallensmith · 2017-08-10T12:55:49Z

Correction - 2.7.9 - Travis seems to be a bit behind. Here's one from 3.6.2:

Traceback (most recent call last):
  File "/opt/python/3.6.2/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/opt/python/3.6.2/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/travis/build/drallensmith/neat-python/tests/test_distributed.py", line 412, in run_secondary
    de.start(secondary_wait=3, exit_on_stop=True)
  File "/home/travis/build/drallensmith/neat-python/neat/distributed.py", line 414, in start
    self._secondary_loop(reconnect_max_time=reconnect_max_time)
  File "/home/travis/build/drallensmith/neat-python/neat/distributed.py", line 616, in _secondary_loop
    self.outqueue.put(res)
  File "<string>", line 2, in put
  File "/opt/python/3.6.2/lib/python3.6/multiprocessing/managers.py", line 772, in _callmethod
    raise convert_to_error(kind, result)
AssertionError
Process Child evaluation process (multiple workers):
Traceback (most recent call last):
  File "/opt/python/3.6.2/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/opt/python/3.6.2/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/travis/build/drallensmith/neat-python/tests/test_distributed.py", line 412, in run_secondary
    de.start(secondary_wait=3, exit_on_stop=True)
  File "/home/travis/build/drallensmith/neat-python/neat/distributed.py", line 414, in start
    self._secondary_loop(reconnect_max_time=reconnect_max_time)
  File "/home/travis/build/drallensmith/neat-python/neat/distributed.py", line 616, in _secondary_loop
    self.outqueue.put(res)
  File "<string>", line 2, in put
  File "/opt/python/3.6.2/lib/python3.6/multiprocessing/managers.py", line 772, in _callmethod
    raise convert_to_error(kind, result)
AssertionError

drallensmith · 2017-08-10T12:58:15Z

The section in question is:

def convert_to_error(kind, result):
    if kind == '#ERROR':
        return result
    elif kind == '#TRACEBACK':
        assert type(result) is str
        return  RemoteError(result)
    elif kind == '#UNSERIALIZABLE':
        assert type(result) is str
        return RemoteError('Unserializable message: %s\n' % result)
    else:
return ValueError('Unrecognized message type')

berkerpeksag · 2017-08-10T13:49:42Z

For those who want to triage this issue, the test case can be found at https://github.com/drallensmith/neat-python/blob/e4aeb39eccefbd73babfb61bb13fd23feef2a102/tests/test_distributed.py#L234

drallensmith · 2017-08-10T15:03:53Z

Well, it looks like I was incorrect in where the AssertionError is coming from - the attached patches, while an improvement IMO on the current code, did not result in a change in behavior. Unfortunately, the combination of bare asserts (with no messages) with remote raising of errors makes it rather hard to debug - I will see if pdb will do it, but given that it's involving a subprocess I will be very impressed if it does. (Coverage.py frequently misses lines that are only executed in subprocesses, for instance.)

drallensmith · 2017-08-10T15:04:40Z

Here is the patch for 2.7.

drallensmith · 2017-08-10T15:20:59Z

pdb is not currently working for debugging a subprocess. I suspect I will need to put "import pdb" and "pdb.set_trace()" into managers.py.

BTW, a thank-you to berker.peksag; while the link is present in Travis, I should still have put it in.

drallensmith · 2017-08-10T15:34:40Z

I've updated the title to be more accurate.

On second thought, putting in a pdb.set_trace() would require that I know where the assertion failure is taking place... sigh. BTW, is there some way to edit earlier messages? I am used to github in that regard...

drallensmith · 2017-08-12T12:47:20Z

Pull request equivalent for the patch submitted. This will not, of course, fix this particular bug, but it should help others with debugging such errors.

pitrou · 2017-08-12T15:37:12Z

New changeset 48d9823 by Antoine Pitrou (Allen W. Smith, Ph.D) in branch 'master':
bpo-5001, bpo-31169: Fix two uninformative asserts in multiprocessing/managers.py (bpo-3078)
48d9823

pitrou · 2017-08-12T15:39:35Z

Now fixed in git master thanks to drallensmith.

drallensmith mannequin added 3.7 (EOL) end of life stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error labels Aug 10, 2017

drallensmith mannequin changed the title ~~convert_to_error assertion failure in multiprocessing/managers.py~~ Unknown-source assertion failure in multiprocessing/managers.py Aug 10, 2017

pitrou closed this as completed Aug 12, 2017

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unknown-source assertion failure in multiprocessing/managers.py #75352

Unknown-source assertion failure in multiprocessing/managers.py #75352

drallensmith mannequin commented Aug 10, 2017

drallensmith mannequin commented Aug 10, 2017

drallensmith mannequin commented Aug 10, 2017

drallensmith mannequin commented Aug 10, 2017

drallensmith mannequin commented Aug 10, 2017

berkerpeksag commented Aug 10, 2017

drallensmith mannequin commented Aug 10, 2017

drallensmith mannequin commented Aug 10, 2017

drallensmith mannequin commented Aug 10, 2017

drallensmith mannequin commented Aug 10, 2017

drallensmith mannequin commented Aug 12, 2017

pitrou commented Aug 12, 2017

pitrou commented Aug 12, 2017

Unknown-source assertion failure in multiprocessing/managers.py #75352

Unknown-source assertion failure in multiprocessing/managers.py #75352

Comments

drallensmith mannequin commented Aug 10, 2017

drallensmith mannequin commented Aug 10, 2017

drallensmith mannequin commented Aug 10, 2017

drallensmith mannequin commented Aug 10, 2017

drallensmith mannequin commented Aug 10, 2017

berkerpeksag commented Aug 10, 2017

drallensmith mannequin commented Aug 10, 2017

drallensmith mannequin commented Aug 10, 2017

drallensmith mannequin commented Aug 10, 2017

drallensmith mannequin commented Aug 10, 2017

drallensmith mannequin commented Aug 12, 2017

pitrou commented Aug 12, 2017

pitrou commented Aug 12, 2017