classification
Title: Diff for visually comparing actual with expected in mock.assert_called_with.
Type: enhancement Stage:
Components: Tests Versions: Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Eli Rose, Pavel Savchenko, cjw296, mariocj89, michael.foord, rhettinger, xtreak
Priority: normal Keywords: patch

Created on 2016-09-09 22:44 by Eli Rose, last changed 2018-12-11 06:33 by cjw296.

Files
File name Uploaded Description Edit
issue28054.diff xtreak, 2018-12-09 14:07
Messages (7)
msg275481 - (view) Author: Eli Rose (Eli Rose) Date: 2016-09-09 22:44
When I call unittest.TestCase.assertEqual(a, b) on e.g. two unequal dictionaries, I get a nice diff pointing me to the differences.

>>> class A(unittest.TestCase):
...     def test_foo(self):
...         self.assertEqual(dict(foo='bar', zab='zar'), dict(foo='bar', zab='zab'))
>>> unittest.main()
======================================================================
FAIL: test_foo (__main__.A)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "<stdin>", line 3, in test_foo
AssertionError: {'foo': 'bar', 'zab': 'zar'} != {'foo': 'bar', 'zab': 'zab'}
- {'foo': 'bar', 'zab': 'zar'}
?                          ^

+ {'foo': 'bar', 'zab': 'zab'}
?                          ^


----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (failures=1)

But when unittest.mock.Mock.assert_called_with fails, I don't get this nice diff output.

This would be very helpful in my present case (asserting that a function with many keyword arguments is called correctly).
msg279319 - (view) Author: Pavel Savchenko (Pavel Savchenko) Date: 2016-10-24 16:47
An implementation of this exists in pytest-mock. Recently an idea was brought up to shift the development of PR #57 into mock directly for everyone's benefit.

https://github.com/pytest-dev/pytest-mock
https://github.com/pytest-dev/pytest-mock/pull/58#issuecomment-253556301
msg279394 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2016-10-25 11:42
I like the idea and would be happy for it to be added to mock.
msg329636 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2018-11-10 18:46
I think this can be useful for keyword arguments as in the original suggestion. I tried an initial implementation as below to use difflib to get the difference like unittest if there are keyword args to be checked against the caller list since the absence of one or other generates a diff with empty {} which I find little distracting during my initial iterations. I would like to know if there is still sufficient interest in getting this to core only for keyword arguments given that there is pytest-mock with specialized error reporting handling more cases : https://github.com/pytest-dev/pytest-mock#improved-reporting-of-mock-call-assertion-errors


$ ./python.exe -q
>>> from unittest.mock import Mock
>>> m = Mock()
>>>
>>> m(foo='bar', bar='baz')
<Mock name='mock()' id='4484887184'>
>>>
>>> m.assert_called_with(bar='baz', foo='car')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/unittest/mock.py", line 838, in assert_called_with
    raise AssertionError(_error_message()) from cause
AssertionError: Expected call: mock(bar='baz', foo='car')
Actual call: mock(bar='baz', foo='bar')
- {'bar': 'baz', 'foo': 'car'}
?                        ^

+ {'bar': 'baz', 'foo': 'bar'}
?                        ^


diff --git a/Lib/unittest/mock.py b/Lib/unittest/mock.py
index a9c82dcb5d..8603b4ac4c 100644
--- a/Lib/unittest/mock.py
+++ b/Lib/unittest/mock.py
@@ -749,6 +749,20 @@ class NonCallableMock(Base):
         call_args = self.call_args
         if len(call_args) == 3:
             call_args = call_args[1:]
+
+        diffMsg = ''
+        if kwargs and self.call_args[1]:
+            import difflib
+
+            seq1 = kwargs
+            seq2 = self.call_args[1]
+            diffMsg = '\n' + '\n'.join(
+                difflib.ndiff(pprint.pformat(seq1).splitlines(),
+                              pprint.pformat(seq2).splitlines()))
+
+        if diffMsg:
+            message += diffMsg
+
         actual_string = self._format_mock_call_signature(*call_args)
         return message % (expected_string, actual_string)
msg329653 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2018-11-11 00:03
+1 for extending the unittest display niceties to mock.
msg331433 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2018-12-09 14:07
It seems that TestCase in unittest.case accepts failureException attribute that is raised on assertion failure. My initial approach is to subclass TestCase and add a custom failure exception like MockException that is caught to add the diff to the original message. I am adding @cjw296 and @mariocj89 for feedback since I hope this can reuse unittest.case.TestCase implementation for diffs. I think this is better done with a flag like verbose=True to display the difference and defaulting to False since sometimes the message can be long depending on args and kwargs. Also I find args (tuple) comparison to be little distracting. Since this is an enhancement I am adding 3.8.

# Sample file

from unittest.mock import Mock

m = Mock()
m(1, 2, foo='bar', bar='baz')
m.assert_called_with(2, 3, bar='baz', foo='car')

# On 3.7

$ python3.7 /tmp/foo_4.py
Traceback (most recent call last):
  File "/tmp/foo_4.py", line 5, in <module>
    m.assert_called_with(2, 3, bar='baz', foo='car')
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/mock.py", line 820, in assert_called_with
    raise AssertionError(_error_message()) from cause
AssertionError: Expected call: mock(2, 3, bar='baz', foo='car')

# With patch

$ ./python.exe /tmp/foo_4.py
Traceback (most recent call last):
  File "/tmp/foo_4.py", line 5, in <module>
    m.assert_called_with(2, 3, bar='baz', foo='car')
  File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/unittest/mock.py", line 844, in assert_called_with
    raise AssertionError(_error_message()) from cause
AssertionError: Expected call: mock(2, 3, bar='baz', foo='car')
Actual call: mock(1, 2, bar='baz', foo='bar')
args: Tuples differ: (2, 3) != (1, 2)

First differing element 0:
2
1

- (2, 3)
+ (1, 2)
kwargs: {'bar': 'baz', 'foo': 'car'} != {'foo': 'bar', 'bar': 'baz'}
- {'bar': 'baz', 'foo': 'car'}
?                        ^

+ {'bar': 'baz', 'foo': 'bar'}
?                        ^
msg331583 - (view) Author: Chris Withers (cjw296) * (Python committer) Date: 2018-12-11 06:33
This is a tricky one as there's plenty of prior art, with pytest's assertion rewriting [1], testfixtures compare [2] and the stuff that unittest already does [3].

I don't think any solution should rely on a TestCase being used as pytest, which is the most prevalent testing framework now, doesn't want to rely on them (they do come with a lot of baggage ;-)).

Can we make use of the pretty diffing stuff in unittest without explicitly making it a requirement?

[1] https://docs.pytest.org/en/latest/assert.html

[2] https://testfixtures.readthedocs.io/en/latest/comparing.html#dicts

[3] https://docs.python.org/3.8/library/unittest.html#unittest.TestCase.assertDictEqual
History
Date User Action Args
2018-12-11 06:33:15cjw296setmessages: + msg331583
2018-12-09 14:07:36xtreaksetfiles: + issue28054.diff
versions: + Python 3.8, - Python 2.7, Python 3.7
nosy: + cjw296, mariocj89

messages: + msg331433

keywords: + patch
2018-11-11 00:03:35rhettingersetnosy: + rhettinger
messages: + msg329653
2018-11-10 18:46:40xtreaksetnosy: + xtreak
messages: + msg329636
2016-10-25 11:42:35michael.foordsetmessages: + msg279394
2016-10-24 16:47:44Pavel Savchenkosetnosy: + Pavel Savchenko
messages: + msg279319
2016-09-10 15:12:54SilentGhostsetnosy: + michael.foord

versions: + Python 3.7, - Python 3.3
2016-09-09 22:44:33Eli Rosecreate