classification
Title: call-matcher breaks if a method is mocked with spec=True
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: David Hoyes, Eli_B, carljm, cjw296, jordan-pittier, mariocj89, michael.foord, pitrou, xtreak
Priority: normal Keywords:

Created on 2016-08-09 02:43 by carljm, last changed 2018-12-14 18:11 by xtreak.

Messages (7)
msg272209 - (view) Author: Carl Meyer (carljm) * Date: 2016-08-09 02:43
When constructing call-matchers to match expected vs actual calls, if `spec=True` was used when patching a function, mock attempts to bind the recorded (and expected) call args to the function signature. But if a method was mocked, the signature includes `self` and the recorded call args don't. This can easily lead to a `TypeError`:

```
from unittest.mock import patch

class Foo:
    def bar(self, x):
        return x

with patch.object(Foo, 'bar', spec=True) as mock_bar:
    f = Foo()
    f.bar(7)

mock_bar.assert_called_once_with(7)

```

The above code worked in mock 1.0, but fails in Python 3.5 and 3.6 tip with this error:

```
TypeError: missing a required argument: 'x'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "../mock-method.example.py", line 11, in <module>
    mock_bar.assert_called_once_with(7)
  File "/home/carljm/projects/python/cpython/Lib/unittest/mock.py", line 203, in assert_called_once_with
    return mock.assert_called_once_with(*args, **kwargs)
  File "/home/carljm/projects/python/cpython/Lib/unittest/mock.py", line 822, in assert_called_once_with
    return self.assert_called_with(*args, **kwargs)
  File "/home/carljm/projects/python/cpython/Lib/unittest/mock.py", line 811, in assert_called_with
    raise AssertionError(_error_message()) from cause
AssertionError: Expected call: bar(7)
Actual call: bar(<__main__.Foo object at 0x7fdca80b7550>, 7)
```
```

If you try to pass in the instance as an expected call arg, the error goes away but it just fails to match:

```
AssertionError: Expected call: bar(<__main__.Foo object at 0x7f5cbab35fd0>, 7)
Actual call: bar(7)
```

So AFAICT there is no way to successfully use `spec=True` when patching a method of a class.

Oddly, using `autospec=True` instead of `spec=True` _does_ record the instance as an argument in the recorded call args, meaning that you have to pass it in as an argument to e.g. `assert_called_with`. But in many (most?) cases where you're patching a method of a class, your test doesn't have access to the instance, elsewise you'd likely just patch the instance instead of the class in the first place.

I don't see a good reason why `autospec=True` and `spec=True` should differ in this way (if both options are needed, there should be a separate flag to control that behavior; it doesn't seem related to the documented differences between autospec and spec). I do think a) there needs to be some way to record call args to a method and assert against those call args without needing the instance (or resorting to manual assertions against a sliced `call_args`), and b) there should be some way to successfully use `spec=True` when patching a method of a class.
msg272210 - (view) Author: Carl Meyer (carljm) * Date: 2016-08-09 02:44
(This bug is also present in Python 3.4.4.)
msg272211 - (view) Author: Carl Meyer (carljm) * Date: 2016-08-09 02:59
It seems likely that this regression originated with https://hg.python.org/cpython/rev/b888c9043566/ (can't confirm via bisection as the commits around that time fail to compile for me).
msg272212 - (view) Author: Carl Meyer (carljm) * Date: 2016-08-09 03:13
`hg clean --all` resolved the compilation issues; confirmed that https://hg.python.org/cpython/rev/b888c9043566/ is at fault.

Also, the exception trace I provided above looks wrong; it must be from when I was messing about with `autospec=True` or passing in the instance. The actual trace from the sample code in the original report  has no mention of the instance:

```
TypeError: missing a required argument: 'x'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "../mock-method.example.py", line 11, in <module>
    mock_bar.assert_called_once_with(7)
  File "/home/carljm/projects/python/cpython/Lib/unittest/mock.py", line 822, in assert_called_once_with
    return self.assert_called_with(*args, **kwargs)
  File "/home/carljm/projects/python/cpython/Lib/unittest/mock.py", line 811, in assert_called_with
    raise AssertionError(_error_message()) from cause
AssertionError: Expected call: bar(7)
Actual call: bar(7)

```
msg286769 - (view) Author: (jordan-pittier) Date: 2017-02-02 12:00
I stumbled onto this today. I can confirm the issue.
msg299319 - (view) Author: David Hoyes (David Hoyes) Date: 2017-07-27 15:44
I came across a different failing test case, which looks a lot like the same issue:

```
from unittest import mock

class Foo(object):
    def __call__(self, x):
        return x

m = mock.create_autospec(Foo, instance=True)
m(7)
m.assert_called_once_with(7)
```

In mock 1.0.1 this passes, but in Python 3.5 we get this error:

```
TypeError: missing a required argument: 'x'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "euhpc/tmp/mockbug.py", line 12, in <module>
    m.assert_called_once_with(7)
  File "/usr/lib/python3.5/unittest/mock.py", line 803, in assert_called_once_with
    return self.assert_called_with(*args, **kwargs)
  File "/usr/lib/python3.5/unittest/mock.py", line 792, in assert_called_with
    raise AssertionError(_error_message()) from cause
AssertionError: Expected call: mock(7)
Actual call: mock(7)
```
msg331846 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python triager) Date: 2018-12-14 18:11
I think the original issue with patch.object reported by Carl is different from the one reported by David for autospec. Analyzed the report by David and When we call autospec on a class with instance=True then the spec is modeled on the signature of __call__ instead of __init__ where __call__ has the signature of (self, x) and self is  discarded with _eat_self passed as True. But mock also stores _spec_signature that is not aware of skipping self and has the signature as (self, x) and is used for checking signature in mock.assert_called_with. When instance=True then kwargs['_spec_as_instance']=True so does it makes sense to set kwargs['_eat_self'] = True at [0] ? I applied the change and there are no test failures so this deserves a test.

This makes __call__ to have a different signature when it's called from mock and when the call is checked with assert_called_with_once. But it's not the case for methods of the class where self is skipped both for mock and _spec_signature. Since __signature__ is set for mock with my PR this makes it little easy to debug. Can this be dealt as a separate issue?

I would also love to see if the assertion message can be improved since expected_call and actual_call are printed with repr version of the call object in spite of the signature failure. This has caused confusion here and in other places like issue26752 and issue25312.

Sample program to demonstrate difference in signatures :  

import inspect
from unittest import mock

class Foo:

    def __call__(self, x):
        return x

    def bar(self, x):
        pass

m = mock.create_autospec(Foo, instance=True)
m(7)
m.bar(7)
print(inspect.signature(m))
print(m._spec_signature)
print(inspect.signature(m.bar))
print(m.bar._spec_signature)
m.bar.assert_called_once_with(7) # 7 passed as self with no value for x
m.assert_called_once_with(7) # Fails due to self

(x)
(self, x) # self is not skipped in _spec_signature
(x)
(x)
TypeError: missing a required argument: 'x'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "../backups/bpo27715_1.py", line 20, in <module>
    m.assert_called_once_with(7)
  File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/unittest/mock.py", line 840, in assert_called_once_with
    return self.assert_called_with(*args, **kwargs)
  File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/unittest/mock.py", line 827, in assert_called_with
    raise AssertionError(_error_message()) from cause
AssertionError: Expected call: mock(7)
Actual call: mock(7)


[0] https://github.com/python/cpython/blob/f8e9bd568adf85c1e4aea1dda542a96b027797e2/Lib/unittest/mock.py#L2199
History
Date User Action Args
2018-12-14 18:11:33xtreaksetnosy: + cjw296, mariocj89

messages: + msg331846
versions: - Python 3.6
2018-12-01 19:10:17xtreaksetnosy: + xtreak

versions: + Python 3.7, Python 3.8, - Python 3.4, Python 3.5
2018-07-11 07:47:37serhiy.storchakasettype: crash -> behavior
2017-07-27 15:44:15David Hoyessetnosy: + David Hoyes
messages: + msg299319
2017-02-02 12:00:30jordan-pittiersetnosy: + jordan-pittier
messages: + msg286769
2016-09-11 16:24:39Eli_Bsetnosy: + Eli_B
2016-08-09 03:13:07carljmsetmessages: + msg272212
2016-08-09 03:11:30carljmsetfiles: - mock-method.example.py
2016-08-09 02:59:49carljmsetnosy: + pitrou, michael.foord
messages: + msg272211
2016-08-09 02:44:50carljmsettype: crash
messages: + msg272210
versions: + Python 3.4
2016-08-09 02:43:25carljmcreate