classification
Title: mock_open is not compatible with read(n) (and pickle.load)
Type: behavior Stage: patch review
Components: Library (Lib) Versions: Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: michael.foord Nosy List: cjw296, mariocj89, michael.foord, remi.lapeyre, ron.rothman, xtreak
Priority: normal Keywords: patch, patch, patch

Created on 2017-10-23 21:03 by ron.rothman, last changed 2019-01-11 16:14 by remi.lapeyre.

Pull Requests
URL Status Linked Edit
PR 11521 open remi.lapeyre, 2019-01-11 16:14
PR 11521 open remi.lapeyre, 2019-01-11 16:14
PR 11521 open remi.lapeyre, 2019-01-11 16:14
Messages (4)
msg304841 - (view) Author: Ron Rothman (ron.rothman) Date: 2017-10-23 21:03
mock.mock_open works as expected when reading the entire file (read()) or when reading a single line (readline()), but it seems to not support reading a number of bytes (read(n)).

These work as expected:

    from mock import mock_open, patch

    # works: consume entire "file"
    with patch('__main__.open', mock_open(read_data='bibble')) as m:
        with open('foo') as h:
            result = h.read()

    assert result == 'bibble'  # ok

    # works: consume one line
    with patch('__main__.open', mock_open(read_data='bibble\nbobble')) as m:
        with open('foo') as h:
            result = h.readline()

    assert result == 'bibble\n'  # ok

But trying to read only a few bytes fails--mock_open returns the entire read_data instead:

    # consume first 3 bytes of the "file"
    with patch('__main__.open', mock_open(read_data='bibble')) as m:
        with open('foo') as h:
            result = h.read(3)

    assert result == 'bib', 'result of read: {}'.format(result)  # fails

Output:

    Traceback (most recent call last):
      File "/tmp/t.py", line 25, in <module>
        assert result == 'bib', 'result of read: {}'.format(result)
    AssertionError: result of read: bibble

The unfortunate effect of this is that mock_open cannot be used with pickle.load.

    with open('/path/to/file.pkl', 'rb') as f:
        x = pickle.load(f)  # this requires f.read(1) to work
msg304846 - (view) Author: Ron Rothman (ron.rothman) Date: 2017-10-23 21:39
Confirmed that the behavior exists in Python 3.6 as well.
msg331920 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python triager) Date: 2018-12-16 08:16
Internally mock_open implementation uses line based iteration [0] to keep track of the state change between read calls. So readline too ignores the argument. There is issue25690 for an alternate mock_open implementation but the code change is large and adds a lot of features. A simpler approach would be to use StringIO or BytesIO to keep track of state changes and they provide read, readline and readlines API. mock_open docs mentions about using a customized mock for complex cases but I don't know if worthy enough to make this enhancement given the internal implementation change to support the API or to document this behavior. 

Looking further there also seems to be a test case for it [1] which will fail if this is fixed since this returns all characters instead of first 10 like using open().read(10).


    def test_mock_open_read_with_argument(self):
        # At one point calling read with an argument was broken
        # for mocks returned by mock_open
        some_data = 'foo\nbar\nbaz'
        mock = mock_open(read_data=some_data)
        self.assertEqual(mock().read(10), some_data)


$ echo -n 'foo\nbar\nbaz' > /tmp/a.txt
$ ./python.exe
Python 3.8.0a0 (heads/master:f5107dfd42, Dec 16 2018, 13:41:57)
[Clang 7.0.2 (clang-700.1.81)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> with open('/tmp/a.txt') as f:
...     actual = f.read(10)
...     actual, len(actual)
...
('foo\nbar\nba', 10)
>>> with open('/tmp/a.txt') as f:
...     from unittest.mock import mock_open
...     mock = mock_open(read_data=f.read())
...     mock_data = mock().read(10)
...     mock_data, len(mock_data)
...
('foo\nbar\nbaz', 11)


[0] https://github.com/python/cpython/blob/f5107dfd42121ef40b13eb678705802f0ff02cf9/Lib/unittest/mock.py#L2349
[1] https://github.com/python/cpython/blob/f5107dfd42121ef40b13eb678705802f0ff02cf9/Lib/unittest/test/testmock/testwith.py#L284
msg333484 - (view) Author: Rémi Lapeyre (remi.lapeyre) * Date: 2019-01-11 16:13
The off-by-one error in a test added for an unrelated issue (#17467) makes me think @michael.foord just made a mistake in the test.

> mock_open docs mentions about using a customized mock for complex cases
I think it's more for complex things like fetching data using a callback for example, this seems like a normal use case for mock_open(). Even more so, leaving the behavior as it is now may lead someone to think there is a bug in its code.

I opened a new PR to fix this issue: https://github.com/python/cpython/pull/11521
History
Date User Action Args
2019-01-11 16:14:24remi.lapeyresetkeywords: + patch
stage: patch review
pull_requests: + pull_request11100
2019-01-11 16:14:16remi.lapeyresetkeywords: + patch
stage: (no value)
pull_requests: + pull_request11099
2019-01-11 16:14:07remi.lapeyresetkeywords: + patch
stage: (no value)
pull_requests: + pull_request11098
2019-01-11 16:13:38remi.lapeyresetnosy: + remi.lapeyre
messages: + msg333484
2018-12-16 08:16:24xtreaksetnosy: + cjw296, xtreak, mariocj89

messages: + msg331920
versions: + Python 3.7, Python 3.8, - Python 2.7, Python 3.6
2017-10-24 04:32:25rhettingersetassignee: michael.foord

nosy: + michael.foord
2017-10-23 21:39:26ron.rothmansetmessages: + msg304846
versions: + Python 3.6
2017-10-23 21:03:10ron.rothmancreate