Customizing unittest.mock.mock_open for iteration

Question

How should I customize unittest.mock.mock_open to handle this code?

file: impexpdemo.py
def import_register(register_fn):
    with open(register_fn) as f:
        return [line for line in f]

My first attempt tried read_data.

class TestByteOrderMark1(unittest.TestCase):
    REGISTER_FN = 'test_dummy_path'
    TEST_TEXT = ['test text 1\n', 'test text 2\n']

    def test_byte_order_mark_absent(self):
        m = unittest.mock.mock_open(read_data=self.TEST_TEXT)
        with unittest.mock.patch('builtins.open', m):
            result = impexpdemo.import_register(self.REGISTER_FN)
            self.assertEqual(result, self.TEST_TEXT)

This failed, presumably because the code doesn't use read, readline, or readlines. The documentation for unittest.mock.mock_open says, "read_data is a string for the read(), readline(), and readlines() methods of the file handle to return. Calls to those methods will take data from read_data until it is depleted. The mock of these methods is pretty simplistic. If you need more control over the data that you are feeding to the tested code you will need to customize this mock for yourself. read_data is an empty string by default."

As the documentation gave no hint on what kind of customization would be required I tried return_value and side_effect. Neither worked.

class TestByteOrderMark2(unittest.TestCase):
    REGISTER_FN = 'test_dummy_path'
    TEST_TEXT = ['test text 1\n', 'test text 2\n']

    def test_byte_order_mark_absent(self):
        m = unittest.mock.mock_open()
        m().side_effect = self.TEST_TEXT
        with unittest.mock.patch('builtins.open', m):
            result = impexpdemo.import_register(self.REGISTER_FN)
            self.assertEqual(result, self.TEST_TEXT)

Martijn Pieters · Accepted Answer · 2017-01-14T22:48:53.813

38

The mock_open() object does indeed not implement iteration.

If you are not using the file object as a context manager, you could use:

m = unittest.mock.MagicMock(name='open', spec=open)
m.return_value = iter(self.TEST_TEXT)

with unittest.mock.patch('builtins.open', m):

Now open() returns an iterator, something that can be directly iterated over just like a file object can be, and it'll also work with next(). It can not, however, be used as a context manager.

You can combine this with mock_open() then provide a __iter__ and __next__ method on the return value, with the added benefit that mock_open() also adds the prerequisites for use as a context manager:

# Note: read_data must be a string!
m = unittest.mock.mock_open(read_data=''.join(self.TEST_TEXT))
m.return_value.__iter__ = lambda self: self
m.return_value.__next__ = lambda self: next(iter(self.readline, ''))

The return value here is a MagicMock object specced from the file object (Python 2) or the in-memory file objects (Python 3), but only the read, write and __enter__ methods have been stubbed out.

The above doesn't work in Python 2 because a) Python 2 expects next to exist, not __next__ and b) next is not treated as a special method in Mock (rightly so), so even if you renamed __next__ to next in the above example the type of the return value won't have a next method. For most cases it would be enough to make the file object produced an iterable rather than an iterator with:

# Python 2!
m = mock.mock_open(read_data=''.join(self.TEST_TEXT))
m.return_value.__iter__ = lambda self: iter(self.readline, '')

Any code that uses iter(fileobj) will then work (including a for loop).

There is a open issue in the Python tracker that aims to remedy this gap.

edited Jan 14 '17 at 22:48

answered Jul 16 '14 at 11:42

Martijn Pieters

1,048,767
296
4,058
3,343

1

Isn't it better to patch `module_under_test.open` than `builtins.open`? – and Jul 17 '14 at 06:42
2

This is based on the OP question and is also used in the [`unittest.mock` examples documentation](https://docs.python.org/3/library/unittest.mock-examples.html#patch-decorators); I wanted to focus on the iterable aspect of the `mock_open()` return value. `module_under_test.open` would be slightly more specific, yes. – Martijn Pieters Jul 17 '14 at 07:47
`with unittest.mock.patch('impexpdemo.open', m):` produces the error message: `AttributeError: does not have the attribute 'open'` – lemi57ssss Jul 17 '14 at 10:06
@lemi57ssss: right, so `mock.patch()` tests to see if the object already exists, in which case `builtins.open` is the way to go here. Does simply mocking `open()` to return the list or augmenting the `mock_open()` return value work for you? – Martijn Pieters Jul 17 '14 at 10:23
Thank you for a most excellent answer which solves the problem. The program from which this is drawn opens the file, silently ignores a utf-8 byte order mark if present, and then reads a csv file. It was the byte order mark handling I was trying to test. – lemi57ssss Jul 17 '14 at 10:36
@Martijn Pieters: I used mock_open exactly as your second example. I was unable to get anything with NagicMock although I'll continue to spend some time on trying to understand why this works. If I discover anything interesting I'll post it here. – lemi57ssss Jul 17 '14 at 10:57
1

@lemi57ssss: it could be that your actual code-under-test does more than just iterate. I've updated the first snippet to return `iter(listobj)` instead to return an iterator, not just an iterable. – Martijn Pieters Jul 17 '14 at 11:00
I reported that `m.return_value = iter(self.TEST_TEXT) also works perfectly.` This was incorrect. In fact `with open...` fails with an `__exit__` error. – lemi57ssss Jul 18 '14 at 12:49
@lemi57ssss: ah, yes, that's because you are using it as a context manager. I'll add a note. – Martijn Pieters Jul 18 '14 at 12:52
+1 for pointing out that mock_open() object does indeed not implement iteration, this should've been added to the documentation, http://www.voidspace.org.uk/python/mock/helpers.html#mock-open – zyxue Jul 17 '15 at 00:05
Can anyone explain the function of 'lambda self: self' in the above code? – gwilymh Jul 12 '16 at 20:37
1

I cannot get the mock_open code above to work. Does the code only work for python 3? I get the following error message: TypeError: iter() returned non-iterator of type 'MagicMock' – gwilymh Jul 12 '16 at 21:10
@gwilymh in Python 2 iterations must have a `next()` method instead of `__next__()`. I can't test this right now but try that. – Martijn Pieters Jul 12 '16 at 21:18
@gwilymh at any rate, on iterators the `__iter__()` *must* return `self` which is what that lambda does. – Martijn Pieters Jul 12 '16 at 21:19
@gwilymh: updated with a work-around, if you *must* have `next(fileobj)` work then that'll not be enough though; you'd have to subclass `MagicMock` and add support for `next` there. – Martijn Pieters Jul 13 '16 at 11:08
hm, what about `StopIteration()`? As-is your `__next__` lambda never raises it - i.e. the loop iterates endlessly. Fix: raise this exception if `readline()` returns the empty string (`StopIteration()` is part of the iterator protocol; `readline()` returns `'\n'` for empty lines and `''` for EOF). – maxschlepzig Jan 14 '17 at 22:16
@maxschlepzig: sorry, yes, fixed that now. – Martijn Pieters Jan 14 '17 at 22:49
This seems unrelated here, in my (python 2.7) test runs, the mock file object was missing `readline` method. In my tests in python console however, it worked as expected(`readline` worked as expected). I had to mock readline() (`lambda: next(iterator_created_from_test_data_list)`). It's working now, but just posted out of curiosity, as I haven't understood what caused the lack of that method, maybe you can throw some light. Also my attempt to help, if anyone gets bitten by the same strange behaviour. – 0xc0de Jan 19 '17 at 07:07
@0xc0de: the `mock_open()` call produces a rather minimal mock for file objects, you indeed have to add 'extra' methods as needed. – Martijn Pieters Jan 19 '17 at 07:18
The same method was available in console tests, but unavailable in a running test suite. I literally copy-pasted the code to console to ensure code similarity, checked (with `is`) that my mocked method is the one being called in the actual code(being tested). The value of `opened_file_obj.readline` was something like `Mock().open() object at xxxxxxx`. – 0xc0de Jan 19 '17 at 07:35

maxschlepzig · Answer 2 · 2017-01-15T15:18:38.383

As of Python 3.6, the mocked file-like object returned by the unittest.mock_open method doesn't support iteration. This bug was reported in 2014 and it is still open as of 2017.

Thus code like this silently yields zero iterations:

f_open = unittest.mock.mock_open(read_data='foo\nbar\n')
f = f_open('blah')
for line in f:
  print(line)

You can work around this limitation via adding a method to the mocked object that returns a proper line iterator:

def mock_open(*args, **kargs):
  f_open = unittest.mock.mock_open(*args, **kargs)
  f_open.return_value.__iter__ = lambda self : iter(self.readline, '')
  return f_open

Bug is still open as of 2018 – Luis Meraz May 02 '18 at 06:20 — Luis Meraz, May 02 '18 at 06:20

score 2 · Answer 3 · answered May 02 '18 at 06:55

I found the following solution:

text_file_data = '\n'.join(["a line here", "the second line", "another 
line in the file"])
with patch('__builtin__.open', mock_open(read_data=text_file_data), 
create=True) as m:
    # mock_open doesn't properly handle iterating over the open file with for line in file:
    # but if we set the return value like this, it works.
    m.return_value.__iter__.return_value = text_file_data.splitlines()
    with open('filename', 'rU') as f:
        for line in f:
            print line

Customizing unittest.mock.mock_open for iteration

3 Answers3

Linked

Related