How to mock os.walk in python with a temporary filesystem?

Question

I'm trying to test some code that uses os.walk. I want to create a temporary, in-memory filesystem that I can populate with sample (empty) files and directories that os.walk will then return. This should save me the complexity of mocking os.walk calls to simulate recursion.

Specifically, the code I want to test is:

if recursive:
    log.debug("Recursively searching for files under %s" % path)

    for (dir_path, dirs, files) in os.walk(path):
        log.debug("Found %d files in %s: %s" % (len(files), path, files))
        for f in [os.path.join(dir_path, f) for f in files
                  if not re.search(exclude, f)]:
            yield f
else:
    log.debug("Non-recursively searching for files under %s" % path)

    for (dir_path, dirs, files) in os.walk(path):
        log.debug("Found %d files in %s: %s" % (len(files), path, files))
        for f in [os.path.join(dir_path, f) for f in files
                    if not re.search(exclude, f)]:
            yield f

Is this possible in python?

Do you want to mock only the lists returned, or full file-like functionality? — Maciej Gol, Jul 02 '14 at 14:12

score 54 · Accepted Answer · edited Apr 04 '18 at 18:04

54

No. os.walk() is constructed entirely around os.listdir(), with assistance of os.path.islink() and os.path.isdir(). These are essentially system calls, so you'd have to mock your filesystem at the system level. Unless you want to write a FUSE plugin this is not going to be easy to mock.

All os.walk() needs to return is a list of tuples, really. Unless you are testing manipulating the dirs component, it couldn't be more simple:

with mock.patch('os.walk') as mockwalk:
    mockwalk.return_value = [
        ('/foo', ('bar',), ('baz',)),
        ('/foo/bar', (), ('spam', 'eggs')),
    ]

This would mock the following directory structure:

/foo
 ├── baz
 └── bar 
     ├── spam
     └── eggs

edited Apr 04 '18 at 18:04

7yl4r

4,788
4
34
46

answered Jul 02 '14 at 14:14

Martijn Pieters

1,048,767
296
4,058
3,343

I thought it'd be more difficult. Thanks – jbrown Jul 02 '14 at 15:36
3

Thank you! I wrote my solution a little more condensed but this helped me with my solution as well. Remember "os.walk() needs to return a **list of tuples**". My solution: `@patch('test_module.os.walk') def test_walk(self, os_walk): os.walk.return_value[('/foo', ('',), ('file.txt',))] ` – levibostian Jul 06 '14 at 17:38
I think a `=` is missing there; `os.walk.return_value` is not indexable. :-) – Martijn Pieters Jul 06 '14 at 17:41
2

actually `(), ('spam', 'eggs')` could be `[], ['spam', 'eggs']`. not that yours won't work here but it wouldn't hurt to resemble the real one more closely either. – n611x007 Aug 17 '15 at 11:07
@naxa: it would, actually, because if you change your code-under-test to manipulate the `dir` component especially, the test would need updating. You'd easily forget about that if you were to use lists. :-) – Martijn Pieters Aug 17 '15 at 11:32

How to mock os.walk in python with a temporary filesystem?

1 Answers1